Documentation Ship releases

Atomic project releases

Release every service of a project together, with readiness checks and schema contracts.

~10 min read
On this page

An atomic project release ships every service of a project as one unit. Check readiness before uploading, guard schema changes, and rotate secrets with the release.

Adding services to an atomic project

Atomic projects can opt into adding independent HTTP source services with gregale deploy --project --atomic --add-services --verify --yes. Run gregale deploy check --project --atomic --add-services --verify first to inspect prerequisites without reserving apps. This is not enabled for every account yet; contact support to use it. Every existing serving service must remain present with unchanged settings. New services pass startup readiness before the graph switches; --verify checks existing critical routes and restores the original membership on regression. New services have no baseline probe history. Failed or withdrawn additions keep their app IDs and consume quota but have no default route. An identical retry can reuse them after the earlier attempt finishes. Removal, dependencies, declared environment keys, schedules, workers, images, migration commands, configuration snapshots and progressive rollout are unsupported in this first version.

Check project deployment readiness

Run a read-only check before uploading source or starting builds:

sh
gregale deploy check --project --project-slug shop --json
gregale deploy check --project --project-slug shop --atomic --verify --config-file release.json
gregale deploy check --project --project-slug shop --from staging --env production --sync-config --verify --workers

The report collects missing runtime keys, invalid dependencies, quota and rollback retention limits, incompatible release options, worker consumer restrictions, unavailable verification policy and missing rollout qualification. It uses the same repository detector and source-selection policy as deployment; use --source=worktree to include local changes. A --secrets-file supplies proposed key names for the check; its values stay on this machine. The command never submits a deployment, stages credentials, records qualification or executes probes.

CI exits 0 for ready, 1 for blocked and 2 for unknown/unavailable. Each check includes a stable code, affected workload when applicable and a remedy. An older server produces unknown and a nonzero exit. Protected targets report that the final deployment/promotion plan needs approval; this check does not issue, consume or validate a separate approval credential.

Save state_hash and pass --expected-state HASH on a subsequent check to detect changed intent or configuration. Unreadable required state has no comparison hash. A ready result covers the declared inputs and current control-plane prerequisites; deployment still validates source bytes, capacity and runtime health.

Verified atomic project deploys can opt into one existing push-queue worker with gregale deploy --project --atomic --verify --workers. The first milestone requires HTTP workloads plus exactly one production worker, one enabled push binding and one matching consumer. It also accepts --config-file and --migrate. The worker starts with its candidate configuration while production queue work continues to use the previous graph. Startup/configuration proof gates the shared HTTP/worker cutover; the previous generation then receives bounded graceful shutdown (default 30 seconds, maximum accepted 60 seconds).

Rollback restores the prior graph and paired variables. Completed jobs and external or database effects remain. Post-cutover jobs retain normal retry/lease semantics. This is not enabled for every account yet; contact support to use it.

For real queue health verification, add --worker-health:

sh
gregale deploy --project --atomic --verify --workers --worker-health

This requires at least five completed delivery attempts in each candidate verification window and five baseline attempts in the five minutes before cutover. It compares failures, retries, processing time, queue wait and due backlog. Low traffic reports insufficient_evidence; it never finalizes a release as healthy. Repeated regressions restore the retained release, worker admission and paired configuration. Job side effects remain committed. The existing ten-minute verification deadline applies. The baseline needs real delivery evidence before opting in. Worker health verification is not enabled for every account yet.

Named production project worker jobs now persist their release graph at the first claimed delivery. Retries and operator queue replays retain that graph across cutovers and gateway restarts. New jobs select the current graph independently. Expired graphs fail delivery rather than silently move the job to another release; these pins do not extend the release retention window. Explicit revision pins keep their existing behavior. While an unexpired pin owns pending or dispatching work, the scheduler keeps a separate retained worker pool and can restart a missing worker from that deployment. It protects the last worker while draining excess replicas, then drains the pool after the last job finishes. Existing worker/account/concurrency limits still apply; insufficient capacity delays recovery rather than changing the job's release. Progressive canaries are now available through an explicit opt-in contract:

sh
gregale deploy --project --atomic --verify --progressive --config-file release.json
# Include a production worker only with real delivery verification:
gregale deploy --project --atomic --verify --progressive --config-file release.json --workers --worker-health

The first version requires paired non-secret configuration snapshots. New unpinned requests and first-delivery queue jobs select a whole graph at 5%, then 25%, then 100%. Each stage needs two fresh healthy rounds; low worker traffic cannot promote a release. The baseline graph and configuration head remain active until the 100% transition, when graph, configuration and controller checkpoint commit together. The candidate graph ID remains stable through promotion and controller restarts.

Regression before activation stops new candidate cohorts; regression afterward restores the previous graph/configuration together. Existing job and explicit version pins retain their graph until retention expires. Operator graph/configuration changes stop the attempt rather than being overwritten. Unknown baseline health requires operator action; an expired full-traffic checkpoint fails closed for new unassigned work until recovery. Public receipts show verification.percent and generation.

Progressive project canaries are not enabled for every account yet; contact support to use them. Workers additionally need atomic worker releases enabled.

Inspect or control a progressive attempt using its durable submission ID:

sh
gregale release attempt get ATTEMPT_ID
gregale release attempt cancel ATTEMPT_ID
gregale release recover ATTEMPT_ID

status shows the traffic stage, generation, blocking reason, latest control request and available actions. Abort before 100% leaves the baseline graph/config unchanged. At 100%, abort blocks new unassigned work while fresh baseline health is checked; qualified restoration commits graph and configuration together. recover retries that verification for a blocked attempt within the retained graph's lifetime. It cannot force unhealthy or expired baselines, overwrite newer operator changes, or retarget pinned jobs. Worker restoration still needs real baseline delivery evidence; the command does not generate synthetic jobs.

Control commands return when their decision is durably accepted. Use status to distinguish pending recovery from completed restoration. For an ambiguous response, retry with the printed --request-id ID --generation N; the same most recent decision is idempotent.

For failures that appear after initial rollout success, opt into stabilization:

sh
gregale deploy --project --atomic --verify --progressive --stabilize --config-file release.json

This keeps monitoring critical HTTP probes and real worker health for ten minutes after success. Workers still require --workers --worker-health. Existing revision retention must be at least 21 minutes. The deploy command returns on rollout success; use gregale release attempt get ATTEMPT_ID to inspect the continuing watch. Two confirmed regressions can restore graph/config together while the baseline remains qualified and this attempt still owns production. Newer production intent stops the old monitor. Unknown health never authorizes rollback or becomes a healthy verdict merely because the window ended. Monitoring status and its health verdict are separate fields.

Migration compatibility before cutover

New --migrate atomic project plans check the current production version again after the expansion command finishes and before publishing the candidate. Configure read-only critical probes that exercise the application's database paths. Command success alone does not permit cutover.

Missing, unauthenticated or unavailable probe results keep the deployment waiting within its existing deadline. A complete failed baseline sample stops the attempt and preserves the current production graph. Inspect the existing task logs and database state before submitting another migration; schema changes are not reversed and the completed command is not replayed. The receipt exposes release_command.baseline_verification; gregale release attempt get <attempt-id> also shows this checkpoint.

Passing probes provides evidence for the configured paths, not a proof that every SQL change is compatible. Keep expansion backward compatible with retained code and workers. Destructive cleanup is not part of --migrate; stabilization does not by itself prove that old versions and pinned work have drained.

Guarded schema cleanup

Where schema contracts are enabled for your account, declare both phases for the expansion owner:

yaml
release:
  command: node migrate.js expand
  contract_command: node migrate.js contract

Deploy using --project --atomic --verify --migrate --progressive --stabilize --config-file release.json. For a multi-workload project, keep one Procfile release: owner and declare only release.contract_command in its manifest. Workers additionally require the existing worker/health flags.

After rollout and stabilization succeed:

sh
gregale release attempt get ATTEMPT_ID
gregale release contract run ATTEMPT_ID

Cleanup waits for expired retained versions and code pins, drained work, and old instances that stopped or parked. Legacy fallback traffic weights must also select the current revision at 100%; graph expiry alone does not disable an explicit old revision that still has a positive traffic weight. Unknown health, uncertain old tasks, newer production intent or changed runtime inputs block it. The command runs against the expansion's artifact and runtime fingerprint; credential changes, including rotation, conservatively require a fresh deployment rather than silently choosing a different database target.

The CLI prints the durable request ID and generation. After an ambiguous response, reuse them with --request-id ID --generation N; an admitted attempt keeps the same cleanup task. Use deploy status and the existing app-task logs to check its result. Acceptance is asynchronous. A failed or uncertain cleanup is not replayed, and retired deployment IDs remain fenced, so they cannot be selected for rollback. Schema changes are not reversed. Investigate the database before a new attempt.

Schema contracts are not enabled for every account yet. Passing critical probes checks the configured paths, not every SQL operation or externally managed work.

Roll forward after failed or uncertain cleanup

Inspect gregale release attempt get FAILED_ATTEMPT and its cleanup task logs. Establish that the original execution has stopped, inspect actual database state, and repair SQL explicitly if necessary. Keep that private operational evidence outside the deploy journal. An unknown task exit remains unknown after recovery.

Deploy compatible corrected code as a different atomic verified attempt with paired configuration (--project --atomic --verify --config-file repair.json). Remove obsolete release declarations, or declare a separately reviewed expansion when needed. The original cleanup command is not part of this repair deployment. Wait for verification/stabilization, retire legacy fallback traffic through the usual revision traffic controls, and allow retained graphs, pins, work and old instances to drain.

Then record incident closure:

bash
gregale release contract resolve FAILED_ATTEMPT \
  --replacement-attempt REPAIR_ATTEMPT \
  --database-state-checked --execution-quiescent \
  --evidence-hash SHA256_OF_PRIVATE_EVIDENCE

The CLI binds the inspected task and cleanup generation and prints a durable request ID. Reuse --request-id and the same identities/attestations after an ambiguous response. State and configured critical-route checks must still match before first acceptance. Existing accepted resolutions replay after newer deploys. Retirement fences extend to drained non-current code, while the original cleanup keeps its failed or uncertain outcome. deploy status shows the resolution and replacement attempt. A missing cleanup receipt cannot be overridden with force.

This action remains available for existing incidents when new cleanup admission is disabled. Database and execution checks are explicit operator attestations; configured HTTP probes do not establish arbitrary SQL compatibility or prove that an unobserved external execution has stopped.

At full traffic, a new abort or recover decision requires an unexpired retained baseline. If retention has ended, the server returns HTTP 410 with project_release_baseline_expired; deploy a compatible replacement. Rejection preserves the current graph, configuration and existing control decision. Replaying an already accepted request still returns its historical decision.

Atomic application secret rotation

Encrypted production secret snapshots are opt-in, and not enabled for every account yet:

sh
gregale deploy --project --atomic --verify --yes \
  --config-file release.json --release-secrets-file credentials.json

credentials.json maps existing workload names to complete secret value sets:

json
{"frontend":{"API_KEY":"replacement-value"}}

Use existing application-owned production keys. Omitted workloads inherit their prior secret snapshot; an empty workload map removes its serving secrets. The first version supports HTTP workloads with persistent secrets and excludes managed credentials, workers, migrations, sidecars, explicit secret grants and reload hooks. The CLI excludes this file from source packing and sends values separately over TLS. APID encrypts them before storage and returns version IDs, which are then bound into the scan/apply plan. The candidate uses its proposed secrets while the old release keeps its earlier versions. Cutover and rollback select code, variables and immutable secret references through the same project graph.

If staging loses its response, reuse the printed ID with --release-secrets-request-id ID and the same file contents within 24 hours. Staging is limited to 32 live bundles per project and the shared 64 KiB document bound. Public receipts and local sessions contain version IDs and fingerprints, never secret values or encrypted envelopes. JSON receipts show candidate references under configuration.secrets.versions and rollback references under configuration.secrets.previous_versions.

Once pinned, production secret edits through secrets set, deletion or resealing are rejected; use another atomic secret release. Keep both external credentials valid while rollback remains possible. Host-key rotation of pinned applications requires a future snapshot-aware rekey workflow; retain the old host identities. The secret registry remains the legacy source, while release receipts identify which versions a deployment actually uses.

Contact support to enable release secret snapshots.

Spotted a problem on this page? Tell us