Atomic project releases
Release every service of a project together, with readiness checks and schema contracts.
On this page
An atomic project release ships every service of a project as one unit. Check readiness before uploading, guard schema changes, and rotate secrets with the release.
Adding services to an atomic project
Atomic projects can opt into adding independent HTTP source services with
gregale deploy --project --atomic --add-services --verify --yes. Run
gregale deploy check --project --atomic --add-services --verify first to inspect
prerequisites without reserving apps. This is not enabled for every account yet; contact support to use it.
Every existing serving service must remain present with unchanged settings.
New services pass startup readiness before the graph switches; --verify checks
existing critical routes and restores the original membership on regression.
New services have no baseline probe history. Failed or withdrawn additions keep
their app IDs and consume quota but have no default route. An identical retry
can reuse them after the earlier attempt finishes. Removal, dependencies, declared
environment keys, schedules, workers, images, migration commands, configuration
snapshots and progressive rollout are unsupported in this first version.
Check project deployment readiness
Run a read-only check before uploading source or starting builds:
gregale deploy check --project --project-slug shop --json
gregale deploy check --project --project-slug shop --atomic --verify --config-file release.json
gregale deploy check --project --project-slug shop --from staging --env production --sync-config --verify --workers
The report collects missing runtime keys, invalid dependencies, quota and rollback
retention limits, incompatible release options, worker consumer restrictions,
unavailable verification policy and missing rollout qualification. It uses the
same repository detector and source-selection policy as deployment; use
--source=worktree to include local changes. A --secrets-file supplies proposed
key names for the check; its values stay on this machine. The command never
submits a deployment, stages credentials, records qualification or executes probes.
CI exits 0 for ready, 1 for blocked and 2 for unknown/unavailable. Each check includes a stable code, affected workload when applicable and a remedy. An older server produces unknown and a nonzero exit. Protected targets report that the final deployment/promotion plan needs approval; this check does not issue, consume or validate a separate approval credential.
Save state_hash and pass --expected-state HASH on a subsequent check to detect
changed intent or configuration. Unreadable required state has no comparison hash.
A ready result covers the declared inputs and current control-plane prerequisites;
deployment still validates source bytes, capacity and runtime health.
Verified atomic project deploys can opt into one existing push-queue worker with
gregale deploy --project --atomic --verify --workers. The first milestone
requires HTTP workloads plus exactly one production worker, one enabled push
binding and one matching consumer. It also accepts --config-file and --migrate.
The worker starts with its candidate configuration while production queue work
continues to use the previous graph. Startup/configuration proof gates the shared
HTTP/worker cutover; the previous generation then receives bounded graceful
shutdown (default 30 seconds, maximum accepted 60 seconds).
Rollback restores the prior graph and paired variables. Completed jobs and external or database effects remain. Post-cutover jobs retain normal retry/lease semantics. This is not enabled for every account yet; contact support to use it.
For real queue health verification, add --worker-health:
gregale deploy --project --atomic --verify --workers --worker-health
This requires at least five completed delivery attempts in each candidate verification
window and five baseline attempts in the five minutes before cutover. It compares
failures, retries, processing time, queue wait and due backlog. Low traffic reports
insufficient_evidence; it never finalizes a release as healthy. Repeated regressions
restore the retained release, worker admission and paired configuration. Job side
effects remain committed. The existing ten-minute verification deadline applies.
The baseline needs real delivery evidence before opting in. Worker health
verification is not enabled for every account yet.
Named production project worker jobs now persist their release graph at the first claimed delivery. Retries and operator queue replays retain that graph across cutovers and gateway restarts. New jobs select the current graph independently. Expired graphs fail delivery rather than silently move the job to another release; these pins do not extend the release retention window. Explicit revision pins keep their existing behavior. While an unexpired pin owns pending or dispatching work, the scheduler keeps a separate retained worker pool and can restart a missing worker from that deployment. It protects the last worker while draining excess replicas, then drains the pool after the last job finishes. Existing worker/account/concurrency limits still apply; insufficient capacity delays recovery rather than changing the job's release. Progressive canaries are now available through an explicit opt-in contract:
gregale deploy --project --atomic --verify --progressive --config-file release.json
# Include a production worker only with real delivery verification:
gregale deploy --project --atomic --verify --progressive --config-file release.json --workers --worker-health
The first version requires paired non-secret configuration snapshots. New unpinned requests and first-delivery queue jobs select a whole graph at 5%, then 25%, then 100%. Each stage needs two fresh healthy rounds; low worker traffic cannot promote a release. The baseline graph and configuration head remain active until the 100% transition, when graph, configuration and controller checkpoint commit together. The candidate graph ID remains stable through promotion and controller restarts.
Regression before activation stops new candidate cohorts; regression afterward
restores the previous graph/configuration together. Existing job and explicit version
pins retain their graph until retention expires. Operator graph/configuration changes
stop the attempt rather than being overwritten. Unknown baseline health requires
operator action; an expired full-traffic checkpoint fails closed for new unassigned
work until recovery. Public receipts show verification.percent and generation.
Progressive project canaries are not enabled for every account yet; contact support to use them. Workers additionally need atomic worker releases enabled.
Inspect or control a progressive attempt using its durable submission ID:
gregale release attempt get ATTEMPT_ID
gregale release attempt cancel ATTEMPT_ID
gregale release recover ATTEMPT_ID
status shows the traffic stage, generation, blocking reason, latest control
request and available actions. Abort before 100% leaves the baseline graph/config
unchanged. At 100%, abort blocks new unassigned work while fresh baseline health
is checked; qualified restoration commits graph and configuration together.
recover retries that verification for a blocked attempt within the retained
graph's lifetime. It cannot force unhealthy or expired baselines, overwrite newer
operator changes, or retarget pinned jobs. Worker restoration still needs real
baseline delivery evidence; the command does not generate synthetic jobs.
Control commands return when their decision is durably accepted. Use status
to distinguish pending recovery from completed restoration. For an ambiguous
response, retry with the printed --request-id ID --generation N; the same
most recent decision is idempotent.
For failures that appear after initial rollout success, opt into stabilization:
gregale deploy --project --atomic --verify --progressive --stabilize --config-file release.json
This keeps monitoring critical HTTP probes and real worker health for ten minutes
after success. Workers still require --workers --worker-health. Existing revision
retention must be at least 21 minutes. The deploy command returns on rollout
success; use gregale release attempt get ATTEMPT_ID to inspect the continuing watch.
Two confirmed regressions can restore graph/config together while the baseline
remains qualified and this attempt still owns production. Newer production intent
stops the old monitor. Unknown health never authorizes rollback or becomes a
healthy verdict merely because the window ended. Monitoring status and its health
verdict are separate fields.
Migration compatibility before cutover
New --migrate atomic project plans check the current production version again
after the expansion command finishes and before publishing the candidate.
Configure read-only critical probes that exercise the application's database
paths. Command success alone does not permit cutover.
Missing, unauthenticated or unavailable probe results keep the deployment waiting
within its existing deadline. A complete failed baseline sample stops the attempt
and preserves the current production graph. Inspect the existing task logs and
database state before submitting another migration; schema changes are not
reversed and the completed command is not replayed. The receipt exposes
release_command.baseline_verification; gregale release attempt get <attempt-id> also shows this checkpoint.
Passing probes provides evidence for the configured paths, not a proof that every
SQL change is compatible. Keep expansion backward compatible with retained code
and workers. Destructive cleanup is not part of --migrate; stabilization does
not by itself prove that old versions and pinned work have drained.
Guarded schema cleanup
Where schema contracts are enabled for your account, declare both phases for the expansion owner:
release:
command: node migrate.js expand
contract_command: node migrate.js contract
Deploy using --project --atomic --verify --migrate --progressive --stabilize --config-file release.json. For a multi-workload project, keep one Procfile
release: owner and declare only release.contract_command in its manifest.
Workers additionally require the existing worker/health flags.
After rollout and stabilization succeed:
gregale release attempt get ATTEMPT_ID
gregale release contract run ATTEMPT_ID
Cleanup waits for expired retained versions and code pins, drained work, and old instances that stopped or parked. Legacy fallback traffic weights must also select the current revision at 100%; graph expiry alone does not disable an explicit old revision that still has a positive traffic weight. Unknown health, uncertain old tasks, newer production intent or changed runtime inputs block it. The command runs against the expansion's artifact and runtime fingerprint; credential changes, including rotation, conservatively require a fresh deployment rather than silently choosing a different database target.
The CLI prints the durable request ID and generation. After an ambiguous response,
reuse them with --request-id ID --generation N; an admitted attempt keeps the
same cleanup task. Use deploy status and the existing app-task logs to check its
result. Acceptance is asynchronous. A failed or uncertain cleanup is not replayed,
and retired deployment IDs remain fenced, so they cannot be selected for rollback.
Schema changes are not reversed. Investigate the database before a new attempt.
Schema contracts are not enabled for every account yet. Passing critical probes checks the configured paths, not every SQL operation or externally managed work.
Roll forward after failed or uncertain cleanup
Inspect gregale release attempt get FAILED_ATTEMPT and its cleanup task logs. Establish
that the original execution has stopped, inspect actual database state, and repair
SQL explicitly if necessary. Keep that private operational evidence outside the
deploy journal. An unknown task exit remains unknown after recovery.
Deploy compatible corrected code as a different atomic verified attempt with
paired configuration (--project --atomic --verify --config-file repair.json).
Remove obsolete release declarations, or declare a separately reviewed expansion
when needed. The original cleanup command is not part of this repair deployment.
Wait for verification/stabilization, retire legacy fallback traffic through the
usual revision traffic controls, and allow retained graphs, pins, work and old
instances to drain.
Then record incident closure:
gregale release contract resolve FAILED_ATTEMPT \
--replacement-attempt REPAIR_ATTEMPT \
--database-state-checked --execution-quiescent \
--evidence-hash SHA256_OF_PRIVATE_EVIDENCE
The CLI binds the inspected task and cleanup generation and prints a durable
request ID. Reuse --request-id and the same identities/attestations after an
ambiguous response. State and configured critical-route checks must still match
before first acceptance. Existing accepted resolutions replay after newer deploys.
Retirement fences extend to drained non-current code, while the original cleanup
keeps its failed or uncertain outcome. deploy status shows the resolution and
replacement attempt. A missing cleanup receipt cannot be overridden with force.
This action remains available for existing incidents when new cleanup admission is disabled. Database and execution checks are explicit operator attestations; configured HTTP probes do not establish arbitrary SQL compatibility or prove that an unobserved external execution has stopped.
At full traffic, a new abort or recover decision requires an unexpired retained
baseline. If retention has ended, the server returns HTTP 410 with
project_release_baseline_expired; deploy a compatible replacement. Rejection
preserves the current graph, configuration and existing control decision. Replaying
an already accepted request still returns its historical decision.
Atomic application secret rotation
Encrypted production secret snapshots are opt-in, and not enabled for every account yet:
gregale deploy --project --atomic --verify --yes \
--config-file release.json --release-secrets-file credentials.json
credentials.json maps existing workload names to complete secret value sets:
{"frontend":{"API_KEY":"replacement-value"}}
Use existing application-owned production keys. Omitted workloads inherit their prior secret snapshot; an empty workload map removes its serving secrets. The first version supports HTTP workloads with persistent secrets and excludes managed credentials, workers, migrations, sidecars, explicit secret grants and reload hooks. The CLI excludes this file from source packing and sends values separately over TLS. APID encrypts them before storage and returns version IDs, which are then bound into the scan/apply plan. The candidate uses its proposed secrets while the old release keeps its earlier versions. Cutover and rollback select code, variables and immutable secret references through the same project graph.
If staging loses its response, reuse the printed ID with
--release-secrets-request-id ID and the same file contents within 24 hours.
Staging is limited to 32 live bundles per project and the shared 64 KiB document
bound. Public receipts and local sessions contain version IDs and fingerprints,
never secret values or encrypted envelopes. JSON receipts show candidate references
under configuration.secrets.versions and rollback references under
configuration.secrets.previous_versions.
Once pinned, production secret edits through secrets set, deletion or resealing
are rejected; use another atomic secret release. Keep both external credentials
valid while rollback remains possible. Host-key rotation of pinned applications
requires a future snapshot-aware rekey workflow; retain the old host identities.
The secret registry remains the legacy source, while release receipts identify
which versions a deployment actually uses.
Contact support to enable release secret snapshots.
Spotted a problem on this page? Tell us