Release process

Shipping is a single pipeline: release-train.yml runs on every push to main, builds the runtime once, and promotes that identical wasm through the networks. A second workflow, watch-mainnet-release.yml, watches the chain and publishes release artifacts once the mainnet upgrade actually executes.

The release train

push to main
  └─ srtool build (once per train)
       └─ deploy devnet ───── smoke-check devnet
            └─ deploy testnet ── smoke-check testnet ── publish SDK rc to PyPI
                 └─ propose mainnet upgrade (multisig)

Build once (srtool)

The train builds the runtime with srtool — a pinned Docker build environment that makes the wasm byte-reproducible. The artifact (subtensor.wasm + digest) is uploaded once and reused by every deploy job; promotion never rebuilds. Anyone can verify the hash independently by running scripts/srtool/build-srtool-image.sh followed by scripts/srtool/run-srtool.sh locally (see Repository scripts).

The ship lever: spec_version

Every deploy job first compares the built runtime's spec_version (runtime/src/lib.rs) with what the target chain is running, and no-ops unless the build is newer. Consequences:

• A merge without a spec_version bump builds and then does nothing — this is deliberate, and it's why the Spec Version Check on PRs exists (label no-spec-version-bump to opt out knowingly).
• Stale queued trains are harmless: they no-op at each guard.
• Rollback is not a concept here — you ship a newer fixed version.

Promotion gates

Human approval lives in GitHub environment settings, not workflow code:

Deploys to devnet and testnet are direct setCode transactions via the CI deploy key. After the on-chain spec_version is verified, the train moves the corresponding network mirror branch. That branch push triggers both Docker publishers from the exact deployed commit, updating the matching :devnet or :testnet tag in ghcr.io/raofoundation/subtensor and ghcr.io/raofoundation/subtensor-localnet. It does not move either package's :latest tag. The smoke suite then validates the live network while the independent image builds run.

The mainnet multisig ceremony

CI never upgrades mainnet directly. Before any on-chain action, the train reserves the immutable v<spec_version> tag at the exact source commit. Its final job then submits a multisig proposal: the CI key is one half of a 2-of-2 deployment multisig that holds a SudoUncheckedSetCode proxy on the sudo key. The triumvirate then approves the proposal 2-of-3 out-of-band — no GitHub credential can unilaterally change the mainnet runtime. Until they sign, nothing happens on chain.

For rehearsal, the mainnet-clone PR label spins up a live clone of mainnet running your runtime with a public endpoint (mainnet clone testing).

The release watcher

watch-mainnet-release.yml polls mainnet every 10 minutes. When an upgrade executes, it resolves the release-train artifact whose commit matches the immutable tag and whose runtime hash matches the finalized on-chain :code. It moves the protected mainnet mirror to that exact commit before cutting the release:

1. GitHub release v<spec_version>
2. Production Docker images tagged v<spec_version> and latest
3. Python SDK + bittensor-core wheels to PyPI (trusted publishing, with PEP 740 provenance attestations)
4. Publishable Rust crates to crates.io (auto-discovered; workspace crates are publish = false while they depend on the patched polkadot-sdk fork)
5. Production website/docs to Vercel

This ordering means the release always reflects what is actually running on mainnet, not what was merged.

PyPI is the terminal state for stable Python publication. The watcher requires the SDK wheel and source distribution plus the full bittensor-core Linux and macOS wheel matrix and source distribution. Every file must carry a PEP 740 attestation from this repository's mainnet release workflow. A failed or partial upload is retried on the next watcher run even if the GitHub release already exists or the release-train artifact has expired. Retries require the release target, immutable tag, protected mainnet mirror, and finalized runtime bytes to agree. The SDK's committed X.Y.Z.dev0 version is stamped to X.Y.Z for this build. The release train rejects a base SDK or core version that already exists on PyPI before publishing release candidates. Releases through v432 predate automated stable Python publication and are handled as historical state.

The watcher explicitly dispatches docker.yml and docker-localnet.yml because releases created with the default GITHUB_TOKEN do not emit another workflow event. The production build publishes both the immutable release tag and :latest. Independently, every merge to main publishes the moving production :main tag, so developers can prepare against merged features before they reach mainnet without changing the deployed :latest image. The localnet build publishes the immutable release tag in its separate package; localnet :latest remains an alias for the current :main branch image, while :devnet and :testnet follow their respective branches.

Hotfixes

The same pipeline applies: land the fix on main with a spec_version bump and let the train promote it. There is no side channel that skips devnet and testnet validation.