Release Checklist¶

Step-by-step process for cutting a doppler release. Run these steps in order — each one is a gate for the next.

1. Confirm main is green and the tree is clean¶

main's required CI already ran the full suite (test-all's equivalent) plus the docs build on every commit that landed there — step 5's PR merge is the real correctness gate (see the callout below), and step 7's verify-ci explicitly does not re-test, only confirms. Re-running make test-all / make docs locally here just repeats work CI already did on the exact same tree; skip it unless you have a specific reason to distrust CI (a suspected environment-specific flake, or extra confidence before an unusually disruptive release).

git status                    # nothing uncommitted
git checkout main && git pull  # sync to the latest green main
gh pr checks <last-merged-pr>  # spot-check CI was actually green, if in doubt

2. Check examples¶

If any gallery example scripts in src/doppler/examples/ changed since the last release, regenerate the plots:

git switch -c gallery/vX.Y.Z
make gallery
git add docs/assets/
git commit -m "docs: update gallery plots for vX.Y.Z"

Land it through its own PR before §2b. §2b measures a commit main already has; a plots commit left on local main is exactly the commit it must not measure (v0.57.0's set was stamped with one, and that tree never reached main).

If you added a new plot-generating script, add it to GALLERY_SCRIPTS in the Makefile before running make gallery.

2a. When freshness asks for work that is not there¶

make release-freshness-check asks, per gallery script, whether the script moved since the last tag while its own plot under docs/assets/ did not, and whether a perf-relevant path moved with no snapshot. It cannot see what moved inside one, so a change confined to a platform this release does not measure — Windows, for doppler's Linux-measured benchmarks and art — still reads as stale. Nor can it tell a plot that re-rendered byte-identically from one nobody rendered: neither leaves a diff, so the identical re-render is waived in writing too.

Answer it in writing rather than with a re-render that changes nothing: add release-waivers/v<ver>.md (repo root) with one line per item,

- gallery: path/to/script.py -- why this needs no re-render
- benchmarks: native/inc/header.h -- why the numbers still describe the tree

Each line waives exactly that item, must carry at least the first words of its reason on the same line as -- (a line without one fails the gate by name), and is printed by the gate while it runs, so a skipped check lands in the release log. A line that matches nothing stale fails the gate, so a waiver cannot outlive its reason. v0.55.0.md is the worked example: four Windows-only items.

2b. Refresh the benchmarks page¶

The published Benchmarks page is rendered from committed snapshots under benchmarks/published/v<ver>/ — representative numbers measured by hand on a real machine, deliberately not from CI (shared runners aren't hardware-representative). Each release is measured in two builds so the page shows the from-source upside: portable (the wheel) and native (-DDOPPLER_NATIVE=ON).

On the same representative machine each release (so the history stays comparable), measure both builds interleaved and publish.

The bench box needs the pinned uv before bench-interleaved (#1940). Every build and run in it goes through uv, and a uv other than pyproject.toml's [tool.uv] required-version refuses with exit 2 before anything is measured. uv --version must print what make -s print-uv-version does; when it does not, Adding a Module has the update for each way uv gets installed.

Measure a commit that is already on origin/main. The set stamps it as doppler_meta.commit, a reader checks it out to reproduce the numbers, and make bench-commits-check refuses one main cannot reach (#1322). bench-interleaved says which commit it measures, and whether origin/main has it, before and after the run.

Measure from one clone per box, not a clone per release. Each run leaves its snapshot in the clone, untracked until its PR merges, and docs/benchmarks.md modified. A clone per release kept every copy: 26 clones, about 10 GB, by 2026-10-14. In one clone, git checkout of the next release commit refuses while the previous copy is there. make bench-preflight clears it once origin/main has the same content, compared the way git status compares (line endings converted). It refuses, without touching anything, over any result main lacks or holds differently, one staged and not committed, or a symlink: publish that through its PR, or remove it. bench-interleaved runs the same check first.

First put the CPU in a peak, repeatable state — the published doppler_meta records the governor either way, but powersave understates the numbers:

sudo cpupower frequency-set -g performance   # peak, repeatable; quiesce other load

cd ~/doppler-bench                     # the box's one clone (any path)
git fetch origin
make bench-preflight                   # clears the last release's outputs
git checkout --detach origin/main      # measure what main has
make bench-interleaved VERSION=X.Y.Z   # builds portable + native, runs them
                                       # alternately, keeps the per-bench best
make bench-labels-fix                  # delete label lines the new set retired
make bench-docs                        # render docs/benchmarks.md (two columns)
git switch -c bench/vX.Y.Z
git add benchmarks/published docs/benchmarks.md benchmarks/
git commit -m "docs: publish benchmarks for vX.Y.Z (<cpu>)"

make bench-docs refuses until every row the new set publishes has a label in benchmarks/labels.toml or a line in benchmarks/.labels-exempt (the label gate, #2372), and each refusal says what to do. Two kinds are routine at a release, and make bench-labels-fix clears both: a benchmarks/legacy-units.toml entry whose row records its own unit now (the first set measured after #2314 recorded every C row's unit, and the fix deleted the whole file then), and an exempt line whose row the new set no longer measures -- read the lines it prints, because a row missing from this one set only (a Python bench whose broker was down) loses its exempt line too, and needs a label when it comes back. A row the set publishes for the first time needs a label written, not an exempt line -- the exempt list only shrinks -- and it should already have one: a new row is labelled in the PR that adds it, where the gate holds the label as pending until a release measures the row. Nothing checks that at PR time yet (#2390), so this is where a forgotten or mistyped label first shows. A label the new set was measured with but that names no row it recorded is reported as dead: a retired row, or a mistyped ID.

The run takes about two hours (2 h 02 min measured 2026-11-05, pinned to the fastest core class), so over ssh it is launched in the background, and it leaves its outcome where a watcher can wait on it: the log at build/bench-runs/X.Y.Z.log and, written last, the exit code at build/bench-runs/X.Y.Z.rc -- on success and on every failure (#2681). Wait on that file, with a cap, never on the process: a remote pgrep -f matches the ssh shell that carries its own pattern, so it never sees the run end -- a failed release run once read "running" for hours that way. The cap is for the one case no file can cover: a run killed outright writes nothing.

ssh <box> 'bash -lc "cd ~/doppler-bench && \
  (nohup make bench-interleaved VERSION=X.Y.Z >/dev/null 2>&1 &)"'
run=doppler-bench/build/bench-runs/X.Y.Z   # relative to the box's $HOME
for _ in $(seq 90); do                      # 3 h cap; a run takes about 2 h
  ssh <box> "test -f $run.rc" && break
  sleep 120
done
# 0 = measured. No .rc after the cap: the run was killed (SIGKILL, OOM,
# reboot) before it could write one -- read where its log stopped.
ssh <box> "cat $run.rc || tail -20 $run.log"

If a set was measured on a branch head instead, it can be restamped after the merge, but only if that exact tree landed on main unchanged (the branch rebase-merged, so the measured commit lands as a commit of its own). A squash-merge folds it into one commit with everything after it, leaves no tree-identical commit, and the restamp refuses:

git fetch origin                     # the restamp matches against origin/main
make bench-restamp VERSION=X.Y.Z     # doppler_meta.commit -> main's tree twin

bench-interleaved builds both flavours in throwaway git worktrees, runs the suite alternately K times (default 5; K=N to override), and keeps each benchmark's lowest-mean run — so the %Δ column reflects the real build difference, not cross-run system drift. The measurement (not the build) is pinned to the CPU's fastest core class, read from cpuinfo_max_freq: on a heterogeneous part an unpinned benchmark is bimodal — the same binary measured 3.5 µs on a Zen 5 core and 5.6 µs on a Zen 5c one — and min-of-K only usually finds the fast mode. The snapshot records the pinned CPUs and the page prints them. Each snapshot is stamped with the compiler + flags it read from compile_commands.json, so the page is self-describing. Skip only if no perf-relevant code changed since the last release.

3. Cut the release branch + bump the version¶

One command does steps 3, 4 and 5's mechanics:

make release-pr VERSION=X.Y.Z

It first asks the tag-time questions that only the tree can answer: whether benchmarks and the gallery are fresh for this version, and whether the notes fit (sizes and entry lengths read from the fragments, BEFORE assembly empties them). Then it branches off origin/main, bumps the five version sites, promotes the changelog fragments and cuts the section, regenerates the comparison links, checks the versions and that nothing is left unassembled, sizes the body release-notes.sh will actually publish, commits, pushes and opens the PR — so a merged, green release commit passes make ship's own checks without a surprise at the tag. Read the sections below for what each part is doing and why — then review the changelog prose it assembled, which is the one part no command can judge.

To drive it by hand instead (or to stop after the bump):

make release-branch VERSION=X.Y.Z   # branches chore/release-X.Y.Z, then bumps

bump-version updates five files atomically, from one declaration in scripts/version_sites.py — there is no second list, and version-check reads the same table:

(uv.lock is re-synced by the uv lock in the same command, and CHANGELOG.md is prose you write in step 4 — neither is a probe.)

The last two joined on 2026-08-24, having never been bumped at all: bootstrap.toml was frozen at 0.3.7 and just-makeit.toml at 0.1.0, the value it was given in the initial commit. Nothing read either, which is why nothing noticed — dp_doppler_version() returns DOPPLER_VERSION, stamped by CMake from PROJECT_VERSION. make version-check now probes all five, so a missed one is a red gate rather than a number nobody reads.

4. Assemble the changelog¶

Entries are written as fragments under changelog.d/ — one file per change, which is what stopped CHANGELOG.md conflicting on every rebase — so ## [Unreleased] stays near-empty between releases and the release is where they get promoted. This step did not exist in this runbook until 2026-08-28, and its absence was not theoretical: cutting from the old step 4 alone would have published the 5 entries already sitting under [Unreleased] and left 62 fragments on the floor.

On the release branch:

make changelog-assemble VERSION=X.Y.Z   # promote, then cut the section
make docs-relink                        # comparison links from the headings

changelog-assemble promotes changelog.d/ into [Unreleased] and STAGES the promotion (the fragments are deleted but still tracked, so an unstaged assemble makes the next make lint fail on paths that are gone). With VERSION= it then renames ## [Unreleased] → ## [X.Y.Z] — YYYY-MM-DD and opens a fresh empty ## [Unreleased] above it.

That rename was a hand step until 2026-09-16, sitting immediately after a command that had already edited the same file — and a hand step beside an automated one is the half that rots. It refuses to cut a version that already has a section, so a second run cannot open a duplicate.

Writing the fragments is prose and stays prose. Renaming a heading was never prose; reviewing what the fragments say still is, and that review is the reason the release is a PR.

5. Open the PR and merge it green (the gate)¶

git commit -am "chore: release vX.Y.Z"
git push -u origin HEAD
gh pr create --fill
gh pr merge --rebase   # (or --squash) ONLY once `CI passed` is green — never bypass it

The release tag will point at this merged commit, so CI passing here is what makes the release safe.

Check the docs walk by hand. The release PR touches pyproject.toml, so the Docs walk (candidate) workflow runs on it: every install path the docs name, walked from a bare container against artifacts built from this tree (both architectures). It is not in CI passed (#2881), so a red run does not block the merge. Until it does, confirm both legs are green before merging (gh pr checks, the two Docs walk, candidate rows); release.yml runs the published face after the tag, and a failure there is a shipped release.

The release PR is a version bump alone, so changes classifies it src=false and the heavy matrix its parent already passed is skipped. The cheap gates (lint, manifest drift, the CI-image pin) still run. See CI.

6. Tag merged main¶

git checkout main && git pull
make tag-release VERSION=X.Y.Z   # verifies on-main + in-sync, tags, pushes the tag

tag-release pushes only the tag (never main), which triggers the release workflow. It refuses to run while any changelog.d/ fragment is still unassembled — step 4 is not optional, and the tag is the irreversible step.

7. Release workflow (automatic)¶

The release.yml workflow runs these jobs in order:

verify-version  ──  tag == pyproject == Cargo == CMakeLists == bootstrap == just-makeit
    │
    ├── verify-ci        ──  poll the "CI passed" aggregator on the tagged SHA
    │                         (the merge to main already ran the full suite — we
    │                          do NOT re-test here, we confirm it was green)
    │
    ├── build-python     ──  manylinux_2_28 x86_64 + aarch64 wheels, one docker run per cp3x/arch
    ├── build-macos      ──  macOS arm64 wheels, `make wheel` per Python version
    ├── build-windows    ──  win_amd64 wheels (clang-cl), `make wheel` per Python version
    ├── build-sdist      ──  source distribution (`make sdist`)
    ├── build-c-linux    ──  C library tarballs (linux-x86_64 + linux-aarch64, matrixed)
    ├── build-c-macos    ──  C library tarball (macos-arm64)
    └── build-c-windows  ──  C library zip (windows-x86_64, clang-cl)
          (all seven build-* jobs run in parallel, gated only on verify-version;
           every C archive is `make package-c-tarball VERSION=x.y.z`, which a
           developer can run before cutting a tag)

smoke-wheel      ──  install the cp312 wheel (x86_64, aarch64, win_amd64)   needs build-python, build-windows
    │                 + run deploy/validation/wfm_e2e.py
    │                 (smoke-tests the ARTIFACT, not the source tree)
wheel-tags       ──  every wheel's name parses as installable by pip         needs the three wheel builds
    │
    ▼
publish-python   ──  PyPI (OIDC trusted publishing, no token needed)
    │                 needs verify-ci, smoke-wheel, wheel-tags, sdist + every wheel build
    ├──────────────────────────────┐
    ▼                              ▼
publish-container                smoke-pypi  ──  install doppler-dsp==X.Y.Z FROM PyPI
    │  runtime, SDK + downstream     and run wfm_e2e.py, per platform x Python
    │  images per arch, pushed by
    │  digest (the runtime image
    │  pip-installs the just-
    │  published wheel, no rebuild)
    ▼
publish-container-merge  ──  multi-arch :X.Y.Z (+ :latest) tags for each image
    │                         (ghcr.io/doppler-dsp/doppler, -sdk, -downstream-jm),
    │                         smoked as a user would pull them
    ▼
github-release   ──  GitHub Release + auto-generated notes + wheel/C-archive attachments
    │                 (also needs the three build-c-* jobs)
    ├──────────────────────────────┐
    ▼                              ▼
smoke-c                          smoke-packages
  download the published C         install THIS arch's published .deb/.rpm
  archive (Linux x2, macOS,        in several distros (x86_64 + aarch64);
  Windows); find_package +         find_package + pkg-config from /usr
  pkg-config

The release pipeline does not re-run the test suite — the bump PR's merge to main is the gate, so release.yml only confirms CI was green on the tagged commit (verify-ci polls the ci-passed aggregator job in ci.yml), then builds, smoke-tests the built wheel, and publishes. This matches the canonical release-process skill.

How each wheel is actually built (no cibuildwheel — just-buildit is a minimal PEP 517 backend driving the whole thing):

1. Linux (build-python) — one docker run per cp3x × arch inside quay.io/pypa/manylinux_2_28_x86_64 or _aarch64 (arch picked by the job matrix), installing cmake/pkg-config (fftw/zeromq were dropped once the FFT was fully vendored and ZMQ was removed — see CHANGELOG) plus numpy/uv/build/just-buildit, then python -m build --no-isolation --wheel. The aarch64 leg runs natively on a ubuntu-24.04-arm runner (free for public repos) — no QEMU or cross-toolchain anywhere.
2. macOS (build-macos, a separate job) — natively on a macOS arm64 runner. astral-sh/setup-uv selects the Python, make install-deps installs the system deps (reads bootstrap.toml, the single source of truth for doppler's system deps), and make wheel (= uv build --wheel) drives the same just-buildit backend as the Linux leg — no hand-rolled uv venv / python -m build bootstrap to drift from CI. The job pins MACOSX_DEPLOYMENT_TARGET=11.0 and _PYTHON_HOST_PLATFORM=macosx-11.0-arm64 so delocate tags the wheel arm64: cmake emits an arm64-only .so on this runner, and without the override delocate fails demanding the absent x86_64 slice.
3. Windows (build-windows) — natively on a windows-latest runner, the same make wheel under the MSVC environment with clang-cl + Ninja (CMAKE_ARGS), the toolchain ci.yml's Windows jobs use on every PR.
4. just-buildit's backend (make just-build → cmake + pyext) assembles the wheel from the build's output dir, detects the platform tag from the .so suffix, and runs auditwheel repair (Linux, via uvx) / delocate-wheel (macOS) / delvewheel repair (Windows) to bundle shared-lib deps into the wheel — this is why the published wheel needs zero system packages to pip install. Each Linux wheel is then scanned for both a leaked -march=native (AVX/AVX-512 on x86_64) and a leaked -mcpu=native (SVE/SVE2 on aarch64) — see Build Internals for both scans.

verify-version checks — it runs make version-check VERSION=<tag>, which fails immediately if any probed file disagrees with the tag or with the others:

• pyproject.toml
• ffi/rust/Cargo.toml
• CMakeLists.txt
• bootstrap.toml ([project] version)
• just-makeit.toml ([project] version)

If it fails, bump the missed file manually, push a fixup commit on main, then re-tag.

A container build that dies on toomanyrequests at a FROM (either Rate exceeded or Data limit exceeded, ECR Public's anonymous per-IP limits): re-run the failed jobs. publish-container builds on buildx's container driver, which pulls inside BuildKit's own container, out of scripts/stock-pull.sh's reach. So its first step, make stock-registry, fetches each FROM's manifest through the helper, pulling nothing. That step retries a rate limit and falls back to STOCK_MIRROR on a quota, and the builds then pull from whichever registry it needed (#2572). A refusal that begins between the probe and the build still fails the build. Re-running the failed jobs probes again. Before that step existed, two releases in a row lost an image job this way after PyPI had published (#1982).

8. Verify the release¶

make ship already did this — there is nothing to type here, which is the point. This step used to be a hand-run pip install in a scratch venv, and a runbook step reports nothing when it is skipped.

Two gates cover it now:

• smoke-pypi (release.yml, after publish-python) installs doppler-dsp==X.Y.Z from PyPI and runs the full wfm_e2e.py against it, once per platform wheel (Linux x86_64 and aarch64, win_amd64, macOS arm64) and per supported Python, read from pyproject.toml's classifiers. It installs a wheel only, never the sdist, so a wheel pip would skip fails the leg instead of quietly building from source (#1817). The install retries itself while PyPI's index catches up with the publish (#1394). smoke-wheel installs a local file before the upload, so this is the only thing in the release that exercises the index — the C library had smoke-c doing exactly this and the wheel did not (#1347).
• release-watch.sh (the watch half of make ship) confirms PyPI serves the version and that latest has caught up, and that the GitHub Release is published, non-draft, with notes and its assets attached.

Neither is a substitute for a look at the GitHub Release page: the asset check counts tarballs, so confirming that every platform wheel (Linux x86_64, Linux aarch64, win_amd64, macOS arm64) plus the four C library archives (three tarballs and the Windows zip) are present is still worth ten seconds of your eyes.

To re-run the published-wheel check by hand — against any released version, which is what makes it rehearsable before touching the script. SMOKE_PYTHON=3.N installs into a venv on that Python, so the wheel tested is that Python's, as one smoke-pypi leg does:

make release-smoke-pypi VERSION=X.Y.Z
make release-smoke-pypi VERSION=X.Y.Z SMOKE_PYTHON=3.13

Version conventions¶

Doppler uses plain Semantic Versioning — MAJOR.MINOR.PATCH, which read positionally is BREAKING.FEATURE.PATCH:

Stable X.Y.Z releases only — no alpha/beta/rc suffixes. main stays at the last released version between releases (no post-release dev bump).

Pre-1.0 (where we are)¶

The MAJOR digit stays 0 until we commit to a stable public API and cut 1.0.0, and that has a consequence sharper than it first looks. SemVer §4 says of 0.y.z: "Anything MAY change at any time. The public API SHOULD NOT be considered stable." So the words "backward compatible" in the table above do not apply pre-1.0 — the qualifier presupposes a stable public API, and there isn't one yet. There is no such thing as a backward-compatible-or-not 0.y.z release.

Which leaves exactly two questions:

Breaking changes do not affect the version. They are not a tiebreaker, not an escalation, and not a reason to hesitate over the bump: 0.y.z has already told users the API is unstable. A release that breaks every signature in the library and adds one feature is a MINOR; one that breaks the same signatures and only fixes bugs is a PATCH. The whole obligation is to say so in a ### Breaking section of CHANGELOG.md, which github-release publishes verbatim as the release notes.

Worked examples: 0.6.0 (waveform generator), 0.7.0 (read_iq), 0.8.0 (the Python composer subsystem), and 0.9.0 (the timing pacing/timestamping subsystem) are all feature bumps. A bug-fix-only release off 0.8.0 would have been 0.8.1.

A release can be a feature bump and carry breakage: the one that added wfm.Reader.header and dp_wfm_kw_check_standard() also changed 38 C kernel signatures. It took the FEATURE digit because it added functionality — the breakage went under ### Breaking in the changelog and had no bearing on which digit moved.