Build & Delivery Internals¶

This page explains how the build and release pipeline works — the stack that turns source into a published wheel and a standalone C library tarball. See Release for the step-by-step checklist.

Overview¶

flowchart TD
    src["Source\nnative/ + src/doppler/"]
    cmake["CMake\nbuild/"]
    so["C extensions\n*.cpython-*.so"]
    bin["wfmgen binary\nsrc/doppler/wfm/_bin/wfmgen"]
    jb["just-buildit\nPEP 517 backend"]
    manylinux["manylinux_2_28\ncontainer (Linux)"]
    auditwheel["auditwheel repair\nbundle shared deps"]
    wheel["Wheel\ndoppler_dsp-X.Y.Z-*.whl"]
    pypi["PyPI\nOIDC trusted publishing"]
    ctarball["C library tarball\ndoppler-X.Y.Z-linux.tar.gz"]

    src --> cmake --> so --> jb --> manylinux --> auditwheel --> wheel --> pypi
    cmake --> bin --> jb
    manylinux --> ctarball

CMake layer¶

All build output goes to build/ — never to the repo root.

Key flags¶

PYTHON_EXECUTABLE matters¶

The CPython C extension suffix encodes the Python version and ABI tag (e.g., cpython-313-x86_64-linux-gnu). If CMake finds a different interpreter than the one in your active venv, the compiled .so will not be importable. The Makefile resolves this with:

PYTHON_EXECUTABLE ?= $(shell uv run python -c 'import sys; print(sys.executable)')
PYTHON_EXECUTABLE := $(or $(JUST_BUILDIT_PYTHON),$(PYTHON_EXECUTABLE))

JUST_BUILDIT_PYTHON is set by the wheel backend so wheel builds always compile against the right version.

Make targets¶

Build artifacts¶

build/libdoppler.so          shared C library
build/libdoppler.a           static C library (C++-free, symbol-verified)
src/doppler/**/*.cpython-*.so  Python extensions (copied from build/ by pyext)

just-buildit wheel backend¶

pyproject.toml declares just-buildit as the PEP 517 build backend:

[build-system]
requires = ["just_buildit"]
build-backend = "just_buildit"

[tool.just-buildit]
command = "make just-build"
editable_path = "src"

When python -m build --wheel (or uv build) runs:

1. just-buildit sets JUST_BUILDIT_OUTPUT_DIR and JUST_BUILDIT_PYTHON.
2. It calls make just-build, which runs make pyext then copies src/doppler/ (including the freshly built .so files and the examples/ directory) into the output staging area.
3. just-buildit packages the staged tree into a wheel and writes it to dist/.

The --no-group docs flag in UV_SYNC_FLAGS ensures the docs dependency group is never installed during a wheel build — a docs network blip must not break packaging.

OIDC trusted publishing¶

PyPI is configured with GitHub's OIDC trusted publisher for the doppler-dsp/doppler repository. No PYPI_TOKEN secret is needed; release.yml uses id-token: write permission and the publish-python job authenticates via a short-lived OIDC token. If PyPI rejects the upload, check the trusted publisher configuration in the PyPI project settings, not the repository secrets.

manylinux & auditwheel (Linux wheels)¶

Linux wheels must satisfy PEP 600 symbol-version constraints so they run on distributions older than the build host. release.yml builds inside the quay.io/pypa/manylinux_2_28_x86_64 Docker image (glibc 2.28, GCC 12), which sets the minimum runtime floor.

After the build, auditwheel repair inspects the wheel, bundles any shared libraries beyond the manylinux allow-list (e.g., libzmq), and retags the wheel as manylinux_2_28_x86_64.

macOS wheels are built natively on a macos-14 (arm64) GitHub runner using delocate for the equivalent bundle-and-retag step.

Portability gate¶

Every wheel is scanned for non-dispatched wide-SIMD instructions:

bad=$(objdump -d "$so" | awk '
  /vmovaps|vfmadd|vaddps/ && /ymm|zmm/ { print }
' | grep -v '_avx2\|_avx512')
[ -z "$bad" ] || { echo "AVX leak: $bad"; exit 1; }

Runtime-dispatched variants (function names containing _avx2 or _avx512) are exempted. Anything else that touches ymm/zmm registers fails the release.

CI pipeline¶

ci.yml runs on every push to main/develop and on all pull requests.

The ci-passed aggregator job is what release.yml polls. It exists so the release pipeline has a single stable job name to wait on, regardless of how many individual jobs are added or renamed in the future.

Release pipeline¶

release.yml triggers on a version tag push (v*.*.*). It does not re-run tests — it trusts that the PR CI gate was green.

Troubleshooting¶

AVX-512 portability failure in build-python¶

Symptom: objdump scan finds ymm/zmm instructions outside _avx2/_avx512 function names.

Cause: A CMake target or a vendored library was compiled with -march=native (or a flag like -mavx2 applied globally).

Fix:

grep -r "march=native\|mavx2\|mavx512" native/ CMakeLists.txt
# remove the offending flag, or gate it on DOPPLER_NATIVE
cmake -B build -DDOPPLER_NATIVE=OFF && cmake --build build

GLIBC symbol floor broken (glibc-228 job fails)¶

Symptom: ImportError: GLIBC_2.29 not found inside the Debian 10 container.

Cause: A new syscall wrapper or libc function (e.g., close_range, statx) was called directly or pulled in by a dependency.

Find the culprit:

nm build/libdoppler.so | grep 'GLIBC_2\.29\|GLIBC_2\.3[0-9]'

Fix: add a shim or avoid the high-version symbol; check what vendored code added the call.

jm version skew (manifest-drift job fails)¶

Symptom: jm status --check reports drift that you cannot reproduce locally.

Cause: Your local jm CLI version differs from the pinned jm_version in just-makeit.toml.

Fix: always drive doppler with the pinned version:

JM_VERSION=$(grep jm_version just-makeit.toml | grep -oP '"\K[^"]+')
uvx --from "just-makeit==$JM_VERSION" just-makeit status --check

verify-ci times out¶

Symptom: release.yml polls for ci-passed but never finds it.

Cause: The CI run on the tagged commit either never ran, is still in progress, or failed. verify-ci polls the tagged commit SHA — if CI ran on the PR branch's HEAD but the merge produced a different SHA, the poll never finds it.

Fix: check https://github.com/doppler-dsp/doppler/actions filtered to the tagged SHA. If CI failed, fix the issue and re-tag (see below).

Re-running a failed release¶

If release.yml fails before publish-python, it is safe to delete the tag, fix the issue, and re-tag:

git push origin :refs/tags/vX.Y.Z  # delete remote tag
git tag -d vX.Y.Z                   # delete local tag
# fix the issue, then:
make tag-release VERSION=X.Y.Z      # re-tag and push

Replicating the wheel build locally¶

To build a Linux manylinux wheel identical to what CI produces:

# Run in the manylinux container (same image CI uses):
docker run --rm -v $PWD:/src -w /src \
  quay.io/pypa/manylinux_2_28_x86_64 \
  bash -c "
    /opt/python/cp312-cp312/bin/pip install uv &&
    JUST_BUILDIT_PYTHON=/opt/python/cp312-cp312/bin/python3 \
    make just-build JUST_BUILDIT_OUTPUT_DIR=/tmp/out &&
    uvx auditwheel repair /tmp/out/*.whl -w dist/
  "

# Verify the built wheel:
pip install dist/doppler_dsp-*.whl --force-reinstall
python -c "import doppler; print(doppler.__version__)"

For a quick local wheel without manylinux constraints (dev iteration):

uv build --wheel
pip install dist/doppler_dsp-*.whl --force-reinstall