Adding an algorithm — the lifecycle¶

Fifteen pages under docs/dev/ describe how to do each part of this well. None of them says what order the parts go in, or which one owns a thing you are looking for — so this page is the spine, and it owns exactly two things:

• the order the phases happen in, and
• who owns the how for each of them.

It deliberately contains no mechanics. Every step below links to the one page that owns it, and where a step's home is a make target or a script, it names that instead. If you find yourself adding a command here, that command belongs on the page this one links to — a second copy of a procedure is the thing that goes quietly out of date, and this page exists to make the map readable, not to become a sixteenth copy of the territory.

Start with why¶

Before any code, write docs/design/<algo>.md:

• the use cases — who calls this, with what, and what they do with the answer;
• the design goals, and the unknowns — say plainly which numbers you do not know yet. The unknowns are what the characterization later measures, and writing them down first is what stops a sweep being designed to confirm a decision already made;
• a Python prototype to de-risk — throwaway, in a scratch directory, to find out whether the idea works at all. It is not the implementation and it is not committed; see Repository Map for why the algorithm lands in C exactly once;
• a sketch of the implementation plan — the object's shape, its state, and which existing primitive it composes rather than re-implements.

This phase has no gate, and a design doc is enforced by review alone — which is precisely why it goes first: it is the cheapest place to be wrong. The design page is also what section 1 of the eventual validation report links to, rather than restating.

The design page states what is. The dated record — what each step measured, in the order it was measured, the wrong guesses and what corrected them — accrues on a companion, docs/design/<algo>-measurements.md, with section numbers shared between the two so a §12.17 cited in an issue or a code comment is the same entry on both. async-dsss-receiver-measurements.md is the shape. A design page that carries its own history stops stating anything: a reader has to replay it to find out what is true today.

It is not, however, the only ungated phase — phase 4's sabotage is the other, and that one is a gap rather than a design.

The lifecycle at a glance¶

CI runs the merge-guarding set; the per-phase targets above are how you find out sooner. (make gates reproduces that set locally, and is only for debugging a CI red: How We Work.)

The phases, and what is decided in each¶

2 — Declare, and why the header is still the spec¶

The interface is declared in a manifest and jm apply scaffolds from it — the C stub, the binding, the CMake, the stubs, the test and benchmark skeletons. That makes the manifest the thing you edit and the generated files the things you do not.

It does not make the manifest the specification. The C header is the SSOT: it is where the contract is written in prose a caller reads, and it is what the certification later audits claim by claim. The manifest declares a shape; the header states what the object promises.

3 — Implement, C first¶

One algorithm, one implementation, in C. Wrappers are glue and never re-implement logic — and that rule holds inside C too: two modules must not grow private copies of the same primitive.

The header you write here is a deliverable, not a comment. Its Doxygen becomes the Python docstring on both faces, so writing it well is cheaper than fixing it twice — Docstring Authoring is the home for that.

Its @code blocks are tests. A >>> block is the binding's example: jm flows it into the .pyi as an Examples section and make test-stubs executes it. A block written in C is compiled -Werror against libdoppler.a and run by make test-snippets (test_c_header_code.py; a fragment with no main is wrapped in one). Either way an example that drifts fails a gate rather than misleading a reader quietly. Write them against a real run — pin the expected output by running the thing, never by reasoning about what it should print. That is phase 9's first deliverable and it is cheapest to write here, while the behaviour is in front of you.

4 — Pin, and prove the pin¶

Write the C tests, then break the implementation and watch each one go red. A test you have not seen fail is not evidence. Object Validation owns this rule and its two failure modes (consistency tests that are blind to a shared defect; reject tests that pass vacuously).

Nothing executes this, and you have to supply the discipline. ctest proves a test passes; no gate proves it can fail. The rule is stated on this page, in Object Validation step 2, and again in Measuring a Receiver — and enforced on none of them, which is the shape validation.md opens by warning about. The cost is measurable: six tests in an already-certified object were later found to pass against a receiver whose carrier discriminator did nothing at all (gh-843). Until a gate exists, treat the sabotage as part of writing the test, not as a review step somebody else performs.

A guard can pass because it never ran. The blob test for rate_hypotheses (the count is a reject key) was first written as "if the two engines' sizes agree, expect DP_ERR_INVALID" -- and they never agreed, so the assertion was skipped and the sabotage that dropped the key read green. It was found only because every guard of that change was sabotaged once. Build the case so the precondition is a DP_REQUIRE (here, an N = 1 engine against an N = 2 one, which land on the same depth), never an if around the assertion. A Bash chain whose hook-blocked tail takes its first half with it did the same thing to a different edit; read the file, not the green.

A guard can pass because the sabotage does not reach it. The capture test for the engine's own rates ran at a noise level far under the burst, so the engine sized itself to D = 1 and detected with or without its de-chirp; zeroing the de-chirp phasors read green. Moving the noise to the design C/N0 made the engine run deep and its rates decide the detection (57 of 60 bursts placed against 6 of 60 without them). A sabotage that stays green says the test is vacuous, not that the code is safe: change the stimulus until it goes red. The clamp test had the converse trap: at a rate of 1e30 the size_t cast wraps to a small number and the unclamped code looked fine; at 1e3 it asks for hundreds of gigabytes and dies, which is what a guard should do.

A guard whose premise is a float overflow is build-dependent. The NaN-pick guard first reached a NaN statistic by feeding samples whose power overflows float. In the Release build it did; in every Debug build (the sanitizers, coverage, macOS, Windows) the constant genuinely detected, and CI went red on one assertion. Pin the rule, not the overflow: the pick is a one-line static inline in a private header (acq_private.h, the det_private.h pattern: nothing exported, nothing promised to a caller) and the test calls it with NaN, infinity and finite values on every build.

Do this before the Python face exists. A test written through the binding measures whatever the binding happens to expose, and the surfaces that matter most are routinely the ones with no binding at all.

5 — Bind, by declaring it¶

The Python face is declared, not written. objects/<obj>.toml says what the binding exposes, and make jm-apply renders it: the _ext fragment, the .pyi and the __init__.py re-exports. Three things follow.

• A behaviour is a manifest key, not a hand patch. Releasing the GIL, an output buffer that grows on demand, a second constructor over the same core, the state triplet and an enum decoded to its name are each a key: nogil, variable_output, [[<obj>.views]], serializable, enum. An adopted fragment carries fragment = "generated", and from then on a hand edit turns make drift-check red. A behaviour jm cannot render yet is a jm issue, not an edit.
• The binding has no prose of its own. Its docstrings are the header's Doxygen on both faces (phase 3), so a wrong docstring is fixed in the header and re-rendered, and make test-stubs runs every >>> example in the stubs.
• A borrowed handle is the binding's to keep alive. A C object that borrows a caller's context, a telemetry context say, writes through it for as long as it is attached. Declaring that argument as a capsule with retain = true makes the Python object hold a reference to it for exactly that long, so a caller cannot free it underneath the C side.

objects/async_dsss_receiver.toml shows all of it in one manifest: set_telemetry takes a retained doppler.telemetry.dp_tlm capsule and turns a refusal into a ValueError (status_return); the lifecycle property decodes the C state through enum = "async_dsss_rx_state"; serializable generates the state triplet; and fragment = "generated" hands the fragment to jm.

The how is Adding a Module (declare, apply, the public surface, the Python tests), Module Layout (what each generated file may contain) and Docstring Authoring.

6 — Instrument¶

Two things every stateful object owes its callers, and both are easy to retrofit badly and hard to retrofit late:

• the state triplet — state_bytes / get_state / set_state, so the object can be checkpointed and resumed. Required for anything carrying running state;
• telemetry probes — the internals a caller cannot otherwise see.

The rule worth knowing here is that a published diagnostic and a useful one are not the same thing: an object can publish two views of its own state that are secretly the same view. Check what a composing receiver would actually need to debug you.

Telemetry is a set_telemetry (state, tlm, prefix, decim) you write once. A leaf registers its probes under the caller's prefix, joining every name through dp_tlm_name_join() before the first registration, and emits at event rate (a gain update, a symbol, a code period), never more than once an input sample, which is the bound a lossless capture is sized from. A composition forwards the attach to each child under a dotted sub-prefix, keeps the request so a rebuilt child is attached again, refuses the whole attach when one name does not fit, and adds the probes only it knows. The pattern, with the AGC as the leaf and AsyncDsssReceiver as the composition, is Telemetry: instrumenting an object. What to decide here:

• Pair a decision with the statistic it thresholds. .locked next to .lock lets a reader see where the rule fired without re-deriving the threshold; a lifecycle state next to the loops says which stage a symptom belongs to.
• Say the units and the cadence in the header, probe by probe. A probe in cycles a sample is read as Hz unless the header says otherwise, and a probe reported once a call reads as one a symbol unless the header says which.
• Pin the list. Every name the attach registers belongs in the header's set_telemetry() doc, and a C test that walks the registry against that list keeps the two together (_test_tlm_names_documented in test_async_dsss_receiver_core.c). No repository-wide gate does this yet (#2478).

The consumer's side, a capture read back into rows and drawn with scripts/plot_capture.py, is the lifecycle gallery page and the Python telemetry API.

A create-time count that changes what a surface means is a reject key, and a constructor argument. rate_hypotheses (the burst engine's Doppler-rate search, burst-capture §11.5) sizes the grid and decides which surface a blob describes, so the blob carries it and refuses a blob with another (acq state v6). rate_hypotheses = 0 asks the engine to choose it: the fewest hypotheses whose residual no longer limits the depth the sizer would pick (burst-capture §11.5). What is stored and compared is the CHOSEN count, read back through the same property, so a blob still names exactly one grid; the choice is made once, at create, from the sizer's own functions, never restated. It is an argument and not a setter on purpose: a setter is a mutator the #2022 gate probes, a view with no row in the serialization matrix cannot be probed, and a mid-stream re-size has to reset everything the engine composes.

A new running field is a blob version bump, and the field you forgot to pack is found by the mid-stream split test, not by the round trip: a split that resumes from get_state / set_state and must match the unsplit run bit-for-bit fails the moment a field that shapes the output is missing from the blob. acq's raw block was found that way (test_acq_core.c); the round trip alone would have restored everything it packed and said nothing about what it did not.

7 — Explore, before you certify¶

This is the phase the word "characterization" names, and it is where a one-time thorough sweep belongs: the whole envelope, enough trials to be statistically meaningful, minutes of runtime. It answers how does this behave, which is a different question from what may a caller rely on.

Two homes, and which one depends on the language of the answer:

• native/validation/<obj>_*.c when the surface is C-only — an inlined discriminator, a static helper, anything with no binding. Run in full by make validate-c, and by ctest as a fast regression subset.
• src/doppler/<module>/tests/characterization/<subject>/ when it is a long Monte-Carlo sweep over an object that does have a binding. Run by make characterize and by nothing else, deliberately.

A probability the caller will rely on -- a Pd, a Pfa -- is reported with its interval, not as a point, and the sweep's rows include the worst case rather than only the average. The exemplar is native/validation/acq_rate_pd.c: Wilson 95% on every row, a rate placed exactly between two hypotheses next to one drawn uniformly, a noise-only Pfa row per configuration, and the same assertions in --check and in the full sweep (a sweep that asserts nothing fails make validate, #1566). Where an effect is smaller than the interval -- two refinements worth 0.02 of Pd against a +-0.04 interval -- the trials are seeded, so the effect is measured paired, by ablating it on the identical trials. Then look for the scenario where it decides the outcome, and guard that deterministically: the two refinements are held in test_burst_capture_core.c by cases a search of 4,536 scenarios under each ablation found, where removing either names the wrong repetition (#2219). Only when no such scenario exists is the seeded point estimate pinned, as a regression pin, not a significance test, and the validator says which it is. A seeded pin can also stay beside exact guards as a broad backstop: the validator's one now watches for any regression of 0.015-0.04 that its intervals cannot see, not only the two the tests hold.

Two more habits the same validator shows. Run the claim on a second signal: the compensation was measured on a Zadoff-Chu preamble, but the downstream preamble is a repeated chirp, so a row swaps the preamble for the generator's chirp (one period of wfm_synth, never a hand-written phase) and asserts the same things -- the capped depth loses, the compensated one beats it outside the intervals, Pfa is not above the target. Gate what the contract claims and print the rest: the reported rate is right only from a block wholly inside the preamble and well above the design point, so the +23 dB rows are asserted (95% of at least 40 such hits) and the design-point rows, which read 30% to 66%, are printed and not claimed. A row that measures the contract beats a prose caveat that nothing runs.

"Deliberately" is the whole point of both rows: neither runs on a push. What CI gets from this phase is the --check subset and the fast tests you write from what the sweep measured — see NO FULL VALIDATION OR CHARACTERIZATION IN CI before you wire anything here into a job.

8 — Certify¶

Now, and only now, the evidence a caller is handed: header claims enumerated, mapped onto the C tests, then measured, reviewed and asserted. Object Validation owns the whole of it, including the report's five sections, the executive summary that opens it, and the two gates.

The order matters and is not negotiable — header, then C tests, then the Python report. Going the other way produces evidence in the wrong language for the gate, and has cost a rewrite before.

An algorithm with no Python face is still certified, and does not grow a binding to become so. conv was the first: a binding built only to be measured is one nobody calls, and the campaign would then be certifying an artifact of its own process. The substitution is one line long — a C harness under native/validation/ measures and emits, and a validator under src/doppler/tests/validation/<obj>/ renders and asserts through the same Report — so the report format, both gates and the five sections are unchanged. Object Validation has the table.

The report's harness call is the C harness's regression subset — the --check --emit run that ctest also makes — and that one run is what the limits gate, the staleness gate and results.md all see, so the evidence and the gate cannot disagree. The population sweep is make validate-c's and the record page's, refreshed on purpose. This is a cost decision, measured: 20 s of ten emitters was 3.3 min on twenty cores, against CI Python jobs of 4–7 min each; a validator runs twice on every push.

9 — Document it, and carry it back¶

A finding that reaches only the report reaches nobody, and an object nobody can find is an object nobody uses. Five deliverables, each with the gate that keeps it honest — none of them optional except where the table says so:

Three things about that table are worth knowing before you trip over them.

The two whens are the only judgement calls. A guide earns its place when the object needs prose a docstring cannot hold — a derivation, a comparison, a decision a caller has to make. A composition of objects earns a user's guide under docs/guide/ on findability rather than on prose: a caller who wants to track a population of emitters does not know to look for a pool whose docstring is split across the objects it composes. A gallery page earns its place when there is a figure that shows something a number cannot. Everything else on the list is owed unconditionally, and "it is obvious from the code" has never once been true for the person reading it six months later.

A benchmark is a claim, and an unmeasured claim is a guess. jm apply scaffolds both files for you, so the deliverable is filling them in, not creating them. Doing it now is also the only cheap moment: a number nobody took before the algorithm was tuned cannot be recovered afterwards.

Both examples are DISCOVERED, and both are owed. src/doppler/examples/*.py is globbed and native/examples/*.c is too, so a new example is gated the moment it exists; opting one out costs an entry in .examples-skip or C_EXAMPLE_SKIPS with a mandatory reason, and an entry naming a deleted script fails a meta-test. The C side used to iterate a hand-written list instead, which left four of thirteen programs compiling, shipping and executed by nothing (gh-863, since fixed) — so discovery is not a convenience here, it is what makes forgetting survivable.

C and Python, not either. This asked for one of the two until 2026-08-26, and one is not enough: the two faces fail differently. A C example is where a caller sees the lifecycle it must actually manage — create, feed, drain, destroy, and who owns the buffer — none of which the Python binding exposes because it does that work for you. A Python example is where the result is legible: a plot, an assertion against a reference, a number a reader can check. An object with only the C one ships an algorithm nobody can see the output of; with only the Python one it ships an API nobody can call from C without reading the header and guessing.

Then carry the findings back. Whatever the certification established — a new limit, a corrected rule, a number a caller has to choose by — goes into the C test and into one of the artifacts above, because those are the two things that keep it true and put it in front of someone.

10 — Land it¶

The CHANGELOG entry lands with the change, not at release time, and it is a fragment, changelog.d/<section>/<slug>.md — one file per entry, so two PRs never touch the same line (Release owns the format and the assembly). A commit must also say what it closes, Closes #N, or say that it closes nothing, No-issue:; make issue-link-check refuses the silence between the two, because silence is how fixed work stays open in the backlog. It reads only the branch's OWN commits: one that is a base commit under a new hash (the same author, date and subject), which is what an --amend after a hook-blocked commit produces, fails outright rather than lending the branch that commit's declaration. Anything found and deliberately not fixed gets filed as an issue before the PR merges, never explained in a docstring — a carve-out written into a comment is invisible to everyone who did not already know to look for it.

Four places evidence lives¶

The single most confusing thing about this repo for a newcomer, and it is worth memorising:

The split between rows three and four is a cost decision: a validator runs twice on every push, so a sweep that needs minutes to say anything taxes every push for an answer nobody asked for on that push.

What each place already gives you¶

The family of shared helpers exists so that no harness renders its own waveform, rolls its own noise, or invents its own assertion — and it is easy to miss, because each member lives beside the tests that use it rather than on a page. A measurement built on a private copy of any of these is not wrong at first; it is wrong the day the shipped one is fixed and the copy is not (native/tests/README.md is the family's own page and the authority on the C side). The table is by phase: reach for the row before writing a line of the thing it names.

Two rules the table implies, stated so they are not inferred:

• A harness renders nothing. If the signal under test is a waveform this library can generate, the harness calls the generator. The first pass of the continuous async-DSSS validators (async-dsss-receiver-measurements.md §12) built its emitter from chips by hand and derived its own noise sigma; the shipped path gave the same numbers to within trial spread, and the hand-rolled one was deleted — not because it was wrong that day, but because nothing would have told anyone when it became wrong.
• A harness asserts through dp_test.h. DP_CHECK counts, reports with file and line, and DP_TEST_END refuses a run that checked nothing. A printf("FAIL ...") and a return code do neither.

One word, three meanings¶

"Characterize" means three different things in this repository, and nothing until now said so:

1. classify the algorithm's I/O shape — can one sample be processed independently with small fixed state? — which is what picks the jm preset (Adding a Module, its step 0);
2. phase 1 of a validation report — measure behaviour, tables and plots, no verdicts (Object Validation);
3. the long-sweep category — tests/characterization/, make characterize, phase 7 above.

Sense 1 is unrelated to the other two and merely shares a name. When writing, prefer "classify the object's shape" for the first and reserve "characterization" for senses 2 and 3.

What this page does not own¶

Nothing on this list is described here, and adding a description of one to this page is the failure mode to watch for:

• how to run jm, what the manifest keys mean, what jm apply regenerates — Adding a Module
• where files go and what .pyi / __init__.py may contain — Module Layout
• how to write the header Doxygen so the docstrings come out right — Docstring Authoring
• how errors cross the C ABI — Error Convention
• what is generated under docs/ and what is hand-owned — Docs Conventions
• how snippets and examples are discovered and tested — Doc Examples
• how to benchmark, and how to compare two snapshots honestly — Benchmarking
• which objects are certified today — Validation Log