Python Streaming API¶
doppler.stream moves sample blocks between processes (or hosts) over
NATS, in three subject-based patterns that share one wire format — a
dp_header_t (sample rate, centre frequency, sample type, an auto
timestamp_ns, and a per-socket sequence counter) followed by the raw
sample bytes. Because the header is shared, a C transmitter and a Python
receiver interoperate freely.
Every pattern needs a nats-server reachable at the endpoint's
host:port — plain nats-server is enough for PUB/SUB and REQ/REP;
PUSH/PULL rides the JetStream work-queue tier, so start it with
nats-server -js. Endpoints are nats://host:port/subject.
Source:
src/doppler/stream/__init__.py
For the C side and a runnable end-to-end walk-through, see the Streaming examples.
Socket patterns¶
| Pattern | Sender | Receiver | NATS tier | Use |
|---|---|---|---|---|
| PUB / SUB | Publisher |
Subscriber |
Core NATS | fan-out broadcast; subscribers may drop |
| PUSH/PULL | Push |
Pull |
JetStream work-queue | load-balanced pipeline; durable, acked |
| REQ / REP | Requester |
Replier |
NATS request/reply | request/response (lock-step) |
Every socket is a context manager and releases the GIL while blocked
waiting on NATS, so receive loops thread cleanly. The sender's
sample_type fixes the NumPy dtype on both ends:
| Constant | dtype | layout |
|---|---|---|
CF32 |
numpy.complex64 |
complex I/Q |
CF64 |
numpy.complex128 |
complex I/Q |
CF128 |
numpy.clongdouble |
extended-precision complex |
CI8 |
numpy.int8 |
interleaved I/Q, length 2n |
CI16 |
numpy.int16 |
interleaved I/Q, length 2n |
CI32 |
numpy.int32 |
interleaved I/Q, length 2n |
TLM16 |
structured rows | telemetry records, not I/Q (Telemetry) |
recv() returns (samples, header) where header is a dict of the decoded
dp_header_t fields (sample_rate, center_freq, sample_type,
timestamp_ns, sequence).
PUB / SUB — broadcast¶
The publisher fans out to every subscriber on the subject; a slow
subscriber drops frames rather than back-pressuring the sender. Core NATS
has no backlog, so a subscriber must already be listening before the
publish — start the Subscriber first (or add a brief warm-up sleep).
import numpy as np
from doppler.stream import Publisher, Subscriber, CF64
# transmitter
with Publisher("nats://127.0.0.1:4222/iq", CF64) as pub:
iq = np.exp(2j * np.pi * 1e3 * np.arange(1000) / 1e6) # complex128
pub.send(iq, sample_rate=1e6, center_freq=2.4e9)
# receiver — in another process, started before the publish above
with Subscriber("nats://127.0.0.1:4222/iq") as sub:
samples, header = sub.recv(timeout_ms=1000)
print(header["sample_rate"], header["sequence"], len(samples))
PUSH / PULL — pipeline¶
PUSH publishes onto a durable JetStream work-queue subject; JetStream
load-balances each frame to exactly one connected PULL worker, giving a
back-pressured, at-least-once work queue. Call Pull.ack() once a frame
has been fully processed. Requires a JetStream-enabled broker
(nats-server -js).
import numpy as np
from doppler.stream import Push, Pull, CF64
with Push("nats://127.0.0.1:4222/work", CF64) as push:
push.send(np.zeros(4096, dtype=np.complex128), sample_rate=1e6)
with Pull("nats://127.0.0.1:4222/work") as pull:
samples, header = pull.recv() # blocks until a frame arrives
pull.ack(samples)
REQ / REP — request/response¶
A Requester sends a sample block and blocks for the Replier's response;
the two alternate strictly, over NATS's native request/reply.
import numpy as np
from doppler.stream import Requester, Replier, CF64
# server: receive a request, send a reply
with Replier("nats://127.0.0.1:4222/ctrl", CF64) as rep:
req, header = rep.recv()
rep.send(req * 2) # echo-and-scale, say
# client: send, then receive the reply
with Requester("nats://127.0.0.1:4222/ctrl", CF64) as req:
req.send(np.ones(256, dtype=np.complex128), sample_rate=1e6)
resp, header = req.recv()
Timestamps¶
get_timestamp_ns() returns the same CLOCK_REALTIME nanosecond stamp the
senders embed in each header, so a receiver can compute end-to-end latency
against header["timestamp_ns"].
PUB / SUB¶
Publisher
¶
NATS core PUB — one-to-many broadcast of signal frames.
Wraps dp_pub_t. Each :meth:send call stages one
[dp_header_t][raw data] buffer and publishes it to the
endpoint's iq.<subject>.<type> NATS subject. Multiple
:class:Subscriber sockets can subscribe to the same subject;
each subscriber receives every frame. NATS core has no
persistence or back-pressure: a slow subscriber simply misses
frames published while it isn't reading.
The endpoint identifies a subject on a NATS broker; both
Publisher and Subscriber connect to the same nats-server (no
bind/connect distinction — the broker mediates fan-out).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding. One of :data: |
...
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If |
Examples:
Construct, use as a context manager, and verify the type:
>>> from doppler.stream import Publisher, CF64
>>> pub = Publisher("nats://127.0.0.1:4222/t19100", CF64)
>>> type(pub).__name__
'Publisher'
>>> pub.close()
Context-manager form (preferred — ensures close on exception):
Full send round-trip with a :class:Subscriber (requires a live
nats-server and a brief warm-up sleep so the subscription is
established before the first publish):
>>> import numpy as np, time
>>> from doppler.stream import Subscriber
>>> pub = Publisher("nats://127.0.0.1:4222/iq", CF64)
>>> sub = Subscriber("nats://127.0.0.1:4222/iq")
>>> time.sleep(0.1)
>>> pub.send(np.array([1+2j, 3+4j],
... dtype=np.complex128),
... sample_rate=int(1e6),
... center_freq=int(2.4e9))
>>> samples, hdr = sub.recv(timeout_ms=2000)
>>> pub.close(); sub.close()
__init__
¶
Create a Publisher and connect to endpoint's NATS broker.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding: :data: |
...
|
__enter__
¶
send
¶
send(
samples: NDArray[Any],
sample_rate: float = 0,
center_freq: float = 0,
timestamp_ns: int | None = None,
) -> None
Broadcast one block of samples to all connected Subscribers.
Constructs a dp_header_t with the supplied metadata (plus a
per-socket monotonically increasing sequence number), stages
one [header][raw sample bytes] buffer (NATS has no scatter/
gather send, so header and data are copied into one contiguous
buffer first), and publishes it. The call releases the GIL
while blocked in the underlying NATS client, so other Python
threads can run concurrently.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
ndarray
|
C-contiguous array whose dtype must match the socket's
|
required |
sample_rate
|
float
|
Samples per second written into the header (default 0). |
0
|
center_freq
|
float
|
Centre frequency in Hz written into the header (default 0). |
0
|
timestamp_ns
|
int
|
UNIX nanoseconds to stamp on |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If |
RuntimeError
|
If the publish fails (e.g. the broker connection dropped). |
Examples:
close
¶
Destroy the underlying NATS handle and release all resources.
Calls dp_pub_destroy(). Safe to call multiple times —
subsequent calls are no-ops. After close() the object
must not be used for sending.
Examples:
Subscriber
¶
NATS core SUB — receives signal frames from one or more Publishers.
Wraps dp_sub_t. Subscribes to the endpoint's iq.<subject>.>
NATS subject, so it receives every frame any :class:Publisher bound
to that subject sends (fan-out: every Subscriber on the subject gets
every frame). There is no "connect to exactly one Publisher" concept
— the broker mediates delivery by subject, not by socket.
For load-balanced consumption (each frame delivered to exactly one
of several workers, with durable redelivery) use :class:Pull
instead, which is backed by NATS JetStream rather than NATS core.
The recv path is zero-copy: the returned NumPy array's memory is
owned by an internal dp_msg_t handle that points into the
received NATS message; the buffer is freed only when the array
(and all views of it) are garbage-collected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If |
Examples:
>>> from doppler.stream import Subscriber
>>> sub = Subscriber("nats://127.0.0.1:4222/t19104")
>>> type(sub).__name__
'Subscriber'
>>> sub.close()
Context-manager form:
Receive one frame (requires a live :class:Publisher):
__init__
¶
Create a Subscriber socket and connect to endpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g.
|
required |
__enter__
¶
recv
¶
Receive one signal frame from the connected Publisher.
Blocks until a frame arrives or the optional timeout expires. The returned NumPy array is a zero-copy view into the received NATS message buffer; the buffer is freed when the array is garbage-collected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_ms
|
int
|
Milliseconds to wait before raising :exc: |
-1
|
Returns:
| Name | Type | Description |
|---|---|---|
samples |
ndarray
|
Decoded sample data. dtype is |
header |
dict
|
Decoded
|
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If |
RuntimeError
|
If the underlying NATS recv fails for any other reason. |
Examples:
close
¶
PUSH / PULL¶
Push
¶
NATS JetStream work-queue producer — durable pipeline sender.
Wraps dp_push_t. Each :meth:send is a synchronous, server-
acked JetStream publish: the frame is persisted (and replicated, on
a clustered broker) before the call returns, so a producer crash
never silently drops a frame. Each persisted frame is later
delivered to exactly one :class:Pull worker (competing-consumers
distribution), unlike :class:Publisher which fans out to every
Subscriber.
Use the PUSH/PULL pattern when you have a pool of workers pulling
from a shared, durable queue and want at-least-once delivery with
redelivery on worker crash (see :meth:Pull.ack).
Both ends address the same NATS work-queue subject; there is no bind/connect distinction (the broker mediates delivery).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding. One of :data: |
...
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If |
Examples:
>>> from doppler.stream import Push, CF64
>>> push = Push("nats://127.0.0.1:4222/t19108", CF64)
>>> type(push).__name__
'Push'
>>> push.close()
Context-manager form:
Round-trip with a :class:Pull worker (requires a live connection):
>>> import numpy as np, time
>>> from doppler.stream import Pull
>>> push = Push("nats://127.0.0.1:4222/work", CF64)
>>> pull = Pull("nats://127.0.0.1:4222/work")
>>> time.sleep(0.05)
>>> push.send(np.ones(4, dtype=np.complex128))
>>> samples, hdr = pull.recv(timeout_ms=2000)
>>> push.close(); pull.close()
__init__
¶
Create a Push socket and bind to endpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding: :data: |
...
|
__enter__
¶
send
¶
send(
samples: NDArray[Any],
sample_rate: float = 0,
center_freq: float = 0,
timestamp_ns: int | None = None,
) -> None
Durably publish one block of samples to the work-queue.
Blocks until the broker acks that the frame is persisted (and replicated, on a clustered broker) — not until a Pull worker is ready; the frame waits in the queue for the next available worker. Releases the GIL while blocked on the broker round trip.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
ndarray
|
C-contiguous array whose dtype must match the socket's
|
required |
sample_rate
|
float
|
Samples per second written into the header (default 0). |
0
|
center_freq
|
float
|
Centre frequency in Hz written into the header (default 0). |
0
|
timestamp_ns
|
int
|
UNIX nanoseconds to stamp on |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If |
RuntimeError
|
If the publish fails (e.g. the broker connection dropped). |
Examples:
>>> from doppler.stream import Push, Pull, CF64
>>> import numpy as np, time
>>> push = Push("nats://127.0.0.1:4222/work", CF64)
>>> pull = Pull("nats://127.0.0.1:4222/work")
>>> time.sleep(0.05)
>>> push.send(np.array([1+2j, 3+4j],
... dtype=np.complex128),
... sample_rate=int(48000))
>>> samples, hdr = pull.recv(timeout_ms=2000)
>>> push.close(); pull.close()
close
¶
Pull
¶
NATS JetStream work-queue consumer — durable pipeline receiver.
Wraps dp_pull_t. Consumes frames persisted by one or more
:class:Push producers. Multiple Pull workers can share the same
durable consumer group; each frame is delivered to exactly one
worker. Call :meth:ack once a frame is fully processed — an
un-acked frame is redelivered if the worker dies first.
The recv path is zero-copy: see :class:Subscriber for the buffer
lifetime semantics.
Both ends address the same NATS work-queue subject; there is no bind/connect distinction (the broker mediates delivery).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If |
Examples:
>>> from doppler.stream import Push, Pull, CF64
>>> _ = Push("nats://127.0.0.1:4222/t19112", CF64) # provisions the queue
>>> pull = Pull("nats://127.0.0.1:4222/t19112")
>>> type(pull).__name__
'Pull'
>>> pull.close()
Context-manager form:
>>> _ = Push("nats://127.0.0.1:4222/t19113", CF64) # provisions the queue
>>> with Pull("nats://127.0.0.1:4222/t19113") as pull:
... type(pull).__name__
'Pull'
Receive one frame (requires a live :class:Push):
__init__
¶
Create a Pull socket and connect to endpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g.
|
required |
__enter__
¶
recv
¶
Receive one signal frame from the connected Push socket.
Blocks until a frame arrives or the optional timeout expires. The returned NumPy array is a zero-copy view into the received NATS message buffer; the buffer is freed when the array is garbage-collected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_ms
|
int
|
Milliseconds to wait before raising :exc: |
-1
|
Returns:
| Name | Type | Description |
|---|---|---|
samples |
ndarray
|
Decoded sample data. dtype is |
header |
dict
|
Decoded |
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If |
RuntimeError
|
If the underlying NATS recv fails for any other reason. |
Examples:
ack
¶
Acknowledge a frame consumed from the JetStream work-queue.
Delivery is at-least-once: a frame stays pending until acked
and is redelivered to another worker if this one dies first.
Pass the array returned by :meth:recv once it has been fully
processed, then drop the array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
ndarray
|
The samples array returned by :meth: |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If the acknowledgement fails. |
Examples:
close
¶
Destroy the underlying NATS handle and release all resources.
Calls dp_pull_destroy(). Safe to call multiple times.
Examples:
REQ / REP¶
Requester
¶
NATS request/reply — sends a request frame, then waits for a reply.
Wraps dp_req_t. Built on a NATS request: :meth:send publishes
to the endpoint's subject with a reply-to address (a dedicated
inbox created for this Requester), and :meth:recv waits on that
same inbox for the :class:Replier's answer. Use this pattern
strictly alternating (send, then recv, then send again) — unlike
ZMQ's REQ socket there is no enforced state machine, but calling
recv before a matching send simply blocks/times out on an
empty inbox rather than raising immediately.
Complements :class:Replier. Use this pattern for control-plane
messages (tuning commands, metadata queries) or synchronous
signal-frame RPC where one peer processes each frame and returns a
result.
Both ends address the same NATS subject; there is no bind/connect distinction (the broker mediates delivery via the reply-to inbox).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding of frames sent by this socket. One of
:data: |
...
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If |
Examples:
>>> from doppler.stream import Requester, CF64
>>> req = Requester("nats://127.0.0.1:4222/t19116", CF64)
>>> type(req).__name__
'Requester'
>>> req.close()
Context-manager form:
Full REQ/REP round-trip (requires a live :class:Replier):
>>> import numpy as np, time
>>> from doppler.stream import Replier
>>> rep = Replier("nats://127.0.0.1:4222/ctrl", CF64)
>>> req = Requester("nats://127.0.0.1:4222/ctrl", CF64)
>>> time.sleep(0.05)
>>> req.send(np.ones(4, dtype=np.complex128),
... sample_rate=int(1e6))
>>> request, hdr = rep.recv(timeout_ms=2000)
>>> rep.send(request)
>>> reply, hdr2 = req.recv(timeout_ms=2000)
>>> req.close(); rep.close()
__init__
¶
Create a Requester socket and connect to endpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g.
|
required |
sample_type
|
int
|
Wire encoding of frames sent by this socket:
:data: |
...
|
__enter__
¶
send
¶
send(
samples: NDArray[Any],
sample_rate: float = 0,
center_freq: float = 0,
timestamp_ns: int | None = None,
) -> None
Publish a request frame with this Requester's reply-to inbox.
Use strictly alternating with :meth:recv (send, recv, send,
recv, ...) — sending again before consuming the previous reply
works mechanically but leaves an unread message in the inbox
for the next :meth:recv to pick up, which will desync the
request/reply pairing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
ndarray
|
C-contiguous array whose dtype must match the socket's
|
required |
sample_rate
|
float
|
Samples per second written into the header (default 0). |
0
|
center_freq
|
float
|
Centre frequency in Hz written into the header (default 0). |
0
|
timestamp_ns
|
int
|
UNIX nanoseconds to stamp on |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If |
RuntimeError
|
If the publish fails (e.g. the broker connection dropped). |
Examples:
recv
¶
Receive the reply frame from the :class:Replier.
Waits on this Requester's reply-to inbox. Call after
:meth:send; calling without a prior send just blocks (or
times out) on an empty inbox rather than raising immediately.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_ms
|
int
|
Milliseconds to wait before raising :exc: |
-1
|
Returns:
| Name | Type | Description |
|---|---|---|
samples |
ndarray
|
Decoded reply data. dtype mirrors the :class: |
header |
dict
|
Decoded |
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If |
Examples:
close
¶
Replier
¶
NATS request/reply — receives a request frame, then sends a reply.
Wraps dp_rep_t. Subscribes to the endpoint's subject; each
:meth:recv captures the request's reply-to inbox, and the next
:meth:send publishes the reply directly to that inbox. Calling
send before recv has captured a request raises
:exc:RuntimeError immediately (there is no reply-to target yet)
— unlike ZMQ's REP FSM, recv itself has no such restriction and
simply waits for the next request.
Complements :class:Requester. Use for control-plane responses or
signal-frame RPC where the Replier processes each frame and returns
a result synchronously.
Both ends address the same NATS subject; there is no bind/connect distinction (the broker mediates delivery via the reply-to inbox).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding of frames sent by this socket (the reply). One
of :data: |
...
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
RuntimeError
|
If |
Examples:
>>> from doppler.stream import Replier, CF64
>>> rep = Replier("nats://127.0.0.1:4222/t19120", CF64)
>>> type(rep).__name__
'Replier'
>>> rep.close()
Context-manager form:
Full REQ/REP server loop (requires a live :class:Requester):
>>> from doppler.stream import Requester
>>> import numpy as np, time
>>> rep = Replier("nats://127.0.0.1:4222/ctrl", CF64)
>>> req = Requester("nats://127.0.0.1:4222/ctrl", CF64)
>>> time.sleep(0.05)
>>> req.send(np.ones(4, dtype=np.complex128))
>>> request, hdr = rep.recv(timeout_ms=2000)
>>> rep.send(request, sample_rate=hdr["sample_rate"])
>>> reply, _ = req.recv(timeout_ms=2000)
>>> req.close(); rep.close()
__init__
¶
Create a Replier socket and bind to endpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str
|
NATS endpoint, e.g. |
required |
sample_type
|
int
|
Wire encoding of reply frames sent by this socket:
:data: |
...
|
__enter__
¶
recv
¶
Receive one request frame from a :class:Requester.
Blocks until a request arrives or the timeout expires. Captures
the request's reply-to inbox so the next :meth:send reaches
the right Requester.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_ms
|
int
|
Milliseconds to wait before raising :exc: |
-1
|
Returns:
| Name | Type | Description |
|---|---|---|
samples |
ndarray
|
Decoded request data. dtype mirrors the
:class: |
header |
dict
|
Decoded |
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If |
RuntimeError
|
If the underlying NATS recv fails for any other reason. |
Examples:
send
¶
send(
samples: NDArray[Any],
sample_rate: float = 0,
center_freq: float = 0,
timestamp_ns: int | None = None,
) -> None
Publish the reply to the request's captured reply-to inbox.
Must be called after :meth:recv; calling without a prior
recv raises :exc:RuntimeError immediately (there is no
reply-to target yet). After this call returns, the Replier is
ready to recv the next request.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
samples
|
ndarray
|
C-contiguous array whose dtype must match the socket's
|
required |
sample_rate
|
float
|
Samples per second written into the reply header (default 0). |
0
|
center_freq
|
float
|
Centre frequency in Hz written into the reply header (default 0). |
0
|
timestamp_ns
|
int
|
UNIX nanoseconds to stamp on |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If |
RuntimeError
|
If |
Examples:
close
¶
Helpers¶
get_timestamp_ns
¶
Current wall-clock time in nanoseconds since the UNIX epoch.
Calls clock_gettime(CLOCK_REALTIME) in the C layer. Useful for
stamping outgoing frames when the caller does not supply its own
timestamp_ns, or for computing round-trip latency from the value
returned in the received header dict.
Returns:
| Type | Description |
|---|---|
int
|
Non-negative nanosecond timestamp. Guaranteed to be
monotonically non-decreasing within a single process on any
POSIX system that supports |
Examples:
Related pages¶
Gallery — Waveform I/O — One Capture, Four File Types Design — Telemetry — zero-cost scalar taps for running pipelines