Streams

A stream sends live video to a deployment and returns the model’s results for every frame as it is processed. The video can come from a camera the deployment connects to (RTSP, or WebRTC over WHEP), or from your own sender over WHIP, such as a browser’s webcam. You can also watch the video with the detections drawn on it.

A stream runs on the deployment’s model and GPU, and is billed as deployment uptime while it runs.

Source: resources/_deployment_stream.py. Full method surface: Deployments API reference.

from pictograph import Client, DetectionResult

client = Client()
deployment = client.deployments.get(
    deployment="prod-detector",
)
infer = client.deployments.connect(
    deployment=deployment,
    api_key="pk_deploy_...",
    task="object_detection",
)

with infer.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    for frame in stream.frames():
        result: DetectionResult = frame.result
        print(frame.index, [p.name for p in result.predictions])

pictograph deployments stream prod-detector rtsp://camera.example.com:554/live --token pk_deploy_...

The stream stops when the with block ends. The CLI prints one JSON line per frame and stops the stream when you press Ctrl-C.

Try it in the app

Open a deployment’s Analytics and choose Test. Pick Camera to stream your camera, Video to stream a video file, Image for a single picture, or Stream URL for an RTSP or WHEP camera. The live video fills the tab, with each frame’s detections under it.

On a phone, the back camera streams first. Use the switch button on the video to change to the other camera while the stream runs, and the full screen button to fill your screen. The stream stops when you press Stop, switch to another source, or leave the tab.

Stream a workflow

A workflow is served live the same way, on the endpoint made with it (client.workflows.create, or New workflow in the app, picks its region): its endpoint runs every block a frame passes through - its models, its filter, its tracker, its lines and zones - and the live view is drawn in the workflow’s look. Each frame also carries measures, each line’s and zone’s counts so far by its id; workflow.endpoint.live.measures names each one.

from pictograph import Client

client = Client()
workflow = client.workflows.get(
    workflow="Gate counter",
)
live = client.workflows.connect(
    workflow=workflow,
    token="pk_deploy_...",
)

with live.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    for frame in stream.frames():
        print(frame.index, frame.measures)

pictograph workflows stream "Gate counter" rtsp://camera.example.com:554/live --token pk_deploy_...

In the app, open the workflow’s Analytics and choose Test: Camera, Video or Stream URL, with the counts beside the detections. A workflow’s endpoint takes streams only; run it over a single image with invoke.

While it streams, the workflow’s webhook and annotations outputs act on its events - a line crossed, a zone entered, a class appearing: they post them to your endpoint and save their frames into a dataset, at most one frame a second. See Live outputs.

A stream takes every frame its camera sends; a run of the same workflow over a video takes the frames its source’s sample_fps asks for (one a second by default). Each tracker runs at the rate of its own frames: a run’s at its sample_fps, a stream’s at the stream’s rate, read from its first frames. So a recording streamed gives the same events as a run of that recording with sample_fps at its frame rate. A run times each frame by its place in the file, a stream by its timestamps, so footage whose frame rate varies is timed differently the two ways. To stream a recording timed exactly as its file is, send it as an RTSP camera with buffer_mode none and reconnect never (below). A live stream keeps only the newest frame when frames arrive faster than the model runs, and counts the rest as dropped in its stats. A recording sent in a burst, for example while the stream is still starting, loses frames that its run keeps.

Video sources

The video must be H.264. A camera’s address must be reachable from the internet. Credentials in its URL are used to connect, and are never stored or logged.

An RTSP camera takes three options:

with infer.open_stream(
    source="rtsp://recorder.example.com:8554/clip", transport="tcp", buffer_mode="none", reconnect="never"
) as stream:
    ...

pictograph deployments stream prod-detector rtsp://recorder.example.com:8554/clip \
  --token pk_deploy_... --transport tcp --buffer-mode none --reconnect never

Frame results

stream.frames() yields one StreamFrame for each frame:

stream.messages() yields every raw message instead, including a stats message each second. If the connection drops, frames() reconnects on its own; frames sent while it was disconnected are skipped.

When the video stalls

A camera that stops sending, or a sender that goes away, does not end a stream: it waits for the video to come back, and by default frames() waits with it. Pass idle_timeout to be told instead. After that many seconds without a frame, frames() raises StreamStalledError and the stream keeps running. Call frames() again to wait another idle_timeout for the video:

from pictograph import StreamStalledError

with infer.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    while stream.ended is None:
        try:
            for frame in stream.frames(idle_timeout=2):
                act_on(frame.result)
        except StreamStalledError as stall:
            hold_still()  # no frame for stall.quiet_s seconds

The time you spend handling a frame does not count toward the timeout. stall.quiet_s is how long the source has been quiet. The stream keeps sending its stats while the video is stalled: when no frame comes, it sends them anyway, marked "heartbeat": true, with each source’s last_frame_s. So stall.alive is True when the stream is up and the camera is quiet, and False when nothing at all came from the stream (the connection may be down). A heartbeat comes about two seconds into a stall, so with an idle_timeout under three seconds, alive can be False before the first one arrives.

# Fail after 5 seconds without a frame (exit code 9, stream_stalled); the stream stops with it.
pictograph deployments stream prod-detector rtsp://camera.example.com:554/live \
  --token pk_deploy_... --idle-timeout 5

Watch the live view

stream.whep_url plays the video with the detections drawn on it, in any WHEP player. Send the deployment token as the bearer. Turn the drawing off or on with stream.set_overlay(on=False) or stream.set_overlay(on=True).

Send video from a browser

With source="whip", you send the video. This is how a browser streams its webcam; the token travels in each request’s Authorization header:

const sessions = endpointUrl.replace(/\/predict$/, '/sessions');
const auth = { Authorization: `Bearer ${token}` };
const session = await (await fetch(sessions, {
  method: 'POST',
  headers: { ...auth, 'Content-Type': 'application/json' },
  body: JSON.stringify({ source: { kind: 'whip' } }),
})).json();
const origin = new URL(sessions).origin;

const camera = await navigator.mediaDevices.getUserMedia({ video: true });
const pc = new RTCPeerConnection({ iceServers: session.ice_servers });
pc.addTransceiver(camera.getVideoTracks()[0], { direction: 'sendonly' });
await pc.setLocalDescription(await pc.createOffer());
await new Promise((done) => {   // the offer carries every ICE candidate: wait for them
  pc.onicegatheringstatechange = () => pc.iceGatheringState === 'complete' && done();
  setTimeout(done, 3000);
});

const answer = await fetch(origin + session.paths.whip, {
  method: 'POST',
  headers: { ...auth, 'Content-Type': 'application/sdp' },
  body: pc.localDescription.sdp,
});
await pc.setRemoteDescription({ type: 'answer', sdp: await answer.text() });

The offer must include H.264, which every major browser supports. To read the results in the browser, open the results WebSocket as shown in Without the SDK. Keep it open while the stream runs: a stream with no request open ends after 30 seconds, and the video alone does not count as one.

For the lowest latency, the results can also come back over the same connection as the video. Open a data channel named pictograph.results.v1 before creating the offer. Each frame’s message then arrives on it as soon as the frame is processed, exactly as the WebSocket sends it:

const channel = pc.createDataChannel('pictograph.results.v1', {
  ordered: false,      // a late result is dropped, never queued behind newer ones
  maxRetransmits: 0,
});
channel.onmessage = (event) => {
  const message = JSON.parse(event.data);
  if (message.type === 'frame') {
    // capture_ns is the frame's capture on this browser's clock: Date.now() minus it is capture to result
    const latency = message.capture_ns ? Date.now() - message.capture_ns / 1e6 : null;
    console.log(message.frame, latency, message.pictograph.annotations);
  }
};
// then createOffer() as above

Frames the browser falls behind on are dropped rather than queued. Each frame arrives on both the channel and the WebSocket, so use its frame number to keep only the newest.

When a stream ends

A stream ends when:

• you call stream.stop(), or the with block ends,
• the camera stops sending,
• it has no request open for 30 seconds (an open results WebSocket counts; the video and the live view, which go straight to the stream, do not), or
• it has run for four hours.

When frames() finishes, stream.ended says why: stopped, source_ended, idle, time_limit, container_stopped, or ended (it had already ended).

Streams per deployment

Each container runs up to four streams, so a deployment runs up to four times its max_containers (deployment.stream_limit). Opening one more fails with 429 stream_limit: stop a stream, or raise max_containers. A deployment created before streams existed has a limit of 0: pause it and resume it once to add them.

Without the SDK

A stream is a few HTTP requests and a WebSocket. Every request carries the deployment token, Authorization: Bearer pk_deploy_.... Streams open at the deployment’s endpoint_url with /sessions in place of /predict (a workflow’s endpoint_url is its /sessions already); every other path comes back in the open’s response.

curl -X POST "https://workflows.pictograph.io/us-west/7f3a91c2/sessions" \
  -H "Authorization: Bearer pk_deploy_..." \
  -H "Content-Type: application/json" \
  -d '{"source": {"kind": "rtsp", "uri": "rtsp://camera.example.com:554/live"}}'

{
  "session": "7955b01ae9212f34",
  "kind": "rtsp",
  "overlay": true,
  "stream_limit": 4,
  "idle_timeout_s": 30,
  "expires_at": "2026-10-26T15:44:44Z",
  "ice_servers": [{"urls": "stun:stun.l.google.com:19302"}],
  "paths": {
    "session": "/us-west/7f3a91c2/sessions/7955b01ae9212f34",
    "whep": "/us-west/7f3a91c2/sessions/7955b01ae9212f34/whep",
    "overlay": "/us-west/7f3a91c2/sessions/7955b01ae9212f34/overlay",
    "results": "/us-west/7f3a91c2/sessions/7955b01ae9212f34/results"
  }
}

source.kind is rtsp (with an rtsp:// or rtsps:// uri, and optionally transport, buffer_mode and reconnect), whep (with an https:// uri) or whip (no uri). Add "overlay": false to start with the drawing off. A whip stream’s paths include whip.

Read the results from the WebSocket. A browser cannot set headers on a WebSocket, so it sends the token as a second subprotocol; other clients can send the Authorization header instead:

const results = new WebSocket(
  origin.replace(/^http/, 'ws') + session.paths.results,
  ['pictograph.results.v1', `bearer.${token}`],
);
results.onmessage = (event) => {
  const message = JSON.parse(event.data);
  if (message.type === 'frame') console.log(message.frame, message.pictograph.annotations);
};

Each frame message carries the frame’s results as Pictograph JSON, with its number, time and latency (shortened here):

{
  "type": "frame",
  "frame": 200,
  "pts": 6.73,
  "capture_ns": 1792125225262529600,
  "latency_ms": {"total": 11.9, "engine": 11.7, "inference": 9.6},
  "pictograph": {
    "image_width": 1280,
    "image_height": 720,
    "annotation_count": 1,
    "annotations": [
      {"id": "0b9c4a6e-...", "name": "car", "type": "bbox", "confidence": 0.67,
       "bounding_box": {"x": 229.7, "y": 524.0, "w": 97.4, "h": 141.1}}
    ]
  }
}

A classifier’s labels are in pictograph.image_tags, with their scores in tag_confidences. capture_ns is when the frame was captured, in nanoseconds since 1970 by the sender’s clock; it is missing until the sender’s first report. A stats message comes about once a second. When no frame comes, it comes anyway, marked "heartbeat": true, with each source’s last_frame_s: the stream is up and its video has been quiet that long. When the stream ends, the WebSocket closes with code 1000 and the reason, such as its source ended or no request for 30 s.

When a call fails

The SDK raises these as typed errors with the error as e.code, and retries deployment_starting on its own. If the video never connects, your network may block the UDP traffic WebRTC uses; some company networks and VPNs do.

Billing

A stream runs on the deployment’s GPU, in its region, at its price: billed per minute while its container runs. The container starts when a stream opens (about 20 seconds when the deployment is asleep) and stops after the deployment’s idle window once the last stream ends. Results and signaling sent through the gateway count as the deployment’s data transfer; the video itself does not.

Next steps

• Deployments - create one, call its /predict URL, and how it is billed
• Pictograph annotation format - what a frame’s results contain
• Deployments API reference - every stream method and field