---
title: "Real-time object detection on live video: RTSP, WebRTC and webcams"
description: "Stream live video through a deployment and get each frame's detections back as it is processed. SDK, CLI, and REST."
section: "Guides"
order: 5
---
A stream sends live video to a [deployment](/docs/deployments.md) and returns the model's
results for every frame as it is processed. The video can come from a camera the deployment
connects to (RTSP, or WebRTC over WHEP), or from your own sender over WHIP, such as a
browser's webcam. You can also watch the video with the detections drawn on it.

A stream runs on the deployment's model and GPU, and is billed as deployment uptime while it
runs.

Source: [`resources/_deployment_stream.py`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.160/src/pictograph/resources/_deployment_stream.py). Full method surface:
[Deployments API reference](/docs/api-reference/deployments#streams).

```python
from pictograph import Client, DetectionResult

client = Client()
deployment = client.deployments.get(
    deployment="prod-detector",
)
infer = client.deployments.connect(
    deployment=deployment,
    api_key="pk_deploy_...",
    task="object_detection",
)

with infer.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    for frame in stream.frames():
        result: DetectionResult = frame.result
        print(frame.index, [p.name for p in result.predictions])
```

```bash
pictograph deployments stream prod-detector rtsp://camera.example.com:554/live --token pk_deploy_...
```

The stream stops when the `with` block ends. The CLI prints one JSON line per frame and stops
the stream when you press Ctrl-C.

## Try it in the app

Open a deployment's **Analytics** and choose **Test**. Pick **Camera** to stream your camera,
**Video** to stream a video file, **Image** for a single picture, or **Stream URL** for an RTSP
or WHEP camera. The live video fills the tab, with each frame's detections under it.

On a phone, the back camera streams first. Use the switch button on the video to change to
the other camera while the stream runs, and the full screen button to fill your screen. The
stream stops when you press Stop, switch to another source, or leave the tab.

## Stream a workflow

A [workflow](/docs/api-reference/workflows#serve-it-live) is served live the same way, on the
endpoint made with it (`client.workflows.create`, or **New workflow** in the app, picks its
region): its endpoint runs every block a frame passes through - its models, its filter, its tracker, its
lines and zones - and the live view is drawn in the workflow's look. Each frame also carries
`measures`, each line's and zone's counts so far by its id; `workflow.endpoint.live.measures`
names each one.

```python
from pictograph import Client

client = Client()
workflow = client.workflows.get(
    workflow="Gate counter",
)
live = client.workflows.connect(
    workflow=workflow,
    token="pk_deploy_...",
)

with live.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    for frame in stream.frames():
        print(frame.index, frame.measures)
```

```bash
pictograph workflows stream "Gate counter" rtsp://camera.example.com:554/live --token pk_deploy_...
```

In the app, open the workflow's **Analytics** and choose **Test**: Camera, Video or Stream URL,
with the counts beside the detections. A workflow's endpoint takes streams only; run it over a
single image with [invoke](/docs/api-reference/workflows#invoke).

While it streams, the workflow's webhook and annotations outputs act on its events - a line
crossed, a zone entered, a class appearing: they post them to your endpoint and save their
frames into a dataset, at most one frame a second. See
[Live outputs](/docs/api-reference/workflows#live-outputs).

A stream takes every frame its camera sends; a run of the same workflow over a video takes the
frames its source's `sample_fps` asks for (one a second by default). Each tracker runs at the
rate of its own frames: a run's at its `sample_fps`, a stream's at the stream's rate, read from
its first frames. So a recording streamed gives the same events as a run of that recording with
`sample_fps` at its frame rate. A run times each frame by its place in the file, a stream by its
timestamps, so footage whose frame rate varies is timed differently the two ways. To stream a
recording timed exactly as its file is, send it as an RTSP camera with `buffer_mode` `none` and
`reconnect` `never` (below). A live stream keeps only the newest frame when frames arrive faster
than the model runs, and counts the rest as `dropped` in its stats. A recording sent in a burst,
for example while the stream is still starting, loses frames that its run keeps.

## Video sources

| `source` | Where the video comes from |
|---|---|
| `rtsp://...` or `rtsps://...` | An RTSP camera. The deployment connects to it. |
| `https://...` | A WebRTC camera that serves WHEP. The deployment connects to it. |
| `whip` | You send the video to `stream.whip_url` over WHIP, from a browser, OBS or GStreamer. |

The video must be H.264. A camera's address must be reachable from the internet. Credentials
in its URL are used to connect, and are never stored or logged.

An RTSP camera takes three options:

| Option | Values | What it does |
|---|---|---|
| `transport` | `auto` (default), `tcp`, `udp` | How the video comes. `auto` tries UDP first and falls back to TCP. Use `tcp` when the camera is behind NAT or a TCP-only tunnel, where UDP never arrives. |
| `buffer_mode` | `auto` (default), `none` | `auto` smooths the camera's clock onto the deployment's. `none` keeps the sender's own timestamps, so a streamed recording is timed exactly as its file is. |
| `reconnect` | `always` (default), `never` | `always` connects again to a camera that drops or ends its stream. `never` treats the stream as a recording: its end ends the stream (`ended` is `source_ended`), with its last frame decoded. |

```python
with infer.open_stream(
    source="rtsp://recorder.example.com:8554/clip", transport="tcp", buffer_mode="none", reconnect="never"
) as stream:
    ...
```

```bash
pictograph deployments stream prod-detector rtsp://recorder.example.com:8554/clip \
  --token pk_deploy_... --transport tcp --buffer-mode none --reconnect never
```

## Frame results

`stream.frames()` yields one `StreamFrame` for each frame:

| Field | Contents |
|---|---|
| `result` | The model's results, as the same typed result `infer()` returns |
| `index` | The frame number, starting at 0 |
| `pts` | The frame's time in the stream, in seconds |
| `latency_ms` | Milliseconds: `engine` from the frame's arrival to its result, `inference` for the model alone, `total` since capture by the camera's clock |
| `capture_time` | When the frame was captured, by the sender's clock (a camera's, or the machine that sent the video), as a UTC `datetime`; `None` until the sender's first report, about a second into a stream |
| `capture_latency_ms` | Milliseconds from capture to the result reaching you, on your clock: exact when you sent the video, as close as your clock and the camera's agree otherwise |
| `width`, `height` | The frame size in pixels |
| `message` | The raw message, with the frame's [Pictograph JSON](/docs/annotation-format.md) under `"pictograph"` |

`stream.messages()` yields every raw message instead, including a `stats` message each second.
If the connection drops, `frames()` reconnects on its own; frames sent while it was
disconnected are skipped.

## When the video stalls

A camera that stops sending, or a sender that goes away, does not end a stream: it waits for
the video to come back, and by default `frames()` waits with it. Pass `idle_timeout` to be
told instead. After that many seconds without a frame, `frames()` raises
`StreamStalledError` and the stream keeps running. Call `frames()` again to wait another
`idle_timeout` for the video:

```python
from pictograph import StreamStalledError

with infer.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    while stream.ended is None:
        try:
            for frame in stream.frames(idle_timeout=2):
                act_on(frame.result)
        except StreamStalledError as stall:
            hold_still()  # no frame for stall.quiet_s seconds
```

The time you spend handling a frame does not count toward the timeout. `stall.quiet_s` is how
long the source has been quiet. The stream keeps sending its `stats` while the video is
stalled: when no frame comes, it sends them anyway, marked `"heartbeat": true`, with each
source's `last_frame_s`. So `stall.alive` is `True` when the stream is up and the camera is
quiet, and `False` when nothing at all came from the stream (the connection may be down).
A heartbeat comes about two seconds into a stall, so with an `idle_timeout` under three
seconds, `alive` can be `False` before the first one arrives.

```bash
# Fail after 5 seconds without a frame (exit code 9, stream_stalled); the stream stops with it.
pictograph deployments stream prod-detector rtsp://camera.example.com:554/live \
  --token pk_deploy_... --idle-timeout 5
```

## Watch the live view

`stream.whep_url` plays the video with the detections drawn on it, in any WHEP player. Send
the deployment token as the bearer. Turn the drawing off or on with `stream.set_overlay(on=False)`
or `stream.set_overlay(on=True)`.

## Send video from a browser

With `source="whip"`, you send the video. This is how a browser streams its webcam; the token
travels in each request's `Authorization` header:

```javascript
const sessions = endpointUrl.replace(/\/predict$/, '/sessions');
const auth = { Authorization: `Bearer ${token}` };
const session = await (await fetch(sessions, {
  method: 'POST',
  headers: { ...auth, 'Content-Type': 'application/json' },
  body: JSON.stringify({ source: { kind: 'whip' } }),
})).json();
const origin = new URL(sessions).origin;

const camera = await navigator.mediaDevices.getUserMedia({ video: true });
const pc = new RTCPeerConnection({ iceServers: session.ice_servers });
pc.addTransceiver(camera.getVideoTracks()[0], { direction: 'sendonly' });
await pc.setLocalDescription(await pc.createOffer());
await new Promise((done) => {   // the offer carries every ICE candidate: wait for them
  pc.onicegatheringstatechange = () => pc.iceGatheringState === 'complete' && done();
  setTimeout(done, 3000);
});

const answer = await fetch(origin + session.paths.whip, {
  method: 'POST',
  headers: { ...auth, 'Content-Type': 'application/sdp' },
  body: pc.localDescription.sdp,
});
await pc.setRemoteDescription({ type: 'answer', sdp: await answer.text() });
```

The offer must include H.264, which every major browser supports. To read the results in the
browser, open the results WebSocket as shown in [Without the SDK](#without-the-sdk). Keep it
open while the stream runs: a stream with no request open ends after 30 seconds, and the video
alone does not count as one.

For the lowest latency, the results can also come back over the same connection as the video.
Open a data channel named `pictograph.results.v1` before creating the offer. Each frame's
message then arrives on it as soon as the frame is processed, exactly as the WebSocket sends it:

```javascript
const channel = pc.createDataChannel('pictograph.results.v1', {
  ordered: false,      // a late result is dropped, never queued behind newer ones
  maxRetransmits: 0,
});
channel.onmessage = (event) => {
  const message = JSON.parse(event.data);
  if (message.type === 'frame') {
    // capture_ns is the frame's capture on this browser's clock: Date.now() minus it is capture to result
    const latency = message.capture_ns ? Date.now() - message.capture_ns / 1e6 : null;
    console.log(message.frame, latency, message.pictograph.annotations);
  }
};
// then createOffer() as above
```

Frames the browser falls behind on are dropped rather than queued. Each frame arrives on both
the channel and the WebSocket, so use its `frame` number to keep only the newest.

## When a stream ends

A stream ends when:

- you call `stream.stop()`, or the `with` block ends,
- the camera stops sending,
- it has no request open for 30 seconds (an open results WebSocket counts; the video and the
  live view, which go straight to the stream, do not), or
- it has run for four hours.

When `frames()` finishes, `stream.ended` says why: `stopped`, `source_ended`, `idle`,
`time_limit`, `container_stopped`, or `ended` (it had already ended).

## Streams per deployment

Each container runs up to four streams, so a deployment runs up to four times its
`max_containers` (`deployment.stream_limit`). Opening one more fails with `429`
`stream_limit`: stop a stream, or raise `max_containers`. A deployment created before streams
existed has a limit of 0: pause it and resume it once to add them.

## Without the SDK

A stream is a few HTTP requests and a WebSocket. Every request carries the deployment token,
`Authorization: Bearer pk_deploy_...`. Streams open at the deployment's `endpoint_url` with
`/sessions` in place of `/predict` (a workflow's `endpoint_url` is its `/sessions` already);
every other path comes back in the open's response.

| Method | Path | What it does |
|---|---|---|
| `POST` | `{sessions}` | Opens a stream. Returns `201` with its paths |
| `GET` | `{paths.results}` | A WebSocket with each frame's results |
| `POST` | `{paths.whip}` | A WHIP offer (`application/sdp`) that sends your video |
| `POST` | `{paths.whep}` | A WHEP offer (`application/sdp`) that plays the live view |
| `POST` | `{paths.overlay}` | `{"on": false}` turns the drawing off; `{"on": true}` turns it back on |
| `GET` | `{paths.session}` | The stream's state |
| `DELETE` | `{paths.session}` | Stops the stream |

```bash
curl -X POST "https://workflows.pictograph.io/us-west/7f3a91c2/sessions" \
  -H "Authorization: Bearer pk_deploy_..." \
  -H "Content-Type: application/json" \
  -d '{"source": {"kind": "rtsp", "uri": "rtsp://camera.example.com:554/live"}}'
```

```json
{
  "session": "7955b01ae9212f34",
  "kind": "rtsp",
  "overlay": true,
  "stream_limit": 4,
  "idle_timeout_s": 30,
  "expires_at": "2026-10-26T15:44:44Z",
  "ice_servers": [{"urls": "stun:stun.l.google.com:19302"}],
  "paths": {
    "session": "/us-west/7f3a91c2/sessions/7955b01ae9212f34",
    "whep": "/us-west/7f3a91c2/sessions/7955b01ae9212f34/whep",
    "overlay": "/us-west/7f3a91c2/sessions/7955b01ae9212f34/overlay",
    "results": "/us-west/7f3a91c2/sessions/7955b01ae9212f34/results"
  }
}
```

`source.kind` is `rtsp` (with an `rtsp://` or `rtsps://` `uri`, and optionally `transport`,
`buffer_mode` and `reconnect`), `whep` (with an `https://` `uri`) or `whip` (no `uri`). Add `"overlay": false`
to start with the drawing off. A `whip` stream's paths include `whip`.

Read the results from the WebSocket. A browser cannot set headers on a WebSocket, so it sends
the token as a second subprotocol; other clients can send the `Authorization` header instead:

```javascript
const results = new WebSocket(
  origin.replace(/^http/, 'ws') + session.paths.results,
  ['pictograph.results.v1', `bearer.${token}`],
);
results.onmessage = (event) => {
  const message = JSON.parse(event.data);
  if (message.type === 'frame') console.log(message.frame, message.pictograph.annotations);
};
```

Each `frame` message carries the frame's results as [Pictograph JSON](/docs/annotation-format.md),
with its number, time and latency (shortened here):

```json
{
  "type": "frame",
  "frame": 200,
  "pts": 6.73,
  "capture_ns": 1792125225262529600,
  "latency_ms": {"total": 11.9, "engine": 11.7, "inference": 9.6},
  "pictograph": {
    "image_width": 1280,
    "image_height": 720,
    "annotation_count": 1,
    "annotations": [
      {"id": "0b9c4a6e-...", "name": "car", "type": "bbox", "confidence": 0.67,
       "bounding_box": {"x": 229.7, "y": 524.0, "w": 97.4, "h": 141.1}}
    ]
  }
}
```

A classifier's labels are in `pictograph.image_tags`, with their scores in `tag_confidences`.
`capture_ns` is when the frame was captured, in nanoseconds since 1970 by the sender's clock;
it is missing until the sender's first report. A `stats` message comes about once a second.
When no frame comes, it comes anyway, marked `"heartbeat": true`, with each source's
`last_frame_s`: the stream is up and its video has been quiet that long.
When the stream ends, the WebSocket closes with code `1000` and the reason, such as
`its source ended` or `no request for 30 s`.

## When a call fails

| Status | `error` | What to do |
|---|---|---|
| 400 | `invalid_source`, `unsafe_source` | Use an `rtsp://`, `rtsps://` or `https://` camera address that is reachable from the internet, or `whip` |
| 400 | `invalid_source_option` | Give `transport`, `buffer_mode` and `reconnect` only to an `rtsp` source, with one of their values |
| 400 | `invalid_request_body`, `invalid_option` | Send a JSON body; `overlay` must be `true` or `false` |
| 401 | `unauthorized` | Send the deployment's own `pk_deploy_` token |
| 404 | `deployment_not_found`, `wrong_region` | Use the deployment's `endpoint_url` exactly as it is |
| 404, 410 | `session_not_found`, `session_ended` | The stream has ended: open a new one |
| 406 | | The video or the player does not support H.264 |
| 409 | `deployment_not_active` | Resume the deployment, or wait for it to finish provisioning |
| 409 | `streams_unavailable` | Pause and resume the deployment once |
| 413 | `request_too_large` | The request is over 256 KiB |
| 429 | `stream_limit` | Stop a stream, or raise `max_containers` |
| 502 | `session_failed`, `deployment_unreachable`, `deployment_unavailable` | Retry; `detail` has the reason |
| 503 | `deployment_starting` | The container did not start within two minutes: retry after `Retry-After` |
| 503 | `sessions_unavailable`, `gateway_unavailable` | Retry shortly |
| 504 | `session_timeout` | Retry |

The SDK raises these as typed errors with the `error` as `e.code`, and retries
`deployment_starting` on its own. If the video never connects, your network may block the UDP
traffic WebRTC uses; some company networks and VPNs do.

## Billing

A stream runs on the deployment's GPU, in its region, at its price: billed per minute while its
container runs. The container starts when a stream opens (about 20 seconds when the deployment
is asleep) and stops after the deployment's idle window once the last stream ends. Results and
signaling sent through the gateway count as the deployment's data transfer; the video itself
does not.

## Next steps

- [Deployments](/docs/deployments.md) - create one, call its `/predict` URL, and how it is billed
- [Pictograph annotation format](/docs/annotation-format.md) - what a frame's results contain
- [Deployments API reference](/docs/api-reference/deployments#streams) - every stream method and field