How fast is real-time object detection on a T4 or an L4?

On a live 1080p RTSP camera, a YOLOX-s detector returns each frame’s boxes 3.75 ms after the camera captured it on an NVIDIA L4 (p50; 5.35 ms p99) and 8.78 ms after on a T4, both in FP16. RF-DETR small, a transformer detector that is slower and more accurate, takes 7.2 ms on the L4 (TF32) and 19.8 ms on the T4 (FP32). Those numbers come from Pictograph’s realtime engine, the one behind every deployment and live stream, measured on 2026-10-16. The rest of this post is how we measured them, what moved them, and what we learned when the GPU could not keep up.

What “latency” means here

Capture to result: from the moment the camera stamped the frame (read from its RTCP sender reports, the same clock a real IP camera provides) to the moment the frame’s detections are written out as JSON. It includes decode, preprocessing, inference and box decoding. It does not include the network hop from your camera to the GPU, which depends on where both are.

The test camera was a simulated RTSP camera on the same host serving NVIDIA’s 1080p H.264 sample at 30 fps over TCP. Every model runs as a TensorRT engine built for the exact GPU it runs on.

Live RTSP: capture to result

Four YOLOX cameras on one L4 cost almost nothing over one. RF-DETR is a different story: four cameras more than double its latency on the L4, and a T4 can only serve one camera at full rate in FP32 (the next section shows what happens with two).

When the GPU falls behind, drop frames instead of queueing them

At two or four cameras, RF-DETR small in strict FP32 needs more compute than a T4 has. We ran the same workload through NVIDIA DeepStream’s own inference element, nvinfer, configured to do strictly less work per frame than our engine (raw tensors out, no box decoding), as a reference:

Both process the same 49-50 frames per second, the most a T4 can do for this model. The difference is what happens to the rest. A pipeline that queues the excess delivers every frame one to three seconds late for as long as the overload lasts. Our engine gates each live source at the muxer’s input: it infers what the GPU can and drops and reports the rest, so every frame it does infer is at most two batches old.

For a robot, a gate, or an alert, a fresh answer about 50% of frames beats a stale answer about all of them. The dropped count is reported with every result, so you can see when to move to a bigger GPU or FP16.

Batching for throughput costs latency

Before this engine, Pictograph ran video through ONNX Runtime with frames batched eight at a time behind a prefetch queue, a design built for offline throughput. On the same T4 with the same YOLOX-s model:

Most of the 280 ms was the design itself: a frame waited for seven others before it was inferred. A live stream needs each frame processed as it arrives.

FP16 is 3.6-6.1x faster and keeps accuracy, but not every detection

Time per frame (preprocess, TensorRT, copies and the box decode), batch 1, L4, median of 200 runs:

The accuracy question has two answers. Measured as accuracy, FP16 keeps it: box mAP stayed within ±0.5 points of FP32 on every model we tested, on both the L4 and the T4, inside the ±0.7-point noise of comparing two runs. Measured detection by detection, FP16 does not reproduce FP32: an individual detection’s confidence can move by up to 0.3. RF-DETR keeps 300 of about 1,300 proposals, and FP16’s small numeric differences reorder near-ties among them, so query i in FP16 is not always query i in FP32.

We tried ten ways of pinning sensitive layers to FP32 to make the outputs identical. None did better than plain FP16. So FP16 is offered as a choice per deployment, and FP32 stays the default: if a downstream rule depends on an exact confidence threshold, check it on your own images before switching.

A single image over HTTP

Most deployments are called one image at a time. Round trip on loopback on a T4 (request, JPEG decode, inference, JSON answer), one request at a time, p50:

On large frames (2880x1620) the CPU’s JPEG decode, about 37 ms, becomes most of the request whatever the model.

A browser’s webcam over WebRTC

A browser can stream its camera straight to a deployment with WebRTC (WHIP in, WHEP for the live view back). From a home connection in California to a deployment in the US West region:

• The WHIP offer is answered in 208-441 ms.
• The first frame is inferred 0.6-1.2 s after the offer.
• The engine’s own latency, decode to result, is 5.1 ms p50 and 5.2 ms p99 for YOLOX-s, with no frames dropped across 1,005 frames.

Results come back per frame on a WebRTC data channel or a WebSocket, so the browser sees each frame’s detections without polling.

Cold starts

A TensorRT engine is built once per model, GPU and precision (10 to 105 seconds; YOLOX’s convolutions take TensorRT the longest to tune) and then stored. A deployment that has scaled to zero loads the stored engine instead: the server is ready 1.1-4.1 s after it starts, and on a T4 a request that wakes it gets its first answer 6-7 s after it was sent. Keeping one container warm removes that wait, at the cost of paying for the idle time.

Choosing a GPU

• T4: YOLOX at several cameras, or RF-DETR at one camera in FP32 (FP16 is several times faster). The cheapest tier.
• L4: RF-DETR at several cameras, segmentation and keypoints, higher resolutions. About twice the T4’s YOLOX throughput (679 vs 362 fps for one stream) and 2.2 times its RF-DETR throughput in strict FP32 (107 vs 48 fps).
• FP16 on either, once you have checked accuracy on your own data.

Try it

Train a model on Pictograph, or import one from Explore, then deploy it and point a camera at it:

from pictograph import Client, CreatedDeployment, DetectionResult

client = Client()
created: CreatedDeployment = client.deployments.create(
    model="road-signs-detector",
    gpu_type="t4",
    precision="fp16",
)
deployment = client.deployments.wait_until_active(deployment=created.deployment)
infer = client.deployments.connect(
    deployment=deployment,
    api_key=created.auth_token,
    task="object_detection",
)

with infer.open_stream(source="rtsp://camera.example.com:554/live") as stream:
    for frame in stream.frames():
        result: DetectionResult = frame.result
        print(frame.index, [p.name for p in result.predictions])

The deployments guide covers GPUs, regions and scaling, and the streams guide covers RTSP, WebRTC, webcams and the live view.