---
title: Local inference
description: Run your trained Pictograph models on your own hardware - PyTorch, ExecuTorch, ONNX Runtime or TensorRT, with the same typed result from every one - then score them, feed a PyTorch DataLoader, and draw the results.
section: Get Started
order: 3
---
Your trained models are published in every executable form your edge actually runs.
Load one by name and predict: the weights download and cache on first use, and the
result is a per-task type whose `predictions` are already narrowed to the annotations
that task can produce.

Source: [`inference/__init__.py`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/inference/__init__.py)

```python
from pictograph import get_model, DetectionModel, DetectionResult

model: DetectionModel = get_model(name="Woody Whirling Grouse", task="object_detection")
result: DetectionResult = model.predict(
    image="photo.jpg",
)

for p in result.predictions:
    print(p.name, round(p.confidence, 2), p.bounding_box)
```

```bash
pip install 'pictograph[inference]'
```

`task=` is what lets the annotation typecheck without a cast, and it is **verified**:
the loader reads the model's real task and raises if they disagree, so it can never
silently lie. Omit it and you get the `AnyModel` union to narrow with `isinstance`.

## Weight formats and their runtimes

The same weights, five formats, four runtimes. Every one returns the **same task
class** and the **same typed result**, so switching is a one-word change.

**You pick a weight format; the runtime follows from it.** There is no `runtime=`
argument - the format determines which engine can execute it, so asking for both
would let you ask for a contradiction. (`pytorch` and `safetensors` are the same
tensors in different containers, which is why five formats resolve to four runtimes.)

| `format=` | Artifact | Runtime | Install | Best for |
| --- | --- | --- | --- | --- |
| `onnx` *(default)* | `.onnx` | `onnxruntime` | `pip install 'pictograph[inference]'` | The default. Runs anywhere, CPU or GPU |
| `pytorch` | `.pth` | `pytorch` | `pip install 'pictograph[inference]'` | Research, fine-tuning, reaching the live `nn.Module` |
| `safetensors` | `model.safetensors` | `pytorch` | `pip install 'pictograph[inference]'` | The same weights in a safer container |
| `pytorch_engine` | `.pte` | `executorch` | `pip install 'pictograph[inference,executorch]'` | Portable edge - phones, ARM boards, a Jetson's CPU |
| `tensorrt_engine` | `.engine` | `tensorrt` | `pip install 'pictograph[inference,tensorrt]'` | NVIDIA, lowest latency |

`pictograph[inference]` is the whole install for the first three rows - `.onnx`, `.pth`
and `.safetensors` - across every model family. It carries torch, torchvision,
safetensors and the pinned `segmentation-models-pytorch` alongside the ONNX stack, and
the YOLOX and RF-DETR architectures are vendored into the wheel, so there is never a
second `pip install` to run a model you trained here. The two engine runtimes need their
own extra only because they cannot be folded in: ExecuTorch pins one exact `torch` minor,
and `tensorrt` has no non-CUDA distribution at all.

```python
from pictograph import get_model

onnx = get_model(name="Shelf Detector", task="object_detection")
pte = get_model(name="Shelf Detector", task="object_detection", format="pytorch_engine")
torch_model = get_model(name="Shelf Detector", task="object_detection", format="pytorch")

for m in (onnx, pte, torch_model):
    result = m.predict(image="photo.jpg")  # same type, same fields, every time
    print(m.backend, m.device, len(result.predictions))
```

```text
onnxruntime coreml 7
executorch cpu 7
pytorch mps 7
```

### What actually ran

Provenance is reported as measured, never as requested - a CUDA session that fell
back to CPU says `cpu`.

| Attribute | Meaning |
| --- | --- |
| `model.backend` | `pytorch` \| `executorch` \| `onnxruntime` \| `tensorrt` |
| `model.device` | `cpu` \| `cuda` \| `mps` \| `coreml` - the device that ran it |
| `model.providers` | ONNX Runtime providers in resolution order; the `.pte`'s delegate backends; the plan's TensorRT version and GPU target. Empty on `pytorch` |
| `model.classes` | Class names in the model's label order |

Every result carries the same three, so a prediction can be traced back to the
runtime that produced it:

```python
model = get_model(name="Shelf Detector", task="object_detection")
result = model.predict(
    image="photo.jpg",
)
print(result.backend, result.device, result.inference_ms)
```

## Offline loading

`load_model` needs no API key and makes no network call. Point it at the weights and the
`config.json` from the model's **Files** tab. **The runtime comes from the weights
suffix**, so the call shape never changes:

```python
from pictograph import load_model, ClassificationModel

# Artifacts are named after the MODEL, so a directory of downloads stays legible.
for weights in (
    "shelf-detector.onnx",
    "shelf-detector-xnnpack.pte",
    "shelf-detector-sm75-trt10.13.3.9.engine",
):
    model: ClassificationModel = load_model(
        weights=weights,
        config="shelf-detector.config.json",
        task="classification",
    )
    print(model.backend, model.device)
```

## Precision

`fp16` means **fp16 weights with fp32 inputs and outputs** - a drop-in for the fp32
serving path, not a different interface.

```python
from pictograph import get_model, DetectionModel

model: DetectionModel = get_model(
    name="Shelf Detector",
    task="object_detection",
    format="onnx",
    precision="fp16",
)
print(model.backend, model.device)
```

`.onnx`, `.pte` and `.engine` are all published per precision, so `precision=` selects
which artifact is fetched. A native PyTorch checkpoint is the one artifact with no
derived form - it is the raw trained tensors - so asking for a precision the model was
not trained at raises and tells you what to do instead.

## Choosing a device

`device=` names the hardware. The default `"auto"` looks at what your chosen `format=`
can run on and picks the best available. `"cpu"` is reproducible and instant to load,
which is what you want in CI and for numerical-parity work.

```python
from pictograph import get_model

model = get_model(name="My Classifier", task="classification", device="cpu")
```

| `device=` | Meaning |
| --- | --- |
| `auto` *(default)* | Best available for the chosen format |
| `cpu` | CPU only |
| `mps` | Apple acceleration |
| `cuda` / `cuda:N` | NVIDIA, optionally a specific GPU |

**The execution provider follows from the (format, device) pair, so you never name one.**
`mps` is the Apple accelerator family, not just PyTorch's backend: it means CoreML for an
`.onnx` graph and torch-MPS for a checkpoint. `cuda` means the CUDA provider for `.onnx`
and TensorRT for an `.engine`.

Not every device fits every format: a `.engine` is CUDA-only, a `.pte` never runs on
CUDA. Asking for a pair that cannot exist **raises and names what that format can run
on**, rather than quietly falling back to something slower than you asked for.

`model.device` reports what actually **ran**, which can be more specific than what you
asked for. Request `mps` with an `.onnx` and it reports `coreml`.

## TensorRT engines are not portable

**A TensorRT engine is not a model file. It is a compiled, hardware-specific
execution plan.** It is bound to all four of:

1. the **GPU architecture** it was built on - T4 is `sm75`, A100 `sm80`, A10G `sm86`,
   L4 `sm89`, H100 `sm90`;
2. the **exact TensorRT version** - a minor bump invalidates every previously
   serialized plan;
3. the **precision** it was built for - fp32 and fp16 are different plans;
4. its build-time **shape profile**.

An engine built on an A100 does not run on a T4. It fails at **load**, not at accuracy.
The filename records the binding (`sm80-trt10.13.3.9-fp16.engine`) and the loader checks
it against your device *before* deserializing, so a mismatch is an explanation rather
than a crash:

```text
RuntimeError: This TensorRT engine was built for trt-10.13.3.9 on sm80; this
device is sm75. A TensorRT plan is compiled for one GPU architecture and one
TensorRT version and cannot be loaded anywhere else - rebuild the engine for
your device.
```

Because of that, `get_model(format="tensorrt_engine")` defaults to fetching the engine for
**your** GPU. Pass `target=` to fetch a different one deliberately:

```python
from pictograph import get_model

model = get_model(
    name="Shelf Detector",
    task="object_detection",
    format="tensorrt_engine",
    target="sm80",  # build for an A100 from a machine that is not one
)
```

If you rename an engine, keep its manifest row beside it as `<name>.json` so the check
still works.

An ExecuTorch `.pte` has none of this: it is **portable across devices** for its
lowering backend, and the portable-CPU (XNNPACK) build is the default published
artifact precisely because it runs everywhere.

Switching runtimes does not change your results. Every runtime shares the same
preprocessing and postprocessing and differs only in the forward pass, so the same
weights predict the same classes at the same coordinates.

## The five task types

One model class and one result class per task, whichever runtime is behind it.

| `task=` | Model class | Result class | `predictions` |
| --- | --- | --- | --- |
| `object_detection` | `DetectionModel` | `DetectionResult` | `BBoxAnnotation` |
| `instance_segmentation` | `InstanceSegmentationModel` | `InstanceSegmentationResult` | `PolygonAnnotation \| BBoxAnnotation` |
| `semantic_segmentation` | `SemanticSegmentationModel` | `SemanticSegmentationResult` | `PolygonAnnotation` |
| `keypoint_detection` | `KeypointModel` | `KeypointResult` | `KeypointAnnotation` |
| `classification` | `ClassificationModel` | `ClassificationResult` | *(ranked `classes`)* |

Classifiers return ranked classes instead of geometry, and `top` is never `None`:

```python
from pictograph import get_model, ClassificationModel, ClassificationResult

model: ClassificationModel = get_model(name="My Classifier", task="classification")
result: ClassificationResult = model.predict(
    image="cat.jpg",
    top_k=3,
)

print(result.top.name, round(result.top.confidence, 2))
for score in result.classes:
    print(score.name, round(score.confidence, 3))
```

## Batches and cleanup

```python
from pictograph import get_model

with get_model(name="Shelf Detector", task="object_detection") as model:
    for result in model.predict_batch(images=["a.jpg", "b.jpg", "c.jpg"]):
        print(len(result.predictions))
```

Every model is a context manager, and you should close it when you are done: an ONNX
session, a CUDA allocator and an ExecuTorch memory arena all hold memory the garbage
collector will not promptly return.

## Accepted image inputs

`predict` takes a file path, an `http(s)` URL, raw bytes, a **BGR** numpy array (what
`cv2.imread` returns), or a PIL image.

```python
import cv2
from pictograph import get_model

model = get_model(name="Shelf Detector", task="object_detection")
result = model.predict(
    image=cv2.imread("photo.jpg"),
)
```

## Score a model against labelled data

Source: [`metrics/_detection.py`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/metrics/_detection.py)

`pictograph.metrics` runs entirely locally. `evaluate_detections` matches predictions to
ground truth by IoU and returns per-class and overall precision, recall, F1 and average
precision, including **mAP** (`result.mean_average_precision`).

```python
from pictograph import Client
from pictograph.metrics import evaluate_detections

client = Client()
filenames = ["img-001.jpg", "img-002.jpg"]
ground_truth = {
    name: client.annotations.get(dataset_name="road-signs", image=name)
    for name in filenames
}
predictions = {name: run_my_model(name) for name in filenames}

result = evaluate_detections(predictions, ground_truth, iou_threshold=0.5)
print(result.precision, result.recall, result.f1)
for name, m in result.per_class.items():
    print(name, m.precision, m.recall, m.support)
```

`confusion_matrix()` in the same module surfaces cross-class confusion. To have
Pictograph run the inference and store the result instead, use
[`client.model_evaluations`](/docs/api-reference/model-evaluations.md) - the metric
math is identical, so a server run and a local run on the same data agree.

## Feed a PyTorch DataLoader

Source: [`_torch_dataset.py`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/_torch_dataset.py)

`client.datasets.as_pytorch(name)` returns a map-style `torch.utils.data.Dataset`. Each
item is an `(image, target)` pair following the torchvision detection convention
(`boxes` in xyxy, integer `labels`, `area`, `iscrowd`, `image_id`, and the raw
annotations). Images download lazily on first access and are cached.

```python
from torch.utils.data import DataLoader
from pictograph import Client

client = Client()
dataset = client.datasets.as_pytorch(
    name="road-signs",
)
loader = DataLoader(dataset, batch_size=8, collate_fn=lambda batch: tuple(zip(*batch)))

for images, targets in loader:
    ...  # your training step
```

## Draw annotations

Source: [`viz.py`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/viz.py)

`draw_annotations` renders any Pictograph annotations onto an image using only Pillow, a
base dependency. All four annotation types render, and each class gets a stable, distinct
colour.

```python
from pictograph import Client, draw_annotations

client = Client()
annotations = client.annotations.get(
    dataset_name="road-signs",
    image="img-001.jpg",
)
annotated = draw_annotations(
    image="photo.jpg",
    annotations=annotations,
)
annotated.save("photo.annotated.png")
```