Local inference

Load a trained model by name and predict. Weights download once and are cached; the result is typed by task.

Source: inference/__init__.py

pip install 'pictograph[inference]'

from pictograph import DetectionModel, DetectionResult, get_model

model: DetectionModel = get_model(name="Shelf Detector", task="object_detection")
result: DetectionResult = model.predict(image="photo.jpg")

for p in result.predictions:
    print(p.name, round(p.confidence, 2), p.bounding_box)

task= types the result and must match the model, or loading raises. Without it you get AnyModel, narrowed with isinstance.

What you get back

Each task has one model class and one result class, whatever the runtime:

A classifier returns ranked classes instead of shapes, and its top is never None:

from pictograph import ClassificationModel, ClassificationResult, get_model

model: ClassificationModel = get_model(name="Aisle Classifier", task="classification")
result: ClassificationResult = model.predict(
    image="aisle.jpg",
    top_k=3,
)

print(result.top.name, round(result.top.confidence, 2))
for score in result.classes:
    print(score.name, round(score.confidence, 3))

Inputs and batches

predict takes a file path, an http(s) URL, bytes, a BGR NumPy array (what cv2.imread returns) or a PIL image.

import cv2
from pictograph import get_model

with get_model(name="Shelf Detector", task="object_detection") as model:
    print(len(model.predict(image=cv2.imread("photo.jpg")).predictions))
    for result in model.predict_batch(images=["a.jpg", "b.jpg", "c.jpg"]):
        print(len(result.predictions))

Close a model when you’re done (with does it): it holds memory until closed.

Weight formats and runtimes

format= picks the weights, and the runtime follows:

from pictograph import get_model

onnx = get_model(name="Shelf Detector", task="object_detection")
pte = get_model(name="Shelf Detector", task="object_detection", format="pte")
torch_model = get_model(name="Shelf Detector", task="object_detection", format="pytorch")

for m in (onnx, pte, torch_model):
    result = m.predict(image="photo.jpg")  # same type, same fields
    print(m.backend, m.device, len(result.predictions))

onnxruntime coreml 7
executorch cpu 7
pytorch mps 7

Every runtime shares the same pre- and post-processing, so the same weights give the same results.

What actually ran

These report what ran, not what you asked for: a CUDA session that fell back to the CPU reports cpu.

Every result carries backend, device and inference_ms too.

Precision

precision="fp16" fetches fp16 weights with fp32 inputs and outputs, so your code doesn’t change. .onnx, .pte and .engine files exist per precision; a .pth only at the precision it trained at (asking for another raises).

Device

device="auto" (the default) picks the best device the format can use; "cpu" gives identical numbers every run, which suits CI.

A device the format can’t use raises, naming the ones it can.

TensorRT engines are not portable

A TensorRT engine runs only where it was compiled. It is bound to:

1. the GPU architecture it was built on: T4 sm75, A100 sm80, A10G sm86, L4 sm89, H100 sm90;
2. the exact TensorRT version: a minor version change invalidates it;
3. its precision: fp32 and fp16 are different engines;
4. its input shape profile.

The filename records all four (sm80-trt10.13.3.9-fp16.engine), and the loader checks them against your GPU first:

RuntimeError: This TensorRT engine was built for trt-10.13.3.9 on sm80; this
device is sm75. A TensorRT plan is compiled for one GPU architecture and one
TensorRT version and cannot be loaded anywhere else - rebuild the engine for
your device.

get_model(format="engine") fetches the engine for your GPU; target= fetches another:

from pictograph import get_model

model = get_model(
    name="Shelf Detector",
    task="object_detection",
    format="engine",
    target="sm80",  # fetch the A100 engine from a machine that is not one
)

If you rename an engine, keep its manifest beside it as <name>.json. A .pte has none of these limits: its default CPU build (XNNPACK) runs anywhere.

Load offline

load_model needs no API key and no network. Give it the weights and the config.json from the model’s Files tab; the file suffix picks the runtime:

from pictograph import DetectionModel, load_model

for weights in (
    "shelf-detector.onnx",
    "shelf-detector-xnnpack.pte",
    "shelf-detector-sm75-trt10.13.3.9.engine",
):
    model: DetectionModel = load_model(
        weights=weights,
        config="shelf-detector.config.json",
        task="object_detection",
    )
    print(model.backend, model.device)

Run a public model

Import a public model into your organization, then load your copy by name:

from pictograph import Client, DetectionModel, get_model

client = Client()
copy = client.models.fork(
    organization="acme-vision",
    model="stop-sign-detector",
)
model: DetectionModel = get_model(name=copy.name, task="object_detection")

Other local tools

Score a model against labelled data

Source: metrics/_detection.py

evaluate_detections matches predictions to ground truth by IoU, locally, and returns per-class and overall precision, recall, F1 and average precision (mean_average_precision).

from pictograph import Client
from pictograph.metrics import evaluate_detections

client = Client()
filenames = ["img-001.jpg", "img-002.jpg"]
ground_truth = {
    name: client.annotations.get(dataset="road-signs", image=name)
    for name in filenames
}
predictions = {name: run_my_model(name) for name in filenames}

result = evaluate_detections(predictions, ground_truth, iou_threshold=0.5)
print(result.precision, result.recall, result.f1)
for name, m in result.per_class.items():
    print(name, m.precision, m.recall, m.support)

confusion_matrix() shows which classes get confused. client.model_evaluations runs the same math on Pictograph.

Feed a PyTorch DataLoader

Source: _torch_dataset.py

client.datasets.as_pytorch returns a torch.utils.data.Dataset of (image, target) pairs in torchvision’s detection layout (boxes as xyxy, integer labels, area, iscrowd, image_id). Images download on first use and are cached.

from pictograph import Client
from torch.utils.data import DataLoader

client = Client()
dataset = client.datasets.as_pytorch(dataset="road-signs")
loader = DataLoader(dataset, batch_size=8, collate_fn=lambda batch: tuple(zip(*batch)))

for images, targets in loader:
    ...  # your training step

Draw annotations

Source: viz.py

draw_annotations draws in the app’s annotation look, using Pillow only. Pass a model’s result as it is:

from pictograph import DetectionModel, DetectionResult, draw_annotations, get_model

model: DetectionModel = get_model(name="Shelf Detector", task="object_detection")
result: DetectionResult = model.predict(image="aisle.jpg")
annotated = draw_annotations(image="aisle.jpg", annotations=result)
annotated.save("aisle.drawn.png")

Stored annotations (client.annotations.get(...)) draw the same way. A classifier’s result goes in classification=, its classes at or above min_confidence (0.5) drawn as badges. An image path is drawn with its EXIF orientation applied, as predict read it.

look= changes any part of the look, for all classes or per class: line widths, fills, the label’s color, text and position, and the badges. Pass a preset (frost, smoke) or only the fields that change. show_confidence=True adds each score to its label:

from pictograph import DetectionModel, DetectionResult, draw_annotations, get_model

model: DetectionModel = get_model(name="Shelf Detector", task="object_detection")
result: DetectionResult = model.predict(image="aisle.jpg")
annotated = draw_annotations(
    image="aisle.jpg",
    annotations=result,
    look={"box": {"fill": 0}, "label": {"position": "tl"}},
    show_confidence=True,
)
annotated.save("aisle.drawn.png")

The CLI draws too: pictograph models predict "Shelf Detector" aisle.jpg --draw aisle.drawn.jpg (CLI).