---
title: Model evaluations
description: Score a trained detection or instance-segmentation model against an export's ground truth - precision, recall, F1, mAP, confusion matrix, worst images.
section: API Reference
order: 9.5
---
An evaluation runs your trained model over an **export** and scores its predictions
against that export's ground truth. The inference runs server-side, so you do not
download weights or images to get metrics, and your annotations are never modified.

The eval set is the export, not the whole dataset. An export records exactly which
images and which classes it contains, so evaluating against one gives you a fixed,
repeatable scoring set and puts evaluation on the same footing as training, which
also runs off exports.

Only **detection** and **instance-segmentation** models can be evaluated. Other
model types return a validation error. For scoring predictions you already hold,
without spending compute, use the offline
[`pictograph.metrics`](/docs/local-inference#score-a-model-against-labelled-data)
helpers - the metric math is identical, so a server run and a local run on the same
data agree.

Evaluation is billed on the trained-model batch-inference path, in compute credits.

There is no `pictograph` CLI group for evaluations; use the Python SDK or raw REST.

## evaluate

The one-call path: start the run and block until it finishes. Use this unless you
need to do something else while it runs.

Source: [`ModelEvaluations.evaluate`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `model` | `str` | required | Trained model name, or a model id |
| `dataset_name` | `str` | required | Dataset name. |
| `export_name` | `str` | required | Completed export whose images and class filter are the eval set. Export names are unique within a dataset, not globally, which is why the pair is needed |
| `iou_threshold` | `float` | `0.5` | Minimum IoU for a prediction to count as matching a ground-truth box |
| `confidence_threshold` | `float` | `0.5` | Minimum detection confidence to keep a prediction |
| `poll_interval` | `float` | `3.0` | Seconds between status checks |
| `timeout` | `float` | `1800.0` | Seconds before `PollTimeoutError` |

```python
evaluation = client.model_evaluations.evaluate(
    model="Swift Falcon",
    dataset_name="road-signs",
    export_name="v1",
    iou_threshold=0.5,
    confidence_threshold=0.5,
)

overall = evaluation.overall_metrics
print(overall.precision, overall.recall, overall.f1, overall.map)

for row in evaluation.per_class_metrics or []:
    print(row.class_name, row.precision, row.recall, row.support, row.ap)
```

```bash
# REST has no combined call: create, then poll get.
curl -s -X POST "https://api.pictograph.io/api/v1/developer/model-evaluations" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" -H "Content-Type: application/json" \
  -d '{"model_id": "<model-uuid>", "export_id": "<export-uuid>"}'
```

**Returns** `ModelEvaluation`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

Raises `ApiError` if the run failed or was cancelled, `PollTimeoutError` if `timeout` elapsed first (the run keeps going, so fetch it later with `get`).

## create

Start an evaluation and return immediately with status `pending`.

Source: [`ModelEvaluations.create`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `model` | `str` | required | Trained model name, or a model id |
| `dataset_name` | `str` | required | Dataset name. |
| `export_name` | `str` | required | Completed export defining the eval set |
| `iou_threshold` | `float` | `0.5` | Match threshold |
| `confidence_threshold` | `float` | `0.5` | Prediction cutoff |

```python
evaluation = client.model_evaluations.create(
    model="Swift Falcon",
    dataset_name="road-signs",
    export_name="v1",
)
print(evaluation.id, evaluation.status)
```

```bash
curl -s -X POST "https://api.pictograph.io/api/v1/developer/model-evaluations" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model_id": "6f1c2f0e-6a1e-4a55-9f3a-2f2d3b4c5d6e",
    "export_id": "b4d1c7e2-3f58-4a90-8c6b-1d2e3f4a5b6c",
    "iou_threshold": 0.5,
    "confidence_threshold": 0.5
  }'
```

**Returns** `ModelEvaluation`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

The SDK resolves the model name and the dataset-plus-export pair into ids for you;
REST takes the ids directly. The export must be `completed` and must carry a
recorded image set, so exports created before per-export image tracking must be
re-created before they can be scored.

## get

Fetch one evaluation, including its metrics once it has completed.

Source: [`ModelEvaluations.get`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `evaluation_id` | `str` | required | Evaluation id from `create` or `list` |

```python
evaluation = client.model_evaluations.get(
    evaluation_id="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
)
print(evaluation.status, evaluation.progress, evaluation.evaluated_images)
```

```bash
curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations/9d8b3c17-2f45-4e6a-b810-5c7e2a94f036" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
```

`overall_metrics` carries micro-averaged `tp` / `fp` / `fn`, `precision`, `recall`,
`f1`, `macro_f1` and `map`. Each entry in `per_class_metrics` adds `class_name`,
`support` and per-class `ap`. `confusion_matrix` gives `labels` and a dense `grid`
where rows are ground truth and columns are predictions; the final label is
`__background__`, so the last column holds false negatives and the last row holds
false positives. `worst_images` lists the images with the most errors, each with
`filename`, `tp`, `fp`, `fn`, `gt_count` and `pred_count`.

```python
matrix = evaluation.confusion_matrix
print(matrix.labels)
for label, row in zip(matrix.labels, matrix.grid):
    print(label, row)

for image in evaluation.worst_images or []:
    print(image.filename, image.fp, image.fn)
```

**Returns** `ModelEvaluation`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

## wait_for_completion

Poll an evaluation you started earlier until it reaches a terminal status.

Source: [`ModelEvaluations.wait_for_completion`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `evaluation_id` | `str` | required | Evaluation id |
| `poll_interval` | `float` | `3.0` | Seconds between status checks |
| `timeout` | `float` | `1800.0` | Seconds before `PollTimeoutError` |
| `sleep` | `Callable \| None` | `None` | Override the sleep function, for tests |

```python
evaluation = client.model_evaluations.wait_for_completion(
    evaluation_id="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
    poll_interval=5.0,
    timeout=3600.0,
)
print(evaluation.overall_metrics.f1)
```

```bash
# Poll until status is terminal.
curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations/<evaluation-id>" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
```

**Returns** `ModelEvaluation`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

A timeout does not stop the run. The evaluation keeps going server-side and can be
read later with `get`.

## list

The organization's evaluations, newest first, optionally narrowed to one model or
one dataset.

Source: [`ModelEvaluations.list`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `model` | `str \| None` | `None` | Only this model's evaluations, by name or id |
| `dataset_name` | `str \| None` | `None` | Only this dataset's evaluations |
| `limit` | `int` | `50` | Page size, maximum 200 |

```python
history = client.model_evaluations.list(
    model="Swift Falcon",
    limit=20,
)
for run in history:
    score = run.overall_metrics.f1 if run.overall_metrics else None
    print(run.id, run.status, score)
```

```bash
curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations?model_id=6f1c2f0e-6a1e-4a55-9f3a-2f2d3b4c5d6e&limit=20" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
```

**Returns** `list[ModelEvaluation]`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

The REST filters take ids: `model_id`, and `project_id` for the dataset. This
endpoint returns a single page and takes no offset, so raise `limit` rather than
paging.

## cancel

Stop a `pending` or `running` evaluation. First terminal status wins, so cancelling
an already-finished run is a 409 rather than a silent no-op.

Source: [`ModelEvaluations.cancel`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `evaluation_id` | `str` | required | Evaluation id |

```python
evaluation = client.model_evaluations.cancel(
    evaluation_id="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
)
print(evaluation.status)
```

```bash
curl -s -X POST "https://api.pictograph.io/api/v1/developer/model-evaluations/9d8b3c17-2f45-4e6a-b810-5c7e2a94f036/cancel" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
```

**Returns** `ModelEvaluation`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

## iter

Auto-paging iterator over the org's evaluations.

Source: [`ModelEvaluations.iter`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.67/src/pictograph/resources/model_evaluations.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `model` | `str \| None` | `None` | Model name or id. |
| `dataset_name` | `str \| None` | `None` | Dataset name. |
| `page_size` | `int` | `50` | Rows fetched per underlying request. Tuning only - the iterator yields every item either way. |

```python
for evaluation in client.model_evaluations.iter(model="Swift Falcon"):
    score = evaluation.overall_metrics.f1 if evaluation.overall_metrics else None
    print(evaluation.id, evaluation.status, score)
```

```bash
# The endpoint pages with limit + offset; `iter` walks it for you.
curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations?limit=50&offset=50" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
```

**Returns** `OffsetPager[ModelEvaluation]`

<details>
<summary><code>ModelEvaluation</code> &middot; 22 fields</summary>

```python
class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalWorstImage] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
```

</details>

## Common errors

| Status | Exception | Cause |
|---|---|---|
| 400 | `ValidationError` | Model is not ready, is not a detection or instance-segmentation model, has no weights, or the export is not completed or has no recorded image set |
| 402 | `PaymentRequiredError` | Not enough compute credits to start the run |
| 404 | `NotFoundError` | Model, export, dataset, or evaluation not found in your organization |
| 409 | `ConflictError` | `cancel` on an evaluation that already finished |