Model evaluations

Score a trained model against an export's labels: precision, recall, F1 and mAP per class, a confusion matrix, and its best and worst images.

View as Markdown

On this page

evaluate
create
compute_options
wait_for_completion
get
report
list
iter
cancel
Reading the metrics
Common errors

The export fixes the images, labels and classes, so a score is repeatable.

Model type	Scored by
Object detection, instance segmentation	Precision, recall, F1 and mAP, overall and per class
Classification	Top-1 accuracy
Semantic segmentation	Pixel mIoU and Dice
Keypoints	Not supported

Engine - it runs your model’s TensorRT engine, the one a deployment serves, at the precision and on the GPU you choose (compute_options).
First run - a model’s first evaluation on a GPU and precision builds that engine first; the evaluation waits as pending, with engine_build set, and the build isn’t charged.
Cost - the GPU time it takes, from your credits; a failed or cancelled evaluation is free.
Free alternative - to score predictions you already have, use pictograph.metrics; the math is the same.

evaluate

Start an evaluation and wait for it to finish.

Source: ModelEvaluations.evaluate

Arg	Type	Default	Notes
`model`	`str`	required	Model name or id
`dataset`	`str`	required	Dataset name or id
`export`	`str`	required	Name or id of a completed export of the dataset
`iou_threshold`	`float`	`0.5`	Minimum overlap (IoU) for a prediction to match a label
`confidence_threshold`	`float`	`0.5`	Drop predictions below this confidence
`precision`	`Literal["fp32", "fp16"] \| None`	`None`	Engine precision (default `fp32`)
`gpu`	`Literal["t4", "l4"] \| None`	`None`	GPU to run on (default `t4`)
`poll_interval`	`float`	`3.0`	Seconds between checks
`timeout`	`float`	`1800.0`	Max seconds to wait

evaluation = client.model_evaluations.evaluate(
    model="Swift Falcon",
    dataset="road-signs",
    export="v1",
    iou_threshold=0.5,
    confidence_threshold=0.5,
    precision="fp16",
    gpu="l4",
)

overall = evaluation.overall_metrics
print(overall.precision, overall.recall, overall.f1, overall.map)

for row in evaluation.per_class_metrics or []:
    print(row.class_name, row.precision, row.recall, row.support, row.ap)

pictograph models evaluate "Swift Falcon" --dataset road-signs --export v1 --precision fp16 --gpu l4 --wait

# REST has no combined call: create, then poll get.
curl -s -X POST "https://api.pictograph.io/api/v1/developer/model-evaluations" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "Swift Falcon", "dataset": "road-signs", "export": "v1", "precision": "fp16", "gpu": "l4"}'

Returns ModelEvaluation

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

Raises ApiError if the evaluation fails or is cancelled.

create

Start an evaluation and return at once: running, or pending while its engine builds.

Source: ModelEvaluations.create

Arg	Type	Default	Notes
`model`	`str`	required	Model name or id
`dataset`	`str`	required	Dataset name or id
`export`	`str`	required	Name or id of a completed export of the dataset
`iou_threshold`	`float`	`0.5`	As in `evaluate`
`confidence_threshold`	`float`	`0.5`	As in `evaluate`
`precision`	`Literal["fp32", "fp16"] \| None`	`None`	As in `evaluate`
`gpu`	`Literal["t4", "l4"] \| None`	`None`	As in `evaluate`

evaluation = client.model_evaluations.create(
    model="Swift Falcon",
    dataset="road-signs",
    export="v1",
)
print(evaluation.id, evaluation.status)

# Prints the new evaluation's id at once; add --wait to block until it finishes.
pictograph models evaluate "Swift Falcon" --dataset road-signs --export v1 --iou 0.5 --confidence 0.5

curl -s -X POST "https://api.pictograph.io/api/v1/developer/model-evaluations" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "Swift Falcon",
    "dataset": "road-signs",
    "export": "v1",
    "iou_threshold": 0.5,
    "confidence_threshold": 0.5,
    "precision": "fp32",
    "gpu": "t4"
  }'

Returns ModelEvaluation

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

compute_options

What an evaluation can run on: each GPU with its memory and hourly price (µUSD), each precision, and the default (fp32 on a T4).

Source: ModelEvaluations.compute_options

options = client.model_evaluations.compute_options()
for gpu in options.gpus:
    print(gpu.gpu, gpu.vram_gb, gpu.cost_per_hour_micro_usd)
for precision in options.precisions:
    print(precision.precision, precision.label, precision.description)

pictograph models evaluate-options

curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations/compute-options" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns EvaluationComputeOptions

EvaluationComputeOptions · 3 fields

class EvaluationComputeOptions(BaseModel):
    """What an evaluation may run on: each GPU with its memory and hourly price, each precision, the default."""
    gpus: list[EvaluationComputeGpu] = []
    precisions: list[EvaluationComputePrecision] = []
    default: EvaluationCompute = ...

wait_for_completion

Wait for an evaluation you started with create to finish.

Source: ModelEvaluations.wait_for_completion

Arg	Type	Default	Notes
`evaluation`	`str`	required	Evaluation id
`poll_interval`	`float`	`3.0`	Seconds between checks
`timeout`	`float`	`1800.0`	Max seconds to wait

evaluation = client.model_evaluations.wait_for_completion(
    evaluation="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
    poll_interval=5.0,
    timeout=3600.0,
)
print(evaluation.overall_metrics.f1)

pictograph models evaluation 9d8b3c17-2f45-4e6a-b810-5c7e2a94f036 --wait --poll-interval 5 --timeout 3600

# Poll until status is terminal.
curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations/<evaluation-id>" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns ModelEvaluation

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

get

One evaluation, with its metrics once it has completed (reading them).

Source: ModelEvaluations.get

Arg	Type	Default	Notes
`evaluation`	`str`	required	Evaluation id

evaluation = client.model_evaluations.get(
    evaluation="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
)
print(evaluation.status, evaluation.progress, evaluation.evaluated_images)

pictograph models evaluation 9d8b3c17-2f45-4e6a-b810-5c7e2a94f036 --json

curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations/9d8b3c17-2f45-4e6a-b810-5c7e2a94f036" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns ModelEvaluation

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

report

A completed evaluation as one JSON report: the model, dataset and export, every metric, the PR curves and confusion matrix, the best and worst images and any warnings, under a schema_version.

Source: ModelEvaluations.report

Arg	Type	Default	Notes
`evaluation`	`str`	required	Id of a completed evaluation

import json
from pathlib import Path

report = client.model_evaluations.report(
    evaluation="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
)
Path("evaluation.json").write_text(json.dumps(report, indent=2))

# The CLI writes the JSON report; the HTML report is made in the app.
pictograph models evaluation 9d8b3c17-2f45-4e6a-b810-5c7e2a94f036 --report evaluation.json

curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations/9d8b3c17-2f45-4e6a-b810-5c7e2a94f036/export" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns dict[str, Any]

The REST response wraps the report as {"success": true, "report": {...}}.

list

One page of evaluations, newest first.

Source: ModelEvaluations.list

Arg	Type	Default	Notes
`model`	`str \| None`	`None`	Only this model’s evaluations, by name or id
`dataset`	`str \| None`	`None`	Only this dataset’s evaluations
`limit`	`int`	`50`	Page size (max 200)

history = client.model_evaluations.list(
    model="Swift Falcon",
    limit=20,
)
for run in history:
    score = run.overall_metrics.f1 if run.overall_metrics else None
    print(run.id, run.status, score)

pictograph models evaluations "Swift Falcon" -n 20

curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations?model=Swift%20Falcon&limit=20" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns list[ModelEvaluation]

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

REST also pages with offset.

iter

Every evaluation, fetched page by page.

Source: ModelEvaluations.iter

Arg	Type	Default	Notes
`model`	`str \| None`	`None`	As in `list`
`dataset`	`str \| None`	`None`	As in `list`
`page_size`	`int`	`50`	Items fetched per request

for evaluation in client.model_evaluations.iter(model="Swift Falcon"):
    score = evaluation.overall_metrics.f1 if evaluation.overall_metrics else None
    print(evaluation.id, evaluation.status, score)

# The CLI pages for you up to --limit.
pictograph models evaluations "Swift Falcon" --limit 1000

curl -s "https://api.pictograph.io/api/v1/developer/model-evaluations?limit=50&offset=50" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns OffsetPager[ModelEvaluation]

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

cancel

Stop a pending or running evaluation.

Source: ModelEvaluations.cancel

Arg	Type	Default	Notes
`evaluation`	`str`	required	Evaluation id

evaluation = client.model_evaluations.cancel(
    evaluation="9d8b3c17-2f45-4e6a-b810-5c7e2a94f036",
)
print(evaluation.status)

pictograph models evaluation 9d8b3c17-2f45-4e6a-b810-5c7e2a94f036 --cancel

curl -s -X POST "https://api.pictograph.io/api/v1/developer/model-evaluations/9d8b3c17-2f45-4e6a-b810-5c7e2a94f036/cancel" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns ModelEvaluation

ModelEvaluation · 28 fields

class ModelEvaluation(BaseModel):
    """A model-evaluation run + its metric summary."""
    id: str
    organization_id: str
    model_id: str
    dataset_id: str
    export_id: str | None = None
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled']
    progress: int = 0
    iou_threshold: float = 0.5
    confidence_threshold: float = 0.5
    total_images: int = 0
    evaluated_images: int = 0
    failed_images: int = 0
    overall_metrics: EvalOverallMetrics | None = None
    per_class_metrics: list[EvalClassMetrics] | None = None
    confusion_matrix: EvalConfusionMatrix | None = None
    worst_images: list[EvalImagePerformance] | None = None
    best_images: list[EvalImagePerformance] | None = None
    seg_metrics: dict[str, Any] | None = None
    config: dict[str, Any] | None = None
    error_message: str | None = None
    created_at: datetime | None = None
    updated_at: datetime | None = None
    started_at: datetime | None = None
    completed_at: datetime | None = None
    precision: str = 'fp32'
    gpu_type: str = 't4'
    version_id: str | None = None
    engine_build: dict[str, Any] | None = None

Reading the metrics

overall_metrics - micro-averaged tp, fp, fn, precision, recall, f1, macro_f1 and map
per_class_metrics - one entry per class, adding class_name, support and ap
confusion_matrix - labels and a dense grid (rows are ground truth, columns predictions); the last label is __background__, so the last column counts false negatives and the last row false positives
worst_images - the images with the most errors, each with filename, tp, fp, fn, gt_count and pred_count

matrix = evaluation.confusion_matrix
print(matrix.labels)
for label, row in zip(matrix.labels, matrix.grid):
    print(label, row)

for image in evaluation.worst_images or []:
    print(image.filename, image.fp, image.fn)

Common errors

Status	Exception	Cause
400	`ValidationError`	A model that isn’t ready or is a keypoint model, or an export that isn’t completed or is too old to evaluate
402	`PaymentRequiredError`	The evaluation costs more than your balance
403	`ForbiddenError`	A key below `member` (starting an evaluation)
404	`NotFoundError`	The model, dataset, export or evaluation doesn’t exist
409	`ConflictError`	`report` on an evaluation that hasn’t completed, or `cancel` on one that has finished
-	`PollTimeoutError`	The wait passed `timeout`; the evaluation keeps going