Auto-annotate

SAM3 single-prompt (point, box, text) and async batch endpoints for AI-generated annotations.

Pictograph runs SAM3 (Segment Anything Model 3) on T4 GPUs for auto-annotation. Three single-image prompt modes plus an async batch endpoint for many images at once.

Every example below shows the Python SDK call and the equivalent raw REST request. The REST examples authenticate with an X-API-Key header; set PICTOGRAPH_API_KEY in your shell to copy-and-run them.

from pictograph import Client
client = Client()  # reads PICTOGRAPH_API_KEY

point — “click here, segment that”

Best when the user knows the object’s location. Returns one polygon annotation per prompt.

Arg	Type	Default	Notes
`dataset_name`	`str`	required	Project name within your org
`image_filename`	`str`	required	Image filename (not the UUID)
`x`, `y`	`int`	required	Anchor point in absolute pixels
`name`	`str`	`"object"`	Class name for the annotation
`positive_points`	`list[(x,y)]`	`None`	Extra positive anchors
`negative_points`	`list[(x,y)]`	`None`	Excluded regions
`score_threshold`	`float`	`0.75`	Minimum SAM3 score (0–1)

result = client.auto_annotate.point(
    dataset_name="my-dataset",
    image_filename="img-1.jpg",
    x=320, y=240,
    name="car",
    positive_points=[(310, 250)],   # optional extra positives
    negative_points=[(100, 100)],   # exclude regions
    score_threshold=0.75,
)
# result.annotations[0] is a polygon — call client.annotations.save() to persist.

curl -s -X POST "https://api.pictograph.io/api/v1/developer/auto-annotate/sam3/point" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_name": "my-dataset", "image_filename": "img-1.jpg", "x": 320, "y": 240, "name": "car", "positive_points": [[310, 250]], "negative_points": [[100, 100]], "score_threshold": 0.75}'

Returns PromptResult with status ∈ {"success", "no_detection", "below_threshold"}. On success, annotations[0] is a PolygonAnnotation.

box — “segment everything in this box”

Best when the user has drawn a rough bounding box. return_polygon=False returns only the refined bbox.

Arg	Type	Default	Notes
`dataset_name`	`str`	required	Project name
`image_filename`	`str`	required	Image filename
`box`	`{x,y,w,h}`	required	Bounding box in absolute pixels
`name`	`str`	required	Class name
`confidence_threshold`	`float`	`0.5`	Minimum SAM3 confidence (0–1)
`return_polygon`	`bool`	`True`	Also include a polygon, not just the bbox
`negative_boxes`	`list[{x,y,w,h}]`	`None`	Exclusion zones

result = client.auto_annotate.box(
    dataset_name="my-dataset",
    image_filename="img-1.jpg",
    box={"x": 100, "y": 200, "w": 200, "h": 150},
    name="car",
    return_polygon=True,           # also include polygon (not just bbox)
    confidence_threshold=0.5,
    negative_boxes=[{"x": 50, "y": 50, "w": 30, "h": 30}],
)

curl -s -X POST "https://api.pictograph.io/api/v1/developer/auto-annotate/sam3/box" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_name": "my-dataset", "image_filename": "img-1.jpg", "box": {"x": 100, "y": 200, "w": 200, "h": 150}, "name": "car", "return_polygon": true, "confidence_threshold": 0.5, "negative_boxes": [{"x": 50, "y": 50, "w": 30, "h": 30}]}'

text — “find all ”

Open-vocabulary text prompt. Best for many objects in one image.

Arg	Type	Default	Notes
`dataset_name`	`str`	required	Project name
`image_filename`	`str`	required	Image filename
`text_prompt`	`str`	required	Natural language description
`output_type`	`str`	`"polygon"`	`"polygon"` or `"bbox"`
`confidence_threshold`	`float`	`0.3`	Minimum confidence (0–1)
`max_detections`	`int`	`50`	Cap on result count (1–100)

result = client.auto_annotate.text(
    dataset_name="my-dataset",
    image_filename="img-1.jpg",
    text_prompt="red cars",
    output_type="polygon",         # or "bbox"
    confidence_threshold=0.3,
    max_detections=50,
)

curl -s -X POST "https://api.pictograph.io/api/v1/developer/auto-annotate/sam3/text" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_name": "my-dataset", "image_filename": "img-1.jpg", "text_prompt": "red cars", "output_type": "polygon", "confidence_threshold": 0.3, "max_detections": 50}'

batch — async, many images

Use for >10 images. Kicks off one job; polls until terminal status. SAM3 jobs (model_id omitted) need at least one class; trained-model jobs (model_id=<uuid>) accept empty classes for classification models. On the REST wire, SAHI is opted in with sahi_enabled / sahi_slice_size.

Arg	Type	Default	Notes
`dataset_name`	`str`	required	Project name
`image_filenames`	`list[str]`	required	Filenames to process
`classes`	`list[BatchClass]`	required for SAM3	`{name, output_type}` per class
`confidence_threshold`	`float`	`0.5`	Minimum confidence
`model_id`	`str \| None`	`None`	Trained-model UUID; `None` routes to SAM3
`top_k`	`int`	`1`	Classifier-only — tags per image
`wait`	`bool`	`True`	Poll until terminal

from pictograph import BatchClass

job = client.auto_annotate.batch(
    dataset_name="my-dataset",
    image_filenames=["img-1.jpg", "img-2.jpg", "..."],
    classes=[
        BatchClass(name="car", output_type="polygon"),
        BatchClass(name="person", output_type="bbox"),
    ],
    confidence_threshold=0.5,
    wait=True,
    poll_interval=5.0,
    timeout=1800.0,                # 30 min default
)
print(job.status, job.processed_images, job.total_annotations_added)

curl -s -X POST "https://api.pictograph.io/api/v1/developer/auto-annotate/batch" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_name": "my-dataset", "image_filenames": ["img-1.jpg", "img-2.jpg"], "classes": [{"name": "car", "output_type": "polygon"}, {"name": "person", "output_type": "bbox"}], "confidence_threshold": 0.5}'

wait=False returns the job immediately — poll later, or cancel:

job = client.auto_annotate.get_batch(job.job_id)
job = client.auto_annotate.wait_for_batch(job.job_id, timeout=600.0)
client.auto_annotate.cancel_batch(job.job_id)

# Poll batch progress.
curl -s "https://api.pictograph.io/api/v1/developer/auto-annotate/batch/{job_id}" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

# Cancel a pending or running batch.
curl -s -X POST "https://api.pictograph.io/api/v1/developer/auto-annotate/batch/{job_id}/cancel" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

SAHI sliced inference (small objects)

For high-resolution images with small objects (drone shots, wide industrial scenes, litter detection), enable SAHI. Each image is sliced into overlapping tiles, every tile runs at near-native resolution alongside one full-image pass, and tile fragments are merged back into whole instances server-side:

job = client.auto_annotate.batch(
    dataset_name="my-dataset",
    image_filenames=["site-4k.jpg"],
    classes=[BatchClass(name="person", output_type="polygon")],
    sahi=True,
    sahi_slice_size=640,           # tile edge in px, 256-1024
)

curl -s -X POST "https://api.pictograph.io/api/v1/developer/auto-annotate/batch" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_name": "my-dataset", "image_filenames": ["site-4k.jpg"], "classes": [{"name": "person", "output_type": "polygon"}], "sahi_enabled": true, "sahi_slice_size": 640}'

SAHI is SAM3-only (a trained-model job with SAHI enabled is rejected with a 400). Smaller slices find smaller objects but run more passes; cost scales with each image’s tile count, which the API computes exactly from stored image dimensions. CLI equivalent: pictograph auto-annotate batch ... --sahi --sahi-slice-size 640.

auto_annotate_dataset workflow

Higher-level helper that paginates a dataset’s image list, runs batch or text mode, and returns a per-image report:

from pictograph.pipelines import auto_annotate_dataset

report = auto_annotate_dataset(
    client,
    "my-dataset",
    classes=[("car", "polygon"), ("person", "bbox")],
    mode="batch",                  # "batch" | "text"
    confidence_threshold=0.5,
    overwrite=False,               # skip already-annotated images
    max_images=None,               # None = all
)
print(report.images_processed, report.annotations_added, len(report.failures))

Choosing a mode

Scenario	Mode
User clicks one spot	`point`
User drags a rough box	`box`
Many images, known classes	`batch` (or `text` per image for small datasets)
Single image, multiple objects	`text`

Cost

SAM3 auto-annotation draws on your USD compute credit (available on every tier). Point and box prompts share one image embedding per session, so follow-up prompts on the same image are near-free; batch is priced per image processed. Prices are USD-denominated — call

client.credits.estimate("sam3_auto_annotation", quantity=N)

for the live cost, and read PaymentRequiredError.required on rejection for the exact ask. Amounts are integer micro-USD (1 USD = 1,000,000 µUSD).

Common errors

Status	Exception	Cause
402	`PaymentRequiredError`	Out of credits
404	`NotFoundError`	Dataset or image missing
408	`PollTimeoutError`	Batch job didn’t finish within `timeout` (job keeps running on the backend)