SAM3 auto-annotation

Auto-annotate images with SAM3 using point, box, and text prompts - one image or a whole dataset, from the editor, SDK, or CLI.

View as Markdown

SAM3 turns a prompt into a mask. Give it a point, a box, or a text phrase and it returns an editable polygon you can refine, export, or train on. Run it on one image, or as an async batch over a whole dataset. Manual annotation is always free.

Source: resources/auto_annotate.py

from pictograph import Client, PromptResult

client = Client()

result: PromptResult = client.auto_annotate.text(
    dataset_name="Road Signs",
    image_filename="img-001.jpg",
    text_prompt="stop sign",
)
client.annotations.save(
    dataset_name="Road Signs",
    image="img-001.jpg",
    annotations=result.annotations,
)

Both address the image by filename - there is no id to look up.

The three prompt modes

All three return a PromptResult - annotations, score, inference_time, status.

Mode	Best for
Point	One specific instance. Positive points mark it, negative points carve regions out.
Box	Many similar objects - drag one box, segment everything matching inside it.
Text	Zero-shot. Name a concept and every match is labelled, with no training.

# Point - (x, y) is the primary click; extra points refine the mask.
client.auto_annotate.point(
    dataset_name="Road Signs",
    image_filename="img-001.jpg",
    x=240,
    y=180,
    name="stop_sign",
    negative_points=[(120, 300)],
)

# Box - x/y/w/h in absolute pixels.
client.auto_annotate.box(
    dataset_name="Road Signs",
    image_filename="img-001.jpg",
    box={"x": 120, "y": 90, "w": 200, "h": 200},
    name="stop_sign",
)

# Text - output_type is "polygon" (default), "bbox", or "tag".
client.auto_annotate.text(
    dataset_name="Road Signs",
    image_filename="img-001.jpg",
    text_prompt="stop sign",
    output_type="polygon",
    confidence_threshold=0.3,
)

Text prompts are zero-shot, so they are the fastest way out of an empty dataset: label common objects before any model exists, or describe a long-tail concept once instead of hand-labelling hundreds of instances. For pixel-precise single instances, point and box prompts give finer control.

Auto-annotate a whole dataset

client.auto_annotate.dataset() enumerates a dataset’s images, runs SAM3 over them, and saves the results. It defaults to batch mode, one async job over many images, which is the right call above roughly 10 images.

from pictograph import Client, AnnotateReport

client = Client()

report: AnnotateReport = client.auto_annotate.dataset(
    dataset_name="Road Signs",
    classes=[("stop_sign", "bbox"), ("yield", "polygon")],
)
print(f"{report.annotations_added} annotations across {report.images_processed} images")

client.auto_annotate.dataset(
    dataset_name: str,
    classes: Sequence[BatchClass | tuple[str, str] | dict[str, str]],
    *,
    mode: AnnotateMode = "batch",
    confidence_threshold: float = 0.5,
    overwrite: bool = False,
    max_images: int | None = None,
    poll_interval: float = 5.0,
    timeout: float = 1800.0,
) -> AnnotateReport

Argument	Default	Purpose
`dataset_name`	required	Project name
`classes`	required	What to detect - see below
`mode`	`"batch"`	`batch` (async multi-image) or `text` (synchronous per-image)
`confidence_threshold`	`0.5`	SAM3 score cutoff (0–1)
`overwrite`	`False`	When `False`, skip images that already have annotations
`max_images`	`None`	Cap the number processed (useful for dry-runs)
`poll_interval`	`5.0`	`batch` mode - seconds between status polls
`timeout`	`1800`	`batch` mode - max seconds to wait

mode="text" runs one synchronous text prompt per image, per class, saving as it goes. It’s slower - no batching - so use it to debug a single image, or when the dataset is too small for the batch warmup to pay for itself.

Naming the classes

Every class carries its own output type, so a bare string is not accepted. Three shapes work - pick whichever is shortest:

# 1. Tuples - (name, output_type). A 1-tuple defaults to polygon.
classes=[("stop_sign", "bbox"), ("yield", "polygon")]

# 2. Dicts
classes=[
    {"name": "stop_sign", "output_type": "bbox"},
    {"name": "yield", "output_type": "polygon"},
]

# 3. BatchClass (canonical)
from pictograph.models.auto_annotate import BatchClass
classes=[BatchClass(name="stop_sign", output_type="bbox")]

output_type is "polygon" (default), "bbox", or "tag" - a mask, its enclosing box, or an image-level label with no geometry.

Skip or overwrite

By default it skips images that already carry at least one annotation, so a re-run only fills the gaps. Pass overwrite=True to re-annotate every image, replacing what is there.

What comes back

@dataclass
class AnnotateReport:
    dataset_name: str
    images_attempted: int
    images_processed: int
    images_skipped: int      # already annotated, and overwrite=False
    images_capped: int       # held back by max_images or the per-batch cap
    annotations_added: int
    failures: list[AnnotationFailure]
    job_id: str | None       # set only when mode="batch"

    @property
    def success(self) -> bool: ...

images_skipped and images_capped are deliberately separate: a run that leaves images unprocessed because of a cap also records a failure entry, so success stays False until the dataset is fully covered.

In batch mode, job_id lets you poll with client.auto_annotate.get_batch(job_id) or stop with client.auto_annotate.cancel_batch(job_id).

Driving the batch job directly

client.auto_annotate.batch is the primitive underneath: N images by M classes in one async job, off the request path. Reach for it when you want to choose the exact image list rather than have the dataset enumerated for you.

from pictograph.models.auto_annotate import BatchClass

job = client.auto_annotate.batch(
    dataset_name="Road Signs",
    image_filenames=["001.jpg", "002.jpg", "003.jpg"],
    classes=[
    BatchClass(name="stop sign", output_type="polygon"),
    BatchClass(name="yield sign", output_type="bbox"),
    ],
)  # waits by default
print(job.processed_images, "of", job.total_images, "images;",
      job.total_annotations_added, "annotations added")

Pass wait=False to return immediately, then poll:

job = client.auto_annotate.batch(
    dataset_name=dataset,
    image_filenames=filenames,
    classes=classes,
    wait=False,
)
job = client.auto_annotate.wait_for_batch(
    job_id=job.job_id,
)

Overlapping detections are de-duplicated per class, so you get clean masks across the set.

Price a batch before you run it

quote returns the cost of a batch without starting one, including for images you have not uploaded yet.

q = client.auto_annotate.quote(
    dataset_name="Road Signs",
    image_filenames=["001.jpg", "002.jpg"],
    classes=[BatchClass(name="stop sign")],
)
print(q.total_images, q.estimated_credits, q.sufficient)

Every result is an ordinary Pictograph annotation - a polygon with paths, or a box with x/y/w/h - so you can refine vertices in the editor, filter by class, export, or train on them directly.

Errors

Status	Exception	Cause
404	`NotFoundError`	Dataset doesn’t exist
402	`PaymentRequiredError`	Insufficient credits
422	`ValidationError`	Class name invalid, or `output_type` not one of `polygon` / `bbox` / `tag`

Per-image failures are recorded in report.failures - they don’t raise.

Next steps

Auto-annotate API reference - point / box / text / batch primitives
Credits - cost estimation per image
Train a model