---
title: "SAM3 API: segment images from a text, box or point prompt"
description: "Run SAM3 on any image - a file, a URL or bytes - with a text, box or point prompt. Nothing is saved."
section: "API Reference"
order: 4.5
---
SAM3 is Pictograph's built-in segmentation model: it needs no training and knows any
class you can name. Send it an image with one prompt and it returns the objects it
finds, as polygons or boxes.

The image is sent with the call. It does not have to be in a dataset, and nothing is
written: no dataset, no image, no annotation. For an image that is already in a
dataset, the [auto-annotate prompts](/docs/api-reference/auto-annotate.md) run on it in
place, and [`batch`](/docs/api-reference/auto-annotate#batch) saves draft labels for
many images.

| Prompt | What comes back |
|---|---|
| `text`: a short noun phrase (`"red car"`) | Every match, named after the phrase |
| `box`: a rough box around one object | That object and every object like it |
| `points`: points on one object | That object's mask, as one annotation |

## predict

Run SAM3 on one image with one prompt. Give exactly one of `text`, `box` or `points`.
The first call after a quiet spell starts a GPU and can take a minute or two; more
prompts on the same image after that are fast.

Source: [`Sam3.predict`](https://github.com/pictograph-io/pictograph-sdk/blob/v1.69.160/src/pictograph/resources/sam3.py)

| Arg | Type | Default | Notes |
|---|---|---|---|
| `image` | `str \| Path \| bytes` | required | A file path, an `http(s)://` URL (fetched on your machine, then uploaded), or the image bytes. JPEG, PNG, WebP, BMP, GIF or TIFF, up to 25 MB and 50 megapixels |
| `text` | `str \| None` | `None` | Text prompt: a short noun phrase |
| `box` | `dict \| tuple \| None` | `None` | Box prompt: `{"x", "y", "w", "h"}` or `(x, y, w, h)` in absolute pixels |
| `points` | `list[(x, y)] \| None` | `None` | Point prompt: points on one object |
| `negative_points` | `list[(x, y)] \| None` | `None` | With `points`: regions to exclude |
| `negative_boxes` | `list[box] \| None` | `None` | With `box`: regions to exclude |
| `name` | `str \| None` | `None` | Class name for the results. Default: the phrase, or `"object"` |
| `output` | `str` | `"polygon"` | `"polygon"` for mask outlines, `"bbox"` for their bounding boxes |
| `confidence` | `float \| None` | `None` | Minimum score, 0 to 1. Default by prompt: 0.3 text, 0.5 box, 0.75 points |
| `max_detections` | `int` | `50` | Text prompts: the most matches to return, best first, 1 to 100 |

```python
result = client.sam3.predict(
    image="shelf.jpg",
    text="price tag",
)
for annotation in result.annotations:
    print(annotation.name, annotation.type, annotation.confidence)
```

```bash
pictograph sam3 predict shelf.jpg --text "price tag" --json
```

```bash
curl -s -X POST "https://api.pictograph.io/api/v1/developer/sam3/predict" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -F "file=@shelf.jpg" \
  -F "text=price tag"
```

**Returns** `Sam3Prediction`

<details>
<summary><code>Sam3Prediction</code> &middot; 10 fields</summary>

```python
class Sam3Prediction(BaseModel):
    """SAM3's answer to one prompt on an image you sent. Nothing was saved."""
    status: Literal['success', 'no_detection', 'below_threshold']
    annotations: list[Annotation] = []
    score: float | None = None
    inference_time: float | None = None
    prompt: Literal['text', 'box', 'point']
    width: int
    height: int
    image_sha256: str | None = None
    confidence_threshold: float | None = None
    degraded: bool = False
```

</details>

`status` is `"success"` when `annotations` holds at least one object,
`"no_detection"` when SAM3 found nothing for the prompt, and `"below_threshold"` when
it found objects but none scored above `confidence_threshold`. `degraded` is `true`
when the image held more matches than fit in GPU memory, so only the most confident
came back: `confidence_threshold` says where the cut fell. The REST response wraps the
result as `{"data": {...}}`.

## Prompts

A box prompt with a region carved out, from a URL:

```python
result = client.sam3.predict(
    image="https://example.com/street.jpg",
    box=(120, 80, 200, 160),
    negative_boxes=[(150, 100, 20, 20)],
    name="car",
    output="bbox",
)
```

```bash
pictograph sam3 predict https://example.com/street.jpg --box 120,80,200,160 --negative-box 150,100,20,20 --name car --output bbox
```

```bash
curl -s https://example.com/street.jpg -o street.jpg
curl -s -X POST "https://api.pictograph.io/api/v1/developer/sam3/predict" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -F "file=@street.jpg" \
  -F "box=120,80,200,160" -F "negative_box=150,100,20,20" \
  -F "name=car" -F "output=bbox"
```

Points on one object, with one point that excludes a region:

```python
result = client.sam3.predict(
    image=frame_bytes,
    points=[(320, 240), (330, 260)],
    negative_points=[(300, 400)],
    name="forklift",
)
```

```bash
cat frame.jpg | pictograph sam3 predict - --point 320,240 --point 330,260 --point 300,400,neg --name forklift
```

```bash
curl -s -X POST "https://api.pictograph.io/api/v1/developer/sam3/predict" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -F "file=@frame.jpg" \
  -F "point=320,240" -F "point=330,260" -F "point=300,400,neg" \
  -F "name=forklift"
```

Over HTTP the image is the multipart `file`, and every prompt field rides beside it:
`text`, `box=x,y,w,h`, `point=x,y` (repeat it; `x,y,neg` excludes a region),
`negative_box=x,y,w,h` (repeatable), `name`, `output`, `confidence_threshold` and
`max_detections`. A field the route does not read is refused, so a typo never runs
with a default instead.

## Many images, a video, or a dataset

The CLI runs the same prompt on several inputs and prints one JSON line per image or
frame: `{"input", "result"}`, or `{"input", "error"}` for one that failed.

```bash
pictograph sam3 predict 'shots/*.jpg' --text forklift --jsonl
pictograph sam3 predict --video clip.mp4 --sample-fps 1 --text person
pictograph sam3 predict --dataset my-dataset --directory /val --text car --limit 20
```

A video is decoded on your machine (`pip install "pictograph[inference]"` brings
OpenCV) and sampled with the same rule a workflow uses, so each line carries the
frame's index and time. Each frame is one call and one image charged; for a long
video, a [workflow](/docs/api-reference/workflows.md) reads it on Pictograph instead.
`--dataset` reads one of your datasets' images and prints the results; it writes
nothing. A batch stops at an error that would fail every remaining image the same
way, such as an exhausted credit balance.

## Coordinates

Every coordinate, in the prompt you send and in the annotations that come back, is an
absolute pixel of the image as it is displayed, with the origin at the top-left. SAM3
applies the EXIF orientation first, so a phone photo taken sideways is the upright
picture. `width` and `height` on the result are that upright size. A prompt outside
the image is refused with the size it was checked against.

## Cost

Each image is charged the SAM3 session fee once, when a prompt on it completes: about a
cent per image, from your compute credits. The image is identified by its bytes
(`image_sha256`), so more prompts on the same image are free, and a prompt that fails is
free. `pictograph tools describe sam3` shows the current price, and
[Pricing](/pricing) lists every operation's cost. An organization whose balance cannot cover one session gets a `402`
before anything runs.

## Common errors

| Error | Cause |
|---|---|
| `ValueError` | No prompt or more than one, or a malformed box or point. Raised before any request |
| `ValidationError` (400) | Not an image SAM3 reads, more than 50 megapixels, a prompt outside the image, or an unknown form field |
| `PaymentRequiredError` (402) | The credits do not cover one session |
| `ForbiddenError` (403) | The API key's role is below `member` |
| `ServerError` (503) | SAM3 could not be reached. Nothing was charged; retry after the `Retry-After` |

## See also

- [Auto-annotate](/docs/api-reference/auto-annotate.md): the same prompts on an image already in a dataset, and batch draft labels
- [SAM3 auto-annotation](/docs/sam3-auto-annotation.md): drafting labels for a dataset, then reviewing them
- [Vision tools](/docs/api-reference/vision-tools.md): `pictograph tools describe sam3` returns this call's card