Vision tools

List the vision tools your organization can call - SAM3, your models, deployments and workflows - and read the card that says exactly how to call each one.

View as Markdown

A vision tool is anything in your organization that takes an image, a video or a dataset and returns predictions: SAM3 (built in, no training), each ready model, each active deployment, and each workflow.

This resource is how an agent learns to use them without reading documentation written for people. It lists what exists, then returns a card for one tool: what it is, which inputs each way of running it accepts, its parameters, the shape of one prediction, the exact CLI, Python and HTTP call, and what each costs.

The habit is three steps: list, describe, then run.

pictograph tools list
pictograph tools describe "model/Shelf Detector"
pictograph models predict "Shelf Detector" photo.jpg --remote --json

A card is generated from what the platform stores about the tool, so it cannot go stale: the classes come from the model, the parameter bounds from the route that enforces them, and the prices from the same table that charges you.

Any API key role can read both calls. Nothing here costs credits or changes anything.

Tool handles

Every tool has a handle, the tool value of a list row. Pass it to describe.

Handle	Tool
`sam3`	SAM3, the built-in promptable segmentation model
`model/<name>`	A ready model, trained here or imported
`deployment/<name>`	An active deployment (a private endpoint for one model)
`workflow/<name>`	A workflow (models plus tracking, counting and outputs, over video or a dataset)

A UUID works in place of a name.

list

Every vision tool you can call right now: SAM3 first, then ready models (newest first), active deployments, and workflows. A model still training and a paused deployment are not listed, because they cannot be called yet.

Source: Tools.list

Arg	Type	Default	Notes
`kind`	`str \| None`	`None`	Only `"sam3"`, `"model"`, `"deployment"` or `"workflow"`

for tool in client.tools.list(kind="model"):
    print(tool.tool, tool.task, tool.summary)

pictograph tools list --kind model --json

curl -s "https://api.pictograph.io/api/v1/developer/tools/vision/?kind=model&limit=100&offset=0" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns list[ToolSummary]

ToolSummary · 7 fields

class ToolSummary(BaseModel):
    """One row of `client.tools.list()`."""
    tool: str
    kind: str
    id: str | None = None
    name: str
    task: str | None = None
    status: str | None = None
    summary: str = ''

The SDK returns every tool, paging for you. The REST response is one page: {"data": [...], "pagination": {"limit", "offset", "total", "has_more"}}, with limit up to 500.

describe

One tool’s card.

Source: Tools.describe

Arg	Type	Default	Notes
`tool`	`str`	required	A handle: `"sam3"`, `"model/<name>"`, `"deployment/<name>"` or `"workflow/<name>"`

card = client.tools.describe(
    tool="model/Shelf Detector",
)
print(card.summary)
for mode in card.modes:
    print(mode.mode, mode.cost.summary)
    print(mode.run.cli)

pictograph tools describe "model/Shelf Detector" --json

curl -s "https://api.pictograph.io/api/v1/developer/tools/vision/model/Shelf%20Detector" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns ToolCard

ToolCard · 15 fields

class ToolCard(BaseModel):
    """Everything an agent needs to call one vision tool."""
    tool: str
    kind: str
    id: str | None = None
    name: str
    task: str | None = None
    status: str | None = None
    summary: str = ''
    about: str | None = None
    classes: list[ToolClass] = []
    facts: dict[str, Any] = {}
    parameters: list[ToolParameter] = []
    output: ToolOutput | None = None
    modes: list[ToolMode] = []
    related: list[ToolRelated] = []
    notes: list[str] = []

Without --json the CLI prints the card as Markdown, for a person. The REST response wraps the card as {"data": {...}}.

A model that exists but is not ready answers 409; a name that matches nothing answers 404.

Reading a card

A card has one entry in modes for each way of running the tool. A model has four:

Mode	What it does	Cost
`hosted`	Uploads one image and returns its predictions. Nothing to install.	Free
`local`	Downloads the weights once and runs them on your machine.	Free
`dataset`	Runs over images in a dataset and saves the predictions as draft labels.	Per image
`deploy`	Creates a deployment: your own always-on endpoint.	Per minute up

Each mode answers the same questions:

inputs says yes or no to each of file, url, bytes, video and dataset. When the answer is no, note says what to do instead. From the CLI a model or a deployment takes a URL, - (bytes on stdin) and several images at once, printing one JSON line per image.
writes is false when the call changes nothing, otherwise a sentence saying what it saves. Labels a tool saves are drafts: have a person review them before you export and train.
cost carries micro_usd (1 USD = 1,000,000) per image, frame or minute. estimate marks a price that is settled from what the run measured.
run holds the exact cli, python and http call, with the tool’s own name filled in.

card = client.tools.describe(
    tool="model/Shelf Detector",
)
hosted = card.mode(name="hosted")
if hosted is not None and hosted.accepts(kind="file"):
    print(hosted.run.python)

Beside the modes, parameters lists what the tool takes with the bounds the API enforces, output describes one prediction (prose, its JSON Schema and an example), classes lists what a model predicts, and related names the tools tied to this one: a model’s active endpoints, the models a workflow runs.

Coordinates are always absolute pixels of the image you sent, with the origin at the top-left.

Vision tools

Tool handles

list

describe

Reading a card

See also