Overview

There are two ways to put an agent on Pictograph. Most people should start with the first: your coding agent drives the pictograph CLI, the same way you would.

Your coding agent (Claude Code, Codex)

Install the CLI and authenticate once, then teach your agent the platform:

pip install 'pictograph[cli]'
pictograph login                    # or: export PICTOGRAPH_API_KEY=pk_live_...
pictograph agents install-skill     # Claude Code: ~/.claude/skills/pictograph-cv/
pictograph init                     # ./AGENTS.md for Codex and other agents

• The pictograph-cv skill ships inside the SDK. It teaches Claude Code the CLI and the SDK: how everything is addressed by name, the draft, review, export, train, deploy order, and pricing a paid step before running it. --target claude-ai zips it for upload at claude.ai; --target both does both. Re-run the install after upgrading the SDK to update it.
• pictograph init writes an AGENTS.md into the current directory: the instructions file Codex and other coding agents read when they work in that repo.

Then tell it what to build:

Upload ./shelf to a dataset called "shelf" and draft labels for every cereal box
with SAM3. Show me the drafts it was least sure of. Once I have reviewed them,
export the dataset, train an RF-DETR detector on it, and deploy it on a T4.

The agent runs one CLI step at a time and reads each result before the next. SAM3’s labels are drafts: you review them in the app before the agent exports the dataset and trains on the export. Nothing paid starts without a price.

Call your model as a tool

Once a model is trained, it is a command your agent can run whenever it needs to see:

# On the agent's own machine (needs pictograph[inference])
pictograph models predict cereal-detector ./aisle-07.jpg --json

# Against a deployed endpoint
pictograph deployments predict cereal-detector ./aisle-07.jpg --token pk_deploy_...

# A saved workflow (model + tracking + counting) over a video
pictograph workflows invoke shelf-counter --file ./aisle-07.mp4

The agent does not have to remember any of this. It can ask which tools exist and how to call each one:

pictograph tools list
pictograph tools describe "model/cereal-detector" --json

tools list names every tool it can call: SAM3, your models, your deployments and your workflows. tools describe returns one tool’s card: the inputs it takes, its parameters, what a prediction looks like, the exact command, and what it costs. See Vision tools.

The CLI reference lists every command; most read commands take --json.

Your own agent (the tool registry)

Building an agent yourself? Every Pictograph operation is also exposed as an agent tool. The 37-tool registry feeds three integration paths.

Source: agents/_registry.py · agents/_toolkit.py

from pictograph.agents import create_toolkit

toolkit = create_toolkit()  # reads PICTOGRAPH_API_KEY

Bundled adapters - Claude / OpenAI

For raw tool-use loops against the Anthropic or OpenAI SDKs, the adapters return ready-to-pass tool dicts. No extra dependencies.

from pictograph.agents import for_anthropic_messages, for_openai_responses

claude_tools = for_anthropic_messages(toolkit)   # → anthropic.messages.create(tools=...)
openai_tools = for_openai_responses(toolkit)     # → openai.responses.create(tools=...)

# Both paths dispatch results the same way:
result = toolkit.dispatch(
    name="list_datasets",
    args={"limit": 10},
)

For the framework SDKs (claude-agent-sdk, openai-agents) install the extra:

pip install 'pictograph[agents]'

from pictograph.agents import for_claude_agent_sdk, for_openai_agents

claude_sdk_tools = for_claude_agent_sdk(toolkit)   # @tool-decorated callables
openai_sdk_tools = for_openai_agents(toolkit)      # FunctionTool objects

Adapter source: agents/claude.py · agents/openai.py. Full cookbooks: Claude · OpenAI.

Dynamic discovery - tools.json

For frameworks without a bundled adapter (Vercel AI SDK, LangChain, custom dispatchers), fetch the JSON Schema registry directly:

curl -H "X-API-Key: pk_live_..." https://api.pictograph.io/api/v1/developer/tools.json

Each entry has name, description, input_schema, plus metadata (required_role, cost_micro_usd, idempotent). Wire it into your dispatcher and route the model’s tool calls to the matching SDK method. See Dynamic discovery for end-to-end examples.

The bundled skill, from Python

The skill installed above is a directory in the wheel (pictograph/skills/pictograph-cv/): a SKILL.md, reference docs, and bash-callable Python scripts. Any agent that reads skill folders can load it from there.

What’s in the registry

All 37 tools, grouped by category. Full JSON schemas at /docs/api-reference/tools.

Guardrails

Three guardrails apply to every dispatch, whichever integration path you use.

• Role gate (required_role) - registry metadata, but the API re-checks the calling key’s role on every request. A viewer key gets 403 ForbiddenError on any write tool.
• Credit gate (cost_micro_usd) - paid tools (auto_annotate_dataset, train_pipeline) carry a known cost. Pre-flight with get_credit_balance + estimate_credit_cost and refuse to start when the balance is short.
• Response cap (max_response_tokens) - results over the cap (default 25k tokens) are truncated with a _truncated marker, so the agent re-calls with narrower filters. Override with create_toolkit(max_response_tokens=N).

Recommended system prompt patterns

These four rules keep agents safe, cheap, and honest about their labels. Drop them into your system prompt.

1. Before destructive actions (delete_dataset, delete_image,
   cancel_training), restate exactly what will be removed and ask
   for confirmation.

2. Before paid actions (auto_annotate_dataset, train_pipeline),
   call estimate_credit_cost first, then surface the cost and the
   remaining balance. Proceed only if sufficient.

3. For multi-step tasks, prefer the pipeline tools
   (upload_dataset_from_directory, auto_annotate_dataset, train_pipeline)
   over chaining individual resource tools. They handle short-circuit
   on failure and credit gating automatically. Run them as separate
   steps so each result can be checked before paying for the next.

4. Auto-annotations are drafts. Hand them to a person to review
   before you export and train. Never approve your own drafts
   unless the user says to.

See also

• Claude · OpenAI - integration cookbooks
• Dynamic discovery - for framework-agnostic stacks
• Cookbook - recipe-style end-to-end examples
• Tool reference - full JSON schemas for all 37 tools