Datasets

A dataset in Pictograph is a collection of images sharing a class set. Datasets are unique by (organization, name), and the API treats the name as the primary identifier — every single-dataset endpoint exists in two interchangeable forms that return byte-identical payloads:

• /api/v1/developer/datasets/by-name/{name} — the primary, human form
• /api/v1/developer/datasets/{dataset_id} — the UUID complement

In the SDK, every single-dataset method takes the name positionally or a dataset_id= keyword (exactly one): client.datasets.get("road-signs") and client.datasets.get(dataset_id="a3e12f…") are equivalent.

Response envelope: a single dataset returns {"data": {...}}; collections return {"data": [...], "pagination": {"limit", "offset", "total", "has_more"}} where total is the org-wide count for the filter. The SDK unwraps this for you and returns typed Dataset models.

Every example below shows the Python SDK call and the equivalent raw REST request. The REST examples authenticate with an X-API-Key header; set PICTOGRAPH_API_KEY in your shell to copy-and-run them.

from pictograph import Client
client = Client()  # reads PICTOGRAPH_API_KEY

list

Single-page list of datasets in your organization (active by default).

Arg	Type	Default	Notes
limit	int	100	Backend cap: 1000
offset	int	0	Page offset
archived	bool	False	List archived datasets instead

datasets = client.datasets.list(limit=100)
for ds in datasets:
    print(ds.name, ds.image_count)

curl -s "https://api.pictograph.io/api/v1/developer/datasets/?limit=100&offset=0" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns list[Dataset]. On the wire: {"data": [...], "pagination": {"limit": 100, "offset": 0, "total": 7, "has_more": false}}.

iter

Auto-paging iterator over every dataset. Stops on the server-computed pagination.has_more flag.

Arg	Type	Default	Notes
page_size	int	100	Items per backend round-trip
max_total	int | None	None	Stop after this many items
archived	bool	False	Iterate archived datasets instead

for ds in client.datasets.iter(page_size=100):
    print(ds.name)

# Or materialize:
all_datasets = client.datasets.iter().all()

# Page manually with limit + offset until pagination.has_more is false.
curl -s "https://api.pictograph.io/api/v1/developer/datasets/?limit=100&offset=0" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns OffsetPager[Dataset].

get

Fetch by name (case-sensitive within org) or by UUID.

Arg	Type	Default	Notes
name	str	—	Dataset name (positional)
dataset_id	str	—	UUID — the keyword alternative to name
include_images	bool	False	Embed first images_limit DatasetImage summaries
images_limit	int	1000	Backend cap: 10000
images_offset	int	0	Page the embedded image list

ds = client.datasets.get("road-signs", include_images=True, images_limit=200)
print(ds.image_count, len(ds.images))

# UUID form — same payload, same serializer:
ds = client.datasets.get(dataset_id="a3e12f...")

curl -s "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs?include_images=true&images_limit=200" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

curl -s "https://api.pictograph.io/api/v1/developer/datasets/a3e12f..." \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns Dataset — id, organization_id, name, description, annotation_types, classes, image_count, completed_image_count, archived_image_count, total_size, is_public, is_archived, archived_at, storage_class, created_at, updated_at (+ images when requested). Embedded images use the canonical field names: width, height, content_type, folder_path, absolute image_url / thumbnail_url / annotation_url.

create

Create a dataset + its initial class config. Member+ API key.

Arg	Type	Default	Notes
name	str	required	Unique within the org (409 on collision)
description	str | None	None
annotation_types	list[str] | None	["bbox"]	bbox/box, polygon, polyline, keypoint
classes	list | None	[]	DatasetClass models or raw dicts

ds = client.datasets.create(
    "new-dataset",
    description="Traffic cameras, downtown",
    annotation_types=["bbox", "polygon"],
    classes=[{"name": "car", "type": "bbox", "color": "#e6194b"}],
)

curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "new-dataset", "annotation_types": ["bbox"], "classes": [{"name": "car", "type": "bbox"}]}'

Returns the full Dataset (HTTP 201).

update

Partial update — metadata, annotation types, or the class list. Member+. Rename via the new_name body field (the current name is the address). Class-list updates replace, they don’t merge — fetch, mutate locally, pass the full list back.

client.datasets.update("new-dataset", description="updated")
client.datasets.update("new-dataset", new_name="renamed-dataset")

# Add a class without dropping the others:
ds = client.datasets.get("renamed-dataset")
client.datasets.update(
    "renamed-dataset",
    classes=[*ds.classes, {"name": "truck", "type": "bbox"}],
)

curl -s -X PATCH "https://api.pictograph.io/api/v1/developer/datasets/by-name/new-dataset" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"new_name": "renamed-dataset"}'

Returns the updated Dataset. A new_name collision returns 409.

archive / unarchive

Hide a dataset from the default list without deleting anything (images, exports, models, annotations all stay) — fully reversible. Admin+; idempotent. A public dataset must be unpublished from Explore first (400).

client.datasets.archive("old-dataset")
client.datasets.list(archived=True)          # the archived view
client.datasets.unarchive("old-dataset")

curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/by-name/old-dataset/archive" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/by-name/old-dataset/unarchive" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Both return the full updated Dataset. (Not to be confused with cold storage, which moves the image bytes between storage classes — archive only affects list visibility.)

delete

Permanently delete a dataset + its images + storage. Admin+. Blobs still referenced by forks of the dataset are retained.

summary = client.datasets.delete("old-dataset")
print(summary["images_deleted"], summary["gcs_blobs_deleted"])

curl -s -X DELETE "https://api.pictograph.io/api/v1/developer/datasets/by-name/old-dataset" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns {"data": {"id", "name", "deleted": true, "images_deleted", "folders_deleted", "gcs_blobs_deleted", "gcs_blobs_retained_for_forks"}}.

insights

Dataset Health / Insights — a one-call health check of a dataset’s composition: headline totals, labeling-stage counts, per-class instance + image counts (class balance), per-annotation-type totals, an annotations-per-image density histogram, and image-dimension insights. Every metric is aggregated server-side over denormalized columns (it never scans annotations), so it stays fast even on 100k+ image datasets.

health = client.datasets.insights("road-signs")

print(health.total_images, health.total_annotations)

# Class balance — instances per class, sorted:
for name, count in sorted(health.class_annotation_counts.items(), key=lambda kv: -kv[1]):
    print(f"{name}: {count} annotations in {health.class_image_counts.get(name, 0)} images")

# Labeling progress:
print(health.status_counts.complete, "of", health.total_images, "complete")

# Image dimensions:
d = health.dimensions
print(f"{d.min_width}-{d.max_width} × {d.min_height}-{d.max_height}, "
      f"{d.orientation.landscape} landscape / {d.orientation.portrait} portrait")

# Active-learning: how many images carry a low-confidence model prediction?
mc = health.model_confidence
if mc and mc.flagged:
    print(f"{mc.flagged} images need review (lowest {mc.lowest:.0%})")
    # ...then page them: client.images.iter(dataset_id, min_confidence_lt=0.9)

curl -s "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs/insights" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DatasetInsights. Counts cover non-archived images only. model_confidence (B62 active-learning) is None when the dataset has no model predictions; otherwise it carries flagged (images below 100% confidence), lowest, avg_flagged, and per-band buckets.

near_duplicates

Data curation — find visually near-duplicate images in a dataset so you can keep one per cluster and archive the redundant rest, cutting annotation volume + dataset bloat before labeling. Reuses the dataset’s image embeddings (a k-NN self-join over the vector index) to group images whose cosine similarity is at or above threshold. It’s an on-demand scan (expensive), so it’s a separate call — not part of insights.

dup = client.datasets.near_duplicates("road-signs", threshold=0.92)

print(f"{dup.group_count} duplicate groups, {dup.redundant_count} redundant images "
      f"(analyzed {dup.analyzed} of {dup.total_images})")

# Keep the first image of each cluster; archive the redundant rest:
redundant = [m.id for g in dup.groups for m in g.members[1:]]
for image_id in redundant:
    client.images.delete(image_id)  # soft-delete → Archive tab (restorable)

# Scope the scan to one folder of a large multi-folder dataset:
dup = client.datasets.near_duplicates("road-signs", folder_path="/train")

curl -s "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs/duplicates?threshold=0.92&folder_path=/train" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Parameters (all optional): threshold (min cosine similarity, 0.5–0.9999, default 0.92 — higher is stricter / near-identical), sample (max source images to scan, default 1000, cap 2000), neighbors (max matches per source, default 10), max_pairs (max edges, default 2000), and folder_path (scope the scan to one virtual folder, e.g. /train; None scans the whole dataset). Every bound is clamped server-side; the result reports the analyzed sample plus sample_capped / pairs_capped flags (no silent caps — raise sample to scan more).

Returns NearDuplicatesResult (groups of DuplicateGroup, each with hydrated members, size, and max_similarity). Non-archived images only. Also available as pictograph datasets duplicates <name> and on the async client (await client.datasets.near_duplicates(...)).

download

Bulk-download images and / or annotations to a local directory. The SDK fetches a batch of signed download URLs in one call, then downloads in parallel via a thread pool.

Arg	Type	Default	Notes
mode	str	"full"	"full" · "images_only" · "annotations_only"
status_filter	str | None	None	Restrict to e.g. "complete" images
max_workers	int	10	Parallel download threads

report = client.datasets.download(
    "road-signs",
    output_dir="./dump",
    mode="full",
    status_filter="complete",
    max_workers=10,
    progress=lambda done, total, fn: print(f"{done}/{total} {fn}"),
)
print(report.images_downloaded, report.annotations_downloaded, len(report.failures))

# Returns the manifest of signed URLs the SDK then downloads in parallel.
curl -s "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs/download?mode=full&status_filter=complete" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

The manifest is {"data": {"id", "name", "mode", "items": [...], "total_items", "total_size", "expires_in_minutes": 60}} — each item carries id, filename, file_size, a signed image_url (direct GCS bytes, no auth), and an annotation_url. The SDK returns a DownloadReport; inspect .failures to retry the subset — the call does not raise on individual file errors.

Cold storage

Move a finished dataset to cold storage and its images count half toward your plan’s image and storage limits. Browsing, search, and annotation data stay fully available — only byte-heavy operations pause (uploads, exports, auto-annotation, full-resolution viewing) until you restore.

Moving to cold storage is free. Restoring is instant (no thaw delay) and charged from compute credits by size; restoring before 90 days in cold storage adds a small early-restore component. The exact price is always quoted up front. Public datasets and datasets with forks can’t move to cold storage. Requires an admin/owner API key. The restore charge is idempotent per frozen generation — retrying a failed restore never double-charges.

storage_status

Current storage state plus a restore quote (present while the dataset is cold).

status = client.datasets.storage_status("road-signs")
print(status.storage_class, status.storage_state)
if status.restore_estimate:
    print(f"Restore costs ${status.restore_estimate.total_micro_usd / 1_000_000:.4f}")

curl -s "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs/storage" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

freeze

Move the dataset to cold storage (free, runs as a background job).

client.datasets.freeze("road-signs")
client.datasets.wait_for_storage("road-signs")

curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs/storage/freeze" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

restore

Restore to standard storage (charges compute credits; quote first).

job = client.datasets.restore("road-signs")
client.datasets.wait_for_storage("road-signs")

curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/by-name/road-signs/storage/restore" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

CLI equivalents:

pictograph datasets create road-signs -t bbox
pictograph datasets update road-signs --new-name signs
pictograph datasets archive road-signs && pictograph datasets unarchive road-signs
pictograph datasets storage road-signs     # state + restore quote
pictograph datasets freeze road-signs
pictograph datasets restore road-signs     # confirms the price first

Common errors

Status	Exception	Cause
404	NotFoundError	Name/UUID doesn’t exist (names are case-sensitive) or belongs to another org
409	ConflictError	create / update new_name with a duplicate name
403	ForbiddenError	create/update need member+; archive/unarchive/delete/freeze/restore need admin+
402	PaymentRequiredError	create past the tier’s dataset cap, or restore with insufficient compute credits
400	ValidationError	archive on a public dataset (unpublish from Explore first)