Sign in Get started

Datasets

The full dataset lifecycle - list, fetch, create, update, archive, delete, insights, near-duplicates, bulk download, cold storage.

View as Markdown

A dataset is a collection of images sharing a class set, addressed by its name, which is unique within your organization.

/api/v1/developer/datasets/{dataset}

A UUID is accepted anywhere a name is.

list

Single-page list of datasets in your organization (active by default). Pass archived=True for the archived-only view.

Source: Datasets.list

Arg Type Default Notes
limit int 100 Page size, capped at 1000
offset int 0 Page offset
archived bool False List archived datasets instead 
datasets = client.datasets.list(
    limit=100,
)
for ds in datasets:
    print(ds.name, ds.image_count)
pictograph datasets list --limit 100
curl -s "https://api.pictograph.io/api/v1/developer/datasets/?limit=100&offset=0" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns list[Dataset]

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

The CLI has no --archived flag; use the SDK or REST for the archived view.

iter

Auto-paging iterator over every dataset. Stops on the server-computed pagination.has_more flag.

Source: Datasets.iter

Arg Type Default Notes
page_size int 100 Items per round-trip
max_total int | None None Stop after this many items
archived bool False Iterate archived datasets instead 
for ds in client.datasets.iter(page_size=100):
    print(ds.name)

# Or materialize:
all_datasets = client.datasets.iter().all()
print(len(all_datasets), "datasets")
# The CLI does not auto-page; this is a single page of up to 1000.
pictograph datasets list -n 1000
# Page manually with limit + offset until pagination.has_more is false.
curl -s "https://api.pictograph.io/api/v1/developer/datasets/?limit=100&offset=100" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns OffsetPager[Dataset]

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

get

Fetch one dataset by name (case-sensitive within the org) or by UUID.

Source: Datasets.get

Arg Type Default Notes
name str - Dataset name (positional)
include_images bool False Embed the first images_limit image summaries
images_limit int 1000 Capped at 10000
images_offset int 0 Page the embedded image list 
ds = client.datasets.get(
    name="road-signs",
    include_images=True,
    images_limit=200,
)
print(ds.image_count, len(ds.images))
pictograph datasets get road-signs --include-images --images-limit 200
curl -s "https://api.pictograph.io/api/v1/developer/datasets/road-signs?include_images=true&images_limit=200" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

# Same resource, addressed by UUID:
curl -s "https://api.pictograph.io/api/v1/developer/datasets/a3e12f0b-4c6d-4e88-9a1b-2c3d4e5f6a7b" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns Dataset

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

Embedded images use the canonical field names: width, height, content_type, directory_path, absolute image_url / thumbnail_url / annotation_url.

create

Create a dataset and its initial class config. Requires a member+ API key.

Source: Datasets.create

Arg Type Default Notes
name str required Unique within the org (409 on collision)
description str | None None Up to 2000 characters
annotation_types list[str] | None ["bbox"] bbox/box, polygon, polyline, keypoint
classes list | None [] DatasetClass models or raw dicts 
ds = client.datasets.create(
    name="new-dataset",
    readme="# Road signs\n\nTraffic cameras, downtown.",
    annotation_types=["bbox", "polygon"],
    classes=[{"name": "car", "type": "bbox", "color": "#e6194b"}],
)
pictograph datasets create new-dataset \
  --description "Traffic cameras, downtown" \
  --type bbox \
  --type polygon
curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "new-dataset",
    "readme": "# Road signs",
    "annotation_types": ["bbox", "polygon"],
    "classes": [{"name": "car", "type": "bbox", "color": "#e6194b"}]
  }'

Returns Dataset

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

The CLI creates the dataset and its annotation types; add classes afterwards with update.

update

Partial update of metadata, annotation types, or the class list. Requires a member+ API key. Rename via new_name (the current name is the address). Class-list updates replace, they do not merge - fetch, mutate locally, then pass the full list back.

Source: Datasets.update

Arg Type Default Notes
name str - Current dataset name (positional)
new_name str | None None Rename target, unique within the org
description str | None None Up to 2000 characters
annotation_types list[str] | None None Replaces the allowed types
classes list | None None Replaces the whole class list 
client.datasets.update(
    name="new-dataset",
    readme="# Road signs\n\nUpdated card.",
)
client.datasets.update(
    name="new-dataset",
    new_name="renamed-dataset",
)

# Add a class without dropping the others:
ds = client.datasets.get(
    name="renamed-dataset",
)
client.datasets.update(
    name="renamed-dataset",
    classes=[*ds.classes, {"name": "truck", "type": "bbox"}],
)
pictograph datasets update new-dataset --description "updated"
pictograph datasets update new-dataset --new-name renamed-dataset
curl -s -X PATCH "https://api.pictograph.io/api/v1/developer/datasets/new-dataset" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"new_name": "renamed-dataset"}'

Returns Dataset

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

A new_name collision returns 409. The CLI covers rename, description, and annotation types; class-list edits go through the SDK or REST.

archive

Hide a dataset from the default list without deleting anything - images, exports, models, and annotations all stay, and the operation is fully reversible with unarchive. Requires an admin+ API key and is idempotent. A public dataset must be unpublished from Explore first (400).

Source: Datasets.archive

Arg Type Default Notes
name str - Dataset name (positional) 
client.datasets.archive(
    name="old-dataset",
)
client.datasets.list(
    archived=True,
)
pictograph datasets archive old-dataset
curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/old-dataset/archive" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns Dataset

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

Not to be confused with cold storage, which moves the image bytes between storage classes - archiving only affects list visibility.

unarchive

Bring an archived dataset back into the default list. Requires an admin+ API key and is idempotent.

Source: Datasets.unarchive

Arg Type Default Notes
name str - Dataset name (positional) 
client.datasets.unarchive(
    name="old-dataset",
)
pictograph datasets unarchive old-dataset
curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/old-dataset/unarchive" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns Dataset

Dataset · 17 fields 
class Dataset(BaseModel):
    """A Pictograph dataset - a group of images sharing an annotation config."""
    id: str
    organization_id: str | None = None
    name: str
    description: str | None = None
    annotation_types: list[str] = ['bbox']
    classes: list[DatasetClass] = []
    image_count: int = 0
    completed_image_count: int = 0
    archived_image_count: int = 0
    total_size: int = 0
    is_public: bool = False
    is_archived: bool = False
    archived_at: datetime | None = None
    storage_class: str = 'standard'
    images: list[Image] | None = None
    created_at: datetime
    updated_at: datetime | None = None

delete

Permanently delete a dataset, its images, and its stored bytes. Requires an admin+ API key. Blobs still referenced by forks of the dataset are retained.

Source: Datasets.delete

Arg Type Default Notes
name str - Dataset name (positional) 
summary = client.datasets.delete(
    name="old-dataset",
)
print(summary["images_deleted"], summary["gcs_blobs_deleted"])
pictograph datasets delete old-dataset --yes
curl -s -X DELETE "https://api.pictograph.io/api/v1/developer/datasets/old-dataset" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns dict[str, Any]

insights

Dataset Health - a one-call check of a dataset’s composition: headline totals, labeling-stage counts, per-class instance and image counts (class balance), per-annotation-type totals, an annotations-per-image density histogram, and image-dimension insights. Every metric is aggregated server-side over denormalized columns and never scans annotations, so it stays fast on 100k+ image datasets.

Source: Datasets.insights

Arg Type Default Notes
name str - Dataset name (positional) 
health = client.datasets.insights(
    name="road-signs",
)

print(health.total_images, health.total_annotations)

# Class balance - instances per class, sorted:
for name, count in sorted(health.class_annotation_counts.items(), key=lambda kv: -kv[1]):
    print(f"{name}: {count} annotations in {health.class_image_counts.get(name, 0)} images")

# Labeling progress:
print(health.status_counts.complete, "of", health.total_images, "complete")

# Image dimensions:
d = health.dimensions
print(f"{d.min_width}-{d.max_width} by {d.min_height}-{d.max_height}, "
      f"{d.orientation.landscape} landscape / {d.orientation.portrait} portrait")

# How many images carry a low-confidence model prediction?
mc = health.model_confidence
if mc and mc.flagged:
    print(f"{mc.flagged} images need review (lowest {mc.lowest:.0%})")
    # ...then page them with client.images.iter(..., min_confidence_lt=0.9)
pictograph datasets insights road-signs
pictograph datasets insights road-signs --json
curl -s "https://api.pictograph.io/api/v1/developer/datasets/road-signs/insights" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DatasetInsights

DatasetInsights · 13 fields 
class DatasetInsights(BaseModel):
    """Dataset Health / Insights - headline totals, class balance, and more."""
    total_images: int = 0
    total_annotations: int = 0
    annotated_images: int = 0
    unannotated_images: int = 0
    avg_annotations_per_image: float = 0.0
    total_bytes: int = 0
    status_counts: InsightsStatusCounts = InsightsStatusCounts(new=0, annotate=0, review=0, complete=0)
    class_annotation_counts: dict[str, int] = {}
    class_image_counts: dict[str, int] = {}
    type_counts: dict[str, int] = {}
    annotation_density: dict[str, int] = {}
    dimensions: InsightsDimensions = InsightsDimensions(min_width=None, max_width=None, avg_width=None, min_height=None, max_height=None, avg_height=None, orientation=InsightsOrientation(landscape=0, portrait=0, square=0), sizes=[], distinct_size_count=0, images_with_dimensions=0, images_missing_dimensions=0)
    model_confidence: ModelConfidence | None = None

Counts cover non-archived images only. model_confidence is None when the dataset has no model predictions; otherwise it carries flagged (images below 100% confidence), lowest, avg_flagged, and per-band buckets.

near_duplicates

Data curation - find visually near-duplicate images in a dataset so you can keep one per cluster and archive the redundant rest, cutting annotation volume and dataset bloat before labeling. It reuses the dataset’s image embeddings (a k-NN self-join over the vector index) to group images whose cosine similarity is at or above threshold. This is an on-demand scan and is expensive, so it is a separate call rather than part of insights.

Source: Datasets.near_duplicates

Arg Type Default Notes
name str - Dataset name (positional)
threshold float | None 0.92 Min cosine similarity, 0.5-0.9999. Higher is stricter
sample int | None 1000 Max source images to scan, capped at 2000
neighbors int | None 10 Max matches per source image
max_pairs int | None 2000 Max edges returned
directory_path str | None None Scope the scan to one virtual directory, e.g. /train 
dup = client.datasets.near_duplicates(
    name="road-signs",
    threshold=0.92,
)

print(f"{dup.group_count} duplicate groups, {dup.redundant_count} redundant images "
      f"(analyzed {dup.analyzed} of {dup.total_images})")

# Keep the first image of each cluster; archive the redundant rest:
redundant = [m.id for g in dup.groups for m in g.members[1:]]
for image_id in redundant:
    client.images.delete(
        dataset_name="road-signs",
        image=image_id,
    )

# Scope the scan to one directory of a large multi-directory dataset:
dup = client.datasets.near_duplicates(
    name="road-signs",
    directory_path="/train",
)
pictograph datasets duplicates road-signs --threshold 0.92 --directory /train
curl -s "https://api.pictograph.io/api/v1/developer/datasets/road-signs/duplicates?threshold=0.92&directory_path=/train" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Every bound is clamped server-side; the result reports the analyzed sample plus sample_capped / pairs_capped flags, so a cap is always visible - raise sample to scan more.

Returns NearDuplicatesResult

NearDuplicatesResult · 11 fields 
class NearDuplicatesResult(BaseModel):
    """Near-duplicate clusters for a dataset + headline data-curation counts."""
    groups: list[DuplicateGroup] = []
    group_count: int = 0
    duplicate_image_count: int = 0
    redundant_count: int = 0
    analyzed: int = 0
    total_images: int = 0
    sample_limit: int = 0
    sample_capped: bool = False
    pairs_capped: bool = False
    threshold: float = 0.0
    directory_path: str | None = None

Non-archived images only. Also available on the async client (await client.datasets.near_duplicates(...)).

download

Bulk-download images and / or annotations to a local directory. The SDK fetches a batch of signed download URLs in one call, then downloads in parallel via a thread pool.

Source: Datasets.download

Arg Type Default Notes
name str - Dataset name (positional)
output_dir str | Path required Local directory to write into
mode str "full" "full" · "images_only" · "annotations_only"
status_filter str | None None Restrict to e.g. "complete" images
max_workers int 10 Parallel download threads
progress callable | None None Called with (done, total, filename) 
report = client.datasets.download(
    name="road-signs",
    output_dir="./dump",
    mode="full",
    status_filter="complete",
    max_workers=10,
    progress=lambda done, total, fn: print(f"{done}/{total} {fn}"),
)
print(report.images_downloaded, report.annotations_downloaded, len(report.failures))
pictograph datasets download road-signs --output ./dump --mode full --workers 10
# Returns the manifest of signed URLs the SDK then downloads in parallel.
curl -s "https://api.pictograph.io/api/v1/developer/datasets/road-signs/download?mode=full&status_filter=complete" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DownloadReport

DownloadReport · 5 fields 
class DownloadReport(BaseModel):
    """Result of a Datasets.download invocation."""
    dataset_id: str
    images_downloaded: int = 0
    annotations_downloaded: int = 0
    failures: list[DownloadFailure] = []
    truncated: bool = False

The manifest is {"data": {"id", "name", "mode", "items": [...], "total_items", "total_size", "expires_in_minutes": 60}} - each item carries id, filename, file_size, a signed image_url (fetches the bytes directly, no auth), and an annotation_url. One manifest covers up to 10000 images. The SDK returns a DownloadReport; inspect .failures to retry the subset, since the call does not raise on individual file errors. The CLI has no --status filter; use the SDK or REST to restrict by stage.

Cold storage

Move a finished dataset to cold storage and its images count half toward your plan’s image and storage limits. Browsing, search, and annotation data stay fully available - only byte-heavy operations pause (uploads, exports, auto-annotation, full-resolution viewing) until you restore.

Moving to cold storage is free. Restoring is instant (no thaw delay) and charged from compute credits by size; restoring before 90 days in cold storage adds a small early-restore component. The exact price is always quoted up front. Public datasets and datasets with forks cannot move to cold storage. Requires an admin+ API key. The restore charge is idempotent per frozen generation, so retrying a failed restore never double-charges.

storage_status

Current storage state plus a restore quote (present while the dataset is cold).

Source: Datasets.storage_status

Arg Type Default Notes
name str - Dataset name (positional) 
status = client.datasets.storage_status(
    name="road-signs",
)
print(status.storage_class, status.storage_state)
if status.restore_estimate:
    print(f"Restore costs ${status.restore_estimate.total_micro_usd / 1_000_000:.4f}")
pictograph datasets storage road-signs
curl -s "https://api.pictograph.io/api/v1/developer/datasets/road-signs/storage" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DatasetStorageStatus

DatasetStorageStatus · 7 fields 
class DatasetStorageStatus(BaseModel):
    """Cold-storage state of a dataset (`GET /developer/datasets/{id}/storage`)."""
    storage_class: str = 'standard'
    storage_state: str = 'idle'
    cold_since: datetime | None = None
    cold_bytes: int = 0
    cold_image_count: int = 0
    storage_job_id: str | None = None
    restore_estimate: DatasetRestoreEstimate | None = None

freeze

Move the dataset to cold storage. Free, and runs as a background job.

Source: Datasets.freeze

Arg Type Default Notes
name str - Dataset name (positional) 
client.datasets.freeze(
    name="road-signs",
)
client.datasets.wait_for_storage(
    name="road-signs",
)
pictograph datasets freeze road-signs
curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/road-signs/storage/freeze" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DatasetStorageTransition

DatasetStorageTransition · 3 fields 
class DatasetStorageTransition(BaseModel):
    """Acknowledgement that a freeze/restore background job started."""
    job_id: str
    storage_state: str
    quoted_micro_usd: int | None = None

The CLI blocks until the job finishes; pass --no-wait to return immediately.

restore

Restore to standard storage. Charges compute credits, so quote first with storage_status.

Source: Datasets.restore

Arg Type Default Notes
name str - Dataset name (positional) 
job = client.datasets.restore(
    name="road-signs",
)
client.datasets.wait_for_storage(
    name="road-signs",
)
pictograph datasets restore road-signs
curl -s -X POST "https://api.pictograph.io/api/v1/developer/datasets/road-signs/storage/restore" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DatasetStorageTransition

DatasetStorageTransition · 3 fields 
class DatasetStorageTransition(BaseModel):
    """Acknowledgement that a freeze/restore background job started."""
    job_id: str
    storage_state: str
    quoted_micro_usd: int | None = None

The CLI confirms the price before charging; pass --yes to skip the prompt and --no-wait to return immediately.

wait_for_storage

Poll until a freeze or restore reaches a terminal state.

Source: Datasets.wait_for_storage

Arg Type Default Notes
name str - Dataset name (positional)
timeout float 600.0 Max seconds to wait
poll_interval float 3.0 Seconds between polls 
status = client.datasets.wait_for_storage(
    name="road-signs",
    timeout=600.0,
    poll_interval=3.0,
)
print(status.storage_class, status.storage_state)
# There is no standalone wait command: freeze and restore already block
# until done. This is what --wait (the default) is doing for you.
pictograph datasets restore road-signs --wait
# Poll the storage endpoint until storage_state is "idle".
curl -s "https://api.pictograph.io/api/v1/developer/datasets/road-signs/storage" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

Returns DatasetStorageStatus

DatasetStorageStatus · 7 fields 
class DatasetStorageStatus(BaseModel):
    """Cold-storage state of a dataset (`GET /developer/datasets/{id}/storage`)."""
    storage_class: str = 'standard'
    storage_state: str = 'idle'
    cold_since: datetime | None = None
    cold_bytes: int = 0
    cold_image_count: int = 0
    storage_job_id: str | None = None
    restore_estimate: DatasetRestoreEstimate | None = None

as_pytorch

Adapt a dataset into a map-style torch.utils.data.Dataset.

Source: Datasets.as_pytorch

Arg Type Default Notes
name str required Dataset name (case-sensitive, unique within the org).
root str | Path | None None Local cache dir for downloaded images (created if missing).
transform Callable[[Any], Any] | None None Applied to the PIL.Image (e.g. a torchvision transform).
target_transform Callable[[dict[str, Any]], Any] | None None Applied to the target dict.
class_to_idx dict[str, int] | None None Class-name → integer-label map. Defaults to the dataset’s configured classes, ordered alphabetically.
download bool True Download images on access (set False if root is already populated).
images_limit int 10000 Cap on images pulled into the dataset (backend max 10000).
augment Augmenter | None None Optional Augmenter for on-the-fly augmentation - each item is a freshly-augmented variant with the target boxes remapped to match. 
dataset = client.datasets.as_pytorch(
    name="road-signs",
    root="./cache",
)
# No `as-pytorch` command - it returns an in-process torch Dataset.
pictograph datasets get road-signs
# `as_pytorch` builds a torch Dataset in-process - there is no REST endpoint.
# `as_pytorch` runs locally in the SDK - there is no REST endpoint.

Returns PictographTorchDataset

Common errors

Status Exception Cause
404 NotFoundError Name or UUID does not exist (names are case-sensitive) or belongs to another organization
409 ConflictError create, or update with a new_name that is already taken
403 ForbiddenError create / update need member+; archive, unarchive, delete, freeze, and restore need admin+
402 PaymentRequiredError create past the plan’s dataset cap, or restore with insufficient compute credits
400 ValidationError archive on a public dataset - unpublish it from Explore first
Copied to clipboard