Cloud Storage

Already have a few hundred thousand images sitting in your own bucket? Point Pictograph at it. Connect the bucket once, browse it like a file tree, and import the images you want straight into a dataset — no re-upload, no zip files, no copying anything through your laptop.

Works with Amazon S3, Google Cloud Storage, Cloudflare R2, MinIO, Wasabi, DigitalOcean Spaces, and anything else that speaks the S3 API.

Every example below shows the Python SDK call and the equivalent raw REST request. The REST examples authenticate with an X-API-Key header; set PICTOGRAPH_API_KEY in your shell to copy-and-run them.

from pictograph import Client
client = Client()  # reads PICTOGRAPH_API_KEY

How it works

Pictograph reads from your bucket — it never writes to it. When you import, the images are copied into your Pictograph dataset, where they behave exactly like anything else you upload: auto-annotation, training, export, and thumbnails all work unchanged.

Two things worth knowing before you start:

• Imports are idempotent. An object that is already in the dataset at the same folder and filename is skipped, not duplicated. Running the same import again after adding files to your bucket is therefore a cheap re-sync.
• Your folder structure is preserved. An object at photos/batch1/img.jpg, imported from the root photos/, lands in the dataset folder batch1.

The credentials you hand over are encrypted at rest and are never returned by any endpoint — there is no field on any response that could leak them. Grant a read-only key (s3:ListBucket + s3:GetObject, or the equivalent). Managing a connection requires an admin/owner API key; any member can browse and import through one that already exists.

connect

Register a bucket. Pictograph probes the credentials before storing anything — if the key cannot list the bucket, the call fails and no connection is created, rather than deferring the failure to your first import.

conn = client.storage.connect(
    name="Production imagery",
    provider="aws",
    bucket="acme-vision",
    region="us-east-1",
    access_key_id="AKIA…",
    secret_access_key="…",       # write-only — never returned again
    prefix="datasets/",          # optional: scope the connection to one folder
)
print(conn.id, conn.last_verified_at)

curl -s -X POST "https://api.pictograph.io/api/v1/developer/storage-connections" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "Production imagery",
        "provider": "aws",
        "bucket": "acme-vision",
        "region": "us-east-1",
        "access_key_id": "AKIA…",
        "secret_access_key": "…"
      }'

Provider settings

Provider	provider	endpoint	region
Amazon S3	aws	(omit — derived from the region)	The bucket’s region, e.g. us-east-1
Google Cloud Storage	gcs	(omit — defaults to https://storage.googleapis.com)	The bucket’s location, e.g. us-central1
Cloudflare R2	r2	https://<account-id>.r2.cloudflarestorage.com	auto
MinIO	minio	https://minio.example.com	us-east-1 unless configured otherwise
Wasabi	wasabi	https://s3.<region>.wasabisys.com	e.g. us-east-1
Any S3-compatible	other	Your endpoint	Per your provider

Google Cloud Storage is reached through its S3-interoperability API, so it needs an HMAC key (a service account with Storage Object Viewer), not a JSON key file.

prefix is a hard boundary, not a default: a browse or import through this connection may narrow it, but can never escape it. Scope a connection to datasets/ and nobody can use it to read the rest of the bucket.

browse

Walk the bucket one level at a time — sub-folders and the objects in the current folder. Use page.images to get just the importable ones (a shared bucket holds READMEs too; they are listed, but they are not images).

page = client.storage.browse(conn.id, prefix="datasets/2026/")

for folder in page.folders:
    print("dir ", folder.name)
for obj in page.images:
    print("img ", obj.name, obj.size)

# Large folders page; pass the cursor back to continue.
if page.next_token:
    page = client.storage.browse(conn.id, prefix="datasets/2026/", token=page.next_token)

curl -s "https://api.pictograph.io/api/v1/developer/storage-connections/$CONNECTION_ID/browse?prefix=datasets/2026/" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"

import_images

Import into a dataset. Give it a prefix to pull everything underneath it (recursively), or keys to pull exactly the objects you name.

job = client.storage.import_images(
    conn.id,
    dataset_id,
    prefix="datasets/2026/",     # walks recursively
    dest_folder="raw",           # optional: where in the dataset it lands
)

done = client.storage.wait_for_import(job.id)
print(done.result.imported, "imported")
print(done.result.skipped,  "already in the dataset")
print(done.result.failed,   "failed")

curl -s -X POST "https://api.pictograph.io/api/v1/developer/storage-connections/$CONNECTION_ID/import" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"project_id": "'"$DATASET_ID"'", "prefix": "datasets/2026/"}'

Import exactly the objects you picked:

page = client.storage.browse(conn.id, prefix="datasets/2026/")
chosen = [o.key for o in page.images if o.size < 5_000_000]

job = client.storage.import_images(conn.id, dataset_id, keys=chosen)

wait_for_import blocks until the job finishes, raising ApiError if it ends in error and PollTimeoutError if it outlives your timeout (the import keeps running server-side — poll again with get_import). To track progress yourself:

job = client.storage.get_import(job.id)
print(job.status, job.progress, job.processed_items, "/", job.total_items)

client.storage.cancel_import(job.id)   # ask a running import to stop

A single import job is bounded. If your bucket holds more images than one job may take, the job says so on result.warning rather than silently importing a slice of it — narrow the prefix and run it again.

list · get · delete

for conn in client.storage.list():
    print(conn.name, conn.provider, conn.bucket, conn.last_verified_at)

conn = client.storage.get(connection_id)

client.storage.delete(connection_id)

Deleting a connection makes Pictograph forget the bucket and its credentials. Images you already imported are untouched — they are copies, and they live in your datasets now.

In the app

Everything here is also in the web app: Settings → Cloud storage to connect a bucket, and Add → Cloud storage inside a dataset to browse it and import.