---
title: Cloud Storage
description: Connect a bucket your organization already owns — Amazon S3, Google Cloud Storage, Cloudflare R2, MinIO, Wasabi or any S3-compatible store — browse it, and import images from it into a dataset without re-uploading.
section: API Reference
order: 19
---
Already have a few hundred thousand images sitting in your own bucket? Point
Pictograph at it. Connect the bucket once, browse it like a file tree, and import
the images you want straight into a dataset — no re-upload, no zip files, no
copying anything through your laptop.

Works with **Amazon S3**, **Google Cloud Storage**, **Cloudflare R2**, **MinIO**,
**Wasabi**, **DigitalOcean Spaces**, and anything else that speaks the S3 API.

Every example below shows the Python SDK call and the equivalent raw REST
request. The REST examples authenticate with an `X-API-Key` header; set
`PICTOGRAPH_API_KEY` in your shell to copy-and-run them.

```python
from pictograph import Client
client = Client()  # reads PICTOGRAPH_API_KEY
```

## How it works

Pictograph **reads** from your bucket — it never writes to it. When you import,
the images are copied into your Pictograph dataset, where they behave exactly
like anything else you upload: auto-annotation, training, export, and thumbnails
all work unchanged.

Two things worth knowing before you start:

- **Imports are idempotent.** An object that is already in the dataset at the
  same folder and filename is *skipped*, not duplicated. Running the same import
  again after adding files to your bucket is therefore a cheap **re-sync**.
- **Your folder structure is preserved.** An object at `photos/batch1/img.jpg`,
  imported from the root `photos/`, lands in the dataset folder `batch1`.

The credentials you hand over are encrypted at rest and are **never returned by
any endpoint** — there is no field on any response that could leak them. Grant a
**read-only** key (`s3:ListBucket` + `s3:GetObject`, or the equivalent). Managing
a connection requires an **admin/owner** API key; any member can browse and
import through one that already exists.

## connect

Register a bucket. Pictograph probes the credentials *before* storing anything —
if the key cannot list the bucket, the call fails and no connection is created,
rather than deferring the failure to your first import.

```python
conn = client.storage.connect(
    name="Production imagery",
    provider="aws",
    bucket="acme-vision",
    region="us-east-1",
    access_key_id="AKIA…",
    secret_access_key="…",       # write-only — never returned again
    prefix="datasets/",          # optional: scope the connection to one folder
)
print(conn.id, conn.last_verified_at)
```

```bash
curl -s -X POST "https://api.pictograph.io/api/v1/developer/storage-connections" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "Production imagery",
        "provider": "aws",
        "bucket": "acme-vision",
        "region": "us-east-1",
        "access_key_id": "AKIA…",
        "secret_access_key": "…"
      }'
```

### Provider settings

| Provider | `provider` | `endpoint` | `region` |
|---|---|---|---|
| Amazon S3 | `aws` | *(omit — derived from the region)* | The bucket's region, e.g. `us-east-1` |
| Google Cloud Storage | `gcs` | *(omit — defaults to `https://storage.googleapis.com`)* | The bucket's location, e.g. `us-central1` |
| Cloudflare R2 | `r2` | `https://<account-id>.r2.cloudflarestorage.com` | `auto` |
| MinIO | `minio` | `https://minio.example.com` | `us-east-1` unless configured otherwise |
| Wasabi | `wasabi` | `https://s3.<region>.wasabisys.com` | e.g. `us-east-1` |
| Any S3-compatible | `other` | Your endpoint | Per your provider |

Google Cloud Storage is reached through its **S3-interoperability** API, so it
needs an **HMAC key** (a service account with *Storage Object Viewer*), not a
JSON key file.

`prefix` is a hard boundary, not a default: a browse or import through this
connection may narrow it, but can never escape it. Scope a connection to
`datasets/` and nobody can use it to read the rest of the bucket.

## browse

Walk the bucket one level at a time — sub-folders and the objects in the current
folder. Use `page.images` to get just the importable ones (a shared bucket holds
READMEs too; they are listed, but they are not images).

```python
page = client.storage.browse(conn.id, prefix="datasets/2026/")

for folder in page.folders:
    print("dir ", folder.name)
for obj in page.images:
    print("img ", obj.name, obj.size)

# Large folders page; pass the cursor back to continue.
if page.next_token:
    page = client.storage.browse(conn.id, prefix="datasets/2026/", token=page.next_token)
```

```bash
curl -s "https://api.pictograph.io/api/v1/developer/storage-connections/$CONNECTION_ID/browse?prefix=datasets/2026/" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY"
```

## import_images

Import into a dataset. Give it a `prefix` to pull everything underneath it
(recursively), or `keys` to pull exactly the objects you name.

```python
job = client.storage.import_images(
    conn.id,
    dataset_id,
    prefix="datasets/2026/",     # walks recursively
    dest_folder="raw",           # optional: where in the dataset it lands
)

done = client.storage.wait_for_import(job.id)
print(done.result.imported, "imported")
print(done.result.skipped,  "already in the dataset")
print(done.result.failed,   "failed")
```

```bash
curl -s -X POST "https://api.pictograph.io/api/v1/developer/storage-connections/$CONNECTION_ID/import" \
  -H "X-API-Key: $PICTOGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"project_id": "'"$DATASET_ID"'", "prefix": "datasets/2026/"}'
```

Import exactly the objects you picked:

```python
page = client.storage.browse(conn.id, prefix="datasets/2026/")
chosen = [o.key for o in page.images if o.size < 5_000_000]

job = client.storage.import_images(conn.id, dataset_id, keys=chosen)
```

`wait_for_import` blocks until the job finishes, raising `ApiError` if it ends in
error and `PollTimeoutError` if it outlives your timeout (the import keeps running
server-side — poll again with `get_import`). To track progress yourself:

```python
job = client.storage.get_import(job.id)
print(job.status, job.progress, job.processed_items, "/", job.total_items)

client.storage.cancel_import(job.id)   # ask a running import to stop
```

A single import job is bounded. If your bucket holds more images than one job may
take, the job says so on `result.warning` rather than silently importing a slice of
it — narrow the prefix and run it again.

## list · get · delete

```python
for conn in client.storage.list():
    print(conn.name, conn.provider, conn.bucket, conn.last_verified_at)

conn = client.storage.get(connection_id)

client.storage.delete(connection_id)
```

Deleting a connection makes Pictograph forget the bucket and its credentials.
Images you already imported are untouched — they are copies, and they live in
your datasets now.

## In the app

Everything here is also in the web app: **Settings → Cloud storage** to connect a
bucket, and **Add → Cloud storage** inside a dataset to browse it and import.