Pictograph annotation format

Every annotation in Pictograph follows the same schema. Snake-case field names, no shorthand: bounding boxes are objects {x, y, w, h}, polygons are multi-ring paths, polylines are ordered point lists, keypoints are single points.

The class-label field is name (not class). Do not improvise.

Discriminator

type	Geometry container	Notes
bbox	bounding_box: {x, y, w, h}	Axis-aligned rectangle.
polygon	polygon: {paths: [[{x, y}, ...], ...]}	Multi-ring (holes via even-odd).
polyline	polyline: {path: [{x, y}, ...]}	Open path, doesn’t close.
keypoint	keypoint: {x, y}	Single landmark.
obb	oriented_box: {cx, cy, w, h, angle}	Oriented (rotated) box. angle is degrees, clockwise, [0, 360). Also carries a derived polygon (its four corners) and bounding_box (its enclosure) — see below.
skeleton	skeleton: {nodes: [{name, x, y, visibility}], edges: [[i, j]]}	Keypoint skeleton (pose). Named joints plus the edges linking them, as one object. Also carries a derived bounding_box — see below.

Required fields

Field	Type	Notes
id	non-blank string	Unique within the image. UUIDs preferred.
name	non-blank string	Class label. Must match a class in project_config.classes (case-sensitive).
type	one of bbox/polygon/polyline/keypoint/obb/skeleton	Discriminator.
<geometry>	see table above	Field name is determined by type.

Optional fields

Field	Default	Notes
confidence	1.0	Range [0, 1]. SAM3 sets this; manual annotations get 1.0.
created_by	null	UUID of the creator. Backend fills this for SDK uploads.
attributes	{}	Per-annotation ontology attributes as a {name: value} map (e.g. {"occluded": "true", "pose": "standing"}). Exported natively to COCO + Datumaro (see below). The legacy opaque-list form is still accepted but not exported.
bounding_box (polygon/polyline)	computed	Backend auto-computes the enclosing rectangle if omitted.
polygon + bounding_box (obb)	computed	Derived from oriented_box — send neither. The backend regenerates both on every write, so they can never disagree with the angle; a hand-written pair is overwritten.
bounding_box (skeleton)	computed	Derived from the labelled joints — send nothing. The backend regenerates the enclosure on every write.

Examples

Bounding box

{
  "id": "ann-1",
  "name": "person",
  "type": "bbox",
  "bounding_box": {"x": 100, "y": 200, "w": 50, "h": 80}
}

Polygon

{
  "id": "ann-2",
  "name": "car",
  "type": "polygon",
  "polygon": {
    "paths": [[
      {"x": 10, "y": 20}, {"x": 110, "y": 20},
      {"x": 110, "y": 80}, {"x": 10, "y": 80}
    ]]
  }
}

Polygon with hole

{
  "id": "ann-3",
  "name": "donut",
  "type": "polygon",
  "polygon": {
    "paths": [
      [{"x": 0, "y": 0}, {"x": 100, "y": 0}, {"x": 100, "y": 100}, {"x": 0, "y": 100}],
      [{"x": 30, "y": 30}, {"x": 70, "y": 30}, {"x": 70, "y": 70}, {"x": 30, "y": 70}]
    ]
  }
}

Polyline

{
  "id": "ann-4",
  "name": "lane_centerline",
  "type": "polyline",
  "polyline": {
    "path": [
      {"x": 0, "y": 100}, {"x": 50, "y": 100}, {"x": 100, "y": 100}
    ]
  }
}

Keypoint

{
  "id": "ann-5",
  "name": "left_eye",
  "type": "keypoint",
  "keypoint": {"x": 250, "y": 180}
}

Oriented (rotated) box

For aerial, satellite, document, shelf and pose imagery, where an axis-aligned box around a turned object is mostly background. Draw it with the Oriented Box tool (hotkey O): drag it out, then grab the handle above it and turn it (hold Shift to snap to 15°).

{
  "id": "8f2b…",
  "name": "ship",
  "type": "obb",
  "oriented_box": { "cx": 100, "cy": 100, "w": 40, "h": 20, "angle": 30 },
  "polygon": { "paths": [[
    {"x": 87.7, "y": 81.3}, {"x": 122.3, "y": 101.3},
    {"x": 112.3, "y": 118.7}, {"x": 77.7, "y": 98.7}
  ]] },
  "bounding_box": { "x": 77.7, "y": 81.3, "w": 44.6, "h": 37.4 }
}

oriented_box is the source of truth. polygon and bounding_box are derived — both are optional when you write, and the server regenerates them from the angle. Send only oriented_box:

from pictograph import ObbAnnotation, OrientedBoxGeometry

client.annotations.save(image_id=img.id, annotations=[
    ObbAnnotation(
        name="ship",
        oriented_box=OrientedBoxGeometry(cx=100, cy=100, w=40, h=20, angle=30),
    ),
])

The derived keys are persisted, not just computed, and that is deliberate: it means every consumer that predates oriented boxes — the export converters, training data-prep, the grid’s class filter — reads a rotated box as the quadrilateral it geometrically is, instead of dropping a type it has never heard of or flattening it to an enclosure up to √2 too large.

Angle convention. Degrees, clockwise-positive, in image space (x → right, y → down), normalized to [0, 360). w and h are measured along the box’s own axes, so they do not change when it turns. This is the same convention CVAT’s rotation attribute uses, so a rotated CVAT box round-trips with no sign flip.

Export. yolo_obb (Ultralytics YOLO-OBB) and dota (the aerial standard) carry the rotation natively, as does cvat. Every other format receives the four rotated corners as a polygon, so the shape survives losslessly even where the parameterization cannot be expressed. Training an oriented-box model happens outside Pictograph today — export yolo_obb and train yolo obb on it.

Import. Both OBB formats read back in, and the round-trip is exact: export a rotated box and re-import it and you get the same box.

dota is auto-detected. yolo_obb must be selected explicitly — and this is not an oversight. A YOLO-OBB label line (a class id plus 8 normalized coordinates) is byte-for-byte identical to a YOLO segmentation line describing a 4-vertex polygon; nothing in the file distinguishes them. Auto-detecting would silently convert your polygons into rotated boxes, or your boxes into polygons, and the result would look perfectly fine either way. So you tell us which you have.

Keypoint skeleton (pose)

A skeleton is a reusable template of named joints with a connectivity graph — COCO-17 human pose, a hand, a face, a vehicle’s corners — placed and edited as one linked object. This is the pose-estimation annotation type. (A plain keypoint is a single, unnamed, unlinked point; a skeleton is what turns a pile of points into a pose.)

Draw it with the Skeleton tool (hotkey J): drag out the object’s extent and the class’s joint template lands in it, scaled, every joint visible. Then drag the joints that are wrong. Alt+click a joint cycles its visibility.

{
  "id": "3c9a…",
  "name": "person",
  "type": "skeleton",
  "skeleton": {
    "nodes": [
      {"name": "nose",     "x": 100, "y": 40,  "visibility": 2},
      {"name": "left_eye", "x": 92,  "y": 34,  "visibility": 1},
      {"name": "left_ear", "x": 0,   "y": 0,   "visibility": 0}
    ],
    "edges": [[0, 1]]
  },
  "bounding_box": { "x": 92, "y": 34, "w": 8, "h": 6 }
}

Visibility is COCO’s, verbatim:

visibility	Meaning
2	Labelled and visible.
1	Labelled but occluded — you know where it is, you just can’t see it. Its position still supervises the model.
0	Not labelled — never placed. Masked out of the keypoint loss entirely; serializes as 0, 0, 0.

Two invariants worth knowing, because breaking either fails silently:

Node lists are template-complete. Every joint the class declares is stored on every annotation, including ones that were never placed (visibility: 0). That is what keeps each instance’s [x, y, v] triplets positionally aligned to the class’s joint-name array. Compact the list and every joint after the gap shifts by one.

Node order is the class’s, not yours. The order is what COCO’s categories[].keypoints indexes. Every pretrained pose model indexes COCO-17 by position, so a reordered template silently mistrains against published weights. The template belongs to the class — define it once, in the Classes tab (four presets ship: COCO-17 person, hand-21, face-5, vehicle-8), and the drawing tool stamps it out.

bounding_box is derived from the labelled joints and regenerated on every write — send only skeleton:

from pictograph import SkeletonAnnotation, SkeletonGeometry, SkeletonNode

client.annotations.save(image_id=img.id, annotations=[
    SkeletonAnnotation(
        name="person",
        skeleton=SkeletonGeometry(
            nodes=[
                SkeletonNode(name="nose",     x=100, y=40, visibility=2),
                SkeletonNode(name="left_eye", x=92,  y=34, visibility=1),
            ],
            edges=[(0, 1)],
        ),
    ),
])

Export. coco carries it natively — categories[].keypoints (the joint names), categories[].skeleton (the edges, 1-indexed as COCO requires) and per-annotation [x, y, v] triplets with num_keypoints. yolo_pose (Ultralytics) is the directly trainable one: yolo pose train data=data.yaml, with kpt_shape and a flip_idx derived from your left_*/right_* joint names (without it, horizontal-flip augmentation mirrors the image but not the joint identities, and teaches the model that a left wrist is a right one). cvat and darwin also carry skeletons natively. Every other format keeps the object — it exports as the enclosure of its labelled joints — rather than dropping it.

Training a pose model happens outside Pictograph today: export yolo_pose and train yolo pose on it.

Import. V7/Darwin and Roboflow pose datasets import as real skeletons — joints named, occlusion preserved, grouped as one object.

Storage

Annotations are stored in project_images.annotations_json as a plain array — no wrapper:

[
  {"id": "ann-1", "name": "person", "type": "bbox", "bounding_box": {…}},
  {"id": "ann-2", "name": "car",    "type": "polygon", "polygon": {…}}
]

Updating an image’s annotations is a full overwrite: pass the complete list every time. There is no partial-update endpoint.

Common mistakes

• ❌ "class": "person" — must be "name".
• ❌ "polygon": [[10, 20, 30, 40]] — flat array. Must be [{"x": …, "y": …}].
• ❌ "bbox": [x, y, w, h] — array. Must be "bounding_box": {x, y, w, h} object.
• ❌ Class label not in project_config.classes — backend rejects with 400.
• ❌ Polygon ring with < 3 points — Pydantic rejects on save.

Confidence in exports

confidence is carried through the formats that have a standard slot for it, and intentionally dropped from the ones that don’t (so the export stays parseable by the target tool). Human annotations export as 1.0.

Export format	Confidence	Where it goes
Pictograph JSON	✅ carried	verbatim on each annotation
COCO	✅ carried	annotation-level score
CSV	✅ carried	a confidence column
Darwin V7	✅ carried	inference.confidence (model-scored only; omitted for human 1.0)
YOLO	⬜ dropped	a trailing token would break Ultralytics’ task dispatch
Pascal VOC	⬜ dropped	no canonical confidence element
LabelMe	⬜ dropped	no canonical confidence field
CVAT	⬜ dropped	not part of the shape schema (would break import)

Attributes in exports

Per-annotation attributes (a {name: value} map — e.g. {"occluded": "true"}) are carried through the formats that model per-annotation ontology attributes natively. Set them via the SDK (attributes={...} on any annotation) or the REST save; they round-trip into the export.

Export format	Attributes	Where they go
COCO	✅ carried	annotation-level attributes object (the CVAT/Datumaro-COCO convention)
Datumaro	✅ carried	merged into the per-annotation attributes (alongside score)
CVAT	✅ carried	<attribute name=…> children on each shape, declared on the <label> (import-safe)
Pictograph JSON	✅ carried	verbatim on each annotation
YOLO / VOC / LabelMe / Darwin	⬜ dropped	no per-annotation attribute slot in the format

Only a non-empty {name: value} dict is exported; the legacy opaque-list form and absent attributes leave the export byte-unchanged.

SDK helpers

from pictograph import BBoxAnnotation, BoundingBox, PolygonAnnotation, PolygonGeometry, Point

bbox = BBoxAnnotation(
    id="ann-1",
    name="person",
    bounding_box=BoundingBox(x=100, y=200, w=50, h=80),
)

polygon = PolygonAnnotation(
    id="ann-2",
    name="car",
    polygon=PolygonGeometry(paths=[
        [Point(x=10, y=20), Point(x=110, y=20), Point(x=110, y=80)],
    ]),
)

client.annotations.save(image_id, [bbox, polygon])

The SDK Pydantic models are the source of truth — they generate the JSON Schema this page describes. If a backend rejects your payload, diff your dump (.model_dump(mode="json", exclude_none=True)) against the rejection message.