Annotation format

Every annotation follows the same schema. Snake-case field names, no shorthand: bounding boxes are objects {x, y, w, h}, polygons are multi-ring paths, polylines are ordered point lists, keypoints are single points.

The class-label field is name, never class.

Source: models/annotation.py - the Pydantic models are what generate this schema.

Discriminator

type	Geometry container	Notes
bbox	bounding_box: {x, y, w, h}	Axis-aligned rectangle.
polygon	polygon: {paths: [[{x, y}, ...], ...]}	Multi-ring (holes via even-odd).
polyline	polyline: {path: [{x, y}, ...]}	Open path, doesn’t close.
keypoint	keypoint: {x, y}	Single landmark. Carries no bounding_box - a point has no extent. A multi-joint pose is several of these sharing an instance_id - see below.

There are four types, not five. An oriented (rotated) box is a bbox carrying an extra oriented_box field - not its own type. See Oriented box.

Required fields

Field	Type	Notes
id	non-blank string	Unique within the image. UUIDs preferred.
name	non-blank string	Class label. Must match one of the dataset’s classes, case-sensitively.
type	one of bbox/polygon/polyline/keypoint	Discriminator.
<geometry>	see table above	Field name is determined by type.

Optional fields

Field	Default	Notes
confidence	1.0	Range [0, 1]. SAM3 and trained models set this; manual annotations get 1.0.
created_by	null	UUID of the creator, filled in server-side for SDK uploads.
attributes	[]	Per-annotation ontology attributes as a {name: value} map (e.g. {"occluded": "true", "pose": "standing"}). Exported natively to COCO, CVAT and Datumaro (see below). The legacy opaque-list form is accepted but not exported.
bounding_box (polygon/polyline)	computed	The enclosing rectangle, computed for you if omitted.
oriented_box (bbox)	null	Rotated boxes only. {cx, cy, w, h, angle} - see Oriented box. null on a plain axis-aligned box.
instance_id (keypoint)	null	Keypoint only. Which object the point belongs to - a positive integer, 1-based, scoped to the image. Points sharing an instance_id are joints of one object. null means an unassociated lone landmark. See Multi-joint pose below.

Examples

Bounding box

{
  "id": "ann-1",
  "name": "person",
  "type": "bbox",
  "bounding_box": {"x": 100, "y": 200, "w": 50, "h": 80}
}

Polygon

{
  "id": "ann-2",
  "name": "car",
  "type": "polygon",
  "polygon": {
    "paths": [[
      {"x": 10, "y": 20}, {"x": 110, "y": 20},
      {"x": 110, "y": 80}, {"x": 10, "y": 80}
    ]]
  }
}

Polygon with hole

{
  "id": "ann-3",
  "name": "donut",
  "type": "polygon",
  "polygon": {
    "paths": [
      [{"x": 0, "y": 0}, {"x": 100, "y": 0}, {"x": 100, "y": 100}, {"x": 0, "y": 100}],
      [{"x": 30, "y": 30}, {"x": 70, "y": 30}, {"x": 70, "y": 70}, {"x": 30, "y": 70}]
    ]
  }
}

Polyline

{
  "id": "ann-4",
  "name": "lane_centerline",
  "type": "polyline",
  "polyline": {
    "path": [
      {"x": 0, "y": 100}, {"x": 50, "y": 100}, {"x": 100, "y": 100}
    ]
  }
}

Keypoint

{
  "id": "ann-5",
  "name": "left_eye",
  "type": "keypoint",
  "keypoint": {"x": 250, "y": 180}
}

Oriented (rotated) box

For aerial, satellite, document and shelf imagery, where an axis-aligned box around a turned object is mostly background. In the editor it is the Oriented Box tool (hotkey O); hold Shift while turning to snap to 15°.

A rotated box is a bbox carrying an oriented_box, not a separate type. There is no obb type and no ObbAnnotation class.

{
  "id": "8f2b…",
  "name": "ship",
  "type": "bbox",
  "bounding_box": { "x": 77.7, "y": 81.3, "w": 44.6, "h": 37.4 },
  "oriented_box": { "cx": 100, "cy": 100, "w": 40, "h": 20, "angle": 30 }
}

oriented_box is the source of truth; bounding_box is its axis-aligned enclosure and is recomputed on every write, so the two can never disagree with the angle. A bounding_box you send alongside an oriented_box is replaced by the derived one.

from pictograph import BBoxAnnotation, BoundingBox, OrientedBoxGeometry

client.annotations.save(
    dataset_name="road-signs",
    image="img-001.jpg",
    annotations=[
            BBoxAnnotation(
                name="ship",
                bounding_box=BoundingBox(x=77.7, y=81.3, w=44.6, h=37.4),
                oriented_box=OrientedBoxGeometry(
                    cx=100, cy=100, w=40, h=20, angle=30
                ),
            ),
    ],
)

Keeping the axis-aligned bounding_box on every rotated box is what lets an OBB-unaware consumer read it without special-casing a type it has never heard of. A plain axis-aligned box simply leaves oriented_box unset, so the common case stays minimal.

Angle convention. Degrees, clockwise-positive, in image space (x → right, y → down), normalized to [0, 360). w and h are measured along the box’s own axes, so they do not change when it turns. This is the same convention CVAT’s rotation attribute uses, so a rotated CVAT box round-trips with no sign flip.

Export. yolo_obb (Ultralytics YOLO-OBB) and dota (the aerial standard) carry the rotation natively, as does cvat. Every other format receives the four rotated corners as a polygon, so the shape survives losslessly even where the parameterization cannot be expressed. Training an oriented-box model happens outside Pictograph today - export yolo_obb and train yolo obb on it.

Import. Both OBB formats read back in, and the round-trip is exact.

dota is auto-detected; yolo_obb must be selected explicitly, deliberately. A YOLO-OBB label line (a class id plus 8 normalized coordinates) is byte-for-byte identical to a YOLO segmentation line describing a 4-vertex polygon, so auto-detecting would silently turn your polygons into rotated boxes, or the reverse, and the result would look fine either way.

Multi-joint pose

A pose is not its own annotation type. It is several keypoint annotations - one per joint - that share an instance_id.

• A joint is a class. Each point’s name says which joint it is (nose, left_eye, ear_l, …), exactly like any other class label.
• instance_id says which object. It is a positive integer, 1-based and scoped to the image. Three people at seventeen joints each is 51 keypoint annotations with instance_id 1, 2 and 3.

[
  {"id": "a1", "name": "nose",     "type": "keypoint", "keypoint": {"x": 100, "y": 40}, "instance_id": 1},
  {"id": "a2", "name": "left_eye", "type": "keypoint", "keypoint": {"x": 92,  "y": 34}, "instance_id": 1},

  {"id": "b1", "name": "nose",     "type": "keypoint", "keypoint": {"x": 260, "y": 44}, "instance_id": 2},
  {"id": "b2", "name": "left_eye", "type": "keypoint", "keypoint": {"x": 252, "y": 38}, "instance_id": 2}
]

from pictograph import KeypointAnnotation, Point

client.annotations.save(
    dataset_name="road-signs",
    image="img-001.jpg",
    annotations=[
        KeypointAnnotation(name="nose",     keypoint=Point(x=100, y=40), instance_id=1),
        KeypointAnnotation(name="left_eye", keypoint=Point(x=92,  y=34), instance_id=1),
        KeypointAnnotation(name="nose",     keypoint=Point(x=260, y=44), instance_id=2),
    ],
)

Grouping helper: _keypoint.py

Why one field instead of a skeleton primitive. A skeleton’s edge list was redundant, a per-class template identical for every instance, while the one thing it uniquely carried was instance identity. Without a grouping key, 51 points are 51 unassociated points, and multi-instance pose cannot be trained at all: a query-based, top-down keypoint head takes ground-truth grouping as its supervision signal. instance_id keeps that in one field, and skeletons become postprocessing: group the points, then connect them with the class template.

Connectivity lives on the class, once. In the Classes tab a keypoint class can declare a template: the joint names in canonical order plus the edges linking them (four presets ship - COCO-17 person, hand-21, face-5, vehicle-8).

{"name": "person", "type": "keypoint",
 "skeleton": {"nodes": [{"name": "nose", "x": 0.5, "y": 0.06}, ...],
              "edges": [[0, 1], [1, 2]]}}

Node order is the class’s, not yours - it is what COCO’s categories[].keypoints indexes, and every pretrained pose model indexes COCO-17 by position, so a reordered template silently mistrains against published weights. Edges are 0-indexed here; COCO’s own categories[].skeleton is 1-indexed, and that +1 happens exactly once, in the COCO writer.

In the editor, instance_id is assigned for you, per class: the first nose on an image is instance 1, the second is instance 2. Placing every joint of one person takes no bookkeeping, since each is the first of its own class and they all land on instance 1. The ID bubble beside a keypoint in the sidebar cycles 1 → 2 → … → N → 1, which is how you deliberately put two same-class points on one object.

Export. coco carries pose natively: instances are grouped by instance_id, then written as categories[].keypoints (the joint names), categories[].skeleton (the edges, 1-indexed) and per-annotation [x, y, v] triplets with num_keypoints. A joint an instance does not carry serializes as 0, 0, 0, which COCO readers key on to mean absent.

Visibility (v) is COCO’s, verbatim: 0 = not labelled, 1 = labelled but occluded, 2 = labelled and visible. Occlusion is the one fact a point cannot carry in its geometry, so it rides on attributes as {"occluded": "true"} rather than as a new schema key. Set it and the joint exports as v = 1; leave it off and a placed joint is v = 2. It round-trips both ways, so an occluded joint from V7, Roboflow or COCO survives an export-import lap instead of being promoted to plainly visible. An occluded joint is still labelled, so it counts toward num_keypoints.

yolo_pose (Ultralytics) is the directly trainable one: yolo pose train data=data.yaml, with kpt_shape and a flip_idx derived from your left_*/right_* joint names (without it, horizontal-flip augmentation mirrors the image but not the joint identities, and teaches the model that a left wrist is a right one). cvat, darwin and datumaro also carry grouped points natively. Every other format keeps the object - each instance exports as the enclosure of its points - rather than dropping it.

Train a pose model in-platform with the rfdetr_keypoint pipeline, which consumes exactly this grouping.

Import. V7/Darwin and Roboflow pose datasets import as grouped points - joints named, one instance_id per source skeleton.

Storage

An image’s annotations are a plain array, with no wrapper:

[
  {"id": "ann-1", "name": "person", "type": "bbox", "bounding_box": {…}},
  {"id": "ann-2", "name": "car",    "type": "polygon", "polygon": {…}}
]

Saving is a full overwrite: pass the complete list every time. There is no partial-update endpoint.

Common mistakes

Wrong	Right
"class": "person"	"name": "person"
"polygon": [[10, 20, 30, 40]]	"polygon": {"paths": [[{"x": 10, "y": 20}, …]]}
"bbox": [x, y, w, h]	"bounding_box": {"x": …, "y": …, "w": …, "h": …}
A class label the dataset does not declare	400 - create the class first
A polygon ring with fewer than 3 points	rejected on save

Confidence in exports

confidence is carried through the formats that have a standard slot for it, and intentionally dropped from the ones that don’t (so the export stays parseable by the target tool). Human annotations export as 1.0.

Export format	Confidence	Where it goes
Pictograph JSON	carried	verbatim on each annotation
COCO	carried	annotation-level score
CSV	carried	a confidence column
Darwin V7	carried	inference.confidence (model-scored only; omitted for human 1.0)
YOLO	dropped	a trailing token would break Ultralytics’ task dispatch
Pascal VOC	dropped	no canonical confidence element
LabelMe	dropped	no canonical confidence field
CVAT	dropped	not part of the shape schema, and would break import

Attributes in exports

Per-annotation attributes (a {name: value} map, e.g. {"occluded": "true"}) carry through the formats that model them natively. Set them with attributes={...} on any annotation.

Export format	Attributes	Where they go
Pictograph JSON	carried	verbatim on each annotation
COCO	carried	annotation-level attributes object (the CVAT/Datumaro-COCO convention)
Datumaro	carried	merged into the per-annotation attributes, alongside score
CVAT	carried	<attribute name=…> children on each shape, declared on the <label>
YOLO / VOC / LabelMe / Darwin	dropped	no per-annotation attribute slot in the format

Only a non-empty {name: value} dict is exported; the legacy list form and absent attributes leave the export byte-unchanged.

SDK helpers

from pictograph import BBoxAnnotation, BoundingBox, PolygonAnnotation, PolygonGeometry, Point

bbox = BBoxAnnotation(
    id="ann-1",
    name="person",
    bounding_box=BoundingBox(x=100, y=200, w=50, h=80),
)

polygon = PolygonAnnotation(
    id="ann-2",
    name="car",
    polygon=PolygonGeometry(paths=[
        [Point(x=10, y=20), Point(x=110, y=20), Point(x=110, y=80)],
    ]),
)

client.annotations.save(
    dataset_name="road-signs",
    image="img-001.jpg",
    annotations=[bbox, polygon],
)

If a payload is rejected, diff your dump (.model_dump(mode="json", exclude_none=True)) against the rejection message.