Polygon vs bounding box: which annotation should you use?
Use a bounding box when you only need to know where an object is and roughly how big it is. Use a polygon when you need the object’s exact shape, for segmentation or for objects that a rectangle would describe poorly. Boxes are faster to draw and cheaper to store. Polygons are more precise and feed instance and semantic segmentation models.
What is the difference?
A bounding box is an axis-aligned rectangle defined by four numbers: an x, y, a width, and a height. A polygon is an ordered list of vertices that traces the object’s outline, so it can capture curves, holes, and irregular contours that a box cannot.
When should you use a bounding box?
- Object detection and counting. Detectors like YOLOX and RF-DETR detection predict boxes, so box labels are the natural training target.
- Speed. Two clicks per object. When you have hundreds of thousands of objects, that adds up.
- Roughly rectangular objects. Cars from the front, road signs, license plates, packages.
When should you use a polygon?
- Instance or semantic segmentation. Models like Mask R-CNN and RF-DETR segmentation predict masks, which come from polygon labels.
- Irregular or overlapping shapes. A person mid-stride, a winding road, a tree canopy, a coastline.
- Measurement. When you care about area, boundary, or precise overlap rather than a coarse region.
Accuracy and cost tradeoffs
| Dimension | Bounding box | Polygon |
|---|---|---|
| Geometry | Axis-aligned rectangle | Exact outline (N vertices) |
| Time to label | Fast | Slower |
| Spatial precision | Coarse | High |
| Trains | Detection (YOLOX, RF-DETR det) | Segmentation (RF-DETR seg, Segformer) |
| Storage | 4 numbers | One point per vertex |
The honest rule of thumb: start with boxes if detection is enough, and reach for polygons only when the downstream task needs the shape. Mislabeling a segmentation problem as a detection one wastes model capacity; over-labeling a detection problem with polygons wastes annotation budget.
Generate both automatically with SAM3
You do not have to choose at labeling time. Pictograph’s SAM3 auto-annotation returns editable polygons from a point, a box, or a text prompt, and you can keep the polygon or derive a tight bounding box from it. That means you can label once and train either a detector or a segmentation model from the same source.
from pictograph import Client
client = Client()
# A text prompt returns polygons you can edit or reduce to boxes
result = client.auto_annotate.text(
dataset_name="Street Scenes",
image_filename="frame-001.jpg",
text="delivery truck",
name="truck",
)Ready to label a dataset? See auto-annotation, then train a model on the result.