Microsoft COCO: Common Objects in Context

By 2014, the field had two landmark datasets, each with a blind spot:

ImageNet — millions of images, thousands of categories, but each image contained one centered object. Models trained on ImageNet learned to classify but not to locate or segment objects in complex scenes.

PASCAL VOC — introduced bounding boxes and segmentation for 20 categories, but averaged only 2.3 objects per image. Models could succeed without truly understanding clutter, occlusion, or scale variation.

Real-world perception demands all three: what is in the scene (classification), where each instance is (detection), and what shape it has (segmentation) — all in cluttered, overlapping, multi-scale contexts. No existing dataset pushed models toward this complete understanding.

The COCO team made four deliberate design choices that shaped the dataset's character:

Non-iconic images. Instead of searching for "dog" and getting a centered dog portrait, COCO searched for pairs of categories ("dog + frisbee", "person + bicycle") to find images where objects appear in natural, cluttered contexts. The average COCO image contains 7.7 object instances — over 3× more than PASCAL VOC.

91 everyday categories. The categories were chosen so that a 4-year-old child could recognize them: person, car, chair, dog, pizza. No esoteric species or industrial parts. Categories are grouped into 11 supercategories (animal, vehicle, furniture, food, …) to organize the hierarchy.

Per-instance segmentation masks. Every object instance gets its own pixel-level mask — not just a bounding box, not just a semantic label per pixel, but an individual contour for each separate cup, each separate person. This is the critical annotation that enabled instance segmentation as a research field.

Rich contextual statistics. COCO images have more small objects, more occluded objects, and more objects per image than prior datasets, deliberately pushing models toward real-world complexity.

Annotating 2.5 million object instances with pixel-level masks required a carefully designed crowdsourcing pipeline on Amazon Mechanical Turk. The process has three stages, each performed by different workers to maintain quality:

Stage 1 — Category labeling. For each image, workers determine which of the 91 categories are present. Since asking 91 binary questions per image is too expensive, the team used a hierarchical approach: first ask which of 11 supercategories appear, then drill into the specific categories within each supercategory. Eight workers label each image to maximize recall.

Stage 2 — Instance spotting. Workers place a crosshair on every instance of each labeled category — up to 10 instances per category. Again, eight workers per image to catch objects that any single worker might miss.

Stage 3 — Instance segmentation. Workers trace the boundary of each spotted instance by clicking around its outline to form a polygon. This is the most expensive step: about 79 seconds per instance. A separate verification step checks each mask for quality.

The entire pipeline took over 70,000 worker hours. This investment in annotation quality is what made COCO a reliable benchmark for a decade.

Each annotated instance in COCO carries several pieces of information:

Bounding box — the tightest axis-aligned rectangle that encloses the object. Format: [x, y, width, height]. This is the minimum annotation for detection.

Segmentation mask — a polygon (list of [x, y] vertices) that traces the object's outline. Multiple polygons handle disconnected parts (e.g., a person visible through a fence). This enables instance segmentation.

Area — the number of pixels inside the segmentation mask. COCO uses area to split objects into three size categories: small (area < 32²), medium (32² < area < 96²), and large (area > 96²). This split is crucial for evaluation — most detection errors happen on small objects.

iscrowd — a binary flag for densely packed groups (a crowd of people, a pile of bananas) where individual instances cannot be separated. Crowd regions are encoded as RLE (run-length encoding) masks rather than polygons, and are handled specially during evaluation.

COCO introduced an evaluation protocol that became the standard for detection research. The key idea: instead of measuring precision at a single IoU threshold (like PASCAL VOC's AP@0.5), COCO averages over 10 IoU thresholds from 0.50 to 0.95 in steps of 0.05.

Think of it this way: a prediction with IoU = 0.5 means the predicted box overlaps the ground truth by about half — a rough match. At IoU = 0.75, the overlap must be much more precise. At IoU = 0.95, the prediction must almost perfectly align with the ground truth. By averaging across all thresholds, COCO rewards models that produce tight, accurate detections, not just approximate ones.

COCO also reports size-specific metrics: AP_S (small objects, area < 32²), AP_M (medium, 32²–96²), and AP_L (large, > 96²). This breakdown revealed a key insight: detecting small objects is dramatically harder — most detectors at launch achieved less than half their large-object AP on small objects. This finding drove years of research into multi-scale detection, feature pyramid networks, and high-resolution feature maps.

A statistical comparison between COCO, ImageNet Detection, and PASCAL VOC reveals why COCO pushes models harder:

Objects per image: COCO averages 7.7 annotated instances per image, versus 2.3 for PASCAL VOC and 1.0 for ImageNet. More objects per image means more occlusion, more inter-object context, and a harder detection problem.

Object sizes: COCO has a much higher proportion of small objects. About 41% of COCO objects are small (area < 32²), compared to only about 14% in PASCAL VOC. Small objects stress feature resolution and require multi-scale architectures.

Category instances: Unlike ImageNet where each category has roughly the same number of images, COCO's category distribution is naturally imbalanced — "person" dominates, while categories like "toaster" or "hair drier" have far fewer instances. This mirrors real-world frequency distributions and tests model robustness to class imbalance.

COCO's annotation format became a de facto standard. The JSON file has five top-level keys — images, annotations, categories, licenses, and info. Every annotation links to its image via image_id and to its category via category_id. Segmentation masks are stored as polygon vertex lists, and crowd regions use RLE encoding. This relational structure lets you query annotations in any direction: all objects in an image, all instances of a category, or all objects above a given size.

The pycocotools API wraps this format with methods for loading, filtering, and evaluating predictions. Almost every modern detection framework — Detectron2, MMDetection, Ultralytics — expects COCO-format annotations natively.

The paper established baselines using the Deformable Parts Model (DPM), the strongest pre-deep-learning detector. Results were sobering: DPM achieved only about 21% AP for bounding box detection on COCO — far below its PASCAL VOC performance. The gap was largest for small objects, where AP_S was nearly zero.

These modest baselines were intentional: COCO was designed to be a challenge, not a solved benchmark. The low starting numbers gave the research community a decade of runway to push detection accuracy from ~20% to over 60% AP through architectural innovations like R-CNN, Faster R-CNN, Feature Pyramid Networks, and DETR.

While the original paper focused on detection and segmentation, COCO grew to host five distinct tasks that collectively cover the full spectrum of visual understanding:

Object detection — predict a bounding box and category label for every object instance.

Instance segmentation — predict a pixel-level mask for every object instance. This is detection + segmentation: each mask is associated with a specific object, not just a class label.

Keypoint detection — locate anatomical landmarks (17 keypoints: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles) on every person instance. This enabled pose estimation research.

Image captioning — generate a natural language description of the image. Each image has 5 captions written by different annotators.

Panoptic segmentation — assign every pixel a class label (semantic segmentation) and an instance ID (instance segmentation), covering both "things" (countable objects) and "stuff" (uncountable regions like sky, grass).