ImageBind: One Embedding Space to Bind Them All

Previous multimodal models like CLIP aligned two modalities: images and text. To add audio, you would need paired (audio, text) data. To add depth, you would need paired (depth, text) data. Each new modality required its own paired dataset with every existing modality — a combinatorial explosion.

ImageBind's insight is that images are a natural binding modality. In the physical world, images co-occur naturally with almost every other sensory signal: photos have captions (image-text), videos have soundtracks (image-audio), RGB cameras co-capture depth maps (image-depth), thermal cameras photograph the same scenes (image-thermal), and wearable devices record motion alongside ego-centric video (image-IMU).

By aligning each modality to images — and only to images — using contrastive learning, the embedding spaces of all six modalities collapse into one. The transitive property does the rest: if audio is close to images, and text is close to images, then audio and text end up close to each other. This emergent alignment was never explicitly trained.

ImageBind uses a separate encoder for each modality, all based on the Transformer architecture. The key design choices:

• Images/Video: A frozen ViT-H (Vision Transformer Huge) from OpenCLIP. Video is treated as inflated images — 2-frame clips with temporal patch projection. The image and text encoders are kept frozen from CLIP, preserving their strong alignment.
• Text: The frozen OpenCLIP text encoder, unchanged.
• Audio: 2-second audio clips converted to mel spectrograms (128 mel bins), then fed to a ViT-B. The spectrogram is treated as a 2D "image," which is why binding through the image modality works so naturally.
• Depth: Disparity maps (inverse depth) converted to single-channel images, fed to a ViT-S. Disparity is used instead of raw depth for scale invariance.
• Thermal: Single-channel thermal images, also fed to a ViT-S.
• IMU: 5-second clips of accelerometer and gyroscope data (6 channels × 2K samples) projected with a 1D convolution (kernel size 8), then processed by a Transformer.

Each encoder produces a feature vector from its CLS token, which is passed through a modality-specific linear projection head to a shared d-dimensional embedding. The embedding is L2-normalized before computing the contrastive loss.

ImageBind trains using the InfoNCE contrastive loss — the same family of loss used by CLIP. The idea is straightforward: given a batch of image-modality pairs, push matching pairs closer in embedding space and pull non-matching pairs apart.

Consider a batch of $N$ pairs $(I_i, M_i)$ where $I$ is an image and $M$ is another modality (audio, depth, etc.). Let $q_i$ be the normalized embedding of image $i$ and $k_i$ the normalized embedding of modality sample $i$. The loss has two symmetric terms — one from image to modality, one from modality to image:

Think of it like a matchmaking game at a party. Every image is looking for its partner among all the audio clips (or depth maps, etc.) in the batch. The loss penalizes the model whenever it assigns higher similarity to the wrong partner. Over millions of batches, the model learns to place matching pairs at the same point in embedding space and separate everything else.

A critical subtlety: during training, each batch contains pairs from only one non-image modality. A batch might be all (image, audio) pairs or all (image, depth) pairs, but never mixed. The image encoder sees every batch, while each modality encoder only sees its own batches. Because the image encoder is the common thread, it becomes the hub that connects all spokes.

The most remarkable result in the paper is emergent zero-shot classification. The model was never trained with (audio, text) pairs — yet it can classify audio clips using text prompts, achieving state-of-the-art performance on benchmarks like ESC (66.9%) and surpassing AudioCLIP, which was trained on paired audio-text data.

How does this work? Because audio embeddings are aligned to image embeddings, and image embeddings are already aligned to text embeddings (via CLIP), audio and text embeddings end up in the same neighborhood of the shared space. The image modality acts as a bridge — and the bridge is strong enough for zero-shot transfer without any direct audio-text training.

This emergent property also holds for depth-text, thermal-text, and IMU-text. Any two modalities can interact through the shared space, even if they were never paired during training. The authors found that emergent alignment improves with the strength of the image encoder: a stronger ViT produces better emergent cross-modal transfer, confirming that the image anchor is the bottleneck.

Because all modalities live in the same vector space, you can perform arithmetic on embeddings. Adding the embedding of a bird singing audio clip to the embedding of a forest image produces a combined embedding that retrieves images of birds in forests. Subtracting the car engine audio from a highway image retrieves empty roads.

This is analogous to the famous Word2Vec arithmetic (king − man + woman = queen), but across modalities. It works because the shared space preserves semantic relationships: similar concepts cluster together regardless of which sense captured them.

The paper demonstrates this with audio-to-image generation: by feeding audio embeddings directly into a pre-trained DALL·E 2 decoder (which normally expects CLIP text embeddings), ImageBind can generate images from sounds — a capability that was never trained. A barking sound generates a picture of a dog. Rain sounds generate rainy landscapes. The shared embedding space makes this cross-modal generation possible without any additional training.

ImageBind's emergent zero-shot performance is remarkable because it outperforms models that were specifically trained for each task:

• Audio classification (ESC): 66.9% — outperforming AudioCLIP which was trained on paired audio-text data.
• Depth classification (NYU-D): 54.0% — surpassing MultiMAE which used supervised depth training.
• Thermal retrieval (LLVIP): 63.4% — without ever seeing thermal-text pairs.
• IMU classification (Ego4D): 25.0% — the first model to perform zero-shot IMU classification.
• ImageNet (IN1k): 77.7% — matching CLIP performance, since the image/text encoders are inherited.

For few-shot learning, ImageBind outperforms self-supervised AudioMAE on audio and supervised MultiMAE on depth classification across all shot settings (1-shot through 16-shot). A linear classifier on ImageBind's fixed features consistently beats these specialists.

The paper also shows that stronger image encoders improve emergent alignment across the board: scaling from ViT-B to ViT-L to ViT-H steadily improves zero-shot audio, depth, and thermal recognition.

ImageBind's deepest contribution is a principle: you do not need paired data for every modality combination. One binding modality is enough. This dramatically reduces the data requirements for multimodal systems and opens the door to adding new modalities (smell, taste, EEG) with only image-paired data.

The embedding arithmetic shows that the shared space preserves compositional semantics — you can combine senses algebraically and get meaningful results. This is a step toward AI systems that experience the world multimodally, the way biological organisms do.

And the scaling result — stronger image encoders yield better emergent alignment — gives a clear research direction: invest in better vision models and reap benefits across every modality for free.