Text JEPA hits 64% top-5 accuracy in pair-free ImageNet-100 test

Top News · 124 HN points

This paper tests a text JEPA for aligning separately trained text and vision encoders. It reports 64% top-5 accuracy on ImageNet-100 in a pair-free zero-shot comparison, versus at most 56% for cited language-model results up to 8B parameters.

Here is what makes this credible:

• The 109M-parameter text JEPA was trained on 900M image-derived text samples.
• The authors report better alignment with DINOv2 and CLIP vision encoders than every language model evaluated, including models up to 72B parameters.
• Separately trained JEPAs can learn representations that agree up to a rotation when their positive pairs are independent observations of the same world state.
• Captioning combines a text-trained JEPA and causal decoder with a frozen vision encoder, using Gaussian noise to handle residual alignment error.

One catch: the captioning setup fits its rotation using a small set of image-caption pairs. The pair-free result applies to the separate benchmark, and the alignment result does not establish a general-purpose replacement for language models.

Sources: hyper.ai