Most visual document retrievers in production today reuse large vision‑language models that were never built for pure encoding. They keep a separately pretrained vision tower and a causal decoder that never generates a token, adding unnecessary parameters and compute for a task that only needs compact representations. Teams face slow indexing, high memory footprints, and difficulty deploying on modest hardware while still demanding strong multimodal search quality.
NeoMME solves this by providing a single bidirectional Transformer that processes both text tokens and raw 32×32 RGB image patches through the same layers, trained from scratch. There is no vision tower, no decoder, and no extra modules—just one encoder that learns joint text‑image understanding. The 260M‑parameter NeoMME‑Retriever achieves 0.523 nDCG@10 on ViDoRe v3, matching the performance of a 3.75B‑parameter ColQwen model while being more than ten times smaller. The 800M variant reaches 0.556 nDCG@10, closing the gap with similarly sized competitors.
Deployment is straightforward: all checkpoints are released under Apache 2.0 and are immediately usable via Hugging Face Transformers. On a single NVIDIA L40S the 260M model indexes 51.3 pages per second and encodes a query in 78.3 ms on a CPU‑only host, delivering high throughput without GPU‑only reliance.
Storage efficiency is also addressed. Late‑interaction vectors normally consume ~1.5 MB per page in float32. Applying hierarchical token pooling with int8 quantization reduces this to 39 kB (≈40× smaller) while retaining 99 % of the original nDCG@10; a more aggressive binary setting pushes the footprint to 6 kB per page with only a 5 % loss in effectiveness.
In short, NeoMME offers a lean, unified encoder that eliminates overhead, speeds up indexing, cuts storage costs, and remains easy to integrate—delivering strong multimodal retrieval without the baggage of repurposed generative models.
#AI #Product #NLP #ComputerVision #Retrieval #OpenSource #HuggingFace #Efficiency