SPARSEUP is an open‑source learned sparse embedding model released by the Linkup research team. It is built on a 149 million‑parameter ModernBERT backbone and is distributed under the Apache 2.0 license. The team reports an average nDCG@10 of 56.4 on the BEIR‑13 benchmark, which they claim is the strongest public vocabulary‑based sparse encoder available below 150 million parameters.
The model can be deployed immediately. The weights are hosted on Hugging Face and can be loaded with the Transformers or Sentence Transformers libraries by setting trust_remote_code=True. No additional training is required for basic retrieval tasks.
Why choose a sparse model? Unlike dense encoders that output a single vector per text, sparse models assign a weight to each vocabulary token. Each dimension corresponds to a real word or subword, so the resulting vectors fit naturally into inverted indexes and can be inspected by humans. This representation also tends to capture rare words more effectively than dense vectors.
The developers identified three shortcomings of a vanilla SPLADE setup on this backbone and applied targeted fixes. First, they introduced a logit shift, computing log(1+ReLU(x‑15)) to prevent the log function from saturating and to keep only informative dimensions active. Second, they limited each input token to its top‑12 strongest vocabulary dimensions before max‑pooling, which caps per‑token expansion without restricting the overall vector size. Third, they performed case folding, merging BPE variants such as heat, Heat, Ġheat and ĠHeat into a single identifier and retaining the maximum weight, which reduces the output dimensionality from roughly 50 000 to about 34 000.
Queries and documents receive the prefixes [Q] and [D] respectively, and similarity is computed as a dot product. Evaluation uses a maximum length of 128 tokens for queries and 512 tokens for documents. Training relied on contrastive learning with seven hard negatives per query and in‑batch negatives, fitting comfortably on a single H100 GPU; no cross‑encoder distillation was employed.
To start using SPARSEUP, download the model weights, load them with Transformers or Sentence Transformers, encode your texts, and retrieve results via dot‑product search or an inverted index. The approach delivers strong retrieval performance while keeping the representation interpretable and efficient.
#AI #MachineLearning #NLP #InformationRetrieval #OpenSource #Product