AnyJev tackles a common production headache: turning a large language model into a reliable decision picker without any extra training. When you simply read the model’s next‑token logits for a set of answer choices you run into two practical problems. First, the model’s answer can flip if you reorder the options – a position bias that makes results unstable. Second, the raw probabilities are poorly calibrated, so you cannot trust a confidence threshold to decide when to automate a response.
AnyJev fixes both issues with a lightweight two‑level approach. Level 0 (L0) removes order bias by cycling the option list through all possible rotations, scoring each rotation with a shared prefix, and averaging the log‑scores. This geometric‑mean combination cancels additive position effects exactly. It also corrects label bias using a running mean of the model’s predicted distributions over real inputs, dividing it out after a few examples. The cost is modest – about a quarter second per decision on a single H100 when batching 32 requests with twenty options.
When you need finer confidence shaping, Level 1 (L1) adds a small temperature‑scaling step fitted on a modest set of labels (100‑500 per question). L1 leaves the ranking untouched but pushes the calibration error down dramatically, turning a noisy logit readout into a probability you can safely threshold.
The workflow is simple: install the library from PyPI, pick a backend (HuggingFace transformers or vLLM with prefix caching), define your question type – choice, yes/no, or scored bin – and call the decider. The returned object gives you a probability distribution over the fixed answers, a clear prediction, and a calibrated score you can threshold for automatic routing, triage, or any other “pick one from many” task.
Benchmarks on Qwen3‑8B with the BANKING77 intent set show L0 cutting option‑order flip rates from 23 % to under 8 % and boosting accuracy from 74.7 % to 80.3 %. Calibration error drops from 0.24 to 0.18, and the share of traffic you can auto‑decide at a 5 % error threshold jumps from 7.7 % to 46.3 %. Adding L1 pushes calibration error below 0.10 and raises auto‑decidable traffic to over half the test set.
AnyJev gives teams a drop‑in way to make LLM‑based decisions stable, trustworthy, and production ready without retraining or complex pipelines.
#AI #ML #LLM #DecisionMaking #Nokia #AnyJev