Evaluating multimodal models is often frustrating because raw scores hide variability and capability gaps. Teams spend time writing ad‑hoc scripts, struggle to get reliable confidence intervals, and find it hard to compare their results with published leaderboards. The evaluation pipeline presented here solves those pains by providing a single, reproducible function that takes a dataset and a backend, returns per‑example predictions, computes overall accuracy with a bootstrap 95 % CI, and breaks performance down by atomic capability, image difficulty, answer type, and judge decisions. To use it, set CFG with your desired backend (local or API), choose the judge mode (rule or LLM), and call run_eval. The helper bootstrap_ci gives you a stable interval even with few samples, and report prints a clear table, difficulty slices, and a short list of failure cases for quick error analysis. Export functions write predictions, per‑capability CSV, and a JSON meta file so you can archive results or feed them into dashboards. If you want to see how your model stacks up, the leaderboard data frame lets you overlay your overall score and capability profile on published numbers, and the radar function visualizes strengths and weaknesses. Adjust N_PER_CATEGORY, MAX_SCAN, or load the full set to increase statistical power, and run ablations such as image‑resolution sweeps or prompt variations directly by changing CFG flags. This end‑to‑end workflow turns scattered experimentation into a repeatable benchmark, letting you focus on model improvements rather than scripting logistics.
#AI #Product #MachineLearning #DataScience #LLM #Evaluation