Teams building AI coding agents struggle to know if the code they generate actually works in a real Supabase environment. Manual testing is slow, error‑prone, and often misses subtle bugs like wrong row‑level security policies or missed declarative schema usage. Without a shared benchmark it is hard to compare agents, track regressions, or trust that a new model release will not break existing integrations.
Supabase Evals solves these problems by providing an open‑source harness that runs agents against actual containerized Supabase stacks. Each scenario is taken from real support tickets or GitHub issues, covering products such as database, auth, storage, edge functions and topics like RLS, migrations and SDK usage. The harness boots a hosted‑like stack and a local CLI project in Docker, so agents call the genuine MCP server and CLI, not mocks.
Scoring combines deterministic checks (e.g., can a user read a table, does an edge function return the expected JSON) with an LLM‑as‑a‑judge for semantic correctness. Agents get one retry before a final grade, reducing false negatives while keeping runs sustainable. The framework separates benchmark scenarios, which are public and used for broad comparisons, from regression scenarios that are refreshed daily to catch known failure modes without affecting published scores.
Because the code is Apache‑2.0 and runs locally with pnpm, teams can install the repo, point it at their own agent, and immediately see pass/fail rates. The tool also highlights where agents over‑rely on hand‑written migrations or ignore official packages, guiding skill updates that improve performance. In practice, top models already score near 100 % without assistance, while smaller models gain significant boosts when relevant skills are supplied.
By using Supabase Evals, developers gain a repeatable, objective way to validate AI‑generated Supabase code, catch security‑relevant mistakes early, and make data‑driven decisions about which agent or model to adopt for production workloads.
#AI #Product #DevTools #LLM #OpenSource #Cloud