Claude Code Plugins: Test with New Eval Graders & CI Gate

Anthropic’s new plugin eval workflow gives Claude Code users a concrete way to measure whether a plugin actually adds value. Before this tool, developers could only check that a plugin’s manifest was syntactically correct; they had no insight into whether Claude would invoke the skill in a natural prompt, whether the skill would keep working after a prompt tweak or a model upgrade, or whether the plugin outperforms the base model on its own.

The workflow solves these gaps by running each eval case twice—once with the plugin loaded and once without. The difference, Δ, shows the plugin’s contribution. If both arms score the same, the plugin didn’t help; a Δ near zero while a tool_used grader flagged as “with‑only” fails reveals that Claude isn’t picking up the skill on its own phrasing—a problem the older validate command can’t detect.

To get started, developers place an evals/ folder inside their plugin, add prompt.md files with realistic user wording, and create grader markdown files that define checks like regex matches, tool usage, file existence, or LLM‑based judgments. The claude plugin eval init command can scaffold cases and graders automatically, then run them locally or in CI. In continuous integration, a typical invocation looks like:

claude plugin eval . –trust-plugin –json results.json –threshold 0.8 –model claude-sonnet-5 –judge-model claude-haiku-4-5 –no-publish –max-cost-usd 20

Each run produces a detailed HTML report under evals/results//report.html, listing per‑grader verdicts, judge votes, and the Δ metric. Costs are transparent: every agent run and judge call is a real model call billed to the user’s plan, so setting –max-cost-usd prevents surprise expenses.

By making skill triggerability, edit robustness, and baseline comparison visible, the plugin eval workflow turns guesswork into measurable evidence, helping developers ship plugins that truly improve Claude’s behavior.

#AI #Product #DevTools #Claude #LLM #Testing