Getting prompts right remains the biggest obstacle when shipping reliable LLM applications. A tiny wording tweak can swing accuracy by twenty percent, and what works on a handful of examples often collapses at scale. In multi‑step pipelines a wrong final answer forces teams to hunt through intermediate outputs by hand, a slow and error‑prone process that stalls delivery.
FAPO (Fully Automated Prompt Optimization) tackles this bottleneck head‑on. It is an open‑source system driven by Claude Code agents that iteratively improves LLM pipelines from a baseline prompt to a target accuracy. You supply a dataset and an initial prompt; FAPO then evaluates the chain, classifies failures, proposes variants, validates them with an independent reviewer, and accepts only those that improve performance. The loop runs through three escalating levels – prompt tweaks, parameter adjustments, and structural chain changes – stopping as soon as the target is met or the budget is exhausted.
Because FAPO attributes failures to the exact step that caused them, it avoids wasted effort on irrelevant edits. Guardrails keep the optimizer from overfitting: it inspects only the training split, treats every variant as an immutable file, and requires an independent reviewer to sign off each proposal before it runs. In Cisco’s evaluation FAPO outperformed the state‑of‑the‑art GEPA optimizer on fifteen of eighteen model‑benchmark comparisons, delivering a mean gain of +14.1 percentage points. On the two benchmarks where it escalated to structural changes, the mean advantage over GEPA rose to +33.8 points.
To get started, let Claude Code scaffold the tenant files from a plain‑English task description and a JSONL dataset. Run a baseline evaluation to confirm the setup, then invoke the optimization agent with a success criterion such as composite_score >= 90. Every prompt variant, config change, and analysis is written to disk for full auditability, and a local read‑only UI (FAPO Explorer) lets you browse the results afterward.
FAPO turns prompt engineering from a manual guessing game into an automated, reliable workflow, freeing teams to focus on building features rather than fixing prompts.
#AI #LLM #PromptEngineering #Automation #MachineLearning #DevOps