Teams that run coding agents such as Claude Code or Codex CLI constantly hit a wall when the model they actually want to serve lives behind a different inference stack like vLLM NVIDIA NIM or Ollama. The agents speak either the Anthropic Messages API or the OpenAI Chat/Responses formats, while the backend expects its own wire format. Rewriting the agent is not an option, so a translation layer must sit elsewhere.
Switchyard provides that layer. It is a Rust‑based proxy and library that accepts inbound requests in any of the three common formats—OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages—and converts them into a provider‑neutral internal representation. A routing algorithm then selects a target backend, the request is re‑encoded into that backend’s native format, and the call is made. On the way back, Switchyard translates the response—including streaming events—into the exact shape the original client expects, so the agent sees no change in its API.
The tool ships with four routing strategies: passthrough sends every request to a single target; random splits traffic with optional weights and a reproducible seed; llm_classifier calls a classifier model to decide between a weak or strong target; stage_router uses recent tool‑result and progress signals to pick a capable or efficient model without an extra classifier call. All strategies treat strong, weak, capable, and efficient as roles inside a route, not fixed model properties.
Observability is built in: a /metrics endpoint exposes Prometheus counters for requests, errors, token usage, latency, and routing overhead, with a tier label to separate strong and weak paths. Switchyard can be run as a standalone server, invoked via a launcher aimed at coding agents, or used as a library inside a Rust application. Licensed under Apache 2.0, it is currently marked pre‑alpha and experimental, making it ideal for evaluation and prototyping rather than production workloads.
#AI #LLM #DevTools #Productivity #OpenSource #NVIDIA