In this guide we walk through a practical workflow for turning declarative YAML recipes into reproducible SLURM benchmark jobs for distributed LLM serving using NVIDIA’s srt‑slurm framework. First we clone the repository and install it in editable mode so the srtctl command line is available. We inspect the repository layout to understand where CLI tools, typed schemas, backend adapters, templates and ready‑made recipes live. Next we create a local srtslurm.yaml file that defines a mock cluster, container aliases, GPU settings and model paths; this lets us validate recipes without needing a real SLURM system.
We then run a dry‑run of a built‑in mocker recipe to see how srtctl checks the configuration and generates a sample sbatch script. Building on that, we author a custom disaggregated recipe that separates prefill and decode workloads across independent node and worker pools, specifying precise SGLang parameters for each stage. Another dry‑run confirms the generated job scripts match our intent.
To explore hyper‑parameter space we execute a parameter sweep, inspect the expanded configs produced in the dry‑run directory, and use the typed Python API to load our recipe, enumerate supported benchmarks, precisions and GPU types, and programmatically expand sweep templates.
Finally we simulate benchmark results for two chunked‑prefill variants, plot a compute a Pareto frontier, visualizing throughput versus inter‑token latency to identify optimal trade‑offs.
When ready to move to a production cluster we follow the standard steps: run make setup for the target architecture, perform preflight validation, apply the recipe or sweep, monitor execution, launch the Streamlit dashboard for post‑processing, and compare runs with srtctl diff. Adding an identity block with model repo, revision, container URI and framework versions guarantees reproducibility.
#AI #LLM #SLURM #Benchmarking #DevOps #GPU