Fix LLM Vulnerabilities with NVIDIA garak: Build Custom Probes

When you run garak and need to quickly see which probes are failing, the first hurdle is locating the newest report file. Instead of hard‑coding a path, scan the usual garak directories and the current folder for any non‑empty report.jsonl files, then pick the one with the latest modification time. Once you have the file, try to load it with garak.report.Report; if that module is missing, fall back to a manual JSON‑lines read, keep only entries where entry_type is “eval”, and compute a safety percentage as passed / total_evaluated × 100. Convert that to an attack‑success‑rate (ASR) by subtracting from 100. Build a small table with probe, detector, passed, totalevaluated, safe% and ASR_%, rename the score column for clarity, round to one decimal, and sort by ASR descending so the most vulnerable combos appear first. If matplotlib is available, draw a horizontal bar chart: labels are “probe\n detector”, bar length is ASR %, invert the y‑axis so the highest risk is on top, label the x‑axis “Attack Success Rate (%)”, set limits 0‑100, add a tight layout and show the figure. If plotting fails, just note that the chart was skipped and continue with the table output. This workflow gives you a repeatable, zero‑config way to go from raw garak output to a clear vulnerability ranking and visual summary, letting you focus on fixing the weak spots rather than wrestling with file handling.

AI #Product #ML #Security #Garaka #Safety