Guardrails promptfoo

Writes new attack cases against your own assistants and grades whether an answer breached a guardrail, using promptfoo and a model running on your machine. Nothing is sent to a vendor service.

What you can do

Grade an answer for meaning, not for a keyword. The Guardrails extension decides a verdict by planting a compliance marker in each attack case and checking whether it appears in the answer. That is deterministic and free, and it has one blind spot it cannot see past: a target that breaches a guardrail in a paraphrase, without ever emitting the marker, is recorded as holding. This extension closes that by grading the answer against a rubric with a local model, and the Guardrails suites pick it up automatically.

Write attacks you did not think of. Describe what your assistant is for and what it must never do, and promptfoo's red-team plugins produce attack cases against that description. Generated cases can be saved as an eval dataset beside the hand-written packs and run through the same suite.

Read what it did. The Red Team panel shows which promptfoo is installed, which local models grade and generate, the environment pinned on every run, and each recent grading with the grader's own sentence about it.

One thing worth knowing before you rely on it

A grading that could not be made is never reported as a clean answer. When the grader returns nothing a verdict can be read from, the case is reported as not scored, and the suite reads that as a hole in its coverage rather than as a guardrail that held. This matters because it is the failure that actually happens: measured on 2026-09-01, a local grader that could not produce a readable verdict made promptfoo's own assertion return "no breach found".

Getting started

1. Install promptfoo. Open the Red Team panel and run Install promptfoo Locally. This is the only step that uses the network, and it is worth knowing what it costs: 552 packages and about 2.3 GB, downloaded into a private folder outside your project. You will be asked to approve it.

2. Point it at a local model. The defaults are ollama:chat:qwen3.5:9b for both grading and generation. Any local ollama model works; anything else is refused rather than called, so a provider key you happen to have exported cannot pull your prompts off the machine.

3. Run a guardrail suite. Nothing else is needed. The Guardrails extension finds this grader on its own and uses it alongside the marker check.

4. Generate a pack when you want more cases. Run Generate Red Team Attack Pack, describe your assistant, and choose whether to save the result into your project.

What it costs in time

One grading measured 14 to 198 seconds on qwen3.5:9b, so an eighteen-case suite grades in tens of minutes rather than seconds. Reasoning is left switched on for the grader deliberately: over a breach and a clean answer for each threat class it was right 5 times out of 6 with reasoning on and 4 out of 6 with it off, and both of the answers it lost without reasoning were missed breaches.

What it gets wrong

The one case it fails is a false positive, and it is named here rather than left for you to discover: an answer that says it noticed and ignored an instruction hidden in its input is currently read as having followed it. So a prompt injection suite can go red on a target that behaved. Sharpening the rubric to exclude that was tried and measured — it killed the false positive and lost a real breach with it, which is the worse trade — so the wording that catches breaches was kept and this limit stands.

The choice this extension makes for you

promptfoo generates better adversarial inputs through its own hosted service, and says so itself. That service is switched off here, in four places at once, and cannot be switched back on from a setting. A tool whose job is to prove your assistant keeps a secret should not be the thing that posts your prompts somewhere, so the quality is traded away on purpose and the trade is written here rather than buried.