FEAT Add LlamaGuard scorer for safety classification of model outputs

[immu4989]

What this proposes

A new scorer that wraps Meta's LlamaGuard models (LlamaGuard-7B / Llama-Guard-3-8B / Llama-Guard-3-1B) and returns a safe / unsafe verdict on a message, plus the list of violated safety categories (S1–S14 in the MLCommons taxonomy) attached to the score as metadata.

The scorer talks to LlamaGuard through any PromptChatTarget, so users can point it at HuggingFace Inference, Together, Groq, Fireworks, a local vLLM/TGI deployment, or any other OpenAI-compatible endpoint. No local transformers / torch dependency.

Why

LlamaGuard is one of the most widely used open-weight safety classifiers for LLM red-teaming work (Llama-Guard-3, Aug 2024). Today PyRIT has no built-in way to use it:

• pyrit/score/ has no scorer that wraps an externally-hosted safety classifier. The closest existing scorers are azure_content_filter_scorer.py (Azure-specific) and prompt_shield_scorer.py (Microsoft Prompt Shields).
• PyRIT already loads the HarmBench dataset (pyrit/datasets/seed_datasets/remote/harmbench_dataset.py), but scoring HarmBench responses today goes through generic Likert / self-ask scorers rather than a purpose-built safety classifier.
• A clean LlamaGuard scorer also gives PyRIT a reusable pattern for future safety-classifier wrappers (ShieldGemma, WildGuard, the HarmBench-paper classifier, etc.) without each one inventing its own abstraction.

Proposed design

File: pyrit/score/true_false/llamaguard_scorer.py
Class: LlamaGuardScorer(TrueFalseScorer)

Constructor:

LlamaGuardScorer(
    chat_target: PromptChatTarget,
    *,
    model_version: Literal["llamaguard-7b", "llamaguard-3-8b", "llamaguard-3-1b"] = "llamaguard-3-8b",
    categories: Optional[list[str]] = None,   # default = official taxonomy for the chosen version
    validator: Optional[ScorerPromptValidator] = None,
)

Behavior:

1. Format the message into the LlamaGuard prompt using the official chat template for the selected version (the original [INST] format for v1/v2, the new chat-template format for v3).
2. Send the formatted prompt to chat_target.send_prompt_async(...).
3. Parse the response:
   • First line "safe" → score_value=False.
   • First line "unsafe" → score_value=True; second line holds the violated category codes (e.g. S1, S6) which go into Score.score_metadata["violated_categories"].
   • Anything else → handled as a bad output (logged, raises a clear error or marks the score with an explanation).
4. The raw classifier output is preserved in the score's rationale string for auditability.

Test pattern: modeled on tests/unit/score/test_self_ask_general_true_false_scorer.py. Mock the chat target, exercise the three response shapes (safe / unsafe / bad output), and verify the metadata round-trip.

Open design questions

I'd like maintainer input on a few choices before writing code:

1. Base class. TrueFalseScorer with categories on score_metadata feels natural for a binary verdict. Would you prefer the violated categories surfaced as separate Score objects (one per category), so score_aggregator can fold them in?
2. Default model version. llamaguard-3-8b is the most current; llamaguard-3-1b is friendlier for users without paid endpoint access. Any preference?
3. Multimodal. Llama-Guard-3-11B-Vision exists. Out of scope for this PR (text-only first), but is a follow-on PR for image_path / video_path something you'd accept?
4. Prompt template. Vendor the official template inline (auditable, no extra dependency) or fetch it from the model's HF tokenizer at runtime? I'd lean toward vendoring inline with a comment pointing at the upstream source.
5. Endpoint validation. Should the scorer probe the configured chat_target once on init to confirm it's actually serving LlamaGuard, or trust the user?

Implementation plan

• This PR: Llama-Guard-3-8B and LlamaGuard-7B (legacy) text-only, via any PromptChatTarget. Unit tests with mocked target.
• Follow-on PR (separate issue): ShieldGemma scorer using the same pattern.
• Follow-on PR (separate issue): multimodal Llama-Guard-3-11B-Vision.

Happy to split this PR differently if that's too much for one review.

References

• Inan et al., Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (2023), arXiv:2312.06674
• Llama-Guard-3 model card: meta-llama/Llama-Guard-3-8B
• MLCommons AI Safety taxonomy v0.5 / v1.0

[romanlutz]

I have recently thought about this as well! Glad you're bringing it up. Here's the thing that made me stop: you can provide any model endpoint as prompt target even today! Nothing is stopping you. So it really just comes down to your system prompt. That's all supported today with SelfAskTrueFalseScorer See here.

So what are we really missing? Perhaps the system prompts should be added?

[immu4989]

Thanks @romanlutz — I tried this first too. The catch: SelfAskTrueFalseScorer expects the model to return JSON (score_value, description, rationale) and _score_value_with_llm parses that. LlamaGuard won't do that — it's fine-tuned to emit "safe" or "unsafe\n<categories>" and ignores any "respond as JSON" instruction. So just dropping in a new system prompt gives us output the parser can't read.

Two ways around it I can see:

1. Add an optional response_parser to SelfAskTrueFalseScorer (or a parser-pluggable peer) and ship the LlamaGuard prompt as a YAML asset plus a small safe / unsafe + categories parser helper. LlamaGuard becomes config. Future classifiers (ShieldGemma, WildGuard, HarmBench's own) slot in the same way without adding new classes.

2. Small LlamaGuardScorer(TrueFalseScorer) with the prompt, parsing, and category map baked in. Same shape as prompt_shield_scorer.py. More surface, less flexible for the next classifier.

I'd lean (1) — smallest change that actually works, and it makes the next safety classifier basically free. But (2) is what I originally proposed, happy to drop it. Which would you prefer?

[romanlutz]

I agree the flexible approach is better since this won't be the last time we need it. Thanks for the proposals!

[immu4989]

Thanks @romanlutz — going with (1). Scope:

• response_parser Callable kwarg on SelfAskTrueFalseScorer (defaults to current JSON behavior — backwards compatible)
• LlamaGuard system prompt as a YAML asset
• parse_llamaguard_response helper (safe → False, unsafe\n<cats> → True with categories on score_metadata["violated_categories"])
• Tests modeled on test_self_ask_general_true_false_scorer.py
• Phase 1 = Llama-Guard-3-8B + LlamaGuard-7B text only; ShieldGemma / multimodal as separate follow-ups

Will open a PR.