Do your guardrails hold up against a real attack?

AI guardrails are protective layers that limit and control the behaviour of a Large Language Model. They include: content filters (block harmful or inappropriate outputs), jailbreak detectors (detect attempts to circumvent security policies), PII masking (anonymise personal data in outputs), output validators (ensure responses conform to a defined schema or format), constitutional classifiers (check outputs against ethical guidelines) and topic-restriction filters (limit the scope of discussion to permitted topics). Guardrails can be implemented model-internally (training-based, such as RLHF), model-externally (separate classifier models) or rule-based (regex, keyword lists) - and all three layers have different vulnerability profiles.

A guardrail bypass is any technique by which an attacker or user circumvents the protective measures of an AI system and provokes unwanted outputs. Bypass techniques include: roleplay prompts (the system is asked to play a fictional character with no restrictions), token smuggling (special characters or unusual encodings circumvent keyword filters), multilingual exploits (switching to less securely trained languages), many-shot jailbreaking (the model is conditioned with examples), adversarial suffixes (mathematically optimised token sequences), contextual circumvention (the request is reformulated so that the classifier does not recognise it as harmful) and encoding tricks (Base64, ROT13, Leet-Speak). In the Guardrail Assessment we measure the false-negative rate - the proportion of successful bypasses out of all bypass attempts - as a quantitative indicator of your guardrail effectiveness.

Guardrail effectiveness can be quantified by two complementary metrics: The False-Negative Rate (FNR) measures what percentage of harmful requests slip through the guardrails - a high value indicates insufficient protection. The False-Positive Rate (FPR) measures what percentage of legitimate requests are incorrectly blocked - a high value indicates UX problems and user frustration. The tension between FNR and FPR is the central challenge in guardrail design: too restrictive leads to too many false positives; too tolerant leads to too many bypasses. Our Guardrail Assessment provides both metrics in a standardised Guardrail Effectiveness Score, broken down by attack category and guardrail layer.

A content filter is a rule-based or classifier-based system that checks inputs or outputs against a list of prohibited content - fast, deterministic, but easily circumvented by rephrasing. Constitutional AI (Anthropic) is a training-based approach: the model is trained with a "constitution" - a rule set of principles - to evaluate and correct its own outputs. Constitutional AI is harder to bypass than keyword filters because the restrictions are anchored in the model itself, but has its own vulnerabilities: multi-step reasoning, contextual manipulation and cross-linguistic exploits. In the Guardrail Assessment we test both paradigms with specific attack techniques tailored to each implementation.

We test all leading guardrail platforms: Azure AI Content Safety (Microsoft), Amazon Bedrock Guardrails, NVIDIA NeMo Guardrails (Colang-based), Anthropic Constitutional AI and Claude Guardrails, OpenAI Moderation API, Lakera Guard, LLM Guard (open source), and custom guardrail implementations based on proprietary classifier models. For each platform we know the specific bypass techniques and vulnerability classes. Our assessment covers both cloud-hosted guardrail services and self-hosted open-source implementations.

A Guardrail Assessment starts from EUR 10,000. The price depends on the number of guardrail layers, the complexity of the systems to be tested and the desired test scope (single guardrail component vs. complete guardrail architecture). For complex systems with multiple guardrail layers, custom classifiers and quantitative effectiveness measurement across multiple attack categories, the typical effort is between EUR 12,000 and EUR 20,000. You receive a binding fixed-price quote within 48 business hours - no hourly rates, no additional charges.

No - that is the fundamental guardrail dilemma. Any guardrail sensitive enough to detect all bypasses will inevitably also block legitimate requests. Highly restrictive guardrails achieve low false-negative rates (few bypasses) but have high false-positive rates (many legitimate requests blocked), leading to user complaints, productivity losses and ultimately the deactivation of the guardrails. The optimal operating point on the ROC curve depends on your use case: a public chatbot requires different guardrail thresholds than an internal analysis tool. Our assessment not only quantifies the current performance but also recommends the optimal configuration point for your specific deployment context.

Guardrails are not static security measures - they are challenged daily by new bypass techniques. The jailbreak community continuously publishes new attack methods; model updates change the behaviour of the underlying classifiers; and new deployment contexts open up new attack surfaces. Recommendation: after every major model update or guardrail reconfiguration run a quick assessment, and at least every six months a full Guardrail Assessment. For systems in regulated industries (finance, healthcare, critical infrastructure) we recommend a retainer model with quarterly effectiveness measurement and threshold calibration.