Detection Accuracy
We publish how we measure prompt injection detection and the results of the current engine, including the cases where it still fails. No detector catches every attack. These numbers are here to help you decide how to use Maskena in your workflow.
1. How we measure
- Data. Public prompt injection and jailbreak datasets published on Hugging Face, at the revision listed in each row, plus our own set of legitimate texts. We use only the test portion of each dataset, which is never used for training or threshold tuning.
- Training. The Maskena classifiers were trained on the training portions of these same datasets. On attacks written in a different style, results tend to be lower than the ones published here.
- Criterion. An example counts as detected when the verdict is not
safe, the same criterion as theflaggedfield in the API. - Configuration. Depth
standard: rules, a statistical classifier and a language-model classifier. Depthmax, which adds Llama Prompt Guard 2, is not included in these numbers. - Updates. The numbers are updated whenever the engine version changes, for better or worse.
| Metric | Question it answers |
|---|---|
| Detection (recall) | Out of 100 attacks, how many were flagged. |
| Precision | Out of 100 flags, how many were real attacks. |
| False positives | Out of 100 legitimate texts, how many were flagged by mistake. |
2. Short texts and prompts
Each example is analyzed on its own, as a POST /api/v1/check request would be.
| Dataset | Attacks | Legitimate | Detection | Precision | False positives |
|---|---|---|---|---|---|
| ai-safety-brazil/prompt-injection-br-synthetic |
400 | 400 | 93.5% | 99.2% | 0.8% |
| usezn/zn-prompt-injection-bench |
149 | 1,088 | 100% | 98.7% | 0.2% |
| jackhhao/jailbreak-classification |
141 | 259 | 85.1% | 100% | 0% |
| drakkeng/outilsia-injections-prompt-multilingue |
93 | 91 | 83.9% | 90.7% | 8.8% |
| yanismiraoui/prompt_injections |
152 | n/a | 80.3% | 100% | n/a |
| Lakera/gandalf_ignore_instructions |
112 | n/a | 77.7% | 100% | n/a |
| deepset/prompt-injections |
60 | 56 | 40% | 100% | 0% |
"n/a": the dataset contains only attacks, so false positives cannot be measured on it.
3. False positives on legitimate text
1,929 passages of real documentation in Portuguese and English: software help pages (GNOME) and technical documentation (Python and conda). None contains an attack. 3 were flagged by mistake, a rate of 0.16%.
Legal, financial and HR documents use different vocabulary from this set. If your volume is high, we recommend measuring the false positive rate on a sample of your own documents before blocking automatically.
4. Attacks inside documents
The most common real-world case is not an isolated prompt: it is a malicious sentence in the middle of pages of legitimate text, often hidden. To measure this, we inserted attacks from the test sets into real documents of three sizes (about 600, 3,000 and 12,000 characters), at random positions, using different insertion methods. As a control, we inserted legitimate texts in the same way.
| Insertion method | File flagged | Attack identified | Control false positives |
|---|---|---|---|
| Visible text (TXT) | 59.9% | 55.8% | 0.8% |
| Visible text (HTML) | 60.3% | 55.8% | 0% |
| Visible text (Word) | 60.3% | 55.5% | 1.2% |
| Visible text (PDF) | 46.5% | 43.3% | 0.4% |
| Markdown comment | 77.9% | 75% | 4.2% |
| Hidden with CSS (display: none) | 100% | 75% | 2.5% |
| White text in HTML | 100% | 75% | 2.5% |
| Hidden text in Word | 100% | 75% | 2.5% |
| Invisible text in PDF | 100% | 64.1% | 2.5% |
- File flagged: the verdict for the file was not
safe. - Attack identified: the malicious instruction itself was reported as an injection, not only the presence of hidden text.
- With hidden insertion methods, any hidden text already makes the file suspicious, even when it is legitimate. In those rows, the control column therefore shows the cases where legitimate hidden text was mistaken for an attack.
5. Known limitations
- Short, subtle attacks. Indirect requests that avoid the classic "ignore your instructions"
phrasing are the weakest point. On
deepset/prompt-injections, which has many examples of this kind, detection is 40%. - Visible attacks in long documents. When the instruction is visible, with no hiding technique, detection is lower than for the same sentence on its own, because it is diluted by the surrounding text: the file is flagged in 59.9% of cases in TXT and 46.5% in PDF.
- "Suspicious" verdicts. Many detected attacks are rated
suspiciousrather thandangerous. If your application blocks onlydangerous, effective detection will be lower than shown here. - Languages. Rules cover Portuguese, English and Spanish in the most depth. In other languages, detection relies more on the classifiers.
6. Recommendations
- Treat Maskena as one layer of protection, together with least-privilege permissions for tools and agents and human confirmation for sensitive actions.
- Use
flaggedto route content to review anddangerousto block it. - For RAG documents, prefer RAG sanitization, which removes hidden text based on how the page renders, regardless of whether the classifier recognizes the sentence.
- For high-risk content, use depth
max.
Found an attack that Maskena misses? Send it to contact@maskena.com. Reports are used to improve detection.