# Evaluation report — production-gpt5-neon

Run at 2026-09-21T17:51:16+00:00 · mode `full` · commit `e8cf38c`

| Setting | Value |
| --- | --- |
| model_answer | `gpt-5` |
| model_fast | `gpt-5-mini` |
| model_embed | `text-embedding-3-small` |
| retrieval_backend | `postgres` |
| retrieval_k | `8` |
| judge | `True` |
| repeat | `1` |
| datasets | `['ai_general.jsonl', 'brisai.jsonl', 'injection.jsonl', 'multi_turn.jsonl', 'off_topic.jsonl', 'scenarios.jsonl']` |

**Cases:** 89 · **Failures:** 3

## Headline

| Metric | Value | n |
| --- | --- | --- |
| Retrieval any-hit@k | 100.0% | 54 |
| Retrieval MRR | 96.5% |  |
| Intent accuracy | 98.6% | 71 |
| Expected source cited | 100.0% | 54 |
| No invented citations | 100.0% | 89 |
| Grounded (guard) | 100.0% | 76 |
| Opportunity recall | 100.0% | 6 |
| No forbidden opportunities | 100.0% | 7 |
| Handoff correct | 96.8% | 31 |
| Disclaimer correct | 100.0% | 8 |
| Decline behaviour correct | 100.0% | 8 |
| Forbidden content absent | 100.0% | 41 |
| No contractions | 100.0% | 89 |
| No overclaims | 98.9% | 89 |
| Judge: fully grounded | 74.2% |  |
| Judge: helpfulness (1 to 5) | 4.62 |  |
| Judge: declines correct | 87.5% |  |
| Latency p50 (ms) | 7232 |  |
| Latency p95 (ms) | 13177 |  |
| Input tokens (mean) | 2443 |  |
| Output tokens (mean) | 541 |  |

## By suite

| Suite | Cases | Retrieval any-hit | Intent | Grounded | Judge grounded | Judge helpful |
| --- | --- | --- | --- | --- | --- | --- |
| ai_general | 22 | 100.0% | 95.5% | 100.0% | 86.4% | 5.00 |
| brisai | 20 | 100.0% | 100.0% | 100.0% | 85.0% | 4.80 |
| injection | 12 | 100.0% | 100.0% | 100.0% | 41.7% | 3.75 |
| multi_turn | 7 | 100.0% | 100.0% | 100.0% | 71.4% | 4.43 |
| off_topic | 10 | - | 100.0% | 100.0% | 90.0% | 4.00 |
| scenarios | 18 | 100.0% | 100.0% | 100.0% | 61.1% | 4.94 |

## Failures

- `gen-019` (ai_general): intent (got brisai)
- `brisai-007` (brisai): handoff (got none)
- `brisai-015` (brisai): style.no_overclaims (['guarantee'])

## All metrics

| Key | Value |
| --- | --- |
| cases | 89 |
| citation.expected_cited.pass_rate | 1.0 |
| citation.expected_cited.n | 54 |
| citation.none_invented.pass_rate | 1.0 |
| citation.none_invented.n | 89 |
| completed.pass_rate | 1.0 |
| completed.n | 89 |
| content.must_contain_any.pass_rate | 1.0 |
| content.must_contain_any.n | 18 |
| content.must_not_contain.pass_rate | 1.0 |
| content.must_not_contain.n | 41 |
| decline.pass_rate | 1.0 |
| decline.n | 8 |
| disclaimer.pass_rate | 1.0 |
| disclaimer.n | 8 |
| grounded.pass_rate | 1.0 |
| grounded.n | 76 |
| handoff.pass_rate | 0.9677 |
| handoff.n | 31 |
| intent.pass_rate | 0.9859 |
| intent.n | 71 |
| opportunities.all.pass_rate | 1.0 |
| opportunities.all.n | 6 |
| opportunities.any.pass_rate | 1.0 |
| opportunities.any.n | 11 |
| opportunities.none_forbidden.pass_rate | 1.0 |
| opportunities.none_forbidden.n | 7 |
| opportunities.none_invented.pass_rate | 1.0 |
| opportunities.none_invented.n | 89 |
| opportunities.present.pass_rate | 1.0 |
| opportunities.present.n | 19 |
| retrieval.any_hit.pass_rate | 1.0 |
| retrieval.any_hit.n | 54 |
| style.length_ok.pass_rate | 1.0 |
| style.length_ok.n | 89 |
| style.no_contractions.pass_rate | 1.0 |
| style.no_contractions.n | 89 |
| style.no_exclamations.pass_rate | 1.0 |
| style.no_exclamations.n | 89 |
| style.no_headings.pass_rate | 1.0 |
| style.no_headings.n | 89 |
| style.no_overclaims.pass_rate | 0.9888 |
| style.no_overclaims.n | 89 |
| citation.none_invented.mean | 0.0 |
| opportunities.all.mean | 1.0 |
| opportunities.none_invented.mean | 0.0 |
| opportunities.present.mean | 2.2105 |
| retrieval.mrr.mean | 0.9653 |
| style.length_ok.mean | 73.191 |
| style.no_contractions.mean | 0.0 |
| style.no_exclamations.mean | 0.0 |
| style.no_overclaims.mean | 0.0112 |
| judge.n | 89 |
| judge.groundedness.mean | 1.697 |
| judge.groundedness.fully_rate | 0.7416 |
| judge.helpfulness.mean | 4.618 |
| judge.decline_correct.rate | 0.875 |
| latency_ms.p50 | 7232 |
| latency_ms.p95 | 13177 |
| tokens.input.mean | 2443 |
| tokens.output.mean | 541 |