Project case study
When LLMs get it wrong
AI reliability research
A four-model hallucination benchmark and a lightweight response-level detector.
01
The problem
Fluent answers can still be wrong. I examined hallucination, confidence, and refusal together, then tested whether response-level signals could identify failures.
02
What I built
I evaluated four models on 223 prompts, collected 892 responses, and compared Logistic Regression with Random Forest detectors.
03
Build a shared benchmark
TinyLlama, Phi-3 Mini, Mistral 7B, and LLaMA 3.1 received the same easy, hard, trap, and adversarial prompts.
04
Detect after generation
The detector uses confidence, refusal, length, prompt category, and model identity. It scores completed responses rather than verifying claims against external evidence.
05
Read the result carefully
At a 0.40 threshold, Random Forest found 21 of 24 held-out hallucinations with nine false positives—eleven fewer than Logistic Regression at the same recall.
06
Know the limits
This is a demonstration study. Simple prompts, heuristic labels, and a four-model distribution limit generalization; token confidence is not truth probability.
07
The result
On 179 held-out responses, Random Forest reached 93.3% accuracy, 87.5% recall, and 70.0% precision. These are within-distribution benchmark results.