Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models

hallucination-mitigation

decoding

reliability

llms

Introduces Delta, an inference-time contrastive decoding strategy that reduces hallucinations by masking and contrasting prompt variants.

Conceptual graphic illustrating reduced hallucination drift.

Delta combats hallucinations without retraining by masking random spans of the input and comparing output token distributions, suppressing unstable generations.

Evaluation Highlights

Benchmarks: SQuAD v1.1, SQuAD v2, TriviaQA, Natural Questions.
Gains: +3–6 absolute points on SQuAD tasks; +7 and +2 on TriviaQA and NQ under sampling regimes.
Significant boost to no-answer exact match on SQuAD v2 (>10 points), indicating better abstention behavior.

Why It Works

Contrastive signals penalize tokens whose probability sharply diverges across masked contexts—often a hallmark of hallucinated continuations.

Deployment

Drop-in inference tweak; no extra training data or parameter updates required.