Team CyTrees
When repeated evidence outweighs recognized contradiction.
We gave three AI models a set of documents that contradicted each other, and changed only how many times each version of the story appeared. The models followed whichever claim showed up most — even while telling us, in the same breath, that their sources disagreed.
Bridgeland High School, Cypress, Texas · NSRI 2026 · AI, Data Science & Computing
01
Retrieval systems are only as trustworthy as the documents they pull back.
Retrieval-Augmented Generation (RAG) is the default way companies put a language model to work on their own data: query a database, paste the results into the prompt, and let the model answer from them. It fixes hallucination and stale memory — but only if the database is clean.
Real document stores are not clean. They hold old records, superseded reports, repeated extractions and duplicated scrapes. When retrieval returns documents that disagree, the model has to resolve conflicting evidence in context, on its own, with no signal about which source is right.
Borrowed from network science
In social networks, the majority illusion is when a genuinely rare opinion looks universal because it is over-represented among the people you happen to be connected to. Your local neighbourhood misleads you about the whole graph.
The retrieved context is a language model’s neighbourhood. If one claim is disproportionately represented in the documents that came back, the model may treat how often it appears as evidence that it is true — even though document count and truth are not the same thing.
Why it matters in deployment
Picture a company whose knowledge base still contains five copies of a superseded policy and one copy of the corrected version. Retrieval returns all six. The correction is outnumbered.
If the model resolves the disagreement by frequency, the stale claim wins — and it wins confidently. That is an operational and compliance problem, not a theoretical one.
The confound we had to remove
Earlier work on conflicting retrieval mostly used real entities and real events. That introduces a problem: if a model already knows the answer from training, you can no longer tell whether its behaviour came from the documents or from memory.
So we invented everything. Our evaluation set is 75 wholly fabricated entities that cannot appear in any training corpus. Neither competing claim is true, which means there is no correct answer to recall — only documents to weigh.
02
The informal idea, written down so it can be measured.
Majority Illusion occurs when
P(output = C | Q, D) → 1 as r increases
… irrespective of whether C is actually valid. In plain terms: the more copies of a claim you put in front of the model, the more certainly it repeats that claim back — regardless of truth.
03
3 RQs in total
RQ1
What is the effect of the majority-to-minority document ratio in the prompt context on the likelihood that the model adheres to the majority claim, and on its degree of confidence?
Plainly: if you repeat a claim more times, does the model follow it more often — and does it get more sure?
RQ2
Is there an interaction between the spatial position of the minority document (primacy vs. recency) and the majority-to-minority ratio within the prompt context on the output?
Plainly: does it matter whether the odd-one-out document sits first, in the middle, or last?
RQ3
When LLMs receive conflicting retrieved evidence, how does requesting a structured probability distribution, compared with an answer-only control, affect majority selection, conflict recognition and abstention — and how do these effects vary across models and contradiction ratios?
Plainly: if you ask the model to show its uncertainty as numbers, does it become more careful?
04
Everything is fabricated, deterministic and repeated three times.
75
Fictional entities
30 banking · 45 general corporate. Two competing values each.
6
Evidence ratios
From a balanced tie to a unanimous control.
3
Models
Independently trained, three different vendors.
3×
Samples per condition
Identical ordered prompt, collapsed to a modal outcome.
Every entity is invented, so no model can have prior knowledge of it. For each one we define a majority value and a conflicting minority value, then generate documents from standardised templates — “According to [Source], the CEO of [Entity] is [Value].” Each document carries its own mock source: a business journal, a press release, a financial audit.
Standardised templates keep the wording around the target fact constant, so ratio is the thing that varies. One honest caveat: total document count differs between ratio conditions, so evidence ratio and context length are not fully separated.
Banking
Sensitive numeric parameters: interest rate, monthly fee, lending cap, overdraft limit.
General corporate
Non-sensitive business facts: founder name, headquarters, CEO, employee count.
4:0
Control
Four agreeing documents, no conflict. Establishes that the task itself is solvable.
2:2
Tied conflict
Two against two. No majority exists at all.
3:2
Weak majority
Near-balanced disagreement.
2:1
Moderate, low count
Same 2:1 proportion, fewer documents overall.
3:1
Moderate conflict
One dissenting document against three.
4:1
Strong majority
One dissenting document against four.
Document order is fixed by a deterministic pseudo-random seed computed from the run seed, entity ID, ratio and layout index — so every condition is reproducible and all three repeats see the documents in exactly the same order.
All three are fast-tier, commercially significant, and were built by different organisations. Reasoning was explicitly enabled on each, and temperature was set to 1.0 so sampling stayed non-deterministic.
Gemini 3.5 Flash
Google · google-genai SDK
Native thinking enabled via thinking_level = MEDIUM. 2048 output tokens.
Claude Haiku 4.5
Anthropic · via OpenRouter
Reasoning enabled with max_tokens: 2048. 4096 output tokens.
DeepSeek V4 Flash
DeepSeek · via OpenRouter
Reasoning enabled with effort: "high". 2048 output tokens.
Structured elicitation
38 entities
The model must return a probability distribution over Claim A, Claim B and “indeterminate” (summing to 100), plus a separate, independent estimate that the sources conflict.
Answer-only control
37 entities
Identical system instruction and identical documents — but the JSON schema is reduced to just {"answer": …}, with no probability fields.
Entities were deterministically allocated to one arm or the other, so no model ever sees the same entity under both prompts. That removes carryover, at the cost of making this a between-entities comparison. The 4:0 control is excluded from RQ3, since a unanimous context has no minority claim to be cautious about.
Each condition was sampled three times and collapsed into a single modal outcome, so the three calls are never treated as independent observations.
95% Wilson score intervals
Majority-follow rates sit near 0% and 100% a lot, where ordinary intervals break down. Wilson intervals stay well-behaved at the extremes and with small samples.
95% Newcombe hybrid-score intervals
Used for the difference between the structured and answer-only arms, per model and per ratio — the error bars in the RQ3 figure.
The unit of analysis is one entity × ratio × model modal outcome. We had intended clustered GEE interaction models, but several binary outcome cells were 0% or 100%, producing complete or quasi-complete separation, so no interpretable global interaction coefficients could be fit. Self-reported conflict probabilities are analysed separately from behaviour, because a model can report awareness and still pick the majority answer.
05
Build a retrieval context, send it, and see what the three models actually did.
Evidence ratio
Minority document position
1 The retrieved context
2 The question put to the model
When was Nydel Labs founded?
Nydel Labs does not exist. Neither year is true, so there is nothing to recall — only documents to count.
3 What the models actually did
Choose a ratio, then send the context.
First, the sanity check: at the conflict-free 4:0 control, every model returned the unanimously supported value in 100% of conditions. So everything that follows can be attributed to the disagreement we injected — not to task difficulty or parsing errors.
RQ1
More copies of a claim meant more agreement with it.
Majority-following generally rose as the contextual majority strengthened. By 4:1, it exceeded 98% across all three models.
At the balanced 2:2 tie, majority-side selection was 36% for Claude, 41% for DeepSeek and 0% for Gemini. Claude and DeepSeek captured the majority almost immediately as soon as one appeared; Gemini held out near 57% at 3:2 and 2:1 before rising to 95% at 3:1.
Splitting the responses into full categories shows something the headline rate hides. At the 2:2 tie the models adopt qualitatively different strategies. Claude and DeepSeek both split roughly 40% majority / 40% minority — they pick a side. Gemini never commits to either value at all: 0% majority, 0% minority, resolving every tied condition through compromise (52%) or flagging (48%).
Mode instability (TIE) was rare and concentrated almost entirely at the balanced split, which means the three-sample mode was stable once a majority existed. As the ratio grows, all three converge on near-exclusive majority selection.
Reported source-conflict probability stayed high at every conflict ratio — and was largely disconnected from how cautiously they actually behaved.
In the structured arm, average confidence that the chosen answer was the best resolution climbed from 78.6/100 to 90.9/100 as the ratio moved from 2:2 to 4:1, with a dip at 3:2. Over the same range, the models’ independent estimate that the sources conflict barely moved — declining only from a mean of 98 to 87.
Protective action collapsed while awareness stayed flat. This gap is the paper’s central finding.
49% → 0%
Explicit conflict flagging, from the 2:2 tie to the 3:1 and 4:1 ratios
19% → 100%
Majority-following over the same range — moving in inverse lockstep
92.3
Mean reported source-conflict probability in the very conditions where the model went with the majority anyway
Among structured-arm conditions whose modal result was the majority claim, 98.4% reported a source-conflict probability of 50 or more, and 74.1% reported 90 or more. The models were not confused about whether their sources disagreed. They said so, clearly, and then picked the more frequent claim regardless.
We also compared each model’s self-reported confidence against how repeatable its three samples actually were. Claude and DeepSeek were more consistent than their reported confidence suggested, with the gap largest under balanced evidence. Gemini showed the opposite: reported confidence generally met or exceeded its run-to-run stability.
One important nuance: at the 2:2 tie, Gemini’s responses were entirely compromise or flagging — so its high confidence reflects confidence in a cautious resolution, not in either competing claim. Because stability measures repeatability rather than correctness, this should not be read as a conventional calibration error.
RQ2
Where the odd-one-out document sat made no reliable difference.
Method. Restricted to 2:1, 3:1 and 4:1 — the conditions containing exactly one minority document, so its position is unambiguous. A higher minority-selection rate when the dissenting document came first would suggest primacy; higher when last would suggest recency.
Finding. The small, non-monotonic variation yielded no clear primacy or recency association.
Because assignment came from a deterministic shuffle rather than forced balancing, the 225 distinct entity–ratio layouts were unevenly spread: 62 first, 111 middle, 52 last. After three replications across models and dropping two modal ties, that gave 185 first-position, 333 middle-position and 155 last-position samples.
| Minority position | Followed majority | Followed minority | Samples |
|---|---|---|---|
| First | 91.9% | 1.1% | 185 |
| Middle | 94.0% | 1.8% | 333 |
| Last | 90.3% | 0.6% | 155 |
The minority claim was chosen barely ever, wherever it sat — between 0.6% and 1.8% of the time.
RQ3
Asking for probabilities is not a measurement. It is an intervention.
Requesting probabilities changed the answer, and the direction flipped
Pooled across conflict ratios, the answer-only arm produced explicit abstention in 0.0% of conditions, expressing disagreement through compromise instead (18.3%). The structured arm reversed that: flagging rose to 12.3% while compromise nearly vanished (1.6%), and majority selection rose modestly from 74.2% to 79.7%.
At the 2:2 tie
Elicitation helped
It reduced majority selection and sharply increased flagging. Asked to quantify its uncertainty, the model surfaced it.
At 3:2 and 2:1
The sign reversed
Elicitation increased majority selection, with only small changes in flagging. The same intervention accelerated majority capture.
At 3:1 and 4:1
Nothing left to move
Behaviour was already saturated at near-total majority following, so the effect was negligible.
The crossover came with large heterogeneity across models: greatest in Gemini, tie-specific in DeepSeek, and mild in Claude.
06
01
Conflict-probability estimates stayed high throughout, while flagging was confined to the balanced tie and became scarce once a majority formed. Because the model produces both signals itself, this is a dissociation within its own outputs — not a gap between perception and policy. A system that resolves conflict by frequency will propagate whichever claim appears more often, even when that claim is incorrect, duplicated or outdated.
02
Given identical stimuli at the tie, Claude and DeepSeek took a side while Gemini compromised or abstained. We cannot say from this design why they differ — architecture, alignment, reasoning configuration and API behaviour all vary together. The practical consequence stands regardless: a RAG pipeline validated on one model cannot be assumed safe on another.
03
Structured elicitation behaved as a behavioural modifier, not a passive read-out of an internal state. Methodologically that matters: evaluating a model’s conflict-resolution ability with a single prompt format will misrepresent how it actually behaves.
04
At a genuine tie, asking for probabilities surfaced uncertainty and increased flagging. Once even a slight majority existed, it consolidated the model into majority adherence. Whether the intervention helps depends on the evidence ratio it meets.
Implications for RAG architecture
Rather than treating every retrieved document as independent evidence, a pipeline could group or down-weight documents that repeat the same underlying claim — stopping duplication from masquerading as independent corroboration.
Because reported certainty tracks frequency, it cannot be relied on as a guardrail. And structured elicitation may be most useful when competing claims are evenly represented — rather than applied uniformly to every query.
Limitations
Conclusion
Across all three models, the likelihood of selecting the more frequently supported claim generally increased with the majority-to-minority ratio, while the position of the minority document produced no clear effect. Models reported high probabilities of source conflict even while selecting the majority value, revealing a gap between reporting disagreement and responding cautiously. Structured probability elicitation was not a remedy: it increased flagging at the balanced tie but sometimes strengthened majority following under slight imbalances. This tendency represents a potential majority illusion vulnerability in real RAG systems whenever incorrect, duplicated or outdated information becomes the contextual majority.