When models agree, are they right? Receipt extraction on a laptop
The problem
An LLM can read the merchant, date, address and total off a scanned receipt in one call. The hard part is no longer extraction. It’s knowing which answers you can trust without a human checking them.
The usual shortcut is to cross-check: run two or three models or prompts over the same document, and auto-accept any field where they agree. It needs no labels and feels safe. Whether it actually works depends on two cases:
- When the models agree, look at what they agreed from. Agreement only means they’re right if the input actually contains the answer, the models are good enough to read it, and they come from different providers (say, one from Alibaba and one from Google), so they don’t share the same blind spots.
- When the models disagree, look at how the final answer is picked. A simple majority vote treats every model as equally good, so two weak models can outvote the one that got it right. And when none of them got it right, no way of picking will fix it: that field still needs a human.
Setup
Treat every extraction config as a rater, like annotators in a labeling study. Run every rater over every receipt, keep every answer at field level, and measure agreement and accuracy side by side against labelled data.
The key design decision is that every comparison is deterministic. Accuracy and agreement go through the same type-aware comparison, so neither is measured more leniently than the other, and every number traces back to a stored model output.
Find what limits accuracy: it was the input, not the model
The first run gave the models the OCR text that ships with the dataset. The best config scored 56%. Tracing each miss back to its input showed why: the OCR text was missing lines the labels contain. The full gold address was present in it for only 1 of 12 receipts, so no model could have produced it.
Qwen3.5 is natively multimodal, so the fix was to let the same model read the receipt image:
| Model | Input | Fields correct: exact match | Fields correct: ≥ 80% word overlap | Time / receipt |
|---|---|---|---|---|
| Qwen3.5-4B | OCR text | 56.3% | 58.3% | 68 s |
| Qwen3.5-4B | image | 87.5% | 91.7% | 81 s |
| Qwen3.5-4B | image + OCR text | 89.6% | 91.7% | 83 s |
| Gemma 4 E4B | image | 58.3% | 66.7% | 56 s |
| Qwen3.5-27B | image | 93.8% | 97.9% | 8.5 min |
Each percentage is the share of 48 fields (12 receipts x 4 fields) the model got right, scored two ways. Exact match: the answer equals the label after ignoring case, punctuation and common abbreviations (JALAN = JLN). ≥ 80% word overlap: company names and addresses also count if the words they share with the label make up at least 80% of all the distinct words in the two (a Jaccard similarity of 0.8 or more), so one wrong word in a long address still passes; dates and totals must still match exactly.
Changing the input was worth 31 points for 20% more time. Scaling from 4B to 27B was worth 6 more, at 6x the time and about 5x the memory. The 27B made no reading errors: its three misses are a typo in the label itself, a spelling correction, and a registration number kept after a company name.
Compare answers deterministically, with partial credit where it’s safe
Every value is normalized by type first: case, punctuation, JALAN vs JLN, 15/01/2019 11:05 AM vs 2019-01-15. Exact match then decides agreement and votes. On top of that, token-set Jaccard gives graded credit on free text, so an address that differs from the label by one word still scores 0.85. Dates, amounts and IDs stay all-or-nothing: 438.20 for 436.20 is a wrong total, not a nearly-right one.
Jaccard is a score, not a vote. A Jaccard threshold isn’t transitive (A≈B and B≈C doesn’t imply A≈C), so it can’t define the groups that majority vote counts. It also turned out to be a weaker trust signal than exact agreement (AUC 0.76 vs 0.97): Gemma’s addresses overlap heavily with the right ones and are still wrong.
When agreement actually means correctness
The trust rule is “auto-accept a field when every config agrees, review the rest”. Whether it works depends on two independent factors: the input the raters read, and the raters themselves. The text run and the vision run differ in both, so I separated them.
The input, with the model held fixed. Two pairs of Qwen3.5-4B configs with the same prompt, one pair reading the OCR text, the other the receipt image:
| Pair of Qwen3.5-4B configs | Fields where they agreed | Agreed answers that were correct |
|---|---|---|
| OCR text input (differ only in temperature) | 18% (7/40) | 57% (4/7) |
| Image input (differ only in added OCR text) | 83% (40/48) | 100% (40/40) |
Same model, very different signal. A flawed input is a source of correlated errors in its own right: when the OCR text drops the street line, every rater reading it misses the street line together. And two samples of the same model share the same failure modes: in the text run, both answered with the placeholder values Merchant and 2023-10-15, even though the real company name is in that receipt’s OCR text.
Explore it below: toggle the input, and click any cell to see both answers, the label, and which of the label’s words were missing from the OCR text.
The raters, with the input held fixed. All reading the image, each paired with Qwen3.5-4B:
| Paired with Qwen3.5-4B (image) | Fields where they agreed | Agreed answers that were correct |
|---|---|---|
| Qwen3.5-27B (same family, stronger) | 92% (44/48) | 95% (42/44) |
| Gemma 4 E4B (other family, weaker) | 54% (26/48) | 100% (26/26) |
The same-family pair agreed most, but also shared quirks: both kept a company’s registration number, and both read the postcode 81750 where the label has the typo B1750. The cross-family pair agreed less often, but every agreement was right. Diversity lowers coverage and raises the value of each agreement.
Toggle the partner model below: the same-family pair agrees almost everywhere, including on its shared quirks; the cross-family pair leaves more fields for review, but never agrees on a wrong answer.
Both together. Across all configs in each run:
| Text run (OCR text only) | Vision run (receipt image) | |
|---|---|---|
| Coverage: agreed fields / total fields | 4% | 54% |
| Precision: correct agreed fields / agreed fields | 100% (2/2) | 100% (26/26) |
| Discrimination AUC: P(correct field agrees more than wrong field) | 0.56 | 0.97 |
Agreement became a reliable trust signal only when both factors were right: an input that contains the answer, and competent raters.
That settles the fields where the raters agree, and leaves the rest open. In the text run, the configs all agreed on only 2 of 48 fields; the other 46 still needed one final answer. The obvious way to pick it is a vote, and a vote fails in the same situation where agreement did: raters of very unequal quality.
When raters disagree, weight them by reliability, not by votes
In the text run, one config was far better than the other two (56% accurate, against 15% and 11%), yet majority vote gave each of them one equal vote. It scored 31%, worse than simply trusting the best config. On 6 fields the two weak configs outvoted the good one without agreeing on anything real: neither gave a usable value, so “no value” won 2 votes to 1. On receipt X51005200931, for example, the good config read the total as 436.20. One weak config returned the words Total amount, the field’s label rather than its value; the other returned nothing. Neither is a number, so both counted as “no value”, and majority vote threw away the correct total. On 6 more every config answered differently, and the tie was broken arbitrarily in favor of junk like coan or company name.
Dawid-Skene, which estimates each rater’s reliability from how they agree with each other, with no labels, recovered the full 56%. Its estimates (0.999, 0.175, 0.136) tracked the true accuracies (56%, 15%, 11%) closely enough to trust the right rater.
Check the red cells in the Majority vote view below. The outlined ones are the 12 fields it lost even though the good config had them right: 2:1 marks a field outvoted by two unusable answers, tie one decided by the arbitrary tie-break. The chart opens on the X51005200931 total from the example above. Then switch to Dawid-Skene: all 12 outlined cells turn green. Most red cells left over are fields no config got right, including the two fully red rows at the bottom (the receipts where the model returned placeholders), so no way of combining answers could fix them. Two show the cost of trusting one rater: on receipt X51005268275, a weak config got the company and date right where the good one didn’t, and Dawid-Skene sided with the good one.
Implementation
Caveat: 12 receipts is a small sample, so read these as directions, not precise rates.
Key takeaways
- Fix the input before scaling the model. Letting Qwen3.5-4B read the receipt image instead of the OCR text lifted accuracy from 56% to 88% for 20% more time; going from 4B to 27B added only 6 more points at 6x the time.
- Agreement is a trust signal only when the input contains the answer. Reading OCR text, agreement was barely better than chance (AUC 0.56); reading the image, it separated right from wrong answers almost perfectly (AUC 0.97).
- Diverse raters make agreement worth more. Qwen3.5-4B paired with a model from the same family agreed more often but shared its quirks (95% of agreements correct); paired with a model from another family it agreed less often, and every agreement was correct.
- Don’t count votes when raters are unequal. Majority vote scored 31%, below the best single config, because two weak configs outvoted the good one. Dawid-Skene estimated each rater’s reliability without labels and recovered the full 56%.
- Some fields still need a human. When no rater got a field right, no way of combining their answers can fix it.
— Sep 24, 2026