Better Harness, Better Answers

Document Intelligence Without Bigger Models

Nirant Kasliwal

Machine learning and AI engineer. I started Scaled Focus to help companies evaluate, optimize and deploy agents.

Built

FastEmbed, first version

Embedding library on ONNX, used by Qdrant, NVIDIA NeMo Guardrails and 3,000+ more repositories

Contributed to

Ragas

Metrics for evaluating agents

Wrote

NLP in Python: Quickstart Guide

Book for software engineers, over 5,000 copies sold

Maintained

Awesome NLP

Resource list, recommended by Andrew Ng’s CS 230

Earlier talk on harness engineering: The Agent as Compiler. More at nirantk.com

One small model stays locked. Only the harness changes.

Model

Qwen3.5-9B, 4-bit

fp4 weights, Darkbloom through OpenRouter. Temperature 0, three runs, thinking measured apart.

Document processing

OCR, conversion, chunks

Search

Embedding, BM25, Hybrid, grep

How we use search

One pass, or tools in a loop

Score on

20 + 20 questions, same 531 pages

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Corpus holds real public documents

531

pages

documents

20 + 20

validation and test questions

Sources: XL-DocBench, ViDoRe v3 and CUAD. Validation and test differ only in their questions. Answer pages sit among the closest matches by embedding and BM25.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Why is this hard?

The answer is not text

1 The page is a scan

A scanned table holds no text for search to match.

2 The value is in a chart

A bar has a height, not a number.

3 The label is in a drawing

A part’s name exists only as ink on a diagram.

The answer is not where search looks

4 The wording differs

The question says B gas. The report says gaz B.

5 The proof is an absence

No retrieved passage can show a clause is missing.

6 The answer spans pages

One page points to the next. None holds it all.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

The answer is not text

1 THE PAGE IS A SCAN

What is the next Ship Transport drop height after a non-damaging impact?

Scanned Table 516.8-X with Ship Transport drop height 1.5 m highlighted

PDF text on this page: 0 characters

MIL-STD-810H, p. 40

2 THE VALUE IS IN A CHART

What was the average hourly wage in the EU’s finance and insurance sector in 2021?

Graph 2.16: bar chart of hourly wages with finance and insurance reaching about 50 euros

The bar reaches 50 with no data label

EU wage report, p. 77

3 THE LABEL IS IN A DRAWING

Where is the 25mm gun safety pin stored when not in use?

Line drawing of an aircraft access door with the label Safety Pin Stowage Bracket highlighted

PDF text: only the figure’s caption

TO 00-25-172CL-3, p. 7-12

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Read each question aloud and ask how a text search would find the answer. It cannot. The scanned page has zero characters behind it. The chart’s value is a bar height, and LightOnOCR estimates 50 from it. The drawing’s PDF text holds only the caption, yet LightOnOCR lists the labels, including safety pin stowage bracket. These three end the argument about whether you need OCR. The chart question is in our validation set. The scan comes from XL-DocBench and the drawing from ViDoRe v3, and we ran those two separately as a demonstration: none of our 40 scored questions has its answer only on a scan or a drawing, which is a limit of the test sets. Run separately, on Qwen3.5-9B fp4 through Darkbloom: Grep Token Reduction, one run each, over the PDF text layer answered not_found on both the scan and the drawing, and the same agent over the merged Markdown answered both correctly, with 1.5 m and the safety pin stowage bracket.

The answer is not where search looks

4 THE WORDING DIFFERS

Which country supplies all of France’s B gas?

French energy programme page with gaz B and Pays-Bas highlighted

“gaz B”, “Pays-Bas”: no shared word

Programmation pluriannuelle de l’énergie, p. 101

5 THE PROOF IS AN ABSENCE

Does the Cremer-Ultragenyx supply agreement contain a non-compete clause?

Supply agreement clause with the word exclusively highlighted

“Compete” appears nowhere in it

Ultragenyx supply agreement, p. 1

6 THE ANSWER SPANS PAGES

A fuel checklist points to one order for ESDS items. How far from the wire is the heat source held?

Technical order page: crimping tool drawing above text saying hold the heat source four to five inches away

The checklists name this order. It holds the distance.

TO 00-25-234, p. 68

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

LightOnOCR-3 reads the chart

Graph 2.16 from the EU labour report: bars of hourly wages by sector, finance and insurance near 50 euros, with no printed value

LightOnOCR-3 output

| Sector | Real hourly wage 2021 |
| --- | --- |
| Finance & insurance | 50 |
| ICT | 48 |
| Real estates | 33 |
| Construction | 29 |

PDF text: not_found, because the bar has no data label.

Markdown: 50 EUR, the benchmark’s answer.

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Grep agent, three runs: not_found on PDF text, 50 EUR on Markdown. Validation question 3. EU wage report, p. 77. Output shortened.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

PyMuPDF restores lines OCR missed

Docling, OCR off

every page

baseline, words

baseline, numbers

pymupdf4llm

text pages

LightOnOCR-3

scan, chart, drawing pages

3.3x

fewer words lost

1.4x

fewer numbers lost

pymupdf4llm

text pages

LightOnOCR-3

scan, chart, drawing pages

PyMuPDF

restores lines OCR missed

15x

fewer words lost

12x

fewer numbers lost

1x is the worst pipeline. The 531 pages, scored against the text PyMuPDF extracts, so OCR on scans goes unscored. Restoring also runs on pymupdf4llm pages, where it adds little. Results

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Read it left to right. Docling with OCR off, on every page, is our 1x. It is not Docling’s best configuration. Then we split pages by type. pymupdf4llm, a library built on PyMuPDF, reads the characters stored in the PDF and writes them as Markdown for text pages. LightOnOCR-3, a vision model, reads a picture of scan, chart and drawing pages, so it never sees those stored characters. A page goes to the vision model when it holds under 200 characters of text, images over 10 percent of the page, or 40 or more vector drawings. Words improve 3.3 times but numbers only 1.4 times, because OCR misses things: on its pages, 21.9 percent of the numbers PyMuPDF can extract were gone. Last, PyMuPDF extracts each line of the page, we check it against the OCR Markdown, and any line holding a word or number the Markdown lacks is restored to the page. This is plain counting, with no model. That is 15 times fewer words lost and 12 times fewer numbers lost, or about 5 and 9 times against the same pipeline without this step. The block sits on LightOnOCR-3 because that is where this recovers text. pymupdf4llm pages already came from PyMuPDF’s text, so there it adds much less. On the 531 pages, 2,073 of 2,960 lines restored to vision pages had at least half their words absent from the page, against 217 of 1,441 on pymupdf4llm pages. One mismatched word pulls in a whole line, so some restored lines are near-duplicates. A real recovery: on a page of a hydraulic charts document the model left out the printed instruction “Connect with a straightedge the given culvert diameter or height”. In plain terms, words lost go 6.7, 2.1, 0.5 percent and numbers lost 16.4, 12.0, 1.4 percent. The last column keeps 99.6 percent of words and 98.6 percent of numbers, where Docling keeps 93.3 and 83.6, so small losses make these ratios look large. This metric treats PyMuPDF’s text as truth, so it says nothing about charts and scans. For charts we ran ParseBench on its 568 chart pages: LightOnOCR-3 at our 2 megapixel setting passes 58.3 percent of data-point rules, against 1.1 percent for pymupdf4llm alone. Its published 64.7 uses 5 megapixels, so expect a gap at our setting. We chose it from this measurement, not from the test questions.

OCR wins only when the answer is pixels

0 of 20

test questions need OCR

Every answer is in the PDF text.

1 of 20

validation questions need OCR

EU wage chart: 0 of 3 runs right on PDF text, 3 of 3 on Markdown.

2 of 2

separate questions need OCR

Scanned table and line drawing: not_found on PDF text, correct on Markdown.

Count your scans, charts and drawings before you pay for OCR.

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Needs OCR: the answer exists only in the page image. Chart: grep agent, 3 runs. Scan and drawing: Grep Token Reduction, 1 run each, outside the scored question sets.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

So how many answers does OCR add? Fewer than the last two slides suggest. Of the 20 test questions, none: every answer is in the PDF text layer. Of the 20 validation questions, one: the EU wage chart you just saw. Across every search method we ran on it, that question scored none of 9 runs on PDF text and all 12 runs on Markdown. We also built two questions outside the scored sets, a scanned table and a label in a line drawing. Grep Token Reduction answered not_found on both from PDF text, and answered both correctly from Markdown, in one run each. So OCR is all or nothing per question: it adds nothing until the answer exists only as pixels, and then it is the only way in. Our scored questions hold few answers like that, which is a limit of these question sets. Your documents decide how much this matters, so count them first.

Hybrid adds 2 answers for almost no tokens

Strategic alliance agreement clause 1.3 with $19,644,000 highlighted

Embedding top 20

BM25 top 20

Hybrid: RRF

k = 60, keep top 5

BM25 finds the exact string “$19,644,000”, which embedding search misses in every run. Hybrid has a median of 7 of 20 across 3 runs, against 5 on test.

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Three runs on 531 pages. Same chunks, same prompt, same top 5. Tokens: +1% median, +2% total. Exact string: validation question 9. Adaptimmune alliance agreement, p. 2

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Plain RAG answers cybersecurity for biology

Which incident category covers misuse of high-consequence biological material?

One page of the guidance: four incident categories

5.3.1 Incidents directly involving biological agents

“misuse of high-consequence material.”

THE ANSWER. This heading is in no retrieved chunk.

5.3.2 Physical security incidents

5.3.3 Personnel-related biosecurity incidents

5.3.4 Incidents related to information security and cybersecurity

A RETRIEVED CHUNK STARTS HERE.

PLAIN RAG, WRONG IN ALL 3 RUNS

“Incidents related to information security and cybersecurity”

GREP AGENT, READS THE PAGE

“Incidents directly involving biological agents”

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Question shortened. Validation question 18, shown as an example. WHO Laboratory biosecurity guidance, p. 41

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Search runs in one pass or in a loop

Linear pipe aka plain RAG

Question

→

Retrieve top 5

→

Answer

Single fan-out

Question

→

Write terms once

→

Grep each term

→

Answer

Grep agent

Question

→

Call grep or read

→

Read the result

→

Answer

↺ repeats until it can answer

BM25 agent

Question

→

Call BM25 or read

→

Read the result

→

Answer

↺ repeats until it can answer

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Correct out of the 20 test questions: median across 3 runs, thinking off. The BM25 agent was added after the test scores were known.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

BM25 agent adds 2 answers

2 more answers

Median correct out of 20, across 3 runs

Grep agent 7–9 per run

BM25 agent 8–10 per run

Hybrid agent 8–10 per run

BM25: −19% median, +34% total

Bars: median per question. Total: per run of 20.

Grep agent 2.8M total

155k

BM25 agent 3.7M total

126k

Hybrid agent 4.7M total

238k

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. 20 test questions, 531 pages. Tokens: input and output, medians across 3 runs. Hybrid: embedding + BM25 with RRF, top 5. Both loops were added after the test scores were known.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

If the loop matters, does the search inside it matter? We forked the plain grep agent and changed one tool. Same loop, same read tool, same 20-call budget, thinking off. Search now returns the five best passages, by BM25 alone or by Hybrid, in place of every matching line. Across three runs both have a median of 9 of 20 against 7 for grep. The runs overlap, 8–10 against 7–9, and we added these after the test scores were known, so call it a lead and not proof. Neither beats the other on score. Per question it moves both ways again: BM25 gains 5 and loses 5, Hybrid gains 6 and loses 3. On tokens, give both numbers. Against grep, BM25 is −19% median, +34% total: its median question is cheaper and its long questions cost more. Hybrid is +54% median, +67% total. The median answer makes 5.5 searches and 7 reads with grep, 2 and 9.5 with BM25, and 2 and 12 with Hybrid. So if you try one, try BM25: same median as Hybrid, no embedding model and fewer tokens by both measures. If someone asks about the validation questions: BM25 and the grep agent tie at a median of 16. If someone asks about BM25 with our token fixes, the FAQ has it. That is the score. The next section is about cause and cost: what the traces showed and the tokens the fixes saved.

Our tools hide the clause from the grep agent

Contracts under the law of the State of Israel: our first rewrite named 5 of 6, each with a correct quote.

NICE outsourcing agreement, clause 17.11 Law and Jurisdiction: governed by and construed in accordance with the laws of the state of Israel

The missed contract, in the PDF. The converter merged clauses 17.9 to 17.11 into one line of 4,248 characters.

What the model saw of that one line

Read showed 600 characters. Search showed 400. Neither gave a warning.

Clause 17.11 starts at character 2,197

Fix: 200 characters around each match, whole lines on read. The fixed agent named all 6.

Local run: Qwen3.5-9B, nvfp4, Ollama, earlier 222-page set, question 6, NICE outsourcing agreement, CUAD. Hosted on merged Markdown the long line is gone, and a 6,000-character read limit stops the grep agent instead. Bar drawn to scale. NICE outsourcing agreement, p. 25

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

The last section moved the score. This one is about cause and cost, and it is the most useful for your own work. The score tells you that something failed; the trace tells you why. Our first rewrite added a catalogue, an outline, refusals and quote checks. I blamed the model for this one until I checked. The grep agent did the right thing: it searched for Israel, opened the NICE contract, and searched that file for “governed”. The tool printed the first 400 characters of the matching line and stopped. That cut was mine, added to save context, and it gave the model no sign that anything was missing. This is the most reusable lesson in the talk. After the fix, the agent got this question right in that local run. In three hosted runs on the 531 pages the Markdown is merged, so that long line is gone, and the plain grep agent still missed it, for a sibling reason: its one read of the contract stopped at the 6,000-character output limit, with a marker this time, and the clause sits at character 8,449. It never read further. Every truncation in your tools needs to say that it truncated. Check yours tonight.

Fresh traces expose a new loop in our own fix

HELD-BACK SET, QUESTION 4

read(line 204) × 9

each repeat refused by the harness
budget gone, 20 of 20 calls
answer: not_found

HELD-BACK SET, QUESTION 7

final_answer(…) × 3, no such tool

read(line 54) × 9, refused
budget gone, 20 of 20 calls
answer: wrong figure

Three changes to the harness

Adds a real final-answer tool, the one the model kept asking for

Ends the search after two turns of refused calls

Stops charging refused calls to the budget

Local run: Qwen3.5-9B, nvfp4, Ollama, held-back set of 10. Grep agent after our first rewrite, run 1: half of the ten questions used all 20 calls. With those changes, 2 used all 20.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Fixes cut tokens; the score stays within noise

Score within noise

Median correct out of 20, across 3 runs

Grep agent 7–9 per run

Grep Token Reduction 7–8 per run

−37% median, −21% total

Bars: median per question. Total: per run of 20.

Grep agent 2.8M total

155k

Grep Token Reduction 2.2M total

98k

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. 20 test questions, 531 pages. Tokens: input and output, medians across 3 runs. Grep Token Reduction frozen before the test questions.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Every fix on the last few slides solved the trace it came from, so here is the score on the test questions. As a median across three runs, the grep agent gets 7 of 20 and Grep Token Reduction gets 8. The lowest and highest runs overlap, so that one answer is within the noise. The fixes move single questions in both directions: against the grep agent, Grep Token Reduction gains 4 questions and loses 6. So each story is true and the score barely moves. Tokens are where it shows. Against the grep agent the median question costs 37% less, and one run of 20 questions costs 21% less in total. Say both numbers, because long questions shrink less than short ones. Median turns fall from 20 to 13, and this agent’s 15-turn limit accounts for part of that. This agent keeps the catalogue, refusals and quote checks from our first rewrite, and adds a map of the collection, naming each source and how it words things. It reads several pages in one call and names any page left out, says when a term search came back empty and how many lines hold each word alone, and drops the review turn. My first drafts failed: hard turn limits of six and ten stopped reads short and lost questions. One bug is worth telling. The model sometimes answered correctly, then replied with an empty object at the review, and my harness kept the empty one. A later empty reply no longer erases a correct answer. One measurement warning: I tried caching model replies to save time. A run partly replayed from that cache scored four questions higher than the same version run fresh. I removed the cache. If someone asks about the validation questions: Grep Token Reduction has a median of 13 against 16 for the grep agent, and tokens still fall, 8% on the median question and 28% in total.

Same model goes from 5 to 9 on harness alone

How the model uses search

Test, median of 20

Tokens

Total

Turns

Plain RAG

Embedding, top 5, one answer

5 (5)

2.6k

51k

Hybrid RAG

Embedding + BM25 with RRF, top 5, one answer

7 (6–7)

2.7k

52k

Single fan-out

One batch of grep terms, one answer

5 (4–5)

6.6k

141k

Grep agent

Grep and read tools in a loop

7 (7–9)

155k

2.8M

Grep Token Reduction

Adds refusals, quote checks, a collection map and multi-page reads

8 (7–8)

98k

2.2M

Grep Token Reduction + Thinking

Adds reasoning before each reply

8 (8–11)

94k

2.1M

BM25 agent

Swaps grep for the five best BM25 passages

9 (8–10)

126k

3.7M

BM25 agent + Thinking

Adds reasoning before each reply

6 (5–10)

304k

5.5M

Hybrid agent

Swaps grep for the five best Hybrid passages

9 (8–10)

238k

4.7M

19.5

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Medians across 3 runs; brackets show lowest to highest run. Tokens and turns: per question. Total: per run of 20. Loops with BM25 and Hybrid were added after test scores were known. Results

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Here is every row side by side, now that you have seen where each one comes from. The same model goes from a median of 5 with plain RAG to 9 with the BM25 agent, from harness changes alone. Same model, same documents, same converted Markdown in every row. Only the way the model uses search changes. These are the test questions: correct answers out of 20, as the median across three runs, with the lowest and highest run in brackets. At temperature 0 plain RAG scored the same in all three runs; the grep agent’s spread is most likely the hosted endpoint varying on long prompts. Plain RAG and the fan-out each have a median of 5. The grep agent has 7, and so does Hybrid RAG in one pass, so a loop alone is not the whole story. Grep Token Reduction has 8, with 37% fewer median tokens and 21% fewer in total. Adding thinking leaves it at 8, with one run at 11. The BM25 agent and the Hybrid agent each have 9. Adding thinking to BM25 drops it to 6, with one run at 10, at 2.4 times the median tokens and 1.5 times the total. The runs overlap throughout, so read these as leads, not proof. If someone asks about the validation questions: the same lift shows there, plain RAG 9 and the grep agent and the BM25 agent 16. The two thinking rows are the one dial I have not explained yet, so that comes next.

Thinking does not lift the agents’ medians

Test 20

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. 531 pages. Right column: median across 3 runs, thinking off to on. Reasoning shares the output limit: 16,384 tokens on, 4,096 off.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Hover a dot for its run, and click a row to focus it. The same model has one more dial, and here is why I run everything three times. My first run with thinking on scored 13 of 20 on the test set, and I nearly built this slide around it. That run had a 4,096-token output limit, and thinking shares that limit, which also starved the fan-out of any search terms. At 16,384 tokens, Grep Token Reduction scored 11, 8 and 8 against 8, 7 and 8 without thinking. So its median across three runs stays at 8 of 20 and its best run rises to 11, at similar tokens: a median of 94,000 against 98,000 per question, and a total of 2.1 million against 2.2 million per run of 20. The BM25 agent goes the other way. Its median falls from 9 to 6, with runs of 10, 6 and 5, and its tokens rise: a median of 126,000 to 304,000 per question, and a total of 3.7 million to 5.5 million per run. The median answer with thinking makes 20 calls against 12 without. This agent has no refusals or collection map, which Grep Token Reduction has, and we did not test whether that explains the gap. RAG’s median moves from 5 to 7. The fan-out does not gain: a median of 4 against 5, at 3.1 times the median tokens and 2.8 times the total. Read that row with care: even at 16,384 tokens, 5 to 7 of its 20 questions per run ended with no search terms, mostly because reasoning used the whole limit. So part of that drop is a limit failure. Thinking also makes results vary between runs, which plain RAG alone does not.

No fix came from these 20 questions

Plain RAG

Hybrid RAG

Single fan-out

Grep agent

Grep Token Reduction

Grep Token Reduction + Thinking

BM25 agent

BM25 agent + Thinking

Hybrid agent

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. 20 test questions, 531 pages. Dark: correct in all 3 runs. Mid: in some. Right: median correct of 20 across 3 runs.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Loops cost 36–115x plain RAG’s median tokens

Turns per question

Tokens per question, total per run

Plain RAG

2.6k 51k

Single fan-out

6.6k 141k

Grep agent

155k 2.8M

Grep Token Reduction

98k 2.2M

Grep Token Reduction + Thinking

94k 2.1M

BM25 agent

126k 3.7M

BM25 agent + Thinking

304k 5.5M

Hybrid agent

19.5

238k 4.7M

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. $0.08 per million input tokens, $0.13 output. Bars: median per question across 3 runs, test 20. Grey: total tokens per run of 20, median across 3 runs. Each panel has its own scale.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

Run this loop before you buy a bigger model

Before the first loop

Convert scans, charts and drawings to Markdown, with page markers

Measure plain RAG and Hybrid RAG on your own questions

Freeze validation and test questions

Observe

What happened?

Log every tool call and every tool result

Example: read(line 204) nine times, then not_found

Orient

Which layer failed?

Place each failure in documents, search or use of search

Example: a silent 400-character cut hid the Israel clause

Decide

What do we change?

Fix a class of failure, never one question

Example: silent cuts, so tools now label cuts and empty results

Act

Did it work?

Change one part, then retest on fresh questions in three runs

Example: one run scored 13 of 20; three more scored 11, 8 and 8

Same
model

Next failure

OODA loop: John Boyd. Examples come from the traces in this talk.

1 The Challenge

2 Document processing

3 Search

4 How we use search

5 Fixes cut tokens

6 Results and playbook

This is the slide people will photograph, so leave it up. OODA is John Boyd's loop: observe, orient, decide, act. Start at the left: three things come before the first loop, because without one-pass scores and frozen fresh questions you can't tell whether anything helped. Convert to Markdown where answers sit in scans, charts or drawings; on our test questions loop agents on PDF text scored about the same, and plain RAG scored 7 against 5. Teams skip those. Then walk clockwise. Observe: log every call and result, so a stall like read line 204 nine times is visible at all. Orient is the step people rush. Name the layer before you touch anything: documents, search, or how the model uses search. I blamed the model for the Israel clause, and the fault was a silent cut in my own tool. Decide: fix the class, never the one question, and that is why tools now label what they cut or find empty. Act: change one part and retest on fresh questions, in several runs. That step cost me two good stories: one run scored 13 and three more scored 11, 8 and 8, and a run replayed from saved replies scored four questions higher than a fresh one. The fresh traces then show the next failure, and you go round again. The model in the middle never changes.

Change search and the loop before the model

Read the trace to learn why an answer failed.

Questions, pages and traces: nirantk/document-harness-evaluation
Code: github.com/NirantK/better-harness-better-answers

Documents: CUAD (The Atticus Project, CC BY 4.0), ViDoRe v3 (arXiv 2601.08620), XL-DocBench (arXiv 2608.00036).

Questions the details answer

What if BM25 gets the token fixes?

Can it connect facts across pages?

What is in the harness?

How does it choose the file?

Does it handle other languages?

How did the excerpt fix work?

Who decides what the model sees?

Where does the empty-result hint go?

Why does the agent read the wrong lines?

What does refusing a call mean?

What else changed in the harness?

What runs in each component?

Which embedding and index run search?

One full agent run, every message with notes: Inside One Agent Run

FAQ

Token fixes save BM25 a third of its total

Score within noise

Median correct out of 20, across 3 runs

BM25 agent 8–10 per run

BM25 Token Reduction 7–9 per run

With model-chosen count 6–9 per run

−22% median, −34% total

Bars: median per question. Total: per run of 20.

BM25 agent 3.7M total

126k

BM25 Token Reduction 2.5M total

97k

With model-chosen count 2.8M total

111k

Hosted runs: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. 20 test questions, 531 pages. Tokens: input and output, medians across 3 runs. Tuned on validation questions and added after the test scores were known.

FAQ

This is the combination people ask about: BM25 search inside the Grep Token Reduction agent. It keeps the shorter prompt, the catalogue that drops out after two turns, multi-page reads and quote checks. On the test questions BM25 Token Reduction has a median of 8 of 20 against 9 for the BM25 agent, and the runs overlap, so that one answer is within the noise. Against the BM25 agent the token change is −22% median, −34% total. The first version scored 12 of 20 on validation. Its traces showed the same silent cut as the Israel clause: passages stopped at 1,000 characters, sometimes before the match. A 15-turn limit and an early stop also ended searches the BM25 agent finished. So each result is now a 600-character excerpt around the matching words with its line number, the turn limit is gone, and the stop comes after three refused turns. We also let the model choose how many passages to fetch, from 1 to 10. It never asked for fewer than 5: on the test questions it left the count out in 114 of 130 searches, chose 5 in 11 and 10 in 5. That version has a median of 7, so a fixed five is simpler and no worse. If someone asks about the validation questions: BM25 Token Reduction has a median of 17, the model-chosen count 16, and the BM25 agent 16.

Multi-hop question spans three pages

For outlet-control headwater of a flowing-full concrete box culvert, which equations and chart are used?

Federal Highway Administration page: Equation 2 can be solved for H by the use of nomographs, Charts 8 through 14

Equation 2 sets the head; Charts 8 to 14 solve it p. 10

Page text: for tailwater at or above the top of the culvert, find headwater by Equation 3

High tailwater needs Equation 3 as well p. 17

Chart list with Chart 8, Head for Concrete Box Culverts Flowing Full, highlighted

Of those charts, only Chart 8 is concrete box p. 21

Answer: Equation 2 and Equation 3; Chart 8

Plain RAG on PDF text: 2 of 3

Equation 3 and Chart 8

Hybrid RAG: 1 of 3

Chart 8

Trace Fixes: 3 of 3

All three, across 11 calls

Local runs: Qwen3.5-9B, nvfp4, Ollama, earlier corpus. Validation question 16. HEC-5, Hydraulic Charts for the Selection of Highway Culverts, Federal Highway Administration.

FAQ

The harness is a prompt, tools and checks

<role>

Every fact must come from a line you read with a tool.

path | title | language | pages, for every file

Which sources exist and how each one words things.

<file_format>

A printed contents page number is not a line number.

Search terms in the document’s language. Read around every hit. Check every candidate.

<final_answer>

JSON with a quote for each part of the answer.

Tool output

“0 matches in all 98 files. This does not yet show absence. Try one distinctive word, a synonym, or the document’s language.”

Loop checks

Refuses repeated calls, checks each quote against the cited file, and sends early not_found answers back.

Final answer

A real tool: the model kept calling one we had not defined.

Prompt sections use XML tags, as Qwen’s guidance suggests. Full prompt: structured_agent.py

FAQ

Two maps show collection and contents

<catalogue> in the prompt, 98 files

vidore-finance_fr/christian_dior_2023.md | christian_dior_2023 | french | 8 pages

vidore-industrial/TO_00-25-172CL-5.md | TO_00-25-172CL-5 | english | 3 pages

xl-docbench/doc_000143.md | HYDRAULIC CHARTS FOR THE SELECTION OF HIGHWAY CULVERTS | english | 11 pages

<collection> in the prompt

cuad/ Commercial contracts. Governing law reads “governed by” or “laws of”.

vidore-*/ Reports on energy, finance, employment and industry. French reports need French terms.

xl-docbench/ Long reports, standards and books. A question often names the document by its title.

<file_format> in the prompt

A page number printed in a table of contents is not a line number.

outline(TO_00-25-172CL-5.md)

1: page 0

2: # TO 00-25-172CL-5

33: # 4 RELATED PUBLICATIONS.

45: page 1

73: page 2

76: # 6 HARDNESS CRITICAL ITEMS (HCI).

84: # 7 ELECTROSTATIC DISCHARGE SENSITIVE (ESDS) ITEMS.

92: # 8 IMPROVEMENT REPORTS.

read(pages=[2])

88: All ESDS parts shall be handled in accordance with the ESDS device handling procedures in TO 00-25-234.

Hosted run: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Validation question 5, first run, Grep Token Reduction. Outline shown as the tool returns it; this run reached page 2 by search. TO 00-25-172CL-5, p. 11. Excerpts; markup removed.

FAQ

Here are the two maps in full size, because this is where a domain prompt earns its keep. The catalogue comes first: every file with its title, language and page count. Look at the third row. The file name doc_000143 tells the model nothing, and the title tells it everything. Next, the collection section: what each source is and how it words things, such as French reports needing French terms. That is the part you can only write if you know your documents. On the right is the table of contents for one file, built by the outline tool from headings and page markers. The question asks about ESDS items. The outline shows that heading at line 84 on page 2, and line 88 on that page names the next document, TO 00-25-234. That is the first hop of the multi-hop question from earlier. Be honest about this trace: the model did not call outline here. It read the file, asked for pages that do not exist, and the hint sent it to a one-word search, which found line 88. It took 8 model calls in each of the three runs and got the right answer. A printed table of contents is also a trap, since its page numbers are not line numbers, and the prompt says so at the bottom left.

First-rewrite grep agent searches in French

Programmation pluriannuelle de l’énergie page: La totalité du gaz B est importée des Pays-Bas

GREP AGENT, WRONG, 6 CALLS

search("B gas") → 0 matches, 3 times

“The documents do not contain information about… B gas.”

FIRST-REWRITE GREP AGENT, CORRECT, 11 CALLS

search("gaz|fourniture|import…") → 16 lines

“La totalité du gaz B est importée des Pays-Bas”

Local run: Qwen3.5-9B, nvfp4, Ollama, earlier 222-page set, question 4. Source: Programmation pluriannuelle de l’énergie, ViDoRe v3, Etalab 2.0. Programmation pluriannuelle de l’énergie, p. 101

FAQ

Excerpts expose the clause on the same line

BEFORE: FIRST-REWRITE GREP AGENT

search("governed|law of|Governing Law")

13 [page 1]: - 17.9. DISPUTE RESOLUTION. Any dispute arising out of or relating to this Agreement or the breach, termination or validity thereof shall be settled in accordance with one of the following procedures. Contractor and NICE shall use the procedures in the following order of priority. 43 17.9.1. In the event of any dispute, controversy or claim of any kind or nature arising under or in connection with

answer: 5 of 6, NICE missed

AFTER: TRACE FIXES

search("governed by")

13 [page 1]: provided herein, and except that the provisions of Section 8.2. and all its subsections will apply with regard to details included in APPENDIX C. 17.11. LAW AND JURISDICTION. This Agreement shall be governed by and construed in accordance with the laws of the state of Israel, without giving effect to choice of law rules. 44 17.12. SCHEDULES AND APPENDICES. Each Schedule and Appendix hereto is incorporated h [excerpts of a 4248-character line]

answer: 6 of 6, NICE named

Tool

Before

After

First 400 characters, no warning

±200 characters around each match, at most 3, then a length marker

Read

First 600 characters, no warning

Whole line; past 6,000 characters a marker says how many are left

Local run: Qwen3.5-9B, nvfp4, Ollama, earlier 222-page set, question 6, first run of each. The model wrote each search itself, so the patterns differ.

FAQ

This is the same line of the same contract, line 13 of the NICE agreement, in two real runs. On the left the search matched, because that line does hold the clause, but the tool printed the first 400 characters of the line and stopped. Those characters are about dispute resolution. Nothing tells the model there are 3,800 more. So the model moves on, and the answer leaves out NICE. On the right the tool prints the text around the match, and the clause is there. It also ends with a marker that gives the length of the line, so the model knows it saw only a part. One caution: the two runs differ in more than this. The model wrote a different search each time, and Trace Fixes brought its other changes too, so this slide shows the mechanism and does not measure this fix alone. I did check one thing: the old pattern matches that line only at the clause, so the new tool would have shown the clause to the old search too. Whether the model would then have named NICE, we did not test. The next slide gives the exact rule. In the hosted runs on the 531 pages Grep Token Reduction, which keeps the same excerpt rule, gets this question right.

Search cuts a long line around each match

One line of the NICE contract, 4,248 characters. Blue goes to the model. Orange marks each match.

search("governed by")

1 match, 1 excerpt, 411 characters

search("court")

2 matches, 2 excerpts, 810 characters

search("agreement")

15 matches, 3 excerpts, 3,685 characters

Sends short lines whole

A line of 400 characters or fewer goes through uncut.

Finds the model’s pattern

The search the model wrote decides where the excerpts sit.

Keeps 200 characters each side

Excerpts that overlap merge into one.

Stops at 3 excerpts

Later matches drop out. A marker gives the line’s length.

No model call: matching_text in structured_tools.py run on line 13 of the earlier Markdown. Bars drawn to scale.

FAQ

This answers the obvious question: who decides what the model sees? The model does, through the search it writes. These bars are the same line with three searches. A precise phrase keeps 411 characters of excerpt. A rarer word keeps two short excerpts. A common word keeps most of the line, 3,685 characters, because the windows merge. The rule is plain code with no ranking: short lines go whole, long lines get 200 characters either side of each match, overlaps merge, and it stops at three separate excerpts. Be honest about the rough edge. On the agreement search the last match falls outside the three excerpts, that is the orange tick on bare background, and the marker gives the line length without saying a match was dropped. The model can still read the line, which returns all of it up to 6,000 characters, and a single search result also stops at 6,000 characters. If the pattern finds nothing in the line, the tool sends the first 400 characters.

The hint lives in the tool response

Tool response, role=tool

0 matches for 'ESDS.*technical order|TO.*ESDS' in file TO_00-25-172CL-5.md.

This does not yet show absence. Try one distinctive word, a synonym, or the wording of the document’s language.

No single line holds all of these terms. Lines holding each term alone:

ESDS: 4 matching lines in 1 files.

technical order|TO: 21 matching lines in 1 files.

Search one term at a time, then read the page that holds it.

Pseudocode

system = rules # fixed, sent every turn

result = search(pattern, file)

if result is empty:

result += HINT

if pattern joins terms with .*:

result += count_each_term()

messages.append(role=tool, result)

Next call in this run

search("ESDS")

88 [page 2]: All ESDS parts shall be handled…

Hosted run: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Validation question 5, third run, Grep Token Reduction. Output shortened.

FAQ

People ask where the hint goes, so here is the exact setup. The left is one real tool response from the fuel checklist question, third run. The white line is what any search tool returns. The amber text is what we add, and it goes in the tool message, not the system prompt. The rules that hold for every question stay in the system prompt: search in the document’s language, read around each hit. The hint holds facts about this one search, such as ESDS appearing on 4 lines, which nobody could write in advance. The next call was a search for ESDS, which found line 88. To measure it we removed the hint and the term counts, kept everything else, and ran the test set three times. As a median across those three runs of 20 questions, with the hint off and then on: empty searches 28 to 26, repeated searches 0 to 1, correct answers 9 to 8 of 20. Why it goes here: the hint says what empty means at the moment the model picks its next call. Now the honest part. It barely moves these searches, and it does not raise the score. Treat it as a cheap default worth testing on your own failures, not a proven gain.

Grep agent reads a page number as a line

FATF Recommendations contents page with the entry Table of Acronyms, page 140, highlighted

read(1) sees “Table of Acronyms|140”

read(140)

read(240) holds the evidence, unused
read(120) read(143) read(230)

search("Table of Acronyms") finds line 15
read(138) read(135) read(130)

read(138) × 7, until the budget ends

answer: not_found

Local run: Qwen3.5-9B, nvfp4, Ollama, earlier 222-page set, question 17, The FATF Recommendations, grep agent, run 2, 20 of 20 calls. Printed page 140 is not line 140 of a 264-line file. FATF Recommendations, p. 3

FAQ

Harness refuses a repeated call

Question: which official in the 2019 OCC Annual Report oversees “compliance and community affairs”?

read(lines 225-254)
line 230 names Morris R. Morgan
read(lines 165-194)
draft answer: Morris R. Morgan, then a review turn

read(lines 228-237)
Lines 228-237 are already above. Read other lines, search again, or answer.
read(lines 228-237)
You made this exact call at step 5; the result is unchanged. Change the terms or the lines.

harness, tools removed:
Stop searching. Give the final JSON now.
answer: Morris R. Morgan, correct

Skips the tool

The reply says why and what to try instead.

Spares the budget

A refused call uses none of the 20-call budget.

Stops after two turns

After two turns of refused calls, the tools go away and the model must answer.

Hosted run: Qwen3.5-9B, 4-bit (fp4), Darkbloom through OpenRouter. Validation question 13, first run, Trace Fixes. Question and output shortened.

FAQ

A refused call is a tool call the harness declines to run, because the model already has the result. There are two kinds: the exact same call again, or a read of lines already on screen. This one is a real trace from the 2019 OCC annual report. The model finds Morris R. Morgan on line 230 and drafts a correct answer. A review turn asks it to check, so it asks for lines 228 to 237, which sit inside what it already read. The harness says so and does not run the call. The model sends the same call again and is told that too. That makes two turns in a row, so the harness takes the tools away and asks for the answer, and the answer is correct. Before this stop, the same habit used all 20 calls and ended in not_found on a question from an earlier set. The refusal finds nothing new; it ends the loop. This is common: as a median across 3 runs, 15 of the 20 Trace Fixes test answers held a refused call, and 14 ended through this stop. Grep Token Reduction later removed the review turn that prompted the repeat here.

Every fix answers a failure in the traces

Grep agent failure in the trace

Change to the harness

Misses a title on letter case

Ignores case in search

Guesses file paths from titles

Lists paths and titles in the prompt

Reads a contents page number as a line

Adds an outline tool and a page number on every hit

Repeats a read until the budget ends

Refuses repeated calls and reads

Gives up after one or two phrasings

Returns early not_found answers to the model

Answers without a line to back it

Checks quotes against the cited file

Stops at a tool call Ollama cannot parse

Sends the model a correction, not 20 retries

Loses the end of long lines with no warning

Returns excerpts on search, whole lines on read

Prompt sections use XML tags. Tool calls use the native tools API, which renders Qwen’s own format. Sampling settings are unchanged.

FAQ

We measure three components, one at a time

Document processing

OCR: LightOnOCR-3 0.8b, for scans, charts and drawings

Other pages: pymupdf4llm

Merge: PyMuPDF restores lines OCR missed

Chunks: 512 tokens, 50 overlap

Embedding: qwen3-embedding:0.6b

Vector search: FAISS IndexFlatIP

Keyword: BM25, by bm25s

Hybrid: RRF, k = 60, top 5

Grep: ripgrep over every file

How we use search

Linear pipe aka plain RAG: retrieve once, answer once

Single fan-out: one batch of terms, one answer

Grep agent: grep and read tools in a loop

BM25 agent: BM25 search and read tools in a loop

FAQ

Search stack has five exact parts

Embedding

qwen3-embedding:0.6b in Ollama

Qwen/Qwen3-Embedding-0.6B, 1024 dimensions

Vector search

FAISS IndexFlatIP

Exact inner product, no approximation, top 5

Keyword

BM25, by bm25s

Over the same 512-token chunks

Hybrid

Reciprocal rank fusion (RRF)

k = 60, top 20 of each, keep 5

Grep

ripgrep

Every file, line by line, no index

SentenceSplitter builds 1,219 chunks. Voyager HNSW: 0.10 ms, 97% recall. Exact FAISS: 0.13 ms, 100%. Comparison

FAQ