Rank 1 · Library hours · January 2026
obsoleteWords 1–18 · score 0.943 · matched: saturday, library, opening, hours
Cedar Library Saturday hours Saturday opening Saturday timetable. In January 2026 the library opens on Saturday at nine
Learn machine learning · retrieval evaluation
A close match.
Enough evidence?
A search result can repeat the question’s words and still miss the answer. Before adding retrieved passages to a model’s context, check whether they contain the evidence the task needs.
The context windows and retrieval session introduces retrieving passages instead of sending everything. Here you evaluate the retrieval step itself, using a small answer key you can inspect.
01 · define the test before the score
Which words must arrive?
A chunk is a window of words cut from a document. Overlap repeats some words in adjacent windows. Top-k is the maximum number of ranked chunks returned. Smaller chunks may split a useful sentence; larger chunks carry more surrounding text. More returned chunks also cost more context.
Predict what will happen when you raise top-k from one to six. Will the required evidence arrive? Will an obsolete timetable arrive too? Then try smaller chunks with and without overlap.
Live · lexical retrieval on fictional documents
No embeddings or language model run here. The score counts distinct matching query terms, divided by the square root of chunk length. Common words such as “the” are removed. Ties keep source order. Dates and source authority do not affect ranking.
- 1 · query
Choose a question
- 2 · chunks
14 source windows
- 3 · rank
3 positive matches selected
- 4 · evidence
0 / 1 required spans complete
What are the current Saturday library opening hours?
Required-span hit rate · live
0%
Evidence word coverage · live
0%
Context cost · live
39 words
Obsolete source retrieved. Both versions of the hours document are present; their Saturday hours conflict. Inspect the source text and dates.
Ranked context
Selected chunks are the context sent onward in this experiment. Word positions below are one-based and inclusive in each source document.
Rank 2 · Library hours · March 2026
currentWords 43–48 · score 0.816 · matched: opening, hours
or borrow books during opening hours.
Rank 3 · Library hours · January 2026
obsoleteWords 29–43 · score 0.516 · matched: saturday, opening
March 2026. The old Saturday opening notice remains here to demonstrate an obsolete search match.
All positive matches and ranks
| Rank | Source span | Score | Selected |
|---|---|---|---|
| 1 | hours-old · 1–18 | 0.943 | Yes |
| 2 | hours-new · 43–48 | 0.816 | Yes |
| 3 | hours-old · 29–43 | 0.516 | Yes |
| 4 | rooms · 29–43 | 0.516 | No |
| 5 | hours-new · 15–32 | 0.471 | No |
| 6 | hours-new · 29–46 | 0.471 | No |
| 7 | hours-new · 1–18 | 0.236 | No |
| 8 | hours-old · 15–32 | 0.236 | No |
| 9 | borrowing · 1–18 | 0.236 | No |
| 10 | rooms · 1–18 | 0.236 | No |
Check against the answer key
A hit requires every word of an authored evidence span, possibly across multiple chunks. Coverage counts unique required words retrieved. Context cost counts every returned word, including overlap; it is not a model token count or a money estimate.
- hours-new · required words 19–32: 0/14 retrieved · incomplete
Run the whole question set
Live 2/3 required spans retrieved across the two answerable questions. The parking question stays separate.
| Question | Hits | Coverage | Context words |
|---|---|---|---|
| hours | 0/1 | 0% | 39 |
| borrow | 2/2 | 100% | 42 |
| parking | N/A | Not applicable | 51 |
Read all four source documents · fictional corpus
Library hours · March 2026 current
Cedar Library serves the fictional town of Fern. The building has a reading room and a public desk. From March 2026 the library opens on Saturday at ten and closes at four. These Saturday hours replace the January timetable. Visitors may read or borrow books during opening hours.
Library hours · January 2026 obsolete
Cedar Library Saturday hours Saturday opening Saturday timetable. In January 2026 the library opens on Saturday at nine and closes at five. This archived timetable was replaced in March 2026. The old Saturday opening notice remains here to demonstrate an obsolete search match.
Borrowing policy · March 2026 current
Cedar Library welcomes readers from Fern. A library card is required to borrow books. Members can borrow up to six books for fourteen days. Bring the books back to the public desk. Renewals are available once unless another reader has reserved a book. The policy applies to ordinary printed books.
Reading room guide · March 2026 current
Cedar Library has a quiet reading room beside the public desk. Readers may bring notebooks and pencils. The reading room has twelve seats. Room bookings are not offered. The guide contains no information about parking prices, parking permits or car park opening hours.
02 · separate retrieval from answers
Coverage is only one check
Complete evidence coverage says that the labelled source words arrived. It does not prove that a later answer uses them correctly, resolves the old timetable, or declines the parking question. Those require separate answer checks. This experiment deliberately does not generate an answer.
These three authored questions illustrate failure cases; they do not estimate production performance. For your own system, collect representative questions and evidence labels, include missing and conflicting sources, and compare settings on questions held apart from tuning.
For the evaluation method, see Manning, Raghavan and Schütze’s Introduction to Information Retrieval: system evaluation. The required-span metric above is defined for this lesson, rather than a standard benchmark score.