How it's built · v1 · naive and rerank measured September 2026 · the other five are still predictions
Seven retrieval patterns, one corpus that happens to be my own bookshelf, nine fixed questions with reference answers written before any pipeline runs, the same generation model everywhere. This page is the lab notebook: what I decided, what I turned down, and what replaces prediction with measurement as each version ships.
I build the way I learn: make the thing, then find out where it breaks, and publish the finding-out. Version 0 is the design and the predicted traces. The measurements start at v1. Every claim on the comparison page that is currently a prediction is labelled as one, and gets a date here when it becomes a measurement.
Where this stands, September 2026. The ingest described below has run, and its numbers are real. The first two patterns — naive and retrieve-and-rerank — are built and wired behind the same interface, and the ledger's SQL path answers live on the comparison page. The measurement run has not been made yet, so every trace in the explorer is still a prediction and still says so. Nothing here is dated as measured until it has been measured and graded.
The seven-pattern RAG picture is everywhere in 2026, and it is right that these are different designs. What it does not give you is a way to choose. Every pattern is shown on its own demo, so any comparison is really a comparison of demos: different data, different questions, different models, different definitions of "worked".
The question I wanted to answer is the one a team asks before they build: which of these do we actually need, and how would we know? Answering that means holding everything still except the pattern.
The corpus is my reading life: what I read and when, what I rated it, the lines I highlighted, who told me to read it. Not because the world needs another chat-with-your-highlights app, but because evaluation is the hard part of RAG and I can only grade answers in a domain I know cold. When a pipeline says I read a book in 2023 that I read in 2021, or quotes a line back from the wrong book, or recommends something already on my read shelf, I know in one second. In an enterprise-docs demo I would have had to guess, or pay a model to guess for me.
The second reason is that the data has, naturally, the properties that separate the seven patterns. I did not have to invent them; I had to make sure the export kept them.
| property | where it comes from | which pattern it separates |
|---|---|---|
| Numbers in a ledger, not in text | dates read, ratings, page counts in the Goodreads export | patterns with a tool from patterns without |
| Sources that disagree | my review, my own highlights from the same book, and the crowd's average rating | naive from reranked |
| Knowledge in relationships | a highlight in one book naming the author of another; who recommended what | vector from graph |
| Answers only in images | photographed pages and covers | text from multimodal |
| Questions needing several sources | a reading list built from shelf, page counts, recommendations and abandoned books | single-path from multi-agent |
The earlier draft of this site used a handstand-coaching notebook I would have had to write from scratch. Real, exported data won on every count: it took an afternoon instead of a week, it cannot be accused of being built to flatter a pattern, and its failures are legible to anyone who has ever kept a reading list.
| decision | chose | instead of | why |
|---|---|---|---|
| Corpus | My own exports, curated | A fabricated corpus, or scraped reviews | I know every fact in it, and it is real. Cost: it is small, and small corpora flatter naive RAG; the question set compensates. |
| Full text | Highlights only | Whole books | A short quotation with attribution can be published; a book cannot. Highlights are also the more personal signal. |
| Recommenders | Initials, from a hand-written CSV | Full names, or nothing | The people edges make the graph interesting. The people themselves stay off a public page. |
| The ledger | Kept as rows in SQLite, queried by a tool | Embedding every row as text | Embedding a ledger is how a system invents a book. Keeping it as rows is what makes the Q2 column an honest test of "do you have a tool or not". |
| Question set | 9 fixed questions, one per failure mode (six planned, three I added from my own shelf) | 50 random questions | Each question is a diagnostic: it should turn one pattern from wrong to right. Random questions average that out. More questions come at v5, once the pipelines run. |
| v0 as predictions | Predicted traces, labelled as such | Waiting for live pipelines | The design thinking is the product; the numbers are the proof. Shipping the thinking first, honestly labelled, beats shipping nothing. Cost: a reader who skips the label will think these are results. |
| Graph construction | LLM extraction with a fixed schema (Book, Author, Person, Theme; WROTE, RECOMMENDED, ABOUT, MENTIONS, ARGUES_WITH, READ_AFTER) | A hand-built graph | A hand-built graph hides the real failure mode of Graph RAG, which is the extractor missing an edge. I want that failure to be possible, visible, and published as JSON. |
| Reranker | A local cross-encoder | An LLM reranking call | Cheaper per query, and a second LLM in the loop makes it harder to attribute a fix to retrieval rather than to more model. |
| Generation model | One model, one prompt template, every pattern | The "natural" model for each pattern | Change one thing at a time. If the model varies, the comparison is of models. |
| Grading | Hand labels against a written reference, plus an automatic invented-book count | An LLM judge | 6 questions × 7 patterns × 5 runs is 315 answers. I can read 315 answers. A judge needs its own validation set, which at this size costs more than reading. |
| Hosting | Static pages now; a small Python service for the pipelines | A full app from day one | Static pages deploy in seconds and cannot go down. The service only needs to exist for the eval runs and, at v5, the demo box. |
| source | what is used | what is never used |
|---|---|---|
| Goodreads export | title, author, ISBN, pages, publication year, date read, date added, my rating, average rating, shelves, my review, read count | private notes, and any title I remove before ingest |
| Kindle highlights | the highlight text, its book, location, and date; my notes on it | bookmarks; clippings under 20 characters |
| Recommendations | title, who (initials), where, when. Written by hand. | the mapping from initials to people, which stays on my machine |
| Photos | covers from Open Library, a few pages I photograph, one shelf | |
| Extracted graph | edges from the files (who wrote what, who recommended what, what I read after what) and edges a model extracts from highlights and reviews (themes; which books mention or argue with which) | nothing; the whole graph is committed as JSON |
The highlights are the semantic index and the ledger is a table. That split is the single most important design decision on this site: it is what makes the "numbers" question an honest test rather than a trick.
| number | what it is |
|---|---|
| 479 | books in the Goodreads export. 205 read, 250 to read, 24 in progress; 372 have an ISBN. |
| 132 | reviews in my Notion "Book Reviews" database, every one of them matched to a Goodreads row — 129 on title alone, one on title and author, two through typos of mine, two by alias. Goodreads holds what and when; the ratings and the written verdicts are mine, kept in Notion. |
| 125 | books I have rated. I never rated anything on Goodreads, so a "five stars" question would have returned nothing without this. |
| 369 | of my own genre tags across 102 books. They are genres more than themes, and they stop after 2022. Both facts change what a correct answer to Q8 and Q9 looks like. |
| 273 | highlighted passages, 256 of them carrying a real Kindle location. See below: they arrived from somewhere I did not expect. |
| 397 | books with a crowd average from Open Library — 311 matched by ISBN, 86 by a title I checked agrees. 82 have none, and are counted as having none rather than guessed at. Open Library's crowd is small: the most-rated book here has 759 ratings, not Goodreads' millions. |
| 775 | chunks in the vector index, over 55 books: every highlight, and my notes split into passages. No ledger row is in it, and only the passage text is embedded — the book and author ride along as metadata. |
The plan said the highlights would come from My Clippings.txt off the Kindle. That file still has not been plugged in. What I found instead, while writing the ingest, was that ten of my Notion pages already held highlights I had pasted there years ago — five of them raw Kindle clipping dumps, separators, locations, timestamps and all. Splitting them out gave the index 273 passages and gave Q1, the control question, a real target a whole version early.
It also found a mistake. Those pasted dumps were sitting inside the field I was calling "my notes": 236,000 of the 434,000 characters of what the ingest thought were my own words were actually verbatim book text. That is the field the pages print. Splitting highlights from notes properly took the notes down to 243,000 characters of my writing, and moved the book's words into the highlights table, where the rule is short excerpts with attribution and never the passage. It is the kind of error that would have been invisible until the day the site quoted three pages of someone's book back at a reader.
The curated exports themselves stay out of the repo; the photos, the question set and every result CSV go in.
| question | tests | expected to separate | |
|---|---|---|---|
| Q1 | Where did I read the line about …? | a quotation in the index | nothing. A control, unless a same-author highlight from another book outranks it |
| Q2 | What did I read in 2024, rated five stars, under 300 pages? | a filter over the ledger | patterns with a tool (right) from patterns without (invented titles) |
| Q3 | Why did I give The Fellowship of the Ring two stars when everyone else loved it? | my notes vs any admiring passage vs the crowd's 4.34 | naive (wrong) from reranked (right) |
| Q4 | Which of my books argue with each other? | a relationship between two books | vector (partial) from graph and hybrid (right) |
| Q5 | [photo] Which book is this, and what did I highlight near it? | an answer that starts from pixels | text-only (partial) from multimodal (right) |
| Q6 | A two-week holiday list from my to-read shelf, under 400 pages, one from a friend, nothing like the last three I abandoned | composing four sources into a deliverable | every single-path pattern (partial) from multi-agent (right) |
| Q7 | Give me my favourite books I have read about race | a rating filter joined to a theme | text-only (right theme, wrong "favourite") from anything that can see the rating column |
| Q8 | What are the themes I like most? | an aggregate over the whole shelf | patterns that can count (a query, a graph) from patterns that see four chunks |
| Q9 | What were my top themes each year? | an aggregate grouped by time, on tags that stop after 2022 | patterns that can say "unknown" for a year from patterns that answer confidently from nothing |
Reference answers are written before I trace a single pipeline and are shown on the comparison page next to each question. Where a reference can be computed from the ledger (Q2, Q7, Q8, Q9) it is, by a query that is published with it. Q2 and Q3 name real books; Q1 waits for the Kindle file. Three questions (Q7 to Q9) were added after the first six, because they were what I actually wanted to ask my shelf; each brings a failure mode the first six lacked. A second reader grades a sample before v5; the agreement rate gets published as the ceiling, as I did for fixmybanana.
Every pattern on this page is given the same system prompt, the same context format, the same model and the same settings. Only the chunks differ. That is the whole basis of the comparison, so here it is, verbatim:
It was one sentence longer until the first pilot run. The removed line told the model that the context held no ratings, dates or page counts — which is true, but it is also most of the answer to the ledger questions, and a prompt that names the missing field is measuring itself rather than the pattern. It came out, the run was done again, and the patterns still refused rather than inventing. That refusal is theirs, not the prompt's.
| measure | how | why it matters |
|---|---|---|
| Verdict | correct / partial / wrong against the reference, by hand | the headline cell in the grid |
| Invented books | any title in the answer that is not in the library, counted by code | the failure a reader can feel; the check that catches what I miss |
| Retrieval hit | did the chunk or row holding the reference answer reach the model? | separates "retrieval failed" from "the model ignored it" |
| Consistency | same verdict on all 5 runs? | a pattern that is right 3 times in 5 is not right |
| Model calls | counted by the code | structural cost, independent of vendor pricing |
| Latency and $ | wall clock and token usage per run | the columns that decide what ships |
The eval calls the same function the site's demo box will call, so what is measured is what a visitor gets. The question set does not change between versions, so the only thing that moves the grid is the change I made.
Eighty answers came to $1.06 — about nine tenths of a cent each, against the $4–5 I estimated from the list price, because the model thinks less than I assumed on questions it decides it cannot answer. Retrieval was free: the whole index is around 100,000 tokens and Voyage's current generation carries 200 million free.
Latency is published in pieces, because the whole is not honest. Generation ran 2.8–11.6 s, median 5.0 s. The reranker took 293–465 ms, median 361 ms, on the thirteen of forty calls that were not queued. The other twenty-seven waited: an account with no payment method gets three Voyage requests a minute, and one cell sat 58 seconds for a call that takes a third of a second. That waiting is a billing detail, not a property of retrieve-and-rerank, so it is reported separately and never as latency.
Predicted traces. Corpus chosen and its rules written, nine questions fixed (six at first, three added the next day from my own shelf), seven pipelines traced by prediction on all six, the scorecard and decision guide published. No code has run. Every score on the comparison page is a design argument, not a measurement.
Ingest, naive, reranked and the ledger tool, measured. 479 books and 273 highlights ingested, 775 chunks embedded with voyage-4, the cross-encoder and the SQL tool built. Eight of the nine questions run through both patterns, five runs each: 80 answers, $1.06, all of them in eval/results, graded by hand. Q5 is not among them — it needs a photograph and this pipeline takes text. Three things the run changed: no answer invented a book, in any run; naive on Q3 never retrieved the book at all, because only passage text is embedded; and reranking bought margin and better chunks without changing a single verdict. The first two rows of the grid are measurements now, and the ledger route answers Q2 live, in milliseconds, with no model call.
Graph and hybrid. LLM extraction over highlights and reviews with the fixed schema, then traversal. Publish the extracted graph so a missed edge is visible. The Q4 column becomes real.
Multimodal. Multimodal embeddings for the photos, vision-language generation. The Q5 column becomes real.
Router and multi-agent. A small routing model over the five retrievers; then a planner, sub-agents and a synthesizer. The Q6 column, and the cost column, become real.
The numbers. Full grid with latency and cost per cell, consistency across runs, invented-book counts, a second reader's agreement rate, ten more questions drawn from how people actually ask, and a live "ask the librarian" box with the trace panel. The "predicted" label comes off the comparison page.
| part | choice | note |
|---|---|---|
| These pages | Plain HTML, inline CSS, no build step | Cloudflare Pages, deploys on push. Same as my other sites. |
| Pipelines (v1+) | Python service, five retrievers, seven patterns behind one interface: answer(question, pattern, image) → {answer, context, route, calls, ms, usage} | the eval imports the same function the demo box calls |
| Ledger | SQLite, queried by the SQL tool | a few hundred rows; never embedded |
| Embeddings | Voyage voyage-4; the multimodal model for v3; vectors in a SQLite file beside the ledger | one file, no vector database to run. The whole index is about 100,000 tokens, which is inside Voyage's free allowance, so retrieval costs nothing. |
| Reranker | Voyage rerank-3, a cross-encoder | the plan said a local cross-encoder; a remote one is the same kind of model without a two-gigabyte dependency in the deploy. Still not an LLM: a second LLM in the loop would muddy which model earned the answer. |
| Graph | NetworkX in memory, extracted by Claude with a fixed schema, committed as JSON | a graph database would be a fourth thing to run for a graph with under a thousand nodes |
| Generation | Claude Opus 5 with adaptive thinking, one prompt template for every pattern | the router uses Claude Haiku 4.5: it is a classifier and should cost like one |
| Hosting the service | Railway, Cloudflare in front | the setup I already run for fixmybanana |