How the RFP engine was built
Five days, 327 commits, and 79 mistakes written down by the person who made them. The tool that came out answers questions about a solicitation with the clause behind each answer. The method that built it is the more portable thing.
The RFP engine takes a solicitation and its amendments and answers plain questions about them. When was the deadline moved to? What is the deposit now? Can we call the grid operator about the bid? Each answer points to the clause it came from, or says plainly that the document does not cover it.
It enforces six rules in code: it shows its source, numbers match the text exactly, the latest amendment wins, "not covered" is a real answer, it reports facts and never makes bid decisions, and without a model it returns the clauses with no summary. It is standard-library Python and runs locally as an MCP server. That is the product. This note is about the loops that produced it, because they compound and the product does not.
Before a test existed, its expected result went into the flight log with a confidence number. Then the test ran, and the prediction was scored. The order matters. A prediction written after the result is a story; one written before is a measurement of your own judgment.
The first big one was a 30,000-run deterministic fuzz of the amendment resolver. I predicted, at 70%, that it would find at least one new seam the 18 golden questions had missed. It found exactly one, and it was in my test harness, not the system. The resolver held on 30,000 of 30,000 runs. Mean Brier score across the three predictions: 0.172. The lesson was specific: the deterministic layer was sturdier than I believed, and my test code was the weaker link.
A later one went the other way. Testing against real-world addendum language, I put 85% on zero clauses being silently misclassified. One was: a revision read as "no change," and the unresolved clause never reached the answer. Brier 0.72 on that call. Both defects were fixed the same day, and the scored miss stays in the log next to the prediction.
A confidence number you never score is decoration. Scored, it tells you which part of your own head to trust less.
Every error got a row in a ledger: what broke, how it was found, whose fault it was (tool, test, premise, design, or me), the repair, and the heading change. Seventy-nine rows by the end. The rule that made the ledger compound instead of merely confess: no row closes until a check exists that fails on the old behavior and passes on the new.
--check runs in pre-commit. A planted stale number fails.
SeamThe leak scan skipped the shipped history bundle, where 27 internal references still lived.
GateThe packager now scans every commit message and diff that ships, and failed on the old history before it passed on the clean one.
SeamThe local test server trusted any request; any open web page could have triggered paid model calls.
GateHost and Origin checks, a body cap, paid calls off by default. Nine security tests: eight fail on the old server.
SeamAn answer about contacting the operator cited the deposit amendment it never used.
GateCitations limited to clauses the answer actually relies on, with a regression case on the live question.
The ledger is a record of mistakes, and every check in the suite traces back to one of them.
The drifting $0.29 taught the widest lesson. Any figure a person types is stale the moment its source moves. So nothing shown is typed. Cost per question, eval scores, error counts and the timeline map are all rendered from the flight log and the eval runs, with the formula next to the number. When the record changes, the page changes, and a gate refuses the commit if they disagree.
That turns the write-up into evidence of how the work was done. A reader can trace a claim to the run that produced it, and so can the next session of me, without trusting my memory of the week.
The golden set started as 14 questions plus 4 planted traps: a stale deadline unless the amendment is applied, a topic the document never addresses, a request to do something the RFP prohibits, a cap that "may not be revised upward." Every fix that touched answering code had to pass it, and a mutation check confirmed the tests would notice if a guard were removed. Eight injected model faults are caught; five deterministic checks pass identically through MCP and direct calls.
Then the honest part. A real model run on 24 questions on 2026-10-30: 11 correct, 12 withheld, 1 wrong. The target was zero wrong, and it was missed. That result is printed on the tool's own page, not here only. Withholding twelve is the gate working; the one wrong answer is the next row in the ledger.
The loop does not make you right. It makes each wrong cheaper than the last, and it keeps the wrong ones on the page where you have to look at them.
Each loop feeds the others. A scored prediction finds a seam. The seam becomes a check. The check feeds the eval. The eval produces numbers that render into the record, and the record is what the next prediction is written against. By day five, a new change met dozens of checks, each one a mistake that had already happened once.
Still not proven, and marked as such: there is no confidence score, retrieval is keyword-only, figures are checked but meaning is not, and the parser expects one numbering style. Those are in the ledger too, as open rows.