Digest

Digest #3: a teardown of GPT-RAG and two corrections

New public work since Digest #2. Short notes here, the detail at the links.

Four of six test assertions contradicted the manifest, and 82% of one session's cache writes followed an idle gap.

A teardown of Microsoft's GPT-RAG. GPT-RAG is Microsoft's solution accelerator for agentic retrieval on Azure, and I read it for the contract it gives the coding agents working on the repository itself. The decision records are the part worth copying. Each one lists the events that would prove it wrong, and the template makes you write out the case for changing nothing. The gap was that no workflow ran the tests. At the commit I read, the tree held about 300 KB of them, and four of the six assertions in the test guarding the release versions contradicted the manifest beside it. That test was brought back in line on 21 September and is now updated by hand at each release, but still no workflow runs it.

Two teardowns corrected. Before posting the herdr teardown to Hacker News, I re-read herdr's whole repository at the pinned commit, and the page had claims wrong. It credited herdr's maintainers with a design discussion none of them joined, and it said nobody had shipped the accounting layer when community plugins had already built per-pane cost tracking and a run ledger. The 12-Factor Agents teardown said the guide ships nothing reusable, and the repository ships a work-in-progress scaffolder. Both pages now carry dated corrections at the foot. Every teardown now gets a search of the whole repository before it calls anything absent, and a check of who actually posted before it credits a project with anything.

The five teardowns side by side. The teardown index now has a matrix of the five subjects against the eighteen patterns. It's generated from each page's header and checked in CI, so it can't drift from the pages. No pattern is present in all five, and the dead man's switch, the cheapest control in the set, is among the most often missing.

Two additions to the architecture. Pattern 18 gained a keep-alive for the prompt cache. It used to say wrap up and clear the session before any break longer than the cache lasts. Measured on two more sessions, resuming after an idle gap accounted for 44% and 82% of the tokens spent writing the cache. On a model that prices cache reads at 0.025x of input, a refresh costs about 1/80th of a re-write, so the sample orient skill now arms a keep-alive when a session starts. META_ARCHITECTURE.md also documents the new overnight skill, a curated queue of no-input work that runs unattended and pauses near the usage cap until the reset.

Three additions to awesome-agent-workspaces. YYLO runs each coding-agent task in its own branch and worktree, behind typed boundaries for validation, merge and release. YYLO Benchmark evaluates agent runs in fresh private repositories, keeps the evidence immutable and judges each step blind. MulmoTerminal is a browser grid of live Claude Code and Codex sessions, and for Claude Code it reads the CLI's own hooks to show "needs you" separately from "done".

Two new essays. Stop asking the AI to be careful makes the case for mechanical guards over instructions, from a floor of denied actions to an interlock on installs. You can't tell whether your AI setup got better is about why a single run comparing two setups misleads, and the fix any team can copy, a suite of its own real failures run five times each.

Same invitation as before. If a teardown misread something, reply, and the correction goes on the page with a date.

Get new essays by email on Substack. This one is on Substack too.

← All writing