A LIVING ACCOUNT · EXTENDED EVERY BUILD DAY

I'm building an AI
platform that writes code,
reviews it, and runs it

BELLWETHER is an AI layer for an engineering team. Agents that generate code and tests, review pull requests against written standards, and diagnose production incidents from real logs and metrics — all grounded in this team's own decisions rather than generic training data, and every one of them carrying a published score.

An AI layer is only worth what you can test it against. So I am also building the thing it operates on: a small ads platform — four services that decide which advert to serve, record what got delivered, and break in ways I can cause on demand. BELLWETHER is the AI. The ads platform is what it operates. Ad serving is the team's domain, and its real outages are configuration mistakes — the kind that leave an actual cause in an actual database for an agent to find.

I'm building it in thirty days and writing it down as I go: the method I follow each day, every tool and why I picked it over the alternative, the architecture at each stage, every decision and the condition that would reverse it, and each time reality proved me wrong.

This page currently covers Day 0 through Day 09. I write nothing with hindsight — I record the mistakes on the day I make them, and I leave them on the page.

The thirty-day arcDay 09 · Level 1 in progress
09 / 30days written
1 / 6levels complete
5services running
448tests passing
10decisions recorded
11/11level 0 gate

Reel one

Setting the stage

Before a single line of the product exists, three things have to be settled: what problem this is actually solving, how the work will be done every single day, and what it will be built with. Everything afterwards is consequences.

Day 00 · The problem

Where this started

Netflix has been the company I wanted to work for since well before I could have explained what a distributed system was. Not because of the product — because of how openly they think in public. Chaos engineering, the microservice patterns half the industry now copies, the causal-inference work, the agentic tooling they have started writing about: most of it was published while it was still being figured out, with the trade-offs left in rather than tidied away.

That is rare. Most companies publish the polished version, years later, once nothing is at stake. Reading Netflix work through a problem in public taught me more than most of the courses I have paid for.

So: dream company. Which is a pleasant sentence and completely useless on its own.

Wanting to work somewhere is not a plan

What I actually needed was a way to build something real rather than something aspirational. Which meant answering a question I had never thought to ask properly: how do you find out what a company is about to build, before they build it?

The answer turned out to be sitting in the open. A company's engineering blog and its job listings are the two most honest public statements it makes about its own direction.

  • The blog tells you what they have already solved — and, more usefully, how they reason. What they measure. What they will accept as evidence.
  • The listings tell you what they are staffing right now — which is to say, what somebody has already decided is important enough to spend headcount on, before a line of it ships.

So I read both, properly, for weeks. The blog gave me the house style. The listings gave me the target.

One posting stopped me: requisition AJRT30201 — Staff AI Engineer, AI Foundation & Tooling, Ads Platform. I started pulling it apart, and the more I dug the more it read less like a wish list and more like a roadmap somebody had already committed to. Strip out the boilerplate and it describes one specific, current problem:

Engineering teams can now produce code far faster than they can verify it.

That sentence is easy to nod along to and easy to underestimate, so let me show you what it actually means.

What "plausible but wrong" actually looks like

Here is a function from my own domain code. An AI could write it in a second, I could approve it in five, and it is wrong.

# Has this campaign spent its daily budget?
def within_budget(campaign, spent_micros):
    return spent_micros < campaign.daily_budget_micros

Read it again. It compiles. It passes a strict type check. It reads exactly like the comment above it. A test asserting "a campaign that has spent nothing is within budget" passes. So does "a campaign that has spent everything is not."

It is still wrong: it asks whether the campaign has already overspent, not whether serving this advert would cause it to overspend.

daily_budget   = 10,000,000   # $10.00
already_spent  =  9,999,000   # $9.999
this_ad_costs  =    500,000   # $0.50

within_budget(...)  → True     # because 9,999,000 < 10,000,000
                             # so it serves, and the day ends at 10,499,000

→ every campaign overspends its daily cap, every day, forever.

The correct version is one clause longer:

return spent_micros + price_micros <= campaign.daily_budget_micros

Nobody is ever paged for this. There is no crash, no error rate, no red panel on a dashboard. Advertisers are quietly billed slightly more than they agreed to, for months. It is found by an auditor, not by an alert.

Why this class of bug is the whole problem

Code that is obviously broken is cheap. It fails loudly and immediately, and someone fixes it before lunch.

Code that is plausibly broken is expensive. It survives review because it reads correctly. It survives testing because the tests were written from the same misunderstanding that produced it. And AI generates it at a rate no review process was designed to absorb.

Speed went up. Confidence did not. That gap is what the posting is actually about.

The five capabilities the posting asks for

They are not independent. Each depends on the one before it — which is why the build order in this project is not negotiable.

#CapabilityWhat it means concretely
1Centralised context layerAgents answer from this team's code and decisions, not generic training data. Ask why the project uses Compose and it should quote ADR-0001 and name its trigger — not recite blog posts.
2Dev lifecycle agentsGenerate code, generate tests, review pull requests against written standards, validate a deployment after it lands.
3Ops agentsRead logs and metrics during an incident, correlate across services, propose a root cause with cited evidence, suggest the remediation.
4Multi-agent orchestrationSeveral agents on one task at once — coordinating, disagreeing, and stopping to ask a human at the moments that matter.
5Evaluation frameworksProving any of the above works. Numbers, not impressions.

Item five is where nearly every portfolio project quietly gives up — and it is the only one that makes the other four trustworthy.

A demo and an evaluation are not the same thing

The words get used interchangeably. They should not be.

A demoAn evaluation
One input, chosen by the person demoingA fixed dataset, written before the agent existed
Run until it looks good, then recordedRun once per change, results kept whatever they are
Success is "the audience was impressed"Success is a number with a target beside it
Failures are re-shotFailures are published
Proves the example was friendlyProves something about the agent

Concretely, for the pull-request reviewer arriving on Day 13, the evaluation is this:

# Take working code. Plant 10 realistic bugs, one per branch.
# Each is the kind a tired human writes at 5pm on a Friday.

  1. off-by-one in a frequency cap         # < where it should be <=
  2. budget check ignores the pending ad   # the bug shown above
  3. missing guard when targeting is empty
  ...

# Ask the reviewer agent to review each branch, cold, with no hint
# that a bug exists. Score = caught, minus false alarms invented.

catch rate: 7/10 — published, including the run that scored 4

That number can go down when the code changes. That is precisely why it is worth having.

The bet I'm making
Every capability ships with a number

Not "here is an agent that reviews pull requests."

Instead: "here is an agent that caught 7 of 10 deliberately planted bugs, here is the evaluation that measured it, here is the run where it only scored 4, and here is what changed after."

Harder to build, far harder to fake, and the only version of this project a Staff engineer would find interesting rather than familiar.

Why I'm building it in public, with the failures left in

The honest version: I want this job, and I know what a portfolio project usually looks like from the other side of the desk. A repository with a confident README, a demo video where everything works first time, and no way for a reviewer to tell whether any of it survives contact with reality.

I did not want to send that. Anyone can claim they build agentic systems. Almost nobody publishes the number that would prove it, because the number is often disappointing.

So I am building the version that can be checked. In the open, one day at a time, with the mistakes left where they happened. Every figure on this page came from a command I actually ran. Every amber box is something I got wrong and then wrote down — including the day I broke a rule I had written myself, four days earlier, about that exact mistake.

If the agents turn out worse than I hoped, that will be on this page too. It is the only thing that makes the good numbers worth anything.

The call I had to make before Day 1 existed

With a plan sketched out, one genuinely hard decision was left — and getting it wrong would have cost me the entire month.

The job is the AI layer. It is not ad serving. Nobody is hiring me to write a frequency capper. So which do I build first?

Start with the AI layer — rejected Start with the ads platform — chosen ✓
Day 1 is on the actual subject of the jobDay 1 is on something nobody is hiring me for
Something impressive to show by the end of week oneFive days — a sixth of the entire budget — before the AI appears at all
Agents have nothing real to operate on, so their inputs must be inventedAgents operate on a system that genuinely runs, genuinely logs, and genuinely breaks
Every evaluation scores the agent against a fixture I wrote myselfEvery evaluation scores the agent against something that behaves independently of me

The top two rows are about convenience — what is quicker, what looks good sooner. The bottom two are about whether any of it means anything, and that is where the argument actually gets settled.

I chose the ads platform.

Play the other choice forward and picture Day 20. I am recording a video where an ops agent diagnoses an outage. Where did the outage come from? A JSON file of log lines I wrote by hand, describing a failure I invented, in a system that was never running. The agent finds the cause — because I put it there, in the shape I expected it to be found in.

What has that measured? My ability to write a fixture. Any viewer with production experience sees straight through it, and worse, I would never learn whether the thing works.

So I paid five days up front. It cost me a week of build videos in which the AI barely appears, and it is the reason every number later on this page is worth reading.

Which is to say: to show an agent an outage, you need an outage

Most demos fake it. In code, the fake usually looks about like this:

# The staged incident
if os.environ.get("CHAOS") == "true":
    return {"error": "database unavailable"}

Now point an agent at that and ask for a root cause. What is there to find? The only true answer is "an environment variable is set." The agent has learned nothing about databases, nothing about this system, and nothing that transfers to a real incident.

An agent tested against a fake failure can only ever find the fake. The evaluation measures the fixture, not the agent.

Compare what I do on Day 5. My simulator sends genuine PATCH requests rewriting every campaign's targeting to a country with no viewers. The decision service reads that configuration and correctly declines to serve. Fill rate drops from 43% to 2%.

Every part of that is real. The cause is a row in a table an agent can query. The symptom is a metric an agent can chart. The fix — restore the previous targeting — is a real operation with real consequences. That is something worth evaluating an agent against.

So the project is two systems, and the unglamorous one goes first

One system is the point. The other exists only so the first has something real to work on.

The two systems, and the one arrow that matters
flowchart LR
    SUB["SUBSTRATE<br/>a small ads platform<br/>•<br/>4 services · real traffic<br/>real logs · real incidents"]
    BW["BELLWETHER<br/>the AI layer — the point<br/>•<br/>context · dev agents · ops agents<br/>orchestrator · evals"]
    SUB ==>|"its code, logs, metrics<br/>and real failures"| BW
    BW -.->|"operates on"| SUB
    classDef sub fill:#241416,stroke:#b8321f,stroke-width:1px,color:#f3d9d6;
    classDef ai fill:#12211a,stroke:#2f6f4f,stroke-width:1px,color:#d6ece0;
    class SUB sub;
    class BW ai;
        
The unglamorous half gets built first (Days 1–5), because an agent can only be honestly tested against something that genuinely runs. The full internal architecture — every service, the data between them, and one request traced end to end — has its own chapter next.

Why an ads platform specifically

The substrate could have been anything. I chose ad serving for four reasons, none of them "it sounded impressive":

  • It is the target team's actual domain. Using the posting's own vocabulary — targeting, frequency capping, brand safety, pacing — in code, logs and docs means every conversation lands in the right register.
  • Its failures are configuration failures. The realistic outage in ad tech is not a severed cable, it is a bad targeting deploy. Those have a cause sitting in a database — exactly the shape an RCA agent can be honestly evaluated against.
  • It produces genuine telemetry under load. Fill rates, rejection reasons, latency percentiles, spend. Real numbers that move for real reasons.
  • It is small enough to finish. Four services, one machine. The substrate is scaffolding, and scaffolding that takes three weeks has eaten the project.

How "mini" is mini? An honest accounting

I want to be precise about this, because "I built a Netflix-style ads platform" is the kind of claim that deserves immediate suspicion.

What I built is realistic in shape and tiny in scale, and those are very different things. Here is the gap, stated plainly:

CapabilityMineA real oneDoes the AI layer care?
Campaigns, creatives, budgetsyesplus contracts, insertion orders, deal termsyes — agents read config
Targetingcountry, device, ratinghundreds of dimensions, audience segmentsshape, not breadth
Frequency cappingper member, per daycross-device, multi-window, identity graphyes
Brand safetycategory exclusionsML classification, third-party verificationshape, not accuracy
Budget pacingeven daily spendpredictive, forecast-drivenyes
Ad selectionmost budget remainingreal-time auction, bidding, CTR predictionnot yet — see below
Event ingestionHTTP, idempotentstreaming pipeline, billions per dayshape, not throughput
ObservabilityPrometheus + Grafanasame, plus tracing and anomaly detectionyes — this is the input
Request volume~2 per secondmillions per secondno — throughput teaches it nothing
Forecasting & inventoryabsentcore to the businessfuture
Identity resolutionabsentcore to the businessfuture
Measurement & attributionabsentcore to the businessno

Read the right-hand column and the logic of every omission becomes visible. I built exactly the parts the AI layer has to reason about, and skipped the parts it does not.

Scale is the clearest example. Going from two requests a second to two million would be weeks of work and would teach the agents nothing new — a root-cause agent reads the same rejection reason off the same metric either way. Throughput is an interesting engineering problem and an irrelevant one here.

What I did not compromise on is the part that matters: every rejection carries a named reason, every request leaves a structured log line, every failure has a cause sitting in a real database, and every number comes from a real run. Those are the properties an agent operates on. A platform a thousand times larger with none of them would be useless to this project.

What I plan to close, and why

A real auction. Right now the winner is whichever eligible campaign has the most daily budget left — a rule, not a market. Replacing it with second-price bidding against a predicted click-through rate would give the ops agents a genuinely harder question: "fill rate is fine but revenue per thousand impressions dropped 8% — why?" That is the sort of incident real ads teams actually fight, and my current substrate cannot produce it.

A forecasting service. Inventory prediction would introduce a component that is wrong on purpose — models drift — which is a different and more interesting failure class than a bad config deploy.

More realistic targeting breadth. Not for realism's sake, but because a decision path with thirty dimensions is much harder to explain than one with three — and explaining decisions is precisely what Level 3's agents are graded on.

None of these are on the thirty-day plan. All of them make the substrate a harder and more honest test, which is the only reason they are on a list at all.

The thirty days, in six levels

LevelDaysWhat gets builtIts gate
01–5The substrate — four services, observability, injectable failures11/11 ✓
16–10Context layer — ingestion, chunking, retrieval, knowledge graphrelevance >85%
211–15Dev agents — code gen, test gen, PR review, deploy validationcatch rate published
316–20Ops agents — log intelligence, triage, RCA, guided resolutionRCA correct >80%
421–25Orchestration — parallel agents, conflict resolution, human gates3/3 complex tasks
526–30Platform — dev environment, CI gates, eval dashboard, launch<10 min setup

I wrote every target before building its level. That ordering matters more than it looks: a target I invent after seeing the result is not a target, it is a description.

Days 1–5 I build the substrate. From Day 6 I start on the AI layer itself.

The method

How a single day actually runs

This is the loop I run every day. None of it is ceremony — every step is there because skipping it cost me something specific, and most of those costs are named later on this page.

flowchart LR
    A["1 · Read<br/>spec + yesterday's<br/>execution notes"] --> B["2 · Write the plan<br/>into the repo,<br/>task by task"]
    B --> C["3 · Per task:<br/>test first, watch it fail,<br/>then implement"]
    C --> D["4 · Four gates,<br/>run separately"]
    D -->|"any fail"| C
    D -->|"all pass"| E["5 · Commit<br/>conventional message"]
    E --> F["6 · Devlog + running doc<br/>+ execution notes"]
    F -->|"tomorrow"| A
      

Step 1 · The plan is a file in the repository

Before I write any code, I commit a written plan to docs/superpowers/plans/. Not a to-do list. A document naming every file that will exist, the exact function signatures, and the tests I will write first.

Here is the shape of a real task from the Day 7 plan:

### Task 4: The embedder protocol, cost tracking, and the hashing engine

Files: create embedders/base.py, embedders/hashing.py + tests

Produces:
  UsageRecord   — engine, texts, tokens, cost_usd, latency_ms
  EngineSpec    — name, label, dimensions, hosted, cost_per_million_tokens
  Embedder      — Protocol: spec, available() -> (bool, str), embed(texts)

Tests: determinism; unit norm; shared vocabulary scores higher than
       unrelated text; cost_usd == 0; registry lists all four names

- [ ] Write failing tests → run → implement → run → commit

I started writing plans into the repository because I kept discovering, four files into a task, that I had chosen the wrong shape — and by then the wrong shape was load-bearing. It forces me to think the design through while changing it is still free — renaming a function in a plan costs nothing, renaming it across nine files costs me an hour. And it means a fresh session, or someone else entirely, can pick the work up without re-deriving anything I already worked out.

The plans are also, deliberately, part of the corpus. From Day 6 onward the AI layer can read how each day was planned, including what the plan got wrong.

Step 2 · Test first — and watch it fail

Every task I write follows five steps. Here is a real one, start to finish.

2a. Write the test before the code exists. This is from Day 4's event-service — the test that pins down idempotent counting:

def test_the_same_event_reported_twice_is_counted_once():
    first  = post_event(event_id="evt-123", event_type="impression")
    second = post_event(event_id="evt-123", event_type="impression")

    assert first.status_code  == 201
    assert first.json()["status"]  == "recorded"
    assert second.status_code == 200
    assert second.json()["status"] == "duplicate"

    # the point of the whole day
    assert get_delivery()["impressions"] == 1

2b. Run it, and confirm it fails for the right reason.

$ uv run pytest tests/substrate/event_service -q

E   ModuleNotFoundError: No module named 'substrate.event_service.main'
1 error during collection

This is the step I am most tempted to skip. It feels like theatre — I already know it will fail, so why spend the ten seconds watching it?

I do it anyway, every time, because running the test before the code exists is the only proof that the test is capable of failing at all.

A test that has never been seen failing might be asserting nothing at all — a typo in a fixture name, an assertion inside a branch that never runs, a comparison that is always true. A test that always passes is worse than no test, because it produces confidence without evidence, and it will still be passing years later while the behaviour it claims to protect has quietly rotted.

2c. Write the minimum code to pass. Not the elegant version, not the general version — the one that makes this test go green.

2d. Run it again.

$ uv run pytest tests/substrate/event_service -q
............ 12 passed in 0.61s

2e. Commit. Then the next task starts from a green repository, which means any breakage in the next hour was introduced in the next hour.

Planted here — pays off on Day 11

This discipline is not only for the human. From Level 2, agents generate code directly into this repository.

The tests and the gates below become the fence that generated code has to clear before it lands. Build the fence before you need it, and "the AI wrote it" stops being a reason to lower the bar. Build it after, and there is no bar to lower it from.

Step 3 · Four gates, run separately, never piped

GateWhat it checksA real thing it caught
ruff checkUnused imports, risky patterns, style driftA helper typed Any instead of a concrete dict, three separate days
ruff formatEvery file laid out identicallyKeeps diffs about meaning rather than whitespace
mypy --strictEvery value is the type it claimsA function returning an untyped value from json.loads
pytestThe behaviour still holdsA metric collision that only appears when two services load together

I run them as four separate commands on four separate lines. That sounds pedantic until you see what happened when I did not:

# WRONG — reports success even when mypy fails.
# In a pipeline the surviving exit code is tail's, and tail always succeeds.
uv run mypy tests substrate platform bellwether | tail -2 && git commit

# RIGHT — each gate's own exit code is visible and fatal.
uv run ruff check .
uv run ruff format --check .
uv run mypy tests substrate platform bellwether
uv run pytest

This exact mistake shipped nine type errors on Day 7. My own Day 4 notes warn against it. I wrote both.

Step 4 · The tests must pass with everything switched off

A rule I hold myself to: the entire suite passes with Docker stopped and no API keys set. 289 tests, about six seconds.

I built it this way for a selfish reason. A test suite that takes two minutes and needs four containers is a suite I will quietly stop running — not deliberately, just by drifting into running it less often, until the day it fails and I have no idea which of the last twenty changes did it. Six seconds is fast enough that I never have to decide.

This is enforced by design rather than willpower. Business logic never reaches out for what it needs — it is handed it. Databases, HTTP clients, clocks and random number generators all arrive as arguments:

# The decision logic has no I/O of its own. In production `store` is Redis
# and `now` is the wall clock. In tests they are a dict and a fixed datetime.
def decide(request, campaigns, store, now):
    ...

# So a frequency-cap test needs no Redis, and no container:
def test_a_member_at_their_cap_is_rejected():
    store = {"freq:member-0042:camp-1:2026-07-22": 3}
    result = decide(request, [campaign_capped_at(3)], store, now=FIXED_TIME)
    assert result.no_fill_reason == "frequency_capped"

Two payoffs. Speed — a six-second feedback loop instead of a two-minute one, which changes how often I am willing to run it. And honesty — a test that needs a running database is partly testing the database, and will fail for reasons that have nothing to do with the code I just wrote.

The limit of this rule — and it is a real one
Fast tests can all agree with each other and all be wrong

On Day 7 my stand-in for the vector database faithfully reproduced my own incorrect mental model of how the real one behaves. Every test passed. The real database was silently deleting my data on every write.

Substitutes verify my understanding of a system. They cannot verify the system. Anything that talks to real infrastructure needs at least one genuine round-trip before I trust the design — which is exactly how I eventually caught it.

Step 5 · Decisions get written down with an expiry condition

Every significant choice becomes an Architecture Decision Record with a fixed five-part shape. Here is ADR-0003, in full outline:

Context       What was true when this was decided.
              Frequency counts are written on every impression, are
              per-viewer, and are worthless after midnight.

Decision      What was chosen.
              Redis holds decision state under day-scoped keys that
              expire themselves. Postgres stays the system of record.

Alternatives  What was rejected, and honestly why.
              Postgres for everything — rejected: these are counters
              with an expiry, not records; it would mean writing a
              cleanup job to delete yesterday's rows forever.

Consequences  What this now costs.
              Two stores to reason about. Redis can be lost without
              data loss, but the pacer briefly forgets what it spent.

Trigger       The specific condition that reverses this.
              Day 4's event-service, when impressions become durable
              and spend becomes billable.

I add the trigger because of something I have watched happen on every team I have worked on: a decision that was correct in 2023 is still being obeyed in 2026, and nobody can say why. The trigger is the part that makes this worth doing. Without it my decision hardens into folklore — "we don't do that here" — and nobody, including me, remembers whether the original reason still holds. With it, the decision carries its own review date.

And it fired. One day later, event-service made impressions durable. The trigger had named that exact event in advance, so the question was already on the table rather than being noticed six months later. The answer turned out to be narrow it rather than reverse it:

# ADR-0003, Day 4 update
Postgres now holds the auditable record of what was served.
Redis keeps only the hot counters the serving path reads.

They now disagree by design — and the gap between them is
"reporting loss", which became a panel on the dashboard.

A decision I could revisit on schedule turned into a feature. That only happened because I wrote down, a day in advance, the condition under which I would think about it again.

Step 6 · Commits say what changed and why

Every commit I make uses a conventional prefix and imperative mood, and has to leave the repository green — tests, lint and types all passing.

PrefixMeansA real one from this repo
feat:New capabilityfeat: the chunk, its anchor, and the provenance it inherits
fix:Corrects broken behaviourfix: restore mypy to clean after the vector store landed
docs:Documentation onlydocs: ADR-0007, ADR-0008, day-07 devlog
test:Tests onlytest: cover the duplicate-event path
chore: / ci:Tooling and pipelinesci: type-check the platform directory too

The message body carries the reasoning when it is not obvious. That fix: above reads, in full:

fix: restore mypy to clean after the vector store landed

The previous commit passed its gates through a pipe, so mypy's
non-zero exit was swallowed by tail and nine type errors went in
green. Day-04 execution note 6, committed by the person who wrote it.

I write the reasoning into the commit because the commit is the only artefact that survives with the code. Writing that down cost me thirty seconds and makes my mistake findable forever. Hiding it would have cost nothing today and everything the third time I do it.

Step 7 · Reproduce the CI command exactly, before pushing

Running a superset of what CI runs is not the same as running what CI runs, and the difference hides an entire class of failure.

Day 7 made this concrete. My machine had an optional AI dependency installed; the build server would not. So before pushing, I deliberately stripped my environment back to match:

# Strip the optional group so the venv matches the build server exactly
$ uv sync --group dev
 - model2vec==0.8.2
 - numpy==2.4.6

# Then run the four CI commands verbatim
$ uv run ruff check .                                    All checks passed
$ uv run ruff format --check .                           105 files formatted
$ uv run mypy tests substrate platform bellwether        no issues in 105 files
$ uv run pytest                                          289 passed

The test count went up by one without the optional package — a test that skips when the library is present and runs when it is absent. Both environments genuinely covered, and I would never have seen it without reproducing the real one.

Where this rule came from

Day 5, execution note 10: a test imported a module that resolved under pytest but not under the exact command CI runs. Caught by reproducing CI locally, not by CI — because catching it in CI means a red build, a context switch, and a fix commit.

The note ends: "Reproduce the CI command verbatim before every push; running a superset locally hides this class of failure."

Step 8 · The day ends by recording what the plan got wrong

Every plan I write ends with a section titled "Execution notes — what the plan missed." I write it after the work, listing everything reality disagreed with me about. Not a retrospective anyone schedules — a numbered list, in the plan file, committed.

A real one, from Day 7:

1. PUT /points in Qdrant replaces a point outright. The plan's
   two-call write destroys every previously written engine's vector,
   because the payload write is itself a full replace. Every unit test
   passed against the fake. Found only by pointing the store at a
   running Qdrant and asking whether the first vector survived. It did not.

   → Any plan that fakes a datastore must include one live
     round-trip before the design is trusted.

I write the next day's plan with those notes open. This is the single highest-leverage habit I have, and I only added it after Day 3 — because I genuinely do repeat the same mistake in disguise, and I would not have believed that about myself until I had the table below:

DayWhat happenedThe shared root cause
06Line endings differ between Windows and Linux, so identical files fingerprint differentlyA text file written on Windows carries bytes you did not write, in line 1, invisibly
07A byte-order mark turned VOYAGE_API_KEY into an unrecognisable name
04Warning written: a piped verification command hides its failureA pipeline reports the exit code of its last command, not its first
07The warning's own author did it, and shipped nine type errors green

Step 9 · The definition of done, every day

My day is not finished when the code works. It is finished when all of this is true:

  • Every task in the plan is implemented, test-first, and committed separately
  • All four gates pass, run individually and unpiped, with Docker stopped
  • The CI command has been reproduced verbatim in a matching environment
  • Any significant choice has an ADR — with alternatives and a trigger
  • The devlog is written: shipped, decisions, what running it found, what tomorrow does
  • The running doc is updated — day tracker, and the scoreboard if a number was produced
  • The execution notes are written, honestly, including the embarrassing ones

That last clause is doing real work. Every mistake on this page — the piped gate, the destroyed vectors, the gate that measured itself — I reported on myself. A build log that only contains successes is a marketing document, and nobody learns anything from it, least of all me.

The toolkit

Every tool, and what it beat

Nothing here was chosen because it is popular. Each one is doing a specific job, and the alternative it displaced is named.

Language and packaging

Python 3.11+language

The language every AI and machine-learning library targets first. Choosing anything else means writing bridges to it later.

Why 3.11 specifically: it is the floor the project promises to support. That promise has teeth — on Day 7 a dependency shipped type definitions requiring 3.12 syntax, and it was pinned back rather than quietly raising the floor.

uvpackage manager

Installs dependencies and manages the virtual environment. Written in Rust, and roughly two orders of magnitude faster than the traditional tool.

Instead of pip + virtualenv + pip-tools: three tools replaced by one, with a single lockfile pinning exact versions so every machine and the CI server resolve identically. Day 7's optional AI dependencies installed in 903 milliseconds.

The web layer

FastAPIweb framework

The framework where the type annotations are the API contract. Declare what a request looks like as a typed class, and validation, error messages and machine-readable documentation all come free and stay in sync automatically.

Instead of Flask: Flask needs validation written by hand and documentation maintained separately, which drifts from the code within weeks. Instead of Django: far larger, opinionated about databases and templates, and most of it would go unused here.

Pydantic v2validation

Defines the shape of data at every boundary and rejects anything that does not fit, with a precise message naming the offending field. Its core is Rust, so this costs almost nothing.

Instead of hand-written checks: a rule expressed once as gt=0 becomes a runtime guarantee, a documented constraint, and a type the checker understands — three jobs from one declaration.

httpxhttp client

How services talk to each other, and later how the AI layer talks to Gemini, Voyage and the vector database.

Instead of vendor SDKs: each provider ships its own library with its own dependencies and release cadence. Three of them would be three ways to describe the same handful of HTTP calls. One HTTP client, already present, keeps the dependency list honest.

Storage — two kinds, on purpose

PostgreSQLsystem of record

Holds anything that must survive a restart and must never be wrong: campaigns, creatives, and every delivered impression.

Why relational: the data genuinely is relational, and the guarantee that makes Day 4's counting correct — a unique key the database enforces — is exactly what relational databases are for.

Redishot decision state

Holds counts that are read on every single ad request and are worthless after midnight: how many times this viewer has seen this advert today, how much this campaign has spent today.

Why not just Postgres: these are not records, they are counters with an expiry date. Redis deletes them itself when their time is up. Storing them in Postgres would mean writing a cleanup job to delete yesterday's rows forever.

SQLAlchemy 2.0database mapping

Maps Python classes to database tables, fully typed, so the type checker catches a mistyped column name before the code runs.

Instead of raw SQL: the class is both the code's model and the table's definition, so they cannot disagree. Tests run the same models against in-memory SQLite for speed.

Seeing what is happening

Prometheusmetrics

Each service publishes its current counters at a /metrics address. Prometheus visits every few seconds and records them, building a history you can ask questions of.

Why pull, not push: the services do not need to know Prometheus exists. Adding a fifth service means adding one line of Prometheus configuration, not changing the service.

Grafanadashboards

Draws the graphs. Critically, its dashboards are defined as files in the repository, not arranged by clicking in a browser.

Instead of click-configured dashboards: a clicked dashboard exists only in one person's Grafana. A dashboard in version control is reviewable, restorable, and guarded by a test that fails the build if a panel loses its query.

Structured JSON logginglogs

Every log line is a JSON object, not a sentence. Written once as a shared library on Day 2 and reused by every service.

Instead of prose logs: "Decision took 4ms for member-0042" requires a fragile pattern-match to read. {"member_id": "member-0042", "latency_ms": 4} is already data. On Day 16 an agent reads these without a single regular expression — that is the entire reason.

Infrastructure and quality

Docker Composeorchestration

Describes every service, its ports and its startup order in one file. docker compose up -d starts the entire platform.

Instead of Kubernetes: see ADR-0001 below. Briefly — Kubernetes solves problems this project does not have, at a cost measured in days.

rufflint + format

One Rust binary replacing four traditional Python tools, fast enough to run on every file save.

Instead of flake8 + black + isort + pyupgrade: four tools with four configurations that disagree with each other, replaced by one that is roughly a hundred times faster.

mypy --stricttype checking

Verifies that every value is what the code claims. Strict mode additionally refuses to let anything be left untyped.

Instead of default mode: normal type checking is advisory and silently ignores untyped code, so coverage erodes. Strict mode is uncomfortable for two days and then prevents a steady stream of bugs — and it is what makes AI-generated code safe to accept.

pytesttests

Plain functions with plain assertions, plus fixtures for shared setup.

Instead of unittest: far less ceremony, and the fixture system is what makes dependency substitution clean enough that the hermetic rule is actually livable.

The AI layer adds three tools of its own — a vector database and two embedding models. They arrive in Reel three, alongside Day 7, where the ideas they depend on are explained first.

Reel two · Days 1–5

Building the world

Five days to build a small ads platform that behaves like a real one: it serves adverts, records what it delivered, shows you what it is doing, and can be broken on demand in ways that are genuinely broken.

I call this half the substrate — the layer underneath the AI: real enough to serve, log and break, so everything built on top of it is measured against something true.

First the map — the finished architecture in one reference chapter. Then the five days that built it, one at a time.

Reference · the substrate in full

The architecture, in depth

Reel two tells the build one day at a time. This chapter is the map you read it with — the finished platform as a whole, in five short passes:

  • The shape — how the two halves connect, in one diagram
  • The parts — every service and store, one line each
  • One real request — traced end to end, no invented values
  • The data — what is stored where, and what is measured
  • One real failure — injected, measured, recovered

Skim it for the shape now; come back when a later day zooms in on one part.

System architecture · how the two halves connect substrate AI layer evaluation
flowchart TB
    subgraph SUB[" SUBSTRATE — a small ads platform, built Days 1–5 "]
        direction LR
        SIM["traffic-simulator<br/>seeded load + injectable failures"]
        ADS["ad-decision-service<br/>6-filter serving path"]
        CS["campaign-service<br/>campaigns, budgets, targeting"]
        EV["event-service<br/>impressions, clicks, spend"]
        RED[("Redis<br/>freq caps · pacing spend<br/>(hot, expires nightly)")]
        PG[("Postgres<br/>system of record")]
        OBS["Prometheus + Grafana<br/>metrics, dashboards, JSON logs"]
        SIM ==>|"① ad request"| ADS
        ADS -->|"reads active campaigns"| CS
        ADS <-->|"caps + spend"| RED
        ADS ==>|"② returns the ad"| SIM
        SIM ==>|"③ reports impression / click"| EV
        SIM -->|"seeds campaigns ·<br/>PATCHes them to inject a failure"| CS
        CS --> PG
        EV --> PG
        SIM -.-> OBS
        ADS -.-> OBS
        CS -.-> OBS
        EV -.-> OBS
    end

    STR1(["source, docs,<br/>API contracts, ADRs"])
    STR2(["logs, metrics,<br/>dashboards"])
    STR3(["real incidents<br/>with a cause in a table"])

    SUB --> STR1
    OBS --> STR2
    SIM --> STR3

    subgraph BW[" BELLWETHER — the AI layer, built Day 6 onward "]
        direction TB
        CTX["Context layer<br/>grounds every agent in team knowledge"]
        DEV["Dev agents<br/>code, tests, PR review, deploy checks"]
        OPS["Ops agents<br/>triage, log/metric correlation, RCA"]
        ORCH["Orchestrator<br/>coordinates agents, escalates to a human"]
        CTX --> DEV
        CTX --> OPS
        ORCH -.-> DEV
        ORCH -.-> OPS
    end

    STR1 ==>|"ingested, chunked, embedded"| CTX
    STR2 ==>|"parsed, correlated"| OPS
    STR3 ==>|"the thing to diagnose"| OPS

    EVAL{{"Eval harness<br/>a published score on every capability"}}
    DEV -.-> EVAL
    OPS -.-> EVAL
    CTX -.-> EVAL

    classDef sub fill:#241416,stroke:#b8321f,stroke-width:1px,color:#f3d9d6;
    classDef store fill:#1a1116,stroke:#8f5a63,stroke-width:1px,color:#e6d2d6;
    classDef ai fill:#12211a,stroke:#2f6f4f,stroke-width:1px,color:#d6ece0;
    classDef stream fill:#1c1c24,stroke:#6b6b78,stroke-width:1px,color:#eaeaef;
    classDef evalcls fill:#241f14,stroke:#e8a33d,stroke-width:1px,color:#f0d9ac;
    class SIM,ADS,CS,EV,OBS sub;
    class RED,PG store;
    class CTX,DEV,OPS,ORCH ai;
    class STR1,STR2,STR3 stream;
    class EVAL evalcls;
        
Follow the numbers. ① The simulator asks the decision service for an ad. ② The decision service reads campaigns from campaign-service, checks caps and spend in Redis, and returns one ad. ③ The simulator reports the resulting impression to event-service, quoting the same request-id, so decision and delivery tie together.

One edge matters more than all the others: the simulator seeds the campaigns, and it injects failures by genuinely PATCHing them through campaign-service's public API (ADR-0005). Every incident an agent later diagnoses starts on that edge — a genuinely broken configuration, not a fixture.

The substrate then feeds the AI layer three things. Its source and docs become the context layer (Days 6–10). Its logs and metrics are what the ops agents read (Days 16–20). Its injected incidents are the thing to diagnose. The eval harness scores every capability and publishes the number, flattering or not.

What each part is, in one line

The substrate — the ads platform

traffic-simulator:8004 · Day 5
Drives seeded, reproducible ad-request load through the platform, reports the resulting impressions and clicks, and injects failures by genuinely rewriting campaign configuration — the source of every real incident.
ad-decision-service:8002 · Day 3
The serving path. Takes a viewer and a slot, runs six filters in order — eligibility, targeting, brand safety, frequency cap, pacing, creative fit — and returns one ad or a named no-fill reason. Pure rules, injected state.
campaign-service:8001 · Day 2
The source of truth for advertisers: campaigns, creatives, budgets, targeting and brand-safety rules. The only writer of its own schema — everyone else reads it over HTTP.
event-service:8003 · Day 4
Ingests impressions and clicks, counting each exactly once using a caller-supplied id as the primary key. Delivery totals are computed on read, never stored as a rollup.

Storage — two kinds, on purpose

Redishot decision state
Frequency counts and pacing spend, per member and per campaign, under day-scoped keys that delete themselves after 48 hours. Read on every request, written the moment an ad fills.
PostgreSQLsystem of record
Everything that must survive a restart and must never be wrong: campaigns behind campaign-service, and one durable row per delivered impression behind event-service.
Prometheus + Grafana:9090 / :3000
Scrapes metrics from all five services every few seconds and draws two dashboards defined as files in the repo. This is the raw material the ops agents read during an incident.

BELLWETHER — the AI layer

Context layerDays 6–10
Ingests the whole repo — code, docs, contracts, decisions — chunks and embeds it, and stores it in a vector database so any agent can retrieve grounded answers with a citation. Built through Day 7.
Dev agentsDays 11–15
Generate code and tests, review pull requests against the written standards, and validate a deployment after it lands. Each one scored against a fixed evaluation.
Ops agentsDays 16–20
Read logs and metrics during an incident, correlate them across services, and produce a root cause with cited evidence — graded on how often the cause is actually right.
OrchestratorDays 21–25
Coordinates several agents on one task, resolves their disagreements, and stops to ask a human at the moments that matter.

Cross-cutting

Eval harnessevery level
Not bolted on at the end. Every capability's target is written before it is built, and its score is published on the running doc — whether it flatters or not.
Plan vs. reality
The architecture I built is not the one I sketched — and the differences are the point

My Day-0 diagram was a clean serving-path sketch: request in, ad out, event reported. Building it changed the shape in three ways, and every change came from a requirement the sketch had not yet met.

Pacing spend moved into Redis. I had it as private logic inside the decision service; it had to become shared state, because a budget spent on one request has to be visible to the next.

The simulator gained an edge to campaign-service. This one is not on the original diagram at all. It is how a failure gets injected — a real PATCH to real configuration — and it is the single most important thing in the picture, because without it every ops-agent evaluation later would be graded against a fixture I wrote myself. I could not have drawn it on Day 0; the mechanism did not exist until Day 5.

Event totals are computed, never stored. The sketch said "aggregate", which sounds like a maintained running total. A running total is a second copy of the truth that drifts, so I compute on read instead.

The sketch was simpler. The built version is busier and correct. Simpler was not better here — it was hiding the control plane that turned out to hold the whole thesis up.

One real ad request, traced end to end

Now the territory. One ad request moves through the running system, against the three campaigns the platform seeds itself with on first start. Nothing below is invented — every number, field and reason string is what the code produces.

A viewer reaches an ad break. The simulator sends this to ad-decision-service:8002:

POST /ad-request
{
  "member": { "member_id":"member-0042", "country":"US", "device_type":"tv" },
  "slot":   { "slot_id":"slot-118", "duration_seconds":30,
              "content_rating":"TV-14", "content_categories":["drama","comedy"] }
}

The decision service pulls the active campaign set from campaign-service, then runs three candidates through six filters in a fixed order — cheapest and most-eliminating first. Each column is one of the real seed campaigns:

Filter, in order Acme Snacks
US/CA · all devices · cap 3 · $60/day
Northwind Motors
US/CA/GB/DE · TV only · cap 2 · $65/day
Vertex Hydration
US/BR/GB · mobile/tablet · cap 5 · $50/day
1 · eligibility
active & in flight?
2 · targeting
US · tv · TV-14?
✗ tv ∉ {mobile, tablet}
3 · brand safety
excluded ∩ slot cats?
✓ {news,true-crime} ∩ {drama,comedy} = ∅✓ {true-crime} ∩ {drama,comedy} = ∅
4 · frequency cap
seen today < cap?
✓ 0 < 3✓ 0 < 2
5 · pacing
spent < daily budget?
✓ $0 < $60✓ $0 < $65
6 · creative fit
a creative ≤ 30s?
✓ 30s hero✓ 30s cutdown
(60s cinematic too long)
verdicteligible
$60 remaining
eligible → wins
$65 remaining
targeting_mismatch

Two campaigns clear all six filters. The tiebreak is the one line of policy that matters most: the winner is the eligible campaign with the most daily budget remainingmax(eligible, key=budget_remaining) — not the first match. Northwind ($65 left) beats Acme ($60 left). Without that rule, whichever campaign sorted first would drain its whole day by mid-morning.

The response names the winner and — this is the part that pays off for the whole rest of the project — carries a trace of why every candidate landed where it did:

{
  "request_id": "3f9c1a20-...",
  "slot_id": "slot-118",
  "filled": true,
  "ad": {
    "campaign_name": "Premium sedan, connected TV only",
    "advertiser":    "Northwind Motors",
    "creative_name": "30s cutdown",
    "asset_url":     "https://cdn.example/sedan-30.mp4",
    "duration_seconds": 30,
    "price_micros": 2000            // $0.002 per impression
  },
  "no_fill_reason": null,
  "candidates_considered": 3,
  "trace": [
    { "campaign_name":"Premium sedan, connected TV only", "reason":"eligible" },
    { "campaign_name":"Wide-reach snack launch",          "reason":"eligible" },
    { "campaign_name":"Sports drink, mobile takeover",    "reason":"targeting_mismatch" }
  ],
  "decision_latency_ms": 1.2
}

Two things happen the instant this fills, and they go to different stores for different reasons. The decision service writes the frequency count and pacing spend to Redis immediately, so the very next request for member-0042 sees the updated numbers. Separately, the simulator reports the impression to event-service:8003, carrying that same request_id, and it becomes one durable row in Postgres:

POST /events  → event-service         # the durable record
{ "event_id":"7c2...", "event_type":"impression", "request_id":"3f9c1a20-...",
  "campaign_id":"...", "member_id":"member-0042", "price_micros":2000 }
→ 201 recorded

# meanwhile, already written to Redis at decision time:
freq:member-0042:<campaign>:2026-07-23   =  1     # TTL 48h, self-expiring
spend:<campaign>:2026-07-23              =  2000  # micros

That split is deliberate and it is ADR-0003 made concrete: Redis is what the pacer believed at decision time; Postgres is what was actually delivered. They can diverge — a lost Redis write, a dropped event — and the gap between them is not a bug, it is a number on the dashboard called reporting loss.

The data model, in real fields

Two tables carry the durable state. Money is Integer micros everywhere — never a float — and a caller-supplied event_id is the primary key that makes double-counting impossible.

campaigns  (Postgres, behind campaign-service)
  id                     uuid   PK
  name, advertiser       str
  status                 str    # draft | active | paused
  budget_micros          int    # total; 32-bit — capped at $2,147 (found the hard way)
  daily_budget_micros    int
  frequency_cap_per_day  int    # default 3
  targeting              json   # { countries, device_types, content_ratings }
  brand_safety_exclusions json  # ["news", "true-crime", ...]
  starts_at, ends_at     datetime
  ↳ creatives          (name, duration_seconds, asset_url)

ad_events  (Postgres, behind event-service — append-only)
  event_id     uuid   PK     # chosen by the caller — the whole idempotency trick
  event_type   str           # impression | click
  request_id   uuid          # ties the event back to the decision that caused it
  campaign_id, creative_id  uuid
  member_id, slot_id        str
  price_micros int           # impression = 2000, click = 0
  occurred_at  datetime

What the platform measures

Prometheus scrapes every service every few seconds. These are the metrics that matter. The one that does the most work is ad_candidates_filtered_total — because Day 3 gave every rejection a name, a fill-rate collapse becomes a single chart where you read the cause off whichever band grew.

MetricLabelsThe question it answers
ad_candidates_filtered_totalreasonWhy did candidates lose? targeting_mismatch, pacing_throttled, frequency_capped…
ad_decisions_totaloutcomeFill rate — filled vs no-fill
ad_events_totalevent_typeImpressions and clicks ingested → click-through rate
ad_events_duplicate_totalRetries the primary key rejected — dedup working
ad_spend_micros_totalMoney actually delivered
http_request_duration_secondsservicep95 latency — is it load, or a fault?
sim_scenario_infoscenarioWhich failure mode is currently injected

A failure, with the actual numbers

This is where the whole "real, not mocked" argument cashes out. To inject a bad config deploy, the simulator makes a genuine call to campaign-service — for every campaign:

PATCH /campaigns/{id}  → campaign-service      # repeated for all 3
{ "targeting": { "countries":["AQ"], "device_types":[], "content_ratings":[] } }
                    // AQ = Antarctica. No viewer is ever there.

Nothing is faked. The decision service reads the changed configuration and correctly declines to serve, because the targeting genuinely no longer matches anyone. On the running stack, the effect was measured:

fill rate     43%2%          # within seconds of the PATCH
loss reason   the targeting_mismatch band takes over the "why candidates lost" chart

Recovery is equally real: the steady scenario PATCHes the original targeting back. It is not an undo flag — it is the same remediation an ops agent will be asked to recommend on Day 19. And a second failure mode, budget_runaway, is how the platform found a defect in itself on day one:

PATCH  daily_budget_micros  →  9,000,000,000
→ 500  NumericValueOutOfRange   # budget_micros is a 32-bit Integer,
                              # capped at 2,147,483,647 micros = $2,147/day

A real ceiling nobody knew existed, surfaced by pointing real load at a real platform — and, because it first escaped as a bare 500 in violation of the project's own standard, now a typed 422. That is the entire thesis in one incident: a system that is genuinely running finds the things a plan cannot.

Day 01 · Foundation

The boring day that decides the next twenty-nine

Day 1 produces nothing a user could touch. It builds the workshop: how code gets checked, how the databases start, how anyone can tell whether things are working.

Skipping this is the most common way a 30-day build collapses around day nine.

What actually got built

  • The repository, with the four quality gates wired to run on every change
  • A Compose file starting Postgres, Redis, Prometheus and Grafana — all healthy on one command
  • Continuous integration: the same four gates run on every push, on a clean machine
  • The running doc — a live page with the architecture, a 30-day tracker, and an evaluation scoreboard
  • ADR-0001

The scoreboard shipped with every number blank. Publishing empty cells on day one is a commitment: those cells are the entire point of the series, and they only get filled by something that actually ran.

ADR-0001
Docker Compose, not Kubernetes

Kubernetes exists to run hundreds of services across many machines, surviving hardware failure and unpredictable traffic. This is four services on one laptop.

Adopting it would cost days of configuration and prove nothing about AI engineering — which is the actual subject. Compose starts everything with one command and remains readable by someone who has never seen the project.

Choosing infrastructure proportionate to the problem is the senior signal. Reaching for the largest available tool is the opposite, and experienced reviewers read it instantly.

Trigger to revisit: the first time this needs to run across more than one machine.

A small detail that saved hours

Postgres was published on port 5433 rather than the standard 5432, and Redis on 6380 rather than 6379.

Any other database already running on the machine would otherwise silently answer the connection, producing failures that look exactly like code bugs and take an hour to trace.

Day 02 · campaign-service

Where the advertisers live

The first real service. It stores campaigns — an advertiser's instructions about when, where and to whom their advert may be shown.

The vocabulary is deliberate. These are the exact terms the Netflix posting uses, and using them consistently in code, logs and documentation is what lets an agent later connect a question to an answer.

TermIn plain English
targetingWho may see this — which countries, device types, content ratings
frequency capThe most times one viewer may see it in a single day
brand safetyContent this advertiser refuses to appear beside
pacingSpreading a daily budget across the day instead of spending it by 9am
creativeThe actual video file, with its duration
flight windowThe dates between which the campaign runs
flowchart LR
    C["Campaign<br/>total budget · daily budget<br/>frequency cap · status<br/>flight window"]
    T["Targeting<br/>countries<br/>device types<br/>content ratings"]
    B["Brand safety<br/>excluded categories"]
    CR["Creative<br/>name · asset URL<br/>duration seconds"]
    C --> T
    C --> B
    C -->|"one campaign,<br/>many creatives"| CR
    C --> PG[("PostgreSQL")]
      

Money is never a decimal

Budgets are stored as whole numbers of micros — millionths of a currency unit. Ten dollars is 10000000.

This looks awkward and prevents a genuine disaster. Computers cannot represent 0.1 exactly in the decimal format most languages default to:

# Python, and every other language using the same number format
>>> 0.1 + 0.2
0.30000000000000004

>>> sum([0.1] * 10)
0.9999999999999999   # should be 1.0

Spread that across millions of impressions and the books stop balancing, in a way that is nearly impossible to trace afterwards. Whole numbers never drift.

Failures have a fixed shape

Every error — a malformed request, a campaign that does not exist, anything — returns the same envelope:

// A campaign that does not exist
{ "error": { "code": 404, "message": "Campaign 7f3a... not found" } }

// A budget that exceeds what the column can hold
{ "error": { "code": 422, "message": "daily_budget_micros must be <= 2147483647" } }

A generic "500 Internal Server Error" tells the caller nothing and gives them nothing to act on. Every failure here is typed and explained.

Planted here — paid off on Day 5

This standard gets violated three days later, when a database range error escapes as a bare 500. Because the standard was written down, the violation was identifiable rather than merely unfortunate — and the fix generalised to every column, not just the one that failed.

Every request leaves a structured trace

{"service":"campaign-service", "endpoint":"POST /campaigns",
 "status":201, "latency_ms":12.4, "campaign_id":"7f3a..."}

One line of JSON per request. Data, not prose.

ADR-0002
Build the tables directly — no migration tooling, for now

Normally, changing a database's shape requires a migration: a versioned script so an existing database can be upgraded without losing its data. The standard tool is Alembic.

At this point there is one service writing, no data anyone depends on, and a shape that will change every single day through Day 5. Migrations written against something that churns daily are noise wearing the costume of safety.

Trigger to revisit: the first time a second service reads these tables.

Day 2 ended at 23 tests passing.

Day 03 · ad-decision-service

The twenty milliseconds that matter

This is the heart of any ads platform. A viewer reaches an ad break. Something must decide, in milliseconds, which advert to play — or that none is suitable.

// The request: who is watching, and what slot is available
POST /ad-request
{
  "member": { "member_id":"member-0042", "country":"US", "device_type":"tv" },
  "slot":   { "slot_id":"slot-118", "duration_seconds":30,
              "content_rating":"TV-14", "content_categories":["drama","comedy"] }
}

Six filters, in a deliberate order

Every campaign starts as a candidate. Each filter eliminates some. The order is chosen so the cheapest, most eliminating checks run first — there is no point asking Redis about frequency caps for a campaign that expired last week.

flowchart TB
    REQ["Ad request<br/>member + slot"] --> F1
    F1["1 · Eligibility<br/>active, and inside its flight window?"] --> F2
    F2["2 · Targeting<br/>right country, device, content rating?"] --> F3
    F3["3 · Brand safety<br/>does this content offend the advertiser?"] --> F4
    F4["4 · Frequency cap<br/>has this viewer already seen it enough?"] --> F5
    F5["5 · Budget pacing<br/>has it spent its allowance for now?"] --> F6
    F6["6 · Creative fit<br/>is there a video of the right length?"] --> WIN
    WIN["Winner: the campaign with the<br/>most daily budget remaining"] --> OUT["Filled ✓"]
    F1 -.->|"campaign_inactive"| NF["No-fill,<br/>with a named reason"]
    F2 -.->|"targeting_mismatch"| NF
    F3 -.->|"brand_safety_excluded"| NF
    F4 -.->|"frequency_capped"| NF
    F5 -.->|"pacing_throttled"| NF
    F6 -.->|"no_creative_fit"| NF
      

I made the winner the campaign with the most daily budget remaining, not the first one that matches. Otherwise whichever campaign happens to sort first drains its entire day by mid-morning and everything else starves.

Design rule — possibly the most consequential one here
Named reasons, never booleans

A filter that returns true or false tells you nothing at 2am. Every rejection carries a name: pacing_throttled, targeting_mismatch, frequency_capped.

That single string becomes four things at once: a field in the API response, a field in the log line, a label on a Prometheus metric, and a coloured band on a Grafana chart.

// The response, when nothing could be served
{
  "request_id": "44444444-...",
  "filled": false,
  "no_fill_reason": "frequency_capped",
  "candidates_considered": 3,
  "trace": [
    {"campaign":"Wide-reach snack launch", "lost_to":"frequency_capped"},
    {"campaign":"Premium sedan, CTV only",  "lost_to":"targeting_mismatch"},
    {"campaign":"Sports drink, mobile",     "lost_to":"targeting_mismatch"}
  ],
  "decision_latency_ms": 1.2
}

Every response says which campaign lost, and to which rule.

Planted here — pays off on Days 4, 5 and 18

Day 4: these names become a stacked chart called "why candidates lost". Day 5: a deliberately broken deployment is diagnosed by watching one band swallow the chart. Day 18: the root-cause agent cites this exact field as evidence.

None of that is possible if the filters had returned false.

Two kinds of memory

Frequency capping needs to know how many times this viewer has already seen this advert today. That count is written on every impression, is per-viewer, and is worthless after midnight.

ADR-0003
Redis holds the decision state, Postgres holds the truth

A count that expires at midnight is a counter with a timer, not a permanent record. Redis holds these under keys scoped to the day that delete themselves after 48 hours.

Note the discipline: the decision service reads campaigns through campaign-service's public API, never its database tables. That is precisely what keeps ADR-0002's trigger — "the first time a second service reads these tables" — deliberately unfired. Two decisions, written a day apart, holding each other in place.

Trigger to revisit: Day 4's event-service, when impressions become permanent and spend becomes billable.

What reality found
Two services, one global counter, instant collision

Prometheus keeps a single registry per running process. Both services defined a metric called http_requests_total, which was fine until the test suite imported both at once — then it crashed with a duplicate-metric error.

Fixed by defining the shared metrics exactly once in substrate/shared/observability.py, with a service label distinguishing them. The plan had not foreseen it; running the tests did, in about four seconds.

Day 3 ended at 51 tests passing, all with Docker stopped.

Day 04 · event-service + dashboards

Counting things exactly once

An advert played. That is an impression, and it costs the advertiser money. Someone clicked it. That is a click, and it costs nothing. Both must be recorded.

The hard part is not storing them. It is storing them exactly once.

Why counting is genuinely hard

Networks are unreliable. A player reports an impression, the confirmation gets lost in transit, the player reasonably retries. The same impression now arrives twice. Count both and the advertiser is billed twice.

The obvious fix is to check first: have I seen this before? No? Then save it. That fix is broken, and understanding why is worth thirty seconds:

sequenceDiagram
    participant A as Report A
    participant B as Report B, a retry
    participant DB as Database
    Note over A,B: same impression, same moment
    A->>DB: seen event-123?
    DB-->>A: no
    B->>DB: seen event-123?
    DB-->>B: no
    A->>DB: save it
    B->>DB: save it
    Note over DB: counted twice
      

Both checks ran before either save. Neither did anything wrong. The bug lives in the gap between checking and writing — and no amount of careful code closes a gap that is inherent to the approach.

ADR-0004
Delete the gap: let the database refuse it

The caller generates the event's identifier, and that identifier is the table's primary key — a value the database itself guarantees is unique.

So both reports simply try to save. One succeeds. The other is rejected by the database, and the service answers 200 duplicate. There is no gap between checking and writing, because there is no check.

A duplicate returns success, not an error — from the caller's point of view the event is recorded. An error would train clients to retry harder at exactly the wrong moment.

POST /events  { "event_id":"evt-123", "event_type":"impression", ... }
→ 201  { "status": "recorded" }

# the identical body again, twice200  { "status": "duplicate" }
→ 200  { "status": "duplicate" }

GET /delivery  → impressions: 1

The whole idea, in three requests.

Server-generated identifiers were never an option: every retry would receive a fresh one and therefore become a brand-new impression. The caller must own the identifier for any of this to work.

Totals are computed, never stored

Delivery figures come from scanning the events table on demand — one grouped query — rather than being maintained in a separate running-total table.

A running total is a second copy of the truth, and two copies drift the first time anyone corrects historical data. The scan is fast because the table is small, not because the query is clever, and that distinction is written down along with the condition that changes it: when the query exceeds roughly 200 milliseconds under load.

The trigger from yesterday fired, on schedule

ADR-0003 said Redis would hold spend until impressions became durable. Today they did. The decision narrowed rather than reversed:

flowchart LR
    R[("Redis<br/>what the pacer believed<br/>at decision time")]
    P[("Postgres<br/>what was actually<br/>delivered")]
    R -.->|"the gap between them"| G["reporting loss<br/>— a dashboard panel,<br/>not a bug"]
    P -.-> G
      

They now disagree by design, and the disagreement is the interesting number.

Dashboards that live in the repository

Two Grafana dashboards, defined as files under version control:

  • substrate-health — which services are up, request rates, error ratios, 95th-percentile latency
  • ads-delivery — fill rate, why candidates lost, events ingested, click-through rate, spend

"Why candidates lost" is Day 3's payoff. It is a stacked chart with one band per rejection reason. When fill rate collapses, you read the cause off whichever band grew. That is a diagnosis in one glance, and it exists only because every filter was made to name itself.

What reality found
An unnamed data source, and every panel silently loads empty

Grafana generates a random internal identifier for its data connection on first startup. The dashboards referenced whatever it generated. Restart, new identifier, and every panel renders blank — with no error, pointing at a connection that no longer exists.

Fixed by pinning the identifier explicitly, plus a test asserting the dashboards and the data source still agree. The live stack surfaced this in about thirty seconds; no amount of re-reading the plan would have.

Day 4 ended at 87 tests, four services scraped by Prometheus.

Day 05 · traffic-simulator + the quality gate

Breaking it on purpose, for real

Four services exist and nothing uses them. Day 5 builds a simulator that generates realistic viewers, requests adverts, and reports impressions and clicks back — continuously, at a controlled rate, from a fixed random seed so any run can be replayed exactly.

Then it breaks things. Five failure modes, switchable live over an API:

ScenarioWhat it actually doesWhat the dashboards show
steadyNormal traffic — and doubles as the rollbackflat and healthy, ~40% fill rate
error_burst3 in 10 requests are deliberately malformederror ratio climbs; the 422s are the caller's fault, correctly
traffic_surgeTen times the request ratethroughput and latency climb together — load, not fault
bad_config_deployRewrites every campaign's targeting to a country with no viewersfill rate falls off a cliff
budget_runawayInflates one campaign's daily budgetpacing stops throttling; one campaign takes everything
ADR-0005 — the most important decision so far
Real failures, never simulated ones

No service contains a branch that knows it is being tested. There is no CHAOS=true flag, no debug endpoint, no invented data written into the metrics.

A bad config deploy genuinely sends PATCH requests that rewrite every campaign's targeting. The decision service then reads that configuration and correctly declines to serve. The code behaves perfectly against configuration that is wrong — which is precisely what a real outage looks like.

Measured live: fill rate went from 43% to 2%.

flowchart LR
    OP["POST /scenario<br/>bad_config_deploy"] --> SIM["traffic-simulator"]
    SIM -->|"PATCH every campaign<br/>targeting → Antarctica"| CS["campaign-service"]
    SIM -->|"keeps requesting ads"| ADS["ad-decision-service"]
    ADS -->|"reads the changed config"| CS
    ADS --> NF["targeting_mismatch<br/>fill rate 43% → 2%"]
    NF --> PROM["Prometheus"]
    PROM --> GRAF["Grafana<br/>the incident, visible"]
    GRAF -.->|"Level 3"| RCA["ops agent finds<br/>the real cause"]
      

I could have built this in an afternoon with a flag each service checks. I spent the extra day on the honest version because of what happens on Day 18: an agent asked to find the root cause of a fake outage can only ever find the fake, and I would have spent the rest of the month grading it against a fixture I wrote myself.

Because the damage is real, recovery has to be real too. The steady scenario does not flip a flag back — it restores the original configuration through the same public API. Which is exactly the remediation an operations agent should later recommend, making it a genuine rehearsal rather than an undo button.

The first published number

The Level 0 quality gate: eleven automated checks against the running system. Five service health checks, one confirming Prometheus is collecting from all four, and one per failure mode — each verifying the failure produces the signal it claims to.

Its decision logic is pure and separately unit-tested; the live run only supplies the numbers. That is what makes it trustworthy in both directions — it cannot pass with the platform switched off, and it cannot fail because a test fixture drifted from reality.

Result: 11/11 — 100%. The first real figure on the scoreboard, and the gate leaves the platform healthy when it exits.

What reality found — three defects on day one, plus one in the gate itself
A $2,147 ceiling nobody knew existed

Budgets were stored in a 32-bit integer column, capping at 2,147,483,647 micros — about $2,147 of daily budget. The runaway scenario tried to exceed it and Postgres refused.

And it escaped as a bare 500, violating the project's own Day 2 standard. Now a typed 422, with a handler ensuring no column's range error can ever return untyped again.

The seeding guard was all-or-nothing. One leftover campaign from a manual test suppressed the entire seed set, so the simulator aimed all its traffic at a single narrowly-targeted campaign — a 2% fill rate with 828 rejections, looking exactly like a bug in the decision logic. Seeding is now idempotent per campaign, by name.

And the gate failed itself on its first run. The traffic-surge check read its baseline from the response of the very call that caused the surge, so it compared 20 requests per second against 20 requests per second and concluded nothing had happened. Caught, fixed, re-run clean — which is precisely what a gate is for.

Level 0 complete: 141 tests, five services, real traffic, real incidents.

Reel three · Days 6–7

Teaching it to think

The substrate stops being the thing under construction and becomes the thing the AI knows about. Two days to turn a repository into searchable understanding.

Day 06 · Document ingestion

Everything built so far becomes knowledge

Level 1 begins. The source code, the five decision records, the daily logs, the runbook, the API contracts, the Compose and Grafana configuration — all of it becomes the corpus.

This is the context layer, and it is the part of the job posting I was most interested in. Giving an AI your team's actual accumulated knowledge, so it answers from your decisions rather than from generic internet training. Ask a normal AI why this project uses Compose and it will recite general trade-offs. Ask one grounded in this corpus and it should quote ADR-0001 and name its trigger.

flowchart LR
    RULES["14 ordered rules<br/>what belongs in the corpus"] --> DISC["discover<br/>63 files claimed"]
    APPS["4 FastAPI app objects<br/>asked directly for their contracts"] --> SPECS["4 API contracts<br/>generated, not scraped"]
    DISC --> LOAD["load<br/>title · component · attributes"]
    LOAD --> HASH["normalise line endings<br/>then fingerprint (sha256)"]
    SPECS --> HASH
    HASH --> UP["store<br/>added / updated / unchanged"]
    UP --> PRUNE["prune<br/>anything whose source is gone"]
    PRUNE --> CORPUS["corpus.jsonl<br/>67 documents · 550 KiB"]
      
ADR-0006
A reviewable list, not a crawl

The lazy approach walks the folder and takes everything not explicitly excluded. It works on day one and rots quietly: add a directory of scratch notes and the AI starts grounding its answers in them, with nobody reviewing a change that was never made.

Instead, what belongs is fourteen ordered rules written in code. Adding a directory to the corpus is a code change with a test attached. "What do the agents know" becomes a question you can answer by reading a pull request.

The blank decision-record template is deliberately excluded — it is a form, not a decision, and grounding an agent in it teaches it to answer with blanks.

Contracts read from the code, not the network

Each service publishes an API contract. The obvious way to collect them is to ask the running services over HTTP.

That would mean the corpus depends on which containers happen to be running, cannot be rebuilt on a build server, and quietly differs between machines. Instead the contracts are generated by loading the application code and asking it directly — identical bytes on every machine, including one with Docker switched off.

Fingerprints, so nothing is done twice

Every document gets a content hash — a fingerprint that changes completely if a single character changes. Re-run the pipeline on an unchanged repository and the answer is "67 unchanged, nothing written."

That is what makes tomorrow's embedding step affordable to repeat, because tomorrow every changed document costs money.

Provenance, so every answer can be checked

{
  "doc_id": "docs/adr/0005-real-failures-not-mocked.md",
  "content_hash": "sha256:d42a03d4128146c2...",
  "provenance": {
    "source_type": "adr",
    "component":   "docs",
    "title":       "ADR-0005: Failures are injected by changing real...",
    "line_count":  37,
    "attributes":  { "adr_number": "0005" }
  }
}

The identifier is the file path itself — a citation someone can paste into an editor.

What reality found
Line endings would have broken everything, invisibly

Windows ends each line with two invisible characters; Linux uses one. Git warned about the conversion on the very first commit of the day.

Without normalising before fingerprinting, a file hashed on Windows and re-hashed on a Linux build server produces different fingerprints for identical content. Every run would report all 67 documents as changed, every embedding would be recomputed, and "unchanged" would mean nothing at all.

The whole efficiency argument of the day would have been quietly false.

Day 6 ended at 201 tests, 67 documents.

Day 07 · Chunking, embeddings, vectors

Teaching it to find things by meaning

A whole document is the wrong unit for retrieval. One API contract is 851 lines — handing an agent all of it to answer one question wastes most of its attention.

Two problems: cut documents into useful pieces, and make those pieces findable by meaning rather than by keyword.

Cutting where the content already has seams

The naive approach cuts every 2,000 characters, which slices functions in half and separates a heading from the text under it. Instead each kind of document is cut where it already has natural boundaries:

flowchart TB
    DOC["A document"] --> R{"What kind?"}
    R -->|"Python"| PY["Split at functions and classes<br/>decorators stay attached<br/>anchor: the dotted symbol name"]
    R -->|"Markdown"| MD["Split at headings<br/>heading line stays in the piece<br/>anchor: the full heading path"]
    R -->|"API contract"| OA["Split per endpoint<br/>anchor: POST /campaigns"]
    R -->|"anything else"| WIN["Fixed window<br/>anchor: none"]
    PY & MD & OA & WIN --> CAP["Cap oversized pieces,<br/>keeping the parent anchor"]
      

Every piece keeps an anchor — the name of the thing it is:

ADR-0005 › Alternatives considered          // a heading path
substrate.traffic_simulator.driver.tick       // a dotted symbol
POST /ad-request                              // an endpoint

A piece that cannot name itself can be returned but not defended. That turns out to be measurable:

ApproachPiecesMedian sizeCan name itself
structure-aware585854 chars97%
cut every 2,000 chars3491,960 chars0%

More pieces, each smaller and each able to say what it is. The naive baseline scores zero by construction — that is not a rigged comparison, it is the actual cost of splitting on a character count, stated as a number.

What an embedding actually is

An embedding converts text into a long list of numbers — a position in a space with hundreds or thousands of dimensions — arranged so that text with similar meaning lands nearby.

"Frequency capping" and "how often a viewer sees an advert" share no words at all. A good embedding still places them close together. Searching then becomes: convert the question to a position, return whatever is nearest.

Different models do this with very different quality. Rather than picking one on faith, BELLWETHER runs four behind a single interchangeable interface and measures them.

ADR-0007 — the original plan was backwards
Hosted models by default, local for free iteration

The Day 0 plan said run a model locally to save money. Measuring the corpus inverted the reasoning completely.

The whole corpus is about 149,000 tokens — roughly two cents to embed with the best model available. Running locally instead means installing PyTorch: 2.5 GB on disk and a resident process competing for memory with Docker, Postgres, Redis, five services, Prometheus and Grafana on a 16 GB laptop with no graphics card.

The best option turned out to be the lightest one locally, because the computation happens on someone else's hardware. The premise flipped the moment anyone measured instead of assuming.

Trigger to revisit: a corpus past a few million tokens, where per-run cost stops being a rounding error and becomes a line item.

EngineDimensionsCostTimeRole
gemini-embedding-0013072$0.022310.1squality tier — leads the public benchmark
voyage-3.51024written and tested; free tier too limited to run
potion-retrieval-32M512free2.1slocal, offline, no graphics card
hashing256free0.2sthe CI engine — no dependencies, no network

The fourth is worth explaining. hashing is a keyword-only method with no model at all — it exists so the test suite can assert real retrieval behaviour on a build server with no API key and no downloads. It is genuinely useful and genuinely limited, which makes it the perfect control.

ADR-0008
One record per piece, one vector per engine

Comparing engines only means something if each sees identical input. Give each its own copy of the data and any difference might be the data, not the model.

Qdrant stores each text piece as a single record carrying all four engines' vectors at once, alongside one shared set of provenance. Switching engines becomes a parameter on the query.

Fairness stops being a discipline someone has to remember and becomes a property of the storage layout.

What two cents buys

The test: "how often can the same viewer be shown one advert?" — a question containing none of the words frequency, cap, member, or impression.

EngineScoreBest match it returned
gemini0.765Substrate gaps › SG-04 › Multiple frequency-cap windows
potion0.428ad_decision_service.main.decide
hashing0.328an unrelated plan section — noise, exactly as designed

Gemini understood the question. The local model found the right neighbourhood. The keyword-only baseline found nothing meaningful, because not one word of the question appears in the answer.

That gap is the entire argument for paying for embeddings — measured on this corpus, not quoted from a vendor's chart.

The tools this level added, and what each beat

Qdrantvector database

Stores the numerical representations of text and finds the nearest matches. Written in Rust, so it is light on memory.

Instead of ChromaDB (which the original plan named): Qdrant can hold several vectors per record under different names, which is what makes comparing four embedding models on identical input possible at all. See ADR-0008 above.

Gemini & Voyagehosted embeddings

Convert text into vectors, on the provider's hardware. Gemini currently leads the public quality benchmark.

Instead of running a model locally: two cents for the whole corpus, versus 2.5 GB of machine-learning libraries competing for memory with Docker on a 16 GB laptop. See ADR-0007 above — the plan had this backwards until the corpus was measured.

Model2Vec (potion)local embeddings

A model whose word-vectors are precomputed, so using it is a lookup and an average rather than running a neural network.

Instead of sentence-transformers: eleven small packages and no PyTorch, versus 2.5 GB — at roughly 82% of the quality. Free, offline, and needs no graphics card.

What reality found — the most dangerous bug in eight days
The database silently destroyed data, and every test passed

Writing one engine's vector to a record replaced the entire record. So storing Voyage's vector silently deleted Gemini's.

Every unit test passed — because the stand-in used in tests faithfully reproduced the wrong mental model. The comparison would have rendered a beautiful, convincing chart built on data that had been destroyed.

It was caught by pointing the code at a real running database and asking one question: is the first vector still there? It was not.

Two more the same day. A text file written by PowerShell carried an invisible marker that turned VOYAGE_API_KEY into an unrecognisable name — presenting as a missing key while the file plainly contained one. And a verification command was piped through another program, which swallowed its failure signal, so nine type errors were committed while the output looked green.

Day 7 ended at 289 tests, 585 pieces, three engines measured.

Day 08 · Hybrid retrieval, reranking, an honest number

The day the prediction was wrong

Day 7 could find pieces that mean what you asked. It could not reliably find pieces that are named what you asked. The embedding of budget_micros lands near “budget” and “spending” — and no nearer the one piece that defines the field than the twenty that merely discuss money.

So Day 8 added keyword search (BM25) beside the vector search, fused the two rankings, reranked the result — and then did the part the other three exist to justify: measured whether any of it actually helped.

flowchart TB
    Q["A question"] --> D["Vector search<br/>finds meaning"]
    Q --> L["BM25<br/>finds names"]
    D & L --> F["Reciprocal Rank Fusion<br/>rank position only, never the score"]
    F --> RR["Rerank<br/>free heuristic · or an LLM"]
    RR --> TOP["The results"]
      

The prediction, written down before the code

The design spec pre-registered a claim in git, timestamped before a line of retrieval code existed: hybrid’s advantage would come almost entirely from identifier queries — questions like where is budget_micros enforced — and on conceptual questions it would be flat or slightly worse than vector search alone.

Writing the prediction down first is the whole point. A number is only honest if you cannot move the target after you have seen it.

The answer key nobody could rig

Before any system ran, 26 questions were committed. Then every configuration’s top results were pooled together, shuffled, and stripped of any sign of which system produced them, and each of the 475 question–piece pairs was graded 0, 1 or 2 by hand. The grader never saw which engine surfaced a piece, or where it ranked. The file is committed and pinned to the exact corpus revision it was judged against, so anyone can check the marking.

ConfigurationnDCG@10recall@10MRRp50
dense — vector only0.6700.7570.774812 ms
hybrid + LLM rerank0.6560.6820.8709,309 ms
hybrid + heuristic rerank0.6390.6890.791783 ms
hybrid (RRF)0.6280.6740.844842 ms
hybrid (weighted)0.6240.6780.834802 ms
lexical — BM25 only0.3950.4960.5562.9 ms

There was no aggregate hybrid win to explain. Vector search alone beat every hybrid configuration. Adding the keyword side lowered the headline score.

And the prediction failed exactly where it was most specific. BM25 was supposed to own the identifier category — its home ground, the literal-name questions. It came last:

Configurationidentifierconceptualcross-document
dense — vector only0.7110.5980.721
hybrid (RRF)0.6960.5830.589
lexical — BM25 only0.4260.4520.250

Gemini’s embeddings are good enough on this corpus that they win the one category keyword search was built to dominate. What I would have published without the answer key: “hybrid retrieval, with reranking.” It would have been slower, more complex, and worse — and it would have looked like progress.

One honest boundary on the result: this is a finding about this corpus — 708 well-structured pieces of ADRs, devlogs and typed Python, embedded by a frontier model. A larger, messier, more jargon-dense corpus is exactly where keyword search earns its keep. The number is a baseline, not a law.

ADR-0009
Fuse on rank, not on score

Cosine similarity lives between −1 and 1; BM25 is unbounded and routinely reaches 20. Adding them requires normalising first — and every normalisation is a knob that can be turned, consciously or not, until the system you were hoping to promote comes out ahead.

Reciprocal Rank Fusion uses rank position only. It never sees the scores, and its one published constant was not tuned against the answer key. The score-normalising alternative shipped too, as its own measured row — and on conceptual questions it was the best fusion of all, which is why it stays.

ADR-0010
The LLM reranker works, and it is still not the default

It is the best hybrid row, and it holds the best MRR of any configuration — the first genuinely useful answer is nearly always rank one.

It also costs 9.3 seconds per query against 783 milliseconds for the free heuristic, and it still loses to plain vector search. So it stays behind a flag, and a free deterministic reranker stays the default.

Trigger to revisit: a purpose-built reranker (Cohere or Voyage) that beats vector search’s 0.670 at a latency the serving path can afford. It becomes the default the day it exists.

What reality found — a reranker that had never run
Two rows agreed to three decimals, and that was the tell

The LLM-reranked configuration came back byte-identical to the un-reranked one — the same score to three decimals, in every category. That is impossible if a reranker is doing anything, so it wasn’t. Every model call was failing, and the reranker was falling back to the fused order exactly as designed.

Three bugs, stacked, and none was visible to a test suite that fakes the network: the grading schema used a shape the provider rejects outright; the default model id had gone stale to a 404 between writing the plan and running it; and once those were fixed the answer was being truncated mid-sentence on 25 of 26 questions, which reads to the parser exactly like a broken reply and degrades just as quietly.

The degrade path was correct behaviour — an outage should cost a slightly worse ranking, not an error — and it was also an observability hole: a completely broken reranker produced entirely plausible numbers. The only thing that caught it was two rows agreeing too well. The eval now counts every silent degrade and prints the total on every run, so the next one is loud.

Day 8 ended at 410 tests, 708 pieces, six configurations measured — and one prediction, published wrong.

Day 09 · AST graph, reconciliation, click-to-focus

The graph that checks its own claims

Retrieval, from Day 7 and Day 8, answers what does this code say? It has no answer for what breaks if campaign_service changes? — a question about structure, not meaning. So Day 9 stops searching the text and starts reading the shape underneath it.

A walker built on Python’s own ast module reads every service’s source and produces a graph: a node for every service, endpoint, Pydantic model and external dependency, and an edge for every relationship between them — a service exposes an endpoint, defines a model, imports a module, depends_on a package. One edge kind is different from the rest: calls. It is the graph noticing that one service’s HTTP client is pointed at a URL that is supposed to be a live route somewhere else — a claim, not yet a fact.

Reconciliation: the graph auditing itself

So it checks. Reconciliation walks every calls edge and confirms the route it names is an endpoint node that actually exists in the graph, rather than a string that merely looks like one. A call pointed at a route that was renamed, removed, or never existed comes back dangling, printed by name — not buried in a diff nobody reads.

$ python -m bellwether.context.graph --reconcile

7/7 call paths resolved (100%)
isolated: platform

Today: 45 nodes, 74 edges — 25 endpoints, 9 services, 8 external dependencies, 3 models — and among the edges, 4 calls edges covering 7 call paths, all 7 resolved, nothing dangling. One name comes back isolated: platform, the Level 0 gate script from Day 5. It is a command-line entry point, not a module any service imports or calls into, so the extractor correctly finds nothing wired to it. Not a bug — a script and a service are different shapes, and the graph tells the two apart instead of guessing.

The substrate, extracted from the AST — --diagram --scope substrate
graph LR
  subgraph n_service_ad_decision_service_ef0d50ac["ad_decision_service"]
    n_endpoint_ad_decision_service_GET__health_f0a25609(["GET /health"])
    n_endpoint_ad_decision_service_GET__metrics_a43ef786(["GET /metrics"])
    n_endpoint_ad_decision_service_POST__ad_request_41a28ab8(["POST /ad-request"])
  end
  subgraph n_service_campaign_service_e8506251["campaign_service"]
    n_endpoint_campaign_service_DELETE__campaigns____d07f140b(["DELETE /campaigns/{}"])
    n_endpoint_campaign_service_GET__campaigns_caa4df2c(["GET /campaigns"])
    n_endpoint_campaign_service_GET__campaigns____71df29ed(["GET /campaigns/{}"])
    n_endpoint_campaign_service_GET__campaigns____creatives_c2d77467(["GET /campaigns/{}/creatives"])
    n_endpoint_campaign_service_GET__health_1e1ac476(["GET /health"])
    n_endpoint_campaign_service_GET__metrics_e482f5e5(["GET /metrics"])
    n_endpoint_campaign_service_PATCH__campaigns____069be044(["PATCH /campaigns/{}"])
    n_endpoint_campaign_service_POST__campaigns_6d2a9e35(["POST /campaigns"])
    n_endpoint_campaign_service_POST__campaigns____creatives_58e965f9(["POST /campaigns/{}/creatives"])
    n_model_campaign_service_Campaign_c979140a["Campaign"]
    n_model_campaign_service_Creative_7378759d["Creative"]
  end
  subgraph n_service_event_service_de0f72ab["event_service"]
    n_endpoint_event_service_GET__campaigns____delivery_5d703263(["GET /campaigns/{}/delivery"])
    n_endpoint_event_service_GET__delivery_420b8d39(["GET /delivery"])
    n_endpoint_event_service_GET__events_0513aa23(["GET /events"])
    n_endpoint_event_service_GET__health_cec6f042(["GET /health"])
    n_endpoint_event_service_GET__metrics_9d97db47(["GET /metrics"])
    n_endpoint_event_service_POST__events_6b7ade97(["POST /events"])
    n_model_event_service_AdEvent_b8129f81["AdEvent"]
  end
  subgraph n_service_shared_ac61e27b["shared"]
  end
  subgraph n_service_traffic_simulator_a4adaf7e["traffic_simulator"]
    n_endpoint_traffic_simulator_GET__health_647c3354(["GET /health"])
    n_endpoint_traffic_simulator_GET__metrics_ca76d008(["GET /metrics"])
    n_endpoint_traffic_simulator_GET__scenarios_b1a8a38a(["GET /scenarios"])
    n_endpoint_traffic_simulator_GET__status_5d0751d8(["GET /status"])
    n_endpoint_traffic_simulator_POST__control_f34a452b(["POST /control"])
    n_endpoint_traffic_simulator_POST__scenario_739f882d(["POST /scenario"])
    n_endpoint_traffic_simulator_POST__seed_e15bea1a(["POST /seed"])
  end
  n_service_ad_decision_service_ef0d50ac -->|"GET /campaigns"| n_service_campaign_service_e8506251
  n_service_ad_decision_service_ef0d50ac --> n_service_shared_ac61e27b
  n_service_campaign_service_e8506251 --> n_service_shared_ac61e27b
  n_service_event_service_de0f72ab --> n_service_shared_ac61e27b
  n_service_traffic_simulator_a4adaf7e -->|"POST /ad-request"| n_service_ad_decision_service_ef0d50ac
  n_service_traffic_simulator_a4adaf7e -->|"GET /campaigns, PATCH /campaigns/{}, POST /campaigns, POST /campaigns/{}/creatives"| n_service_campaign_service_e8506251
  n_service_traffic_simulator_a4adaf7e -->|"POST /events"| n_service_event_service_de0f72ab
  n_service_traffic_simulator_a4adaf7e --> n_service_shared_ac61e27b
        
Five services, their endpoints and models, and the calls between them — generated straight from --diagram --scope substrate, not redrawn by hand. Click a service below to isolate what it depends on and what depends on it.

7 / 7 call paths resolved — no drift

Why the diagram shows five services, not nine

The adjacency behind the buttons above is the full graph — it also reaches bellwether.context, bellwether.eval and bellwether.llm, the AI layer being built on top of this substrate, plus the still-isolated platform script. The diagram itself is deliberately scoped narrower, to --scope substrate: the four services and the shared package that actually run in production. Conflating “what serves an ad” with “what builds the thing that reads the ads platform” would answer a different question than this chapter is asking. The focus buttons below only render for the services this diagram actually draws — a button pointing at nothing on screen would be worse than no button.

Day 9 ends at 45 nodes, 74 edges, 7 of 7 call paths reconciled — and 448 tests passing, 38 of them new.

Closing

Where it stands today

There is a working ads platform — five services, real traffic, real dashboards, and incidents that can be caused on demand and are genuinely real when they happen. On top of it, the beginnings of an AI layer that has read the entire project and can find things in it by meaning.

flowchart TB
    subgraph L1["LEVEL 1 — context layer (in progress)"]
        COR["73 documents<br/>708 searchable pieces"]
        VEC["Vector store<br/>4 engines, 1 record each"]
        RET["Hybrid retrieval<br/>6 configurations, one answer key"]
        GR["Architecture graph<br/>45 nodes · 74 edges · 7/7 reconciled"]
    end
    subgraph L0["LEVEL 0 — the substrate (complete, 11/11)"]
        CS["campaign-service<br/>:8001"]
        ADS["ad-decision-service<br/>:8002"]
        EV["event-service<br/>:8003"]
        SIM["traffic-simulator<br/>:8004"]
        OBS["Prometheus + Grafana"]
    end
    SIM --> ADS --> CS
    SIM --> EV
    L0 -->|"source · docs · contracts · config"| COR
    COR --> VEC --> RET --> GR
    GR -.->|"Day 10"| NEXT["the console<br/>MCP server"]
    NEXT -.->|"Levels 2-4"| AG["dev agents · ops agents<br/>orchestration"]
      

The setups, and where they pay off

PlantedWhat it wasPaid off
Day 1Four quality gates, before any product codeDay 11 — the fence AI-generated code must clear
Day 2Structured JSON logs, one line per requestDay 16 — log agents read them with no parsing
Day 2"Never a bare 500" written down as a standardDay 5 — the violation was identifiable, not just unlucky
Day 3Every rejection carries a name, not a booleanDay 4 chart → Day 5 diagnosis → Day 18 evidence
Day 3Reading the API, never the tablesDay 4 — kept ADR-0002's trigger unfired
Day 5Failures change real configurationDay 18 — an RCA agent with a real cause to find
Day 6Content fingerprints on every documentDay 7 — re-embedding costs nothing when nothing changed
Day 7The engine is a parameter, not a commitmentDay 8 — six retrieval configurations behind one command
Day 8An adversarial answer key, graded before any system ranDay 9 on — every retrieval change is an experiment, not an argument
Day 9Every service call reconciled against a route that provably existsDay 10 — the console and the MCP server query a graph that already audits itself

The pattern worth noticing

Reading my own nine days back to back, the same shape repeats. I make a decision and write it down with the condition that would reverse it. Then reality tests it — and every single time, running the real thing found something my plan had not:

  • A metric collision that only appeared when two services loaded together
  • A dashboard connection identifier that changed on restart, silently
  • A $2,147 budget ceiling nobody knew existed
  • A quality gate that measured itself and passed
  • Line endings that would have made every fingerprint meaningless
  • A database that quietly deleted data while every test agreed it was fine
  • A reranker that had never once run, hidden behind a degrade path working exactly as designed

I found none of these by thinking carefully. I found every one of them by running the real thing under real conditions and looking hard at what came back.

Which is, in the end, the whole argument I'm making. An AI that writes code quickly is only useful if something can prove the code is right. That proof cannot come from a fixture, a mock, or a demo that worked once on camera. It has to come from the real system, misbehaving in real ways, measured by numbers I publish whether they flatter me or not.

Twenty-one days remain, and this page grows with them. Level 1 finishes with a console and an MCP server that let any AI client query this project's knowledge — the corpus, the retrieval index, and now the graph — directly, each measured against an answer key like the one Day 8 built. Then the agents arrive, and every one of them gets a number next to it.