Reel one
Setting the stage
Before a single line of the product exists, three things have to be settled: what problem this is actually solving, how the work will be done every single day, and what it will be built with. Everything afterwards is consequences.
Day 00 · The problem
Where this started
Netflix has been the company I wanted to work for since well before I could have explained what a distributed system was. Not because of the product — because of how openly they think in public. Chaos engineering, the microservice patterns half the industry now copies, the causal-inference work, the agentic tooling they have started writing about: most of it was published while it was still being figured out, with the trade-offs left in rather than tidied away.
That is rare. Most companies publish the polished version, years later, once nothing is at stake. Reading Netflix work through a problem in public taught me more than most of the courses I have paid for.
So: dream company. Which is a pleasant sentence and completely useless on its own.
Wanting to work somewhere is not a plan
What I actually needed was a way to build something real rather than something aspirational. Which meant answering a question I had never thought to ask properly: how do you find out what a company is about to build, before they build it?
The answer turned out to be sitting in the open. A company's engineering blog and its job listings are the two most honest public statements it makes about its own direction.
- The blog tells you what they have already solved — and, more usefully, how they reason. What they measure. What they will accept as evidence.
- The listings tell you what they are staffing right now — which is to say, what somebody has already decided is important enough to spend headcount on, before a line of it ships.
So I read both, properly, for weeks. The blog gave me the house style. The listings gave me the target.
One posting stopped me: requisition AJRT30201 — Staff AI Engineer, AI Foundation & Tooling, Ads Platform. I started pulling it apart, and the more I dug the more it read less like a wish list and more like a roadmap somebody had already committed to. Strip out the boilerplate and it describes one specific, current problem:
Engineering teams can now produce code far faster than they can verify it.
That sentence is easy to nod along to and easy to underestimate, so let me show you what it actually means.
What "plausible but wrong" actually looks like
Here is a function from my own domain code. An AI could write it in a second, I could approve it in five, and it is wrong.
# Has this campaign spent its daily budget? def within_budget(campaign, spent_micros): return spent_micros < campaign.daily_budget_micros
Read it again. It compiles. It passes a strict type check. It reads exactly like the comment above it. A test asserting "a campaign that has spent nothing is within budget" passes. So does "a campaign that has spent everything is not."
It is still wrong: it asks whether the campaign has already overspent, not whether serving this advert would cause it to overspend.
daily_budget = 10,000,000 # $10.00 already_spent = 9,999,000 # $9.999 this_ad_costs = 500,000 # $0.50 within_budget(...) → True # because 9,999,000 < 10,000,000 # so it serves, and the day ends at 10,499,000 → every campaign overspends its daily cap, every day, forever.
The correct version is one clause longer:
return spent_micros + price_micros <= campaign.daily_budget_micros
Nobody is ever paged for this. There is no crash, no error rate, no red panel on a dashboard. Advertisers are quietly billed slightly more than they agreed to, for months. It is found by an auditor, not by an alert.
Code that is obviously broken is cheap. It fails loudly and immediately, and someone fixes it before lunch.
Code that is plausibly broken is expensive. It survives review because it reads correctly. It survives testing because the tests were written from the same misunderstanding that produced it. And AI generates it at a rate no review process was designed to absorb.
Speed went up. Confidence did not. That gap is what the posting is actually about.
The five capabilities the posting asks for
They are not independent. Each depends on the one before it — which is why the build order in this project is not negotiable.
| # | Capability | What it means concretely |
|---|---|---|
| 1 | Centralised context layer | Agents answer from this team's code and decisions, not generic training data. Ask why the project uses Compose and it should quote ADR-0001 and name its trigger — not recite blog posts. |
| 2 | Dev lifecycle agents | Generate code, generate tests, review pull requests against written standards, validate a deployment after it lands. |
| 3 | Ops agents | Read logs and metrics during an incident, correlate across services, propose a root cause with cited evidence, suggest the remediation. |
| 4 | Multi-agent orchestration | Several agents on one task at once — coordinating, disagreeing, and stopping to ask a human at the moments that matter. |
| 5 | Evaluation frameworks | Proving any of the above works. Numbers, not impressions. |
Item five is where nearly every portfolio project quietly gives up — and it is the only one that makes the other four trustworthy.
A demo and an evaluation are not the same thing
The words get used interchangeably. They should not be.
| A demo | An evaluation |
|---|---|
| One input, chosen by the person demoing | A fixed dataset, written before the agent existed |
| Run until it looks good, then recorded | Run once per change, results kept whatever they are |
| Success is "the audience was impressed" | Success is a number with a target beside it |
| Failures are re-shot | Failures are published |
| Proves the example was friendly | Proves something about the agent |
Concretely, for the pull-request reviewer arriving on Day 13, the evaluation is this:
# Take working code. Plant 10 realistic bugs, one per branch. # Each is the kind a tired human writes at 5pm on a Friday. 1. off-by-one in a frequency cap # < where it should be <= 2. budget check ignores the pending ad # the bug shown above 3. missing guard when targeting is empty ... # Ask the reviewer agent to review each branch, cold, with no hint # that a bug exists. Score = caught, minus false alarms invented. catch rate: 7/10 — published, including the run that scored 4
That number can go down when the code changes. That is precisely why it is worth having.
Every capability ships with a number
Not "here is an agent that reviews pull requests."
Instead: "here is an agent that caught 7 of 10 deliberately planted bugs, here is the evaluation that measured it, here is the run where it only scored 4, and here is what changed after."
Harder to build, far harder to fake, and the only version of this project a Staff engineer would find interesting rather than familiar.
Why I'm building it in public, with the failures left in
The honest version: I want this job, and I know what a portfolio project usually looks like from the other side of the desk. A repository with a confident README, a demo video where everything works first time, and no way for a reviewer to tell whether any of it survives contact with reality.
I did not want to send that. Anyone can claim they build agentic systems. Almost nobody publishes the number that would prove it, because the number is often disappointing.
So I am building the version that can be checked. In the open, one day at a time, with the mistakes left where they happened. Every figure on this page came from a command I actually ran. Every amber box is something I got wrong and then wrote down — including the day I broke a rule I had written myself, four days earlier, about that exact mistake.
If the agents turn out worse than I hoped, that will be on this page too. It is the only thing that makes the good numbers worth anything.
The call I had to make before Day 1 existed
With a plan sketched out, one genuinely hard decision was left — and getting it wrong would have cost me the entire month.
The job is the AI layer. It is not ad serving. Nobody is hiring me to write a frequency capper. So which do I build first?
| Start with the AI layer — rejected | Start with the ads platform — chosen ✓ |
|---|---|
| Day 1 is on the actual subject of the job | Day 1 is on something nobody is hiring me for |
| Something impressive to show by the end of week one | Five days — a sixth of the entire budget — before the AI appears at all |
| Agents have nothing real to operate on, so their inputs must be invented | Agents operate on a system that genuinely runs, genuinely logs, and genuinely breaks |
| Every evaluation scores the agent against a fixture I wrote myself | Every evaluation scores the agent against something that behaves independently of me |
The top two rows are about convenience — what is quicker, what looks good sooner. The bottom two are about whether any of it means anything, and that is where the argument actually gets settled.
I chose the ads platform.
Play the other choice forward and picture Day 20. I am recording a video where an ops agent diagnoses an outage. Where did the outage come from? A JSON file of log lines I wrote by hand, describing a failure I invented, in a system that was never running. The agent finds the cause — because I put it there, in the shape I expected it to be found in.
What has that measured? My ability to write a fixture. Any viewer with production experience sees straight through it, and worse, I would never learn whether the thing works.
So I paid five days up front. It cost me a week of build videos in which the AI barely appears, and it is the reason every number later on this page is worth reading.
Which is to say: to show an agent an outage, you need an outage
Most demos fake it. In code, the fake usually looks about like this:
# The staged incident if os.environ.get("CHAOS") == "true": return {"error": "database unavailable"}
Now point an agent at that and ask for a root cause. What is there to find? The only true answer is "an environment variable is set." The agent has learned nothing about databases, nothing about this system, and nothing that transfers to a real incident.
An agent tested against a fake failure can only ever find the fake. The evaluation measures the fixture, not the agent.
Compare what I do on Day 5. My simulator sends genuine PATCH requests rewriting every campaign's targeting to a country with no viewers. The decision service reads that configuration and correctly declines to serve. Fill rate drops from 43% to 2%.
Every part of that is real. The cause is a row in a table an agent can query. The symptom is a metric an agent can chart. The fix — restore the previous targeting — is a real operation with real consequences. That is something worth evaluating an agent against.
So the project is two systems, and the unglamorous one goes first
One system is the point. The other exists only so the first has something real to work on.
flowchart LR
SUB["SUBSTRATE<br/>a small ads platform<br/>•<br/>4 services · real traffic<br/>real logs · real incidents"]
BW["BELLWETHER<br/>the AI layer — the point<br/>•<br/>context · dev agents · ops agents<br/>orchestrator · evals"]
SUB ==>|"its code, logs, metrics<br/>and real failures"| BW
BW -.->|"operates on"| SUB
classDef sub fill:#241416,stroke:#b8321f,stroke-width:1px,color:#f3d9d6;
classDef ai fill:#12211a,stroke:#2f6f4f,stroke-width:1px,color:#d6ece0;
class SUB sub;
class BW ai;
Why an ads platform specifically
The substrate could have been anything. I chose ad serving for four reasons, none of them "it sounded impressive":
- It is the target team's actual domain. Using the posting's own vocabulary — targeting, frequency capping, brand safety, pacing — in code, logs and docs means every conversation lands in the right register.
- Its failures are configuration failures. The realistic outage in ad tech is not a severed cable, it is a bad targeting deploy. Those have a cause sitting in a database — exactly the shape an RCA agent can be honestly evaluated against.
- It produces genuine telemetry under load. Fill rates, rejection reasons, latency percentiles, spend. Real numbers that move for real reasons.
- It is small enough to finish. Four services, one machine. The substrate is scaffolding, and scaffolding that takes three weeks has eaten the project.
How "mini" is mini? An honest accounting
I want to be precise about this, because "I built a Netflix-style ads platform" is the kind of claim that deserves immediate suspicion.
What I built is realistic in shape and tiny in scale, and those are very different things. Here is the gap, stated plainly:
| Capability | Mine | A real one | Does the AI layer care? |
|---|---|---|---|
| Campaigns, creatives, budgets | yes | plus contracts, insertion orders, deal terms | yes — agents read config |
| Targeting | country, device, rating | hundreds of dimensions, audience segments | shape, not breadth |
| Frequency capping | per member, per day | cross-device, multi-window, identity graph | yes |
| Brand safety | category exclusions | ML classification, third-party verification | shape, not accuracy |
| Budget pacing | even daily spend | predictive, forecast-driven | yes |
| Ad selection | most budget remaining | real-time auction, bidding, CTR prediction | not yet — see below |
| Event ingestion | HTTP, idempotent | streaming pipeline, billions per day | shape, not throughput |
| Observability | Prometheus + Grafana | same, plus tracing and anomaly detection | yes — this is the input |
| Request volume | ~2 per second | millions per second | no — throughput teaches it nothing |
| Forecasting & inventory | absent | core to the business | future |
| Identity resolution | absent | core to the business | future |
| Measurement & attribution | absent | core to the business | no |
Read the right-hand column and the logic of every omission becomes visible. I built exactly the parts the AI layer has to reason about, and skipped the parts it does not.
Scale is the clearest example. Going from two requests a second to two million would be weeks of work and would teach the agents nothing new — a root-cause agent reads the same rejection reason off the same metric either way. Throughput is an interesting engineering problem and an irrelevant one here.
What I did not compromise on is the part that matters: every rejection carries a named reason, every request leaves a structured log line, every failure has a cause sitting in a real database, and every number comes from a real run. Those are the properties an agent operates on. A platform a thousand times larger with none of them would be useless to this project.
A real auction. Right now the winner is whichever eligible campaign has the most daily budget left — a rule, not a market. Replacing it with second-price bidding against a predicted click-through rate would give the ops agents a genuinely harder question: "fill rate is fine but revenue per thousand impressions dropped 8% — why?" That is the sort of incident real ads teams actually fight, and my current substrate cannot produce it.
A forecasting service. Inventory prediction would introduce a component that is wrong on purpose — models drift — which is a different and more interesting failure class than a bad config deploy.
More realistic targeting breadth. Not for realism's sake, but because a decision path with thirty dimensions is much harder to explain than one with three — and explaining decisions is precisely what Level 3's agents are graded on.
None of these are on the thirty-day plan. All of them make the substrate a harder and more honest test, which is the only reason they are on a list at all.
The thirty days, in six levels
| Level | Days | What gets built | Its gate |
|---|---|---|---|
| 0 | 1–5 | The substrate — four services, observability, injectable failures | 11/11 ✓ |
| 1 | 6–10 | Context layer — ingestion, chunking, retrieval, knowledge graph | relevance >85% |
| 2 | 11–15 | Dev agents — code gen, test gen, PR review, deploy validation | catch rate published |
| 3 | 16–20 | Ops agents — log intelligence, triage, RCA, guided resolution | RCA correct >80% |
| 4 | 21–25 | Orchestration — parallel agents, conflict resolution, human gates | 3/3 complex tasks |
| 5 | 26–30 | Platform — dev environment, CI gates, eval dashboard, launch | <10 min setup |
I wrote every target before building its level. That ordering matters more than it looks: a target I invent after seeing the result is not a target, it is a description.
Days 1–5 I build the substrate. From Day 6 I start on the AI layer itself.
The method
How a single day actually runs
This is the loop I run every day. None of it is ceremony — every step is there because skipping it cost me something specific, and most of those costs are named later on this page.
flowchart LR
A["1 · Read<br/>spec + yesterday's<br/>execution notes"] --> B["2 · Write the plan<br/>into the repo,<br/>task by task"]
B --> C["3 · Per task:<br/>test first, watch it fail,<br/>then implement"]
C --> D["4 · Four gates,<br/>run separately"]
D -->|"any fail"| C
D -->|"all pass"| E["5 · Commit<br/>conventional message"]
E --> F["6 · Devlog + running doc<br/>+ execution notes"]
F -->|"tomorrow"| A
Step 1 · The plan is a file in the repository
Before I write any code, I commit a written plan to docs/superpowers/plans/. Not a to-do list. A document naming every file that will exist, the exact function signatures, and the tests I will write first.
Here is the shape of a real task from the Day 7 plan:
### Task 4: The embedder protocol, cost tracking, and the hashing engine Files: create embedders/base.py, embedders/hashing.py + tests Produces: UsageRecord — engine, texts, tokens, cost_usd, latency_ms EngineSpec — name, label, dimensions, hosted, cost_per_million_tokens Embedder — Protocol: spec, available() -> (bool, str), embed(texts) Tests: determinism; unit norm; shared vocabulary scores higher than unrelated text; cost_usd == 0; registry lists all four names - [ ] Write failing tests → run → implement → run → commit
I started writing plans into the repository because I kept discovering, four files into a task, that I had chosen the wrong shape — and by then the wrong shape was load-bearing. It forces me to think the design through while changing it is still free — renaming a function in a plan costs nothing, renaming it across nine files costs me an hour. And it means a fresh session, or someone else entirely, can pick the work up without re-deriving anything I already worked out.
The plans are also, deliberately, part of the corpus. From Day 6 onward the AI layer can read how each day was planned, including what the plan got wrong.
Step 2 · Test first — and watch it fail
Every task I write follows five steps. Here is a real one, start to finish.
2a. Write the test before the code exists. This is from Day 4's event-service — the test that pins down idempotent counting:
def test_the_same_event_reported_twice_is_counted_once(): first = post_event(event_id="evt-123", event_type="impression") second = post_event(event_id="evt-123", event_type="impression") assert first.status_code == 201 assert first.json()["status"] == "recorded" assert second.status_code == 200 assert second.json()["status"] == "duplicate" # the point of the whole day assert get_delivery()["impressions"] == 1
2b. Run it, and confirm it fails for the right reason.
$ uv run pytest tests/substrate/event_service -q E ModuleNotFoundError: No module named 'substrate.event_service.main' 1 error during collection
This is the step I am most tempted to skip. It feels like theatre — I already know it will fail, so why spend the ten seconds watching it?
I do it anyway, every time, because running the test before the code exists is the only proof that the test is capable of failing at all.
A test that has never been seen failing might be asserting nothing at all — a typo in a fixture name, an assertion inside a branch that never runs, a comparison that is always true. A test that always passes is worse than no test, because it produces confidence without evidence, and it will still be passing years later while the behaviour it claims to protect has quietly rotted.
2c. Write the minimum code to pass. Not the elegant version, not the general version — the one that makes this test go green.
2d. Run it again.
$ uv run pytest tests/substrate/event_service -q
............ 12 passed in 0.61s
2e. Commit. Then the next task starts from a green repository, which means any breakage in the next hour was introduced in the next hour.
This discipline is not only for the human. From Level 2, agents generate code directly into this repository.
The tests and the gates below become the fence that generated code has to clear before it lands. Build the fence before you need it, and "the AI wrote it" stops being a reason to lower the bar. Build it after, and there is no bar to lower it from.
Step 3 · Four gates, run separately, never piped
| Gate | What it checks | A real thing it caught |
|---|---|---|
| ruff check | Unused imports, risky patterns, style drift | A helper typed Any instead of a concrete dict, three separate days |
| ruff format | Every file laid out identically | Keeps diffs about meaning rather than whitespace |
| mypy --strict | Every value is the type it claims | A function returning an untyped value from json.loads |
| pytest | The behaviour still holds | A metric collision that only appears when two services load together |
I run them as four separate commands on four separate lines. That sounds pedantic until you see what happened when I did not:
# WRONG — reports success even when mypy fails. # In a pipeline the surviving exit code is tail's, and tail always succeeds. uv run mypy tests substrate platform bellwether | tail -2 && git commit # RIGHT — each gate's own exit code is visible and fatal. uv run ruff check . uv run ruff format --check . uv run mypy tests substrate platform bellwether uv run pytest
This exact mistake shipped nine type errors on Day 7. My own Day 4 notes warn against it. I wrote both.
Step 4 · The tests must pass with everything switched off
A rule I hold myself to: the entire suite passes with Docker stopped and no API keys set. 289 tests, about six seconds.
I built it this way for a selfish reason. A test suite that takes two minutes and needs four containers is a suite I will quietly stop running — not deliberately, just by drifting into running it less often, until the day it fails and I have no idea which of the last twenty changes did it. Six seconds is fast enough that I never have to decide.
This is enforced by design rather than willpower. Business logic never reaches out for what it needs — it is handed it. Databases, HTTP clients, clocks and random number generators all arrive as arguments:
# The decision logic has no I/O of its own. In production `store` is Redis # and `now` is the wall clock. In tests they are a dict and a fixed datetime. def decide(request, campaigns, store, now): ... # So a frequency-cap test needs no Redis, and no container: def test_a_member_at_their_cap_is_rejected(): store = {"freq:member-0042:camp-1:2026-07-22": 3} result = decide(request, [campaign_capped_at(3)], store, now=FIXED_TIME) assert result.no_fill_reason == "frequency_capped"
Two payoffs. Speed — a six-second feedback loop instead of a two-minute one, which changes how often I am willing to run it. And honesty — a test that needs a running database is partly testing the database, and will fail for reasons that have nothing to do with the code I just wrote.
Fast tests can all agree with each other and all be wrong
On Day 7 my stand-in for the vector database faithfully reproduced my own incorrect mental model of how the real one behaves. Every test passed. The real database was silently deleting my data on every write.
Substitutes verify my understanding of a system. They cannot verify the system. Anything that talks to real infrastructure needs at least one genuine round-trip before I trust the design — which is exactly how I eventually caught it.
Step 5 · Decisions get written down with an expiry condition
Every significant choice becomes an Architecture Decision Record with a fixed five-part shape. Here is ADR-0003, in full outline:
Context What was true when this was decided. Frequency counts are written on every impression, are per-viewer, and are worthless after midnight. Decision What was chosen. Redis holds decision state under day-scoped keys that expire themselves. Postgres stays the system of record. Alternatives What was rejected, and honestly why. Postgres for everything — rejected: these are counters with an expiry, not records; it would mean writing a cleanup job to delete yesterday's rows forever. Consequences What this now costs. Two stores to reason about. Redis can be lost without data loss, but the pacer briefly forgets what it spent. Trigger The specific condition that reverses this. Day 4's event-service, when impressions become durable and spend becomes billable.
I add the trigger because of something I have watched happen on every team I have worked on: a decision that was correct in 2023 is still being obeyed in 2026, and nobody can say why. The trigger is the part that makes this worth doing. Without it my decision hardens into folklore — "we don't do that here" — and nobody, including me, remembers whether the original reason still holds. With it, the decision carries its own review date.
And it fired. One day later, event-service made impressions durable. The trigger had named that exact event in advance, so the question was already on the table rather than being noticed six months later. The answer turned out to be narrow it rather than reverse it:
# ADR-0003, Day 4 update Postgres now holds the auditable record of what was served. Redis keeps only the hot counters the serving path reads. They now disagree by design — and the gap between them is "reporting loss", which became a panel on the dashboard.
A decision I could revisit on schedule turned into a feature. That only happened because I wrote down, a day in advance, the condition under which I would think about it again.
Step 6 · Commits say what changed and why
Every commit I make uses a conventional prefix and imperative mood, and has to leave the repository green — tests, lint and types all passing.
| Prefix | Means | A real one from this repo |
|---|---|---|
| feat: | New capability | feat: the chunk, its anchor, and the provenance it inherits |
| fix: | Corrects broken behaviour | fix: restore mypy to clean after the vector store landed |
| docs: | Documentation only | docs: ADR-0007, ADR-0008, day-07 devlog |
| test: | Tests only | test: cover the duplicate-event path |
| chore: / ci: | Tooling and pipelines | ci: type-check the platform directory too |
The message body carries the reasoning when it is not obvious. That fix: above reads, in full:
fix: restore mypy to clean after the vector store landed The previous commit passed its gates through a pipe, so mypy's non-zero exit was swallowed by tail and nine type errors went in green. Day-04 execution note 6, committed by the person who wrote it.
I write the reasoning into the commit because the commit is the only artefact that survives with the code. Writing that down cost me thirty seconds and makes my mistake findable forever. Hiding it would have cost nothing today and everything the third time I do it.
Step 7 · Reproduce the CI command exactly, before pushing
Running a superset of what CI runs is not the same as running what CI runs, and the difference hides an entire class of failure.
Day 7 made this concrete. My machine had an optional AI dependency installed; the build server would not. So before pushing, I deliberately stripped my environment back to match:
# Strip the optional group so the venv matches the build server exactly $ uv sync --group dev - model2vec==0.8.2 - numpy==2.4.6 # Then run the four CI commands verbatim $ uv run ruff check . All checks passed $ uv run ruff format --check . 105 files formatted $ uv run mypy tests substrate platform bellwether no issues in 105 files $ uv run pytest 289 passed
The test count went up by one without the optional package — a test that skips when the library is present and runs when it is absent. Both environments genuinely covered, and I would never have seen it without reproducing the real one.
Day 5, execution note 10: a test imported a module that resolved under pytest but not under the exact command CI runs. Caught by reproducing CI locally, not by CI — because catching it in CI means a red build, a context switch, and a fix commit.
The note ends: "Reproduce the CI command verbatim before every push; running a superset locally hides this class of failure."
Step 8 · The day ends by recording what the plan got wrong
Every plan I write ends with a section titled "Execution notes — what the plan missed." I write it after the work, listing everything reality disagreed with me about. Not a retrospective anyone schedules — a numbered list, in the plan file, committed.
A real one, from Day 7:
1. PUT /points in Qdrant replaces a point outright. The plan's two-call write destroys every previously written engine's vector, because the payload write is itself a full replace. Every unit test passed against the fake. Found only by pointing the store at a running Qdrant and asking whether the first vector survived. It did not. → Any plan that fakes a datastore must include one live round-trip before the design is trusted.
I write the next day's plan with those notes open. This is the single highest-leverage habit I have, and I only added it after Day 3 — because I genuinely do repeat the same mistake in disguise, and I would not have believed that about myself until I had the table below:
| Day | What happened | The shared root cause |
|---|---|---|
| 06 | Line endings differ between Windows and Linux, so identical files fingerprint differently | A text file written on Windows carries bytes you did not write, in line 1, invisibly |
| 07 | A byte-order mark turned VOYAGE_API_KEY into an unrecognisable name | |
| 04 | Warning written: a piped verification command hides its failure | A pipeline reports the exit code of its last command, not its first |
| 07 | The warning's own author did it, and shipped nine type errors green |
Step 9 · The definition of done, every day
My day is not finished when the code works. It is finished when all of this is true:
- Every task in the plan is implemented, test-first, and committed separately
- All four gates pass, run individually and unpiped, with Docker stopped
- The CI command has been reproduced verbatim in a matching environment
- Any significant choice has an ADR — with alternatives and a trigger
- The devlog is written: shipped, decisions, what running it found, what tomorrow does
- The running doc is updated — day tracker, and the scoreboard if a number was produced
- The execution notes are written, honestly, including the embarrassing ones
That last clause is doing real work. Every mistake on this page — the piped gate, the destroyed vectors, the gate that measured itself — I reported on myself. A build log that only contains successes is a marketing document, and nobody learns anything from it, least of all me.
The toolkit
Every tool, and what it beat
Nothing here was chosen because it is popular. Each one is doing a specific job, and the alternative it displaced is named.
Language and packaging
The language every AI and machine-learning library targets first. Choosing anything else means writing bridges to it later.
Why 3.11 specifically: it is the floor the project promises to support. That promise has teeth — on Day 7 a dependency shipped type definitions requiring 3.12 syntax, and it was pinned back rather than quietly raising the floor.
Installs dependencies and manages the virtual environment. Written in Rust, and roughly two orders of magnitude faster than the traditional tool.
Instead of pip + virtualenv + pip-tools: three tools replaced by one, with a single lockfile pinning exact versions so every machine and the CI server resolve identically. Day 7's optional AI dependencies installed in 903 milliseconds.
The web layer
The framework where the type annotations are the API contract. Declare what a request looks like as a typed class, and validation, error messages and machine-readable documentation all come free and stay in sync automatically.
Instead of Flask: Flask needs validation written by hand and documentation maintained separately, which drifts from the code within weeks. Instead of Django: far larger, opinionated about databases and templates, and most of it would go unused here.
Defines the shape of data at every boundary and rejects anything that does not fit, with a precise message naming the offending field. Its core is Rust, so this costs almost nothing.
Instead of hand-written checks: a rule expressed once as gt=0 becomes a runtime guarantee, a documented constraint, and a type the checker understands — three jobs from one declaration.
How services talk to each other, and later how the AI layer talks to Gemini, Voyage and the vector database.
Instead of vendor SDKs: each provider ships its own library with its own dependencies and release cadence. Three of them would be three ways to describe the same handful of HTTP calls. One HTTP client, already present, keeps the dependency list honest.
Storage — two kinds, on purpose
Holds anything that must survive a restart and must never be wrong: campaigns, creatives, and every delivered impression.
Why relational: the data genuinely is relational, and the guarantee that makes Day 4's counting correct — a unique key the database enforces — is exactly what relational databases are for.
Holds counts that are read on every single ad request and are worthless after midnight: how many times this viewer has seen this advert today, how much this campaign has spent today.
Why not just Postgres: these are not records, they are counters with an expiry date. Redis deletes them itself when their time is up. Storing them in Postgres would mean writing a cleanup job to delete yesterday's rows forever.
Maps Python classes to database tables, fully typed, so the type checker catches a mistyped column name before the code runs.
Instead of raw SQL: the class is both the code's model and the table's definition, so they cannot disagree. Tests run the same models against in-memory SQLite for speed.
Seeing what is happening
Each service publishes its current counters at a /metrics address. Prometheus visits every few seconds and records them, building a history you can ask questions of.
Why pull, not push: the services do not need to know Prometheus exists. Adding a fifth service means adding one line of Prometheus configuration, not changing the service.
Draws the graphs. Critically, its dashboards are defined as files in the repository, not arranged by clicking in a browser.
Instead of click-configured dashboards: a clicked dashboard exists only in one person's Grafana. A dashboard in version control is reviewable, restorable, and guarded by a test that fails the build if a panel loses its query.
Every log line is a JSON object, not a sentence. Written once as a shared library on Day 2 and reused by every service.
Instead of prose logs: "Decision took 4ms for member-0042" requires a fragile pattern-match to read. {"member_id": "member-0042", "latency_ms": 4} is already data. On Day 16 an agent reads these without a single regular expression — that is the entire reason.
Infrastructure and quality
Describes every service, its ports and its startup order in one file. docker compose up -d starts the entire platform.
Instead of Kubernetes: see ADR-0001 below. Briefly — Kubernetes solves problems this project does not have, at a cost measured in days.
One Rust binary replacing four traditional Python tools, fast enough to run on every file save.
Instead of flake8 + black + isort + pyupgrade: four tools with four configurations that disagree with each other, replaced by one that is roughly a hundred times faster.
Verifies that every value is what the code claims. Strict mode additionally refuses to let anything be left untyped.
Instead of default mode: normal type checking is advisory and silently ignores untyped code, so coverage erodes. Strict mode is uncomfortable for two days and then prevents a steady stream of bugs — and it is what makes AI-generated code safe to accept.
Plain functions with plain assertions, plus fixtures for shared setup.
Instead of unittest: far less ceremony, and the fixture system is what makes dependency substitution clean enough that the hermetic rule is actually livable.
The AI layer adds three tools of its own — a vector database and two embedding models. They arrive in Reel three, alongside Day 7, where the ideas they depend on are explained first.
Reel two · Days 1–5
Building the world
Five days to build a small ads platform that behaves like a real one: it serves adverts, records what it delivered, shows you what it is doing, and can be broken on demand in ways that are genuinely broken.
I call this half the substrate — the layer underneath the AI: real enough to serve, log and break, so everything built on top of it is measured against something true.
First the map — the finished architecture in one reference chapter. Then the five days that built it, one at a time.
Reference · the substrate in full
The architecture, in depth
Reel two tells the build one day at a time. This chapter is the map you read it with — the finished platform as a whole, in five short passes:
- The shape — how the two halves connect, in one diagram
- The parts — every service and store, one line each
- One real request — traced end to end, no invented values
- The data — what is stored where, and what is measured
- One real failure — injected, measured, recovered
Skim it for the shape now; come back when a later day zooms in on one part.
flowchart TB
subgraph SUB[" SUBSTRATE — a small ads platform, built Days 1–5 "]
direction LR
SIM["traffic-simulator<br/>seeded load + injectable failures"]
ADS["ad-decision-service<br/>6-filter serving path"]
CS["campaign-service<br/>campaigns, budgets, targeting"]
EV["event-service<br/>impressions, clicks, spend"]
RED[("Redis<br/>freq caps · pacing spend<br/>(hot, expires nightly)")]
PG[("Postgres<br/>system of record")]
OBS["Prometheus + Grafana<br/>metrics, dashboards, JSON logs"]
SIM ==>|"① ad request"| ADS
ADS -->|"reads active campaigns"| CS
ADS <-->|"caps + spend"| RED
ADS ==>|"② returns the ad"| SIM
SIM ==>|"③ reports impression / click"| EV
SIM -->|"seeds campaigns ·<br/>PATCHes them to inject a failure"| CS
CS --> PG
EV --> PG
SIM -.-> OBS
ADS -.-> OBS
CS -.-> OBS
EV -.-> OBS
end
STR1(["source, docs,<br/>API contracts, ADRs"])
STR2(["logs, metrics,<br/>dashboards"])
STR3(["real incidents<br/>with a cause in a table"])
SUB --> STR1
OBS --> STR2
SIM --> STR3
subgraph BW[" BELLWETHER — the AI layer, built Day 6 onward "]
direction TB
CTX["Context layer<br/>grounds every agent in team knowledge"]
DEV["Dev agents<br/>code, tests, PR review, deploy checks"]
OPS["Ops agents<br/>triage, log/metric correlation, RCA"]
ORCH["Orchestrator<br/>coordinates agents, escalates to a human"]
CTX --> DEV
CTX --> OPS
ORCH -.-> DEV
ORCH -.-> OPS
end
STR1 ==>|"ingested, chunked, embedded"| CTX
STR2 ==>|"parsed, correlated"| OPS
STR3 ==>|"the thing to diagnose"| OPS
EVAL{{"Eval harness<br/>a published score on every capability"}}
DEV -.-> EVAL
OPS -.-> EVAL
CTX -.-> EVAL
classDef sub fill:#241416,stroke:#b8321f,stroke-width:1px,color:#f3d9d6;
classDef store fill:#1a1116,stroke:#8f5a63,stroke-width:1px,color:#e6d2d6;
classDef ai fill:#12211a,stroke:#2f6f4f,stroke-width:1px,color:#d6ece0;
classDef stream fill:#1c1c24,stroke:#6b6b78,stroke-width:1px,color:#eaeaef;
classDef evalcls fill:#241f14,stroke:#e8a33d,stroke-width:1px,color:#f0d9ac;
class SIM,ADS,CS,EV,OBS sub;
class RED,PG store;
class CTX,DEV,OPS,ORCH ai;
class STR1,STR2,STR3 stream;
class EVAL evalcls;
One edge matters more than all the others: the simulator seeds the campaigns, and it injects failures by genuinely PATCHing them through campaign-service's public API (ADR-0005). Every incident an agent later diagnoses starts on that edge — a genuinely broken configuration, not a fixture.
The substrate then feeds the AI layer three things. Its source and docs become the context layer (Days 6–10). Its logs and metrics are what the ops agents read (Days 16–20). Its injected incidents are the thing to diagnose. The eval harness scores every capability and publishes the number, flattering or not.
What each part is, in one line
The substrate — the ads platform
Storage — two kinds, on purpose
BELLWETHER — the AI layer
Cross-cutting
The architecture I built is not the one I sketched — and the differences are the point
My Day-0 diagram was a clean serving-path sketch: request in, ad out, event reported. Building it changed the shape in three ways, and every change came from a requirement the sketch had not yet met.
Pacing spend moved into Redis. I had it as private logic inside the decision service; it had to become shared state, because a budget spent on one request has to be visible to the next.
The simulator gained an edge to campaign-service. This one is not on the original diagram at all. It is how a failure gets injected — a real PATCH to real configuration — and it is the single most important thing in the picture, because without it every ops-agent evaluation later would be graded against a fixture I wrote myself. I could not have drawn it on Day 0; the mechanism did not exist until Day 5.
Event totals are computed, never stored. The sketch said "aggregate", which sounds like a maintained running total. A running total is a second copy of the truth that drifts, so I compute on read instead.
The sketch was simpler. The built version is busier and correct. Simpler was not better here — it was hiding the control plane that turned out to hold the whole thesis up.
One real ad request, traced end to end
Now the territory. One ad request moves through the running system, against the three campaigns the platform seeds itself with on first start. Nothing below is invented — every number, field and reason string is what the code produces.
A viewer reaches an ad break. The simulator sends this to ad-decision-service:8002:
POST /ad-request { "member": { "member_id":"member-0042", "country":"US", "device_type":"tv" }, "slot": { "slot_id":"slot-118", "duration_seconds":30, "content_rating":"TV-14", "content_categories":["drama","comedy"] } }
The decision service pulls the active campaign set from campaign-service, then runs three candidates through six filters in a fixed order — cheapest and most-eliminating first. Each column is one of the real seed campaigns:
| Filter, in order | Acme Snacks US/CA · all devices · cap 3 · $60/day |
Northwind Motors US/CA/GB/DE · TV only · cap 2 · $65/day |
Vertex Hydration US/BR/GB · mobile/tablet · cap 5 · $50/day |
|---|---|---|---|
| 1 · eligibility active & in flight? | ✓ | ✓ | ✓ |
| 2 · targeting US · tv · TV-14? | ✓ | ✓ | ✗ tv ∉ {mobile, tablet} |
| 3 · brand safety excluded ∩ slot cats? | ✓ {news,true-crime} ∩ {drama,comedy} = ∅ | ✓ {true-crime} ∩ {drama,comedy} = ∅ | — |
| 4 · frequency cap seen today < cap? | ✓ 0 < 3 | ✓ 0 < 2 | — |
| 5 · pacing spent < daily budget? | ✓ $0 < $60 | ✓ $0 < $65 | — |
| 6 · creative fit a creative ≤ 30s? | ✓ 30s hero | ✓ 30s cutdown (60s cinematic too long) | — |
| verdict | eligible $60 remaining | eligible → wins $65 remaining | targeting_mismatch |
Two campaigns clear all six filters. The tiebreak is the one line of policy that matters most: the winner is the eligible campaign with the most daily budget remaining — max(eligible, key=budget_remaining) — not the first match. Northwind ($65 left) beats Acme ($60 left). Without that rule, whichever campaign sorted first would drain its whole day by mid-morning.
The response names the winner and — this is the part that pays off for the whole rest of the project — carries a trace of why every candidate landed where it did:
{
"request_id": "3f9c1a20-...",
"slot_id": "slot-118",
"filled": true,
"ad": {
"campaign_name": "Premium sedan, connected TV only",
"advertiser": "Northwind Motors",
"creative_name": "30s cutdown",
"asset_url": "https://cdn.example/sedan-30.mp4",
"duration_seconds": 30,
"price_micros": 2000 // $0.002 per impression
},
"no_fill_reason": null,
"candidates_considered": 3,
"trace": [
{ "campaign_name":"Premium sedan, connected TV only", "reason":"eligible" },
{ "campaign_name":"Wide-reach snack launch", "reason":"eligible" },
{ "campaign_name":"Sports drink, mobile takeover", "reason":"targeting_mismatch" }
],
"decision_latency_ms": 1.2
}
Two things happen the instant this fills, and they go to different stores for different reasons. The decision service writes the frequency count and pacing spend to Redis immediately, so the very next request for member-0042 sees the updated numbers. Separately, the simulator reports the impression to event-service:8003, carrying that same request_id, and it becomes one durable row in Postgres:
POST /events → event-service # the durable record { "event_id":"7c2...", "event_type":"impression", "request_id":"3f9c1a20-...", "campaign_id":"...", "member_id":"member-0042", "price_micros":2000 } → 201 recorded # meanwhile, already written to Redis at decision time: freq:member-0042:<campaign>:2026-07-23 = 1 # TTL 48h, self-expiring spend:<campaign>:2026-07-23 = 2000 # micros
That split is deliberate and it is ADR-0003 made concrete: Redis is what the pacer believed at decision time; Postgres is what was actually delivered. They can diverge — a lost Redis write, a dropped event — and the gap between them is not a bug, it is a number on the dashboard called reporting loss.
The data model, in real fields
Two tables carry the durable state. Money is Integer micros everywhere — never a float — and a caller-supplied event_id is the primary key that makes double-counting impossible.
campaigns (Postgres, behind campaign-service) id uuid PK name, advertiser str status str # draft | active | paused budget_micros int # total; 32-bit — capped at $2,147 (found the hard way) daily_budget_micros int frequency_cap_per_day int # default 3 targeting json # { countries, device_types, content_ratings } brand_safety_exclusions json # ["news", "true-crime", ...] starts_at, ends_at datetime ↳ creatives (name, duration_seconds, asset_url) ad_events (Postgres, behind event-service — append-only) event_id uuid PK # chosen by the caller — the whole idempotency trick event_type str # impression | click request_id uuid # ties the event back to the decision that caused it campaign_id, creative_id uuid member_id, slot_id str price_micros int # impression = 2000, click = 0 occurred_at datetime
What the platform measures
Prometheus scrapes every service every few seconds. These are the metrics that matter. The one that does the most work is ad_candidates_filtered_total — because Day 3 gave every rejection a name, a fill-rate collapse becomes a single chart where you read the cause off whichever band grew.
| Metric | Labels | The question it answers |
|---|---|---|
| ad_candidates_filtered_total | reason | Why did candidates lose? targeting_mismatch, pacing_throttled, frequency_capped… |
| ad_decisions_total | outcome | Fill rate — filled vs no-fill |
| ad_events_total | event_type | Impressions and clicks ingested → click-through rate |
| ad_events_duplicate_total | — | Retries the primary key rejected — dedup working |
| ad_spend_micros_total | — | Money actually delivered |
| http_request_duration_seconds | service | p95 latency — is it load, or a fault? |
| sim_scenario_info | scenario | Which failure mode is currently injected |
A failure, with the actual numbers
This is where the whole "real, not mocked" argument cashes out. To inject a bad config deploy, the simulator makes a genuine call to campaign-service — for every campaign:
PATCH /campaigns/{id} → campaign-service # repeated for all 3 { "targeting": { "countries":["AQ"], "device_types":[], "content_ratings":[] } } // AQ = Antarctica. No viewer is ever there.
Nothing is faked. The decision service reads the changed configuration and correctly declines to serve, because the targeting genuinely no longer matches anyone. On the running stack, the effect was measured:
fill rate 43% → 2% # within seconds of the PATCH loss reason the targeting_mismatch band takes over the "why candidates lost" chart
Recovery is equally real: the steady scenario PATCHes the original targeting back. It is not an undo flag — it is the same remediation an ops agent will be asked to recommend on Day 19. And a second failure mode, budget_runaway, is how the platform found a defect in itself on day one:
PATCH daily_budget_micros → 9,000,000,000 → 500 NumericValueOutOfRange # budget_micros is a 32-bit Integer, # capped at 2,147,483,647 micros = $2,147/day
A real ceiling nobody knew existed, surfaced by pointing real load at a real platform — and, because it first escaped as a bare 500 in violation of the project's own standard, now a typed 422. That is the entire thesis in one incident: a system that is genuinely running finds the things a plan cannot.
Day 01 · Foundation
The boring day that decides the next twenty-nine
Day 1 produces nothing a user could touch. It builds the workshop: how code gets checked, how the databases start, how anyone can tell whether things are working.
Skipping this is the most common way a 30-day build collapses around day nine.
What actually got built
- The repository, with the four quality gates wired to run on every change
- A Compose file starting Postgres, Redis, Prometheus and Grafana — all healthy on one command
- Continuous integration: the same four gates run on every push, on a clean machine
- The running doc — a live page with the architecture, a 30-day tracker, and an evaluation scoreboard
- ADR-0001
The scoreboard shipped with every number blank. Publishing empty cells on day one is a commitment: those cells are the entire point of the series, and they only get filled by something that actually ran.
Docker Compose, not Kubernetes
Kubernetes exists to run hundreds of services across many machines, surviving hardware failure and unpredictable traffic. This is four services on one laptop.
Adopting it would cost days of configuration and prove nothing about AI engineering — which is the actual subject. Compose starts everything with one command and remains readable by someone who has never seen the project.
Choosing infrastructure proportionate to the problem is the senior signal. Reaching for the largest available tool is the opposite, and experienced reviewers read it instantly.
Trigger to revisit: the first time this needs to run across more than one machine.
Postgres was published on port 5433 rather than the standard 5432, and Redis on 6380 rather than 6379.
Any other database already running on the machine would otherwise silently answer the connection, producing failures that look exactly like code bugs and take an hour to trace.
Day 02 · campaign-service
Where the advertisers live
The first real service. It stores campaigns — an advertiser's instructions about when, where and to whom their advert may be shown.
The vocabulary is deliberate. These are the exact terms the Netflix posting uses, and using them consistently in code, logs and documentation is what lets an agent later connect a question to an answer.
| Term | In plain English |
|---|---|
| targeting | Who may see this — which countries, device types, content ratings |
| frequency cap | The most times one viewer may see it in a single day |
| brand safety | Content this advertiser refuses to appear beside |
| pacing | Spreading a daily budget across the day instead of spending it by 9am |
| creative | The actual video file, with its duration |
| flight window | The dates between which the campaign runs |
flowchart LR
C["Campaign<br/>total budget · daily budget<br/>frequency cap · status<br/>flight window"]
T["Targeting<br/>countries<br/>device types<br/>content ratings"]
B["Brand safety<br/>excluded categories"]
CR["Creative<br/>name · asset URL<br/>duration seconds"]
C --> T
C --> B
C -->|"one campaign,<br/>many creatives"| CR
C --> PG[("PostgreSQL")]
Money is never a decimal
Budgets are stored as whole numbers of micros — millionths of a currency unit. Ten dollars is 10000000.
This looks awkward and prevents a genuine disaster. Computers cannot represent 0.1 exactly in the decimal format most languages default to:
# Python, and every other language using the same number format >>> 0.1 + 0.2 0.30000000000000004 >>> sum([0.1] * 10) 0.9999999999999999 # should be 1.0
Spread that across millions of impressions and the books stop balancing, in a way that is nearly impossible to trace afterwards. Whole numbers never drift.
Failures have a fixed shape
Every error — a malformed request, a campaign that does not exist, anything — returns the same envelope:
// A campaign that does not exist { "error": { "code": 404, "message": "Campaign 7f3a... not found" } } // A budget that exceeds what the column can hold { "error": { "code": 422, "message": "daily_budget_micros must be <= 2147483647" } }
A generic "500 Internal Server Error" tells the caller nothing and gives them nothing to act on. Every failure here is typed and explained.
This standard gets violated three days later, when a database range error escapes as a bare 500. Because the standard was written down, the violation was identifiable rather than merely unfortunate — and the fix generalised to every column, not just the one that failed.
Every request leaves a structured trace
{"service":"campaign-service", "endpoint":"POST /campaigns",
"status":201, "latency_ms":12.4, "campaign_id":"7f3a..."}
One line of JSON per request. Data, not prose.
Build the tables directly — no migration tooling, for now
Normally, changing a database's shape requires a migration: a versioned script so an existing database can be upgraded without losing its data. The standard tool is Alembic.
At this point there is one service writing, no data anyone depends on, and a shape that will change every single day through Day 5. Migrations written against something that churns daily are noise wearing the costume of safety.
Trigger to revisit: the first time a second service reads these tables.
Day 2 ended at 23 tests passing.
Day 03 · ad-decision-service
The twenty milliseconds that matter
This is the heart of any ads platform. A viewer reaches an ad break. Something must decide, in milliseconds, which advert to play — or that none is suitable.
// The request: who is watching, and what slot is available POST /ad-request { "member": { "member_id":"member-0042", "country":"US", "device_type":"tv" }, "slot": { "slot_id":"slot-118", "duration_seconds":30, "content_rating":"TV-14", "content_categories":["drama","comedy"] } }
Six filters, in a deliberate order
Every campaign starts as a candidate. Each filter eliminates some. The order is chosen so the cheapest, most eliminating checks run first — there is no point asking Redis about frequency caps for a campaign that expired last week.
flowchart TB
REQ["Ad request<br/>member + slot"] --> F1
F1["1 · Eligibility<br/>active, and inside its flight window?"] --> F2
F2["2 · Targeting<br/>right country, device, content rating?"] --> F3
F3["3 · Brand safety<br/>does this content offend the advertiser?"] --> F4
F4["4 · Frequency cap<br/>has this viewer already seen it enough?"] --> F5
F5["5 · Budget pacing<br/>has it spent its allowance for now?"] --> F6
F6["6 · Creative fit<br/>is there a video of the right length?"] --> WIN
WIN["Winner: the campaign with the<br/>most daily budget remaining"] --> OUT["Filled ✓"]
F1 -.->|"campaign_inactive"| NF["No-fill,<br/>with a named reason"]
F2 -.->|"targeting_mismatch"| NF
F3 -.->|"brand_safety_excluded"| NF
F4 -.->|"frequency_capped"| NF
F5 -.->|"pacing_throttled"| NF
F6 -.->|"no_creative_fit"| NF
I made the winner the campaign with the most daily budget remaining, not the first one that matches. Otherwise whichever campaign happens to sort first drains its entire day by mid-morning and everything else starves.
Named reasons, never booleans
A filter that returns true or false tells you nothing at 2am. Every rejection carries a name: pacing_throttled, targeting_mismatch, frequency_capped.
That single string becomes four things at once: a field in the API response, a field in the log line, a label on a Prometheus metric, and a coloured band on a Grafana chart.
// The response, when nothing could be served { "request_id": "44444444-...", "filled": false, "no_fill_reason": "frequency_capped", "candidates_considered": 3, "trace": [ {"campaign":"Wide-reach snack launch", "lost_to":"frequency_capped"}, {"campaign":"Premium sedan, CTV only", "lost_to":"targeting_mismatch"}, {"campaign":"Sports drink, mobile", "lost_to":"targeting_mismatch"} ], "decision_latency_ms": 1.2 }
Every response says which campaign lost, and to which rule.
Day 4: these names become a stacked chart called "why candidates lost". Day 5: a deliberately broken deployment is diagnosed by watching one band swallow the chart. Day 18: the root-cause agent cites this exact field as evidence.
None of that is possible if the filters had returned false.
Two kinds of memory
Frequency capping needs to know how many times this viewer has already seen this advert today. That count is written on every impression, is per-viewer, and is worthless after midnight.
Redis holds the decision state, Postgres holds the truth
A count that expires at midnight is a counter with a timer, not a permanent record. Redis holds these under keys scoped to the day that delete themselves after 48 hours.
Note the discipline: the decision service reads campaigns through campaign-service's public API, never its database tables. That is precisely what keeps ADR-0002's trigger — "the first time a second service reads these tables" — deliberately unfired. Two decisions, written a day apart, holding each other in place.
Trigger to revisit: Day 4's event-service, when impressions become permanent and spend becomes billable.
Two services, one global counter, instant collision
Prometheus keeps a single registry per running process. Both services defined a metric called http_requests_total, which was fine until the test suite imported both at once — then it crashed with a duplicate-metric error.
Fixed by defining the shared metrics exactly once in substrate/shared/observability.py, with a service label distinguishing them. The plan had not foreseen it; running the tests did, in about four seconds.
Day 3 ended at 51 tests passing, all with Docker stopped.
Day 04 · event-service + dashboards
Counting things exactly once
An advert played. That is an impression, and it costs the advertiser money. Someone clicked it. That is a click, and it costs nothing. Both must be recorded.
The hard part is not storing them. It is storing them exactly once.
Why counting is genuinely hard
Networks are unreliable. A player reports an impression, the confirmation gets lost in transit, the player reasonably retries. The same impression now arrives twice. Count both and the advertiser is billed twice.
The obvious fix is to check first: have I seen this before? No? Then save it. That fix is broken, and understanding why is worth thirty seconds:
sequenceDiagram
participant A as Report A
participant B as Report B, a retry
participant DB as Database
Note over A,B: same impression, same moment
A->>DB: seen event-123?
DB-->>A: no
B->>DB: seen event-123?
DB-->>B: no
A->>DB: save it
B->>DB: save it
Note over DB: counted twice
Both checks ran before either save. Neither did anything wrong. The bug lives in the gap between checking and writing — and no amount of careful code closes a gap that is inherent to the approach.
Delete the gap: let the database refuse it
The caller generates the event's identifier, and that identifier is the table's primary key — a value the database itself guarantees is unique.
So both reports simply try to save. One succeeds. The other is rejected by the database, and the service answers 200 duplicate. There is no gap between checking and writing, because there is no check.
A duplicate returns success, not an error — from the caller's point of view the event is recorded. An error would train clients to retry harder at exactly the wrong moment.
POST /events { "event_id":"evt-123", "event_type":"impression", ... }
→ 201 { "status": "recorded" }
# the identical body again, twice
→ 200 { "status": "duplicate" }
→ 200 { "status": "duplicate" }
GET /delivery → impressions: 1
The whole idea, in three requests.
Server-generated identifiers were never an option: every retry would receive a fresh one and therefore become a brand-new impression. The caller must own the identifier for any of this to work.
Totals are computed, never stored
Delivery figures come from scanning the events table on demand — one grouped query — rather than being maintained in a separate running-total table.
A running total is a second copy of the truth, and two copies drift the first time anyone corrects historical data. The scan is fast because the table is small, not because the query is clever, and that distinction is written down along with the condition that changes it: when the query exceeds roughly 200 milliseconds under load.
The trigger from yesterday fired, on schedule
ADR-0003 said Redis would hold spend until impressions became durable. Today they did. The decision narrowed rather than reversed:
flowchart LR
R[("Redis<br/>what the pacer believed<br/>at decision time")]
P[("Postgres<br/>what was actually<br/>delivered")]
R -.->|"the gap between them"| G["reporting loss<br/>— a dashboard panel,<br/>not a bug"]
P -.-> G
They now disagree by design, and the disagreement is the interesting number.
Dashboards that live in the repository
Two Grafana dashboards, defined as files under version control:
- substrate-health — which services are up, request rates, error ratios, 95th-percentile latency
- ads-delivery — fill rate, why candidates lost, events ingested, click-through rate, spend
"Why candidates lost" is Day 3's payoff. It is a stacked chart with one band per rejection reason. When fill rate collapses, you read the cause off whichever band grew. That is a diagnosis in one glance, and it exists only because every filter was made to name itself.
An unnamed data source, and every panel silently loads empty
Grafana generates a random internal identifier for its data connection on first startup. The dashboards referenced whatever it generated. Restart, new identifier, and every panel renders blank — with no error, pointing at a connection that no longer exists.
Fixed by pinning the identifier explicitly, plus a test asserting the dashboards and the data source still agree. The live stack surfaced this in about thirty seconds; no amount of re-reading the plan would have.
Day 4 ended at 87 tests, four services scraped by Prometheus.
Day 05 · traffic-simulator + the quality gate
Breaking it on purpose, for real
Four services exist and nothing uses them. Day 5 builds a simulator that generates realistic viewers, requests adverts, and reports impressions and clicks back — continuously, at a controlled rate, from a fixed random seed so any run can be replayed exactly.
Then it breaks things. Five failure modes, switchable live over an API:
| Scenario | What it actually does | What the dashboards show |
|---|---|---|
| steady | Normal traffic — and doubles as the rollback | flat and healthy, ~40% fill rate |
| error_burst | 3 in 10 requests are deliberately malformed | error ratio climbs; the 422s are the caller's fault, correctly |
| traffic_surge | Ten times the request rate | throughput and latency climb together — load, not fault |
| bad_config_deploy | Rewrites every campaign's targeting to a country with no viewers | fill rate falls off a cliff |
| budget_runaway | Inflates one campaign's daily budget | pacing stops throttling; one campaign takes everything |
Real failures, never simulated ones
No service contains a branch that knows it is being tested. There is no CHAOS=true flag, no debug endpoint, no invented data written into the metrics.
A bad config deploy genuinely sends PATCH requests that rewrite every campaign's targeting. The decision service then reads that configuration and correctly declines to serve. The code behaves perfectly against configuration that is wrong — which is precisely what a real outage looks like.
Measured live: fill rate went from 43% to 2%.
flowchart LR
OP["POST /scenario<br/>bad_config_deploy"] --> SIM["traffic-simulator"]
SIM -->|"PATCH every campaign<br/>targeting → Antarctica"| CS["campaign-service"]
SIM -->|"keeps requesting ads"| ADS["ad-decision-service"]
ADS -->|"reads the changed config"| CS
ADS --> NF["targeting_mismatch<br/>fill rate 43% → 2%"]
NF --> PROM["Prometheus"]
PROM --> GRAF["Grafana<br/>the incident, visible"]
GRAF -.->|"Level 3"| RCA["ops agent finds<br/>the real cause"]
I could have built this in an afternoon with a flag each service checks. I spent the extra day on the honest version because of what happens on Day 18: an agent asked to find the root cause of a fake outage can only ever find the fake, and I would have spent the rest of the month grading it against a fixture I wrote myself.
Because the damage is real, recovery has to be real too. The steady scenario does not flip a flag back — it restores the original configuration through the same public API. Which is exactly the remediation an operations agent should later recommend, making it a genuine rehearsal rather than an undo button.
The first published number
The Level 0 quality gate: eleven automated checks against the running system. Five service health checks, one confirming Prometheus is collecting from all four, and one per failure mode — each verifying the failure produces the signal it claims to.
Its decision logic is pure and separately unit-tested; the live run only supplies the numbers. That is what makes it trustworthy in both directions — it cannot pass with the platform switched off, and it cannot fail because a test fixture drifted from reality.
Result: 11/11 — 100%. The first real figure on the scoreboard, and the gate leaves the platform healthy when it exits.
A $2,147 ceiling nobody knew existed
Budgets were stored in a 32-bit integer column, capping at 2,147,483,647 micros — about $2,147 of daily budget. The runaway scenario tried to exceed it and Postgres refused.
And it escaped as a bare 500, violating the project's own Day 2 standard. Now a typed 422, with a handler ensuring no column's range error can ever return untyped again.
The seeding guard was all-or-nothing. One leftover campaign from a manual test suppressed the entire seed set, so the simulator aimed all its traffic at a single narrowly-targeted campaign — a 2% fill rate with 828 rejections, looking exactly like a bug in the decision logic. Seeding is now idempotent per campaign, by name.
And the gate failed itself on its first run. The traffic-surge check read its baseline from the response of the very call that caused the surge, so it compared 20 requests per second against 20 requests per second and concluded nothing had happened. Caught, fixed, re-run clean — which is precisely what a gate is for.
Level 0 complete: 141 tests, five services, real traffic, real incidents.
Reel three · Days 6–7
Teaching it to think
The substrate stops being the thing under construction and becomes the thing the AI knows about. Two days to turn a repository into searchable understanding.
Day 06 · Document ingestion
Everything built so far becomes knowledge
Level 1 begins. The source code, the five decision records, the daily logs, the runbook, the API contracts, the Compose and Grafana configuration — all of it becomes the corpus.
This is the context layer, and it is the part of the job posting I was most interested in. Giving an AI your team's actual accumulated knowledge, so it answers from your decisions rather than from generic internet training. Ask a normal AI why this project uses Compose and it will recite general trade-offs. Ask one grounded in this corpus and it should quote ADR-0001 and name its trigger.
flowchart LR
RULES["14 ordered rules<br/>what belongs in the corpus"] --> DISC["discover<br/>63 files claimed"]
APPS["4 FastAPI app objects<br/>asked directly for their contracts"] --> SPECS["4 API contracts<br/>generated, not scraped"]
DISC --> LOAD["load<br/>title · component · attributes"]
LOAD --> HASH["normalise line endings<br/>then fingerprint (sha256)"]
SPECS --> HASH
HASH --> UP["store<br/>added / updated / unchanged"]
UP --> PRUNE["prune<br/>anything whose source is gone"]
PRUNE --> CORPUS["corpus.jsonl<br/>67 documents · 550 KiB"]
A reviewable list, not a crawl
The lazy approach walks the folder and takes everything not explicitly excluded. It works on day one and rots quietly: add a directory of scratch notes and the AI starts grounding its answers in them, with nobody reviewing a change that was never made.
Instead, what belongs is fourteen ordered rules written in code. Adding a directory to the corpus is a code change with a test attached. "What do the agents know" becomes a question you can answer by reading a pull request.
The blank decision-record template is deliberately excluded — it is a form, not a decision, and grounding an agent in it teaches it to answer with blanks.
Contracts read from the code, not the network
Each service publishes an API contract. The obvious way to collect them is to ask the running services over HTTP.
That would mean the corpus depends on which containers happen to be running, cannot be rebuilt on a build server, and quietly differs between machines. Instead the contracts are generated by loading the application code and asking it directly — identical bytes on every machine, including one with Docker switched off.
Fingerprints, so nothing is done twice
Every document gets a content hash — a fingerprint that changes completely if a single character changes. Re-run the pipeline on an unchanged repository and the answer is "67 unchanged, nothing written."
That is what makes tomorrow's embedding step affordable to repeat, because tomorrow every changed document costs money.
Provenance, so every answer can be checked
{
"doc_id": "docs/adr/0005-real-failures-not-mocked.md",
"content_hash": "sha256:d42a03d4128146c2...",
"provenance": {
"source_type": "adr",
"component": "docs",
"title": "ADR-0005: Failures are injected by changing real...",
"line_count": 37,
"attributes": { "adr_number": "0005" }
}
}
The identifier is the file path itself — a citation someone can paste into an editor.
Line endings would have broken everything, invisibly
Windows ends each line with two invisible characters; Linux uses one. Git warned about the conversion on the very first commit of the day.
Without normalising before fingerprinting, a file hashed on Windows and re-hashed on a Linux build server produces different fingerprints for identical content. Every run would report all 67 documents as changed, every embedding would be recomputed, and "unchanged" would mean nothing at all.
The whole efficiency argument of the day would have been quietly false.
Day 6 ended at 201 tests, 67 documents.
Day 07 · Chunking, embeddings, vectors
Teaching it to find things by meaning
A whole document is the wrong unit for retrieval. One API contract is 851 lines — handing an agent all of it to answer one question wastes most of its attention.
Two problems: cut documents into useful pieces, and make those pieces findable by meaning rather than by keyword.
Cutting where the content already has seams
The naive approach cuts every 2,000 characters, which slices functions in half and separates a heading from the text under it. Instead each kind of document is cut where it already has natural boundaries:
flowchart TB
DOC["A document"] --> R{"What kind?"}
R -->|"Python"| PY["Split at functions and classes<br/>decorators stay attached<br/>anchor: the dotted symbol name"]
R -->|"Markdown"| MD["Split at headings<br/>heading line stays in the piece<br/>anchor: the full heading path"]
R -->|"API contract"| OA["Split per endpoint<br/>anchor: POST /campaigns"]
R -->|"anything else"| WIN["Fixed window<br/>anchor: none"]
PY & MD & OA & WIN --> CAP["Cap oversized pieces,<br/>keeping the parent anchor"]
Every piece keeps an anchor — the name of the thing it is:
ADR-0005 › Alternatives considered // a heading path substrate.traffic_simulator.driver.tick // a dotted symbol POST /ad-request // an endpoint
A piece that cannot name itself can be returned but not defended. That turns out to be measurable:
| Approach | Pieces | Median size | Can name itself |
|---|---|---|---|
| structure-aware | 585 | 854 chars | 97% |
| cut every 2,000 chars | 349 | 1,960 chars | 0% |
More pieces, each smaller and each able to say what it is. The naive baseline scores zero by construction — that is not a rigged comparison, it is the actual cost of splitting on a character count, stated as a number.
What an embedding actually is
An embedding converts text into a long list of numbers — a position in a space with hundreds or thousands of dimensions — arranged so that text with similar meaning lands nearby.
"Frequency capping" and "how often a viewer sees an advert" share no words at all. A good embedding still places them close together. Searching then becomes: convert the question to a position, return whatever is nearest.
Different models do this with very different quality. Rather than picking one on faith, BELLWETHER runs four behind a single interchangeable interface and measures them.
Hosted models by default, local for free iteration
The Day 0 plan said run a model locally to save money. Measuring the corpus inverted the reasoning completely.
The whole corpus is about 149,000 tokens — roughly two cents to embed with the best model available. Running locally instead means installing PyTorch: 2.5 GB on disk and a resident process competing for memory with Docker, Postgres, Redis, five services, Prometheus and Grafana on a 16 GB laptop with no graphics card.
The best option turned out to be the lightest one locally, because the computation happens on someone else's hardware. The premise flipped the moment anyone measured instead of assuming.
Trigger to revisit: a corpus past a few million tokens, where per-run cost stops being a rounding error and becomes a line item.
| Engine | Dimensions | Cost | Time | Role |
|---|---|---|---|---|
| gemini-embedding-001 | 3072 | $0.0223 | 10.1s | quality tier — leads the public benchmark |
| voyage-3.5 | 1024 | — | — | written and tested; free tier too limited to run |
| potion-retrieval-32M | 512 | free | 2.1s | local, offline, no graphics card |
| hashing | 256 | free | 0.2s | the CI engine — no dependencies, no network |
The fourth is worth explaining. hashing is a keyword-only method with no model at all — it exists so the test suite can assert real retrieval behaviour on a build server with no API key and no downloads. It is genuinely useful and genuinely limited, which makes it the perfect control.
One record per piece, one vector per engine
Comparing engines only means something if each sees identical input. Give each its own copy of the data and any difference might be the data, not the model.
Qdrant stores each text piece as a single record carrying all four engines' vectors at once, alongside one shared set of provenance. Switching engines becomes a parameter on the query.
Fairness stops being a discipline someone has to remember and becomes a property of the storage layout.
What two cents buys
The test: "how often can the same viewer be shown one advert?" — a question containing none of the words frequency, cap, member, or impression.
| Engine | Score | Best match it returned |
|---|---|---|
| gemini | 0.765 | Substrate gaps › SG-04 › Multiple frequency-cap windows |
| potion | 0.428 | ad_decision_service.main.decide |
| hashing | 0.328 | an unrelated plan section — noise, exactly as designed |
Gemini understood the question. The local model found the right neighbourhood. The keyword-only baseline found nothing meaningful, because not one word of the question appears in the answer.
That gap is the entire argument for paying for embeddings — measured on this corpus, not quoted from a vendor's chart.
The tools this level added, and what each beat
Stores the numerical representations of text and finds the nearest matches. Written in Rust, so it is light on memory.
Instead of ChromaDB (which the original plan named): Qdrant can hold several vectors per record under different names, which is what makes comparing four embedding models on identical input possible at all. See ADR-0008 above.
Convert text into vectors, on the provider's hardware. Gemini currently leads the public quality benchmark.
Instead of running a model locally: two cents for the whole corpus, versus 2.5 GB of machine-learning libraries competing for memory with Docker on a 16 GB laptop. See ADR-0007 above — the plan had this backwards until the corpus was measured.
A model whose word-vectors are precomputed, so using it is a lookup and an average rather than running a neural network.
Instead of sentence-transformers: eleven small packages and no PyTorch, versus 2.5 GB — at roughly 82% of the quality. Free, offline, and needs no graphics card.
The database silently destroyed data, and every test passed
Writing one engine's vector to a record replaced the entire record. So storing Voyage's vector silently deleted Gemini's.
Every unit test passed — because the stand-in used in tests faithfully reproduced the wrong mental model. The comparison would have rendered a beautiful, convincing chart built on data that had been destroyed.
It was caught by pointing the code at a real running database and asking one question: is the first vector still there? It was not.
Two more the same day. A text file written by PowerShell carried an invisible marker that turned VOYAGE_API_KEY into an unrecognisable name — presenting as a missing key while the file plainly contained one. And a verification command was piped through another program, which swallowed its failure signal, so nine type errors were committed while the output looked green.
Day 7 ended at 289 tests, 585 pieces, three engines measured.
Day 08 · Hybrid retrieval, reranking, an honest number
The day the prediction was wrong
Day 7 could find pieces that mean what you asked. It could not reliably find pieces that are named what you asked. The embedding of budget_micros lands near “budget” and “spending” — and no nearer the one piece that defines the field than the twenty that merely discuss money.
So Day 8 added keyword search (BM25) beside the vector search, fused the two rankings, reranked the result — and then did the part the other three exist to justify: measured whether any of it actually helped.
flowchart TB
Q["A question"] --> D["Vector search<br/>finds meaning"]
Q --> L["BM25<br/>finds names"]
D & L --> F["Reciprocal Rank Fusion<br/>rank position only, never the score"]
F --> RR["Rerank<br/>free heuristic · or an LLM"]
RR --> TOP["The results"]
The prediction, written down before the code
The design spec pre-registered a claim in git, timestamped before a line of retrieval code existed: hybrid’s advantage would come almost entirely from identifier queries — questions like where is budget_micros enforced — and on conceptual questions it would be flat or slightly worse than vector search alone.
Writing the prediction down first is the whole point. A number is only honest if you cannot move the target after you have seen it.
The answer key nobody could rig
Before any system ran, 26 questions were committed. Then every configuration’s top results were pooled together, shuffled, and stripped of any sign of which system produced them, and each of the 475 question–piece pairs was graded 0, 1 or 2 by hand. The grader never saw which engine surfaced a piece, or where it ranked. The file is committed and pinned to the exact corpus revision it was judged against, so anyone can check the marking.
| Configuration | nDCG@10 | recall@10 | MRR | p50 |
|---|---|---|---|---|
| dense — vector only | 0.670 | 0.757 | 0.774 | 812 ms |
| hybrid + LLM rerank | 0.656 | 0.682 | 0.870 | 9,309 ms |
| hybrid + heuristic rerank | 0.639 | 0.689 | 0.791 | 783 ms |
| hybrid (RRF) | 0.628 | 0.674 | 0.844 | 842 ms |
| hybrid (weighted) | 0.624 | 0.678 | 0.834 | 802 ms |
| lexical — BM25 only | 0.395 | 0.496 | 0.556 | 2.9 ms |
There was no aggregate hybrid win to explain. Vector search alone beat every hybrid configuration. Adding the keyword side lowered the headline score.
And the prediction failed exactly where it was most specific. BM25 was supposed to own the identifier category — its home ground, the literal-name questions. It came last:
| Configuration | identifier | conceptual | cross-document |
|---|---|---|---|
| dense — vector only | 0.711 | 0.598 | 0.721 |
| hybrid (RRF) | 0.696 | 0.583 | 0.589 |
| lexical — BM25 only | 0.426 | 0.452 | 0.250 |
Gemini’s embeddings are good enough on this corpus that they win the one category keyword search was built to dominate. What I would have published without the answer key: “hybrid retrieval, with reranking.” It would have been slower, more complex, and worse — and it would have looked like progress.
One honest boundary on the result: this is a finding about this corpus — 708 well-structured pieces of ADRs, devlogs and typed Python, embedded by a frontier model. A larger, messier, more jargon-dense corpus is exactly where keyword search earns its keep. The number is a baseline, not a law.
Fuse on rank, not on score
Cosine similarity lives between −1 and 1; BM25 is unbounded and routinely reaches 20. Adding them requires normalising first — and every normalisation is a knob that can be turned, consciously or not, until the system you were hoping to promote comes out ahead.
Reciprocal Rank Fusion uses rank position only. It never sees the scores, and its one published constant was not tuned against the answer key. The score-normalising alternative shipped too, as its own measured row — and on conceptual questions it was the best fusion of all, which is why it stays.
The LLM reranker works, and it is still not the default
It is the best hybrid row, and it holds the best MRR of any configuration — the first genuinely useful answer is nearly always rank one.
It also costs 9.3 seconds per query against 783 milliseconds for the free heuristic, and it still loses to plain vector search. So it stays behind a flag, and a free deterministic reranker stays the default.
Trigger to revisit: a purpose-built reranker (Cohere or Voyage) that beats vector search’s 0.670 at a latency the serving path can afford. It becomes the default the day it exists.
Two rows agreed to three decimals, and that was the tell
The LLM-reranked configuration came back byte-identical to the un-reranked one — the same score to three decimals, in every category. That is impossible if a reranker is doing anything, so it wasn’t. Every model call was failing, and the reranker was falling back to the fused order exactly as designed.
Three bugs, stacked, and none was visible to a test suite that fakes the network: the grading schema used a shape the provider rejects outright; the default model id had gone stale to a 404 between writing the plan and running it; and once those were fixed the answer was being truncated mid-sentence on 25 of 26 questions, which reads to the parser exactly like a broken reply and degrades just as quietly.
The degrade path was correct behaviour — an outage should cost a slightly worse ranking, not an error — and it was also an observability hole: a completely broken reranker produced entirely plausible numbers. The only thing that caught it was two rows agreeing too well. The eval now counts every silent degrade and prints the total on every run, so the next one is loud.
Day 8 ended at 410 tests, 708 pieces, six configurations measured — and one prediction, published wrong.
Day 09 · AST graph, reconciliation, click-to-focus
The graph that checks its own claims
Retrieval, from Day 7 and Day 8, answers what does this code say? It has no answer for what breaks if campaign_service changes? — a question about structure, not meaning. So Day 9 stops searching the text and starts reading the shape underneath it.
A walker built on Python’s own ast module reads every service’s source and produces a graph: a node for every service, endpoint, Pydantic model and external dependency, and an edge for every relationship between them — a service exposes an endpoint, defines a model, imports a module, depends_on a package. One edge kind is different from the rest: calls. It is the graph noticing that one service’s HTTP client is pointed at a URL that is supposed to be a live route somewhere else — a claim, not yet a fact.
Reconciliation: the graph auditing itself
So it checks. Reconciliation walks every calls edge and confirms the route it names is an endpoint node that actually exists in the graph, rather than a string that merely looks like one. A call pointed at a route that was renamed, removed, or never existed comes back dangling, printed by name — not buried in a diff nobody reads.
$ python -m bellwether.context.graph --reconcile
7/7 call paths resolved (100%)
isolated: platform
Today: 45 nodes, 74 edges — 25 endpoints, 9 services, 8 external dependencies, 3 models — and among the edges, 4 calls edges covering 7 call paths, all 7 resolved, nothing dangling. One name comes back isolated: platform, the Level 0 gate script from Day 5. It is a command-line entry point, not a module any service imports or calls into, so the extractor correctly finds nothing wired to it. Not a bug — a script and a service are different shapes, and the graph tells the two apart instead of guessing.
--diagram --scope substrate
graph LR
subgraph n_service_ad_decision_service_ef0d50ac["ad_decision_service"]
n_endpoint_ad_decision_service_GET__health_f0a25609(["GET /health"])
n_endpoint_ad_decision_service_GET__metrics_a43ef786(["GET /metrics"])
n_endpoint_ad_decision_service_POST__ad_request_41a28ab8(["POST /ad-request"])
end
subgraph n_service_campaign_service_e8506251["campaign_service"]
n_endpoint_campaign_service_DELETE__campaigns____d07f140b(["DELETE /campaigns/{}"])
n_endpoint_campaign_service_GET__campaigns_caa4df2c(["GET /campaigns"])
n_endpoint_campaign_service_GET__campaigns____71df29ed(["GET /campaigns/{}"])
n_endpoint_campaign_service_GET__campaigns____creatives_c2d77467(["GET /campaigns/{}/creatives"])
n_endpoint_campaign_service_GET__health_1e1ac476(["GET /health"])
n_endpoint_campaign_service_GET__metrics_e482f5e5(["GET /metrics"])
n_endpoint_campaign_service_PATCH__campaigns____069be044(["PATCH /campaigns/{}"])
n_endpoint_campaign_service_POST__campaigns_6d2a9e35(["POST /campaigns"])
n_endpoint_campaign_service_POST__campaigns____creatives_58e965f9(["POST /campaigns/{}/creatives"])
n_model_campaign_service_Campaign_c979140a["Campaign"]
n_model_campaign_service_Creative_7378759d["Creative"]
end
subgraph n_service_event_service_de0f72ab["event_service"]
n_endpoint_event_service_GET__campaigns____delivery_5d703263(["GET /campaigns/{}/delivery"])
n_endpoint_event_service_GET__delivery_420b8d39(["GET /delivery"])
n_endpoint_event_service_GET__events_0513aa23(["GET /events"])
n_endpoint_event_service_GET__health_cec6f042(["GET /health"])
n_endpoint_event_service_GET__metrics_9d97db47(["GET /metrics"])
n_endpoint_event_service_POST__events_6b7ade97(["POST /events"])
n_model_event_service_AdEvent_b8129f81["AdEvent"]
end
subgraph n_service_shared_ac61e27b["shared"]
end
subgraph n_service_traffic_simulator_a4adaf7e["traffic_simulator"]
n_endpoint_traffic_simulator_GET__health_647c3354(["GET /health"])
n_endpoint_traffic_simulator_GET__metrics_ca76d008(["GET /metrics"])
n_endpoint_traffic_simulator_GET__scenarios_b1a8a38a(["GET /scenarios"])
n_endpoint_traffic_simulator_GET__status_5d0751d8(["GET /status"])
n_endpoint_traffic_simulator_POST__control_f34a452b(["POST /control"])
n_endpoint_traffic_simulator_POST__scenario_739f882d(["POST /scenario"])
n_endpoint_traffic_simulator_POST__seed_e15bea1a(["POST /seed"])
end
n_service_ad_decision_service_ef0d50ac -->|"GET /campaigns"| n_service_campaign_service_e8506251
n_service_ad_decision_service_ef0d50ac --> n_service_shared_ac61e27b
n_service_campaign_service_e8506251 --> n_service_shared_ac61e27b
n_service_event_service_de0f72ab --> n_service_shared_ac61e27b
n_service_traffic_simulator_a4adaf7e -->|"POST /ad-request"| n_service_ad_decision_service_ef0d50ac
n_service_traffic_simulator_a4adaf7e -->|"GET /campaigns, PATCH /campaigns/{}, POST /campaigns, POST /campaigns/{}/creatives"| n_service_campaign_service_e8506251
n_service_traffic_simulator_a4adaf7e -->|"POST /events"| n_service_event_service_de0f72ab
n_service_traffic_simulator_a4adaf7e --> n_service_shared_ac61e27b
--diagram --scope substrate, not redrawn by hand. Click a service below to isolate what it depends on and what depends on it.7 / 7 call paths resolved — no drift
The adjacency behind the buttons above is the full graph — it also reaches bellwether.context, bellwether.eval and bellwether.llm, the AI layer being built on top of this substrate, plus the still-isolated platform script. The diagram itself is deliberately scoped narrower, to --scope substrate: the four services and the shared package that actually run in production. Conflating “what serves an ad” with “what builds the thing that reads the ads platform” would answer a different question than this chapter is asking. The focus buttons below only render for the services this diagram actually draws — a button pointing at nothing on screen would be worse than no button.
Day 9 ends at 45 nodes, 74 edges, 7 of 7 call paths reconciled — and 448 tests passing, 38 of them new.
Closing
Where it stands today
There is a working ads platform — five services, real traffic, real dashboards, and incidents that can be caused on demand and are genuinely real when they happen. On top of it, the beginnings of an AI layer that has read the entire project and can find things in it by meaning.
flowchart TB
subgraph L1["LEVEL 1 — context layer (in progress)"]
COR["73 documents<br/>708 searchable pieces"]
VEC["Vector store<br/>4 engines, 1 record each"]
RET["Hybrid retrieval<br/>6 configurations, one answer key"]
GR["Architecture graph<br/>45 nodes · 74 edges · 7/7 reconciled"]
end
subgraph L0["LEVEL 0 — the substrate (complete, 11/11)"]
CS["campaign-service<br/>:8001"]
ADS["ad-decision-service<br/>:8002"]
EV["event-service<br/>:8003"]
SIM["traffic-simulator<br/>:8004"]
OBS["Prometheus + Grafana"]
end
SIM --> ADS --> CS
SIM --> EV
L0 -->|"source · docs · contracts · config"| COR
COR --> VEC --> RET --> GR
GR -.->|"Day 10"| NEXT["the console<br/>MCP server"]
NEXT -.->|"Levels 2-4"| AG["dev agents · ops agents<br/>orchestration"]
The setups, and where they pay off
| Planted | What it was | Paid off |
|---|---|---|
| Day 1 | Four quality gates, before any product code | Day 11 — the fence AI-generated code must clear |
| Day 2 | Structured JSON logs, one line per request | Day 16 — log agents read them with no parsing |
| Day 2 | "Never a bare 500" written down as a standard | Day 5 — the violation was identifiable, not just unlucky |
| Day 3 | Every rejection carries a name, not a boolean | Day 4 chart → Day 5 diagnosis → Day 18 evidence |
| Day 3 | Reading the API, never the tables | Day 4 — kept ADR-0002's trigger unfired |
| Day 5 | Failures change real configuration | Day 18 — an RCA agent with a real cause to find |
| Day 6 | Content fingerprints on every document | Day 7 — re-embedding costs nothing when nothing changed |
| Day 7 | The engine is a parameter, not a commitment | Day 8 — six retrieval configurations behind one command |
| Day 8 | An adversarial answer key, graded before any system ran | Day 9 on — every retrieval change is an experiment, not an argument |
| Day 9 | Every service call reconciled against a route that provably exists | Day 10 — the console and the MCP server query a graph that already audits itself |
The pattern worth noticing
Reading my own nine days back to back, the same shape repeats. I make a decision and write it down with the condition that would reverse it. Then reality tests it — and every single time, running the real thing found something my plan had not:
- A metric collision that only appeared when two services loaded together
- A dashboard connection identifier that changed on restart, silently
- A $2,147 budget ceiling nobody knew existed
- A quality gate that measured itself and passed
- Line endings that would have made every fingerprint meaningless
- A database that quietly deleted data while every test agreed it was fine
- A reranker that had never once run, hidden behind a degrade path working exactly as designed
I found none of these by thinking carefully. I found every one of them by running the real thing under real conditions and looking hard at what came back.
Which is, in the end, the whole argument I'm making. An AI that writes code quickly is only useful if something can prove the code is right. That proof cannot come from a fixture, a mock, or a demo that worked once on camera. It has to come from the real system, misbehaving in real ways, measured by numbers I publish whether they flatter me or not.
Twenty-one days remain, and this page grows with them. Level 1 finishes with a console and an MCP server that let any AI client query this project's knowledge — the corpus, the retrieval index, and now the graph — directly, each measured against an answer key like the one Day 8 built. Then the agents arrive, and every one of them gets a number next to it.