BUILT IN PUBLIC · TARGET ROLE: NETFLIX STAFF AI ENGINEER — AI FOUNDATION & TOOLING (AJRT30201)

BELLWETHER

An AI-native engineering platform for ad-tech teams. A mini ads platform as the substrate; a context layer, dev agents, ops agents, and an orchestrator on top — every capability shipped with published eval numbers.

GitHub repo Video series Independent open-source project, not affiliated with or endorsed by Netflix.
The 30-day pod9 shipped · 21 queued

SEQ 01 · SYSTEM OVERVIEW

Two systems, one thesis

The Substrate is a mini Netflix-style ads platform — four small services producing real traffic, logs, metrics, and stageable incidents. Bellwether is the AI foundation that operates on it. The thesis: AI velocity with provable quality — agents are easy to demo and hard to trust, so every capability here ships with numbers.

flowchart TB
    subgraph BELLWETHER["BELLWETHER — AI Foundation"]
        CTX["Context Layer<br/>RAG + Knowledge Graph + MCP"]
        DEV["Dev Lifecycle Agents"]
        OPS["Ops Agents"]
        ORCH["Multi-Agent Orchestrator"]
        EVAL["Eval Harness + Dashboard"]
        LLM["LLM Abstraction + Cost Tracking"]
    end
    subgraph SUB["SUBSTRATE — Mini Ads Platform"]
        CS["campaign-service"]
        ADS["ad-decision-service"]
        EV["event-service"]
        SIM["traffic-simulator + failure injection"]
        OBS["Prometheus + Grafana + JSON logs"]
    end
    CTX --> DEV & OPS
    ORCH --> DEV & OPS
    DEV & OPS --> LLM
    EVAL -.evaluates.-> CTX & DEV & OPS & ORCH
    SIM --> ADS
    ADS --> CS & EV
    SUB --> OBS
    OBS --> OPS
    SUB -. "code, docs, logs, metrics" .-> CTX
    

SEQ 02 · LEVEL 0 — THE SUBSTRATE (DAYS 1–5)

Something real to operate on

Every later AI capability needs real traffic to observe, real logs and metrics to correlate, real incidents to diagnose. Level 0 builds that: a small but production-shaped ads platform. No mock-data theater.

Read the Level 0 deep dive →The whole story, so far →

The ad-request path

sequenceDiagram
    participant SIM as traffic-simulator
    participant ADS as ad-decision-service
    participant CS as campaign-service
    participant R as Redis
    participant EV as event-service
    SIM->>ADS: POST /ad-request {member_ctx, slot}
    ADS->>CS: eligible campaigns (targeting)
    ADS->>R: frequency cap check
    ADS->>ADS: brand-safety + budget pacing
    ADS-->>SIM: selected ad
    SIM->>EV: impression / click events
    EV-->>EV: aggregate, emit metrics
    

Day 1 — infrastructure skeleton

flowchart LR
    DEVBOX["Developer machine<br/>uv + pytest + ruff + mypy"] -->|docker compose up| NET["bellwether network"]
    NET --> PG[("Postgres 16")]
    NET --> RD[("Redis 7")]
    NET --> PROM["Prometheus :9090"]
    NET --> GRAF["Grafana :3000"]
    PROM --> GRAF
    

Day 2 — the campaign data model

erDiagram
    CAMPAIGN ||--o{ CREATIVE : "carries"
    CAMPAIGN {
        uuid id PK
        string name
        string advertiser
        string status "draft|active|paused|completed"
        int budget_micros
        int daily_budget_micros
        int frequency_cap_per_day
        json targeting "countries, device_types, content_ratings"
        json brand_safety_exclusions
        datetime starts_at
        datetime ends_at
    }
    CREATIVE {
        uuid id PK
        uuid campaign_id FK
        string name
        int duration_seconds
        string asset_url
    }
    

Day 3 — the decision path

One ad request in, one decision out. Every candidate campaign walks the same chain, and every rejection carries a named reason — which is what turns "fill rate dropped at 14:20" into an answerable question three levels from now.

flowchart LR
    REQ["POST /ad-request<br/>member + slot"] --> FETCH["active campaigns<br/>(campaign-service HTTP)"]
    FETCH --> F1{"in flight?"}
    F1 -->|no| X1["not_active /<br/>outside_flight_window"]
    F1 -->|yes| F2{"targeting match?"}
    F2 -->|no| X2["targeting_mismatch"]
    F2 -->|yes| F3{"brand safe?"}
    F3 -->|no| X3["brand_safety_excluded"]
    F3 -->|yes| F4{"under frequency cap?"}
    F4 -->|no| X4["frequency_capped"]
    F4 -->|yes| F5{"within pacing allowance?"}
    F5 -->|no| X5["pacing_throttled"]
    F5 -->|yes| WIN["most daily budget remaining wins"]
    WIN --> REC["record impression<br/>(Redis: freq + spend)"]
    REC --> RES["AdDecision + full candidate trace"]
    X1 --> RES
    X2 --> RES
    X3 --> RES
    X4 --> RES
    X5 --> RES
    

Where the decision state lives

flowchart LR
    ADS["ad-decision-service"] -->|"GET /campaigns?status=active"| CS["campaign-service"]
    CS --> PG[("Postgres<br/>system of record<br/>campaigns · budgets · creatives")]
    ADS -->|"freq: / spend: keys, 48h TTL"| RD[("Redis<br/>per-day decision state")]
    ADS --> PROM["/metrics<br/>ad_decisions_total<br/>ad_candidates_filtered_total{reason}"]
    

Day 4 — the event path

Serving an ad is a decision; reporting it is a fact. Events land keyed on an id the caller chose, so a retried report collides with itself instead of inflating the numbers — idempotency enforced by the primary key, not by a check that races. Delivery numbers are then one GROUP BY over that table, never a second copy of the truth.

flowchart LR
    SIM["traffic-simulator<br/>(Day 5)"] -->|"POST /events"| EV["event-service"]
    EV --> PK{"event_id already stored?"}
    PK -->|yes| DUP["200 duplicate<br/>ad_events_duplicate_total"]
    PK -->|no| INS["201 recorded<br/>INSERT into ad_events"]
    INS --> MET["ad_events_total{event_type}<br/>ad_spend_micros_total"]
    INS --> PGE[("Postgres<br/>ad_events — append only")]
    PGE -->|"GROUP BY campaign_id"| DEL["GET /delivery<br/>impressions · clicks · CTR · spend"]
    

Day 4 — what the dashboards read

Three services, one metric vocabulary, two dashboards provisioned from files in the repo. Nothing here was clicked into existence: a fresh docker compose up lands on working graphs, and a hermetic test fails the build if a panel loses its query or graphs a metric no service emits.

flowchart LR
    CS["campaign-service<br/>/metrics"] --> PROM["Prometheus :9090"]
    ADS["ad-decision-service<br/>/metrics"] --> PROM
    EVS["event-service<br/>/metrics"] --> PROM
    PROM --> D1["Grafana · substrate-health<br/>traffic · errors · p95 · targets"]
    PROM --> D2["Grafana · ads-delivery<br/>fill rate · why candidates lost<br/>events · CTR · spend"]
    

Day 5 — the failure injection loop

The simulator does not fake a symptom. It changes real configuration through campaign-service's public API, so the incident that follows has an actual cause sitting in an actual table — which is the only way Level 3's RCA agent can be honestly evaluated. No service contains a branch that knows it is being tested.

flowchart LR
    OP["POST /scenario<br/>{bad_config_deploy}"] --> SIM["traffic-simulator"]
    SIM -->|"PATCH /campaigns/{id}<br/>targeting → AQ"| CS["campaign-service"]
    SIM -->|"POST /ad-request<br/>at scenario rate"| ADS["ad-decision-service"]
    ADS -->|"reads the changed config"| CS
    ADS --> NOFILL["targeting_mismatch<br/>fill rate 43% → 2%"]
    SIM -->|"impression / click"| EV["event-service"]
    NOFILL --> PROM["Prometheus"]
    EV --> PROM
    PROM --> GRAF["Grafana<br/>the incident, visible"]
    GRAF -.->|"Level 3"| RCA["ops agent<br/>finds the real cause"]
    

Day 5 — the Level 0 gate

The spec's Level 0 metric turned into a script that exits zero or names the check that failed. Its decision logic is pure and unit-tested; the live run only supplies numbers, so it cannot pass by accident with the substrate switched off. On its first run it failed itself — the traffic-surge check was comparing the surged rate against the surged rate — which is exactly what a gate is for.

flowchart TB
    G["platform/level0_gate.py"] --> H["5 health endpoints"]
    G --> T["Prometheus scrapes<br/>all 4 services"]
    G --> S1["steady<br/>serves ads"]
    G --> S2["error_burst<br/>produces 422s"]
    G --> S3["traffic_surge<br/>raises the rate"]
    G --> S4["bad_config_deploy<br/>collapses fill rate"]
    G --> S5["budget_runaway<br/>changes config"]
    H & T & S1 & S2 & S3 & S4 & S5 --> R["11/11 · exit 0<br/>platform left healthy"]
    

SEQ 03 · LEVEL 1 — THE CONTEXT LAYER (DAYS 6–10)

The substrate stops being the thing we build and becomes the thing the AI knows

Everything Level 0 produced — four services' source, five ADRs, the devlogs, the runbook, the coding standards, the design spec, the plans, the Compose and Prometheus and Grafana config — is now a corpus. From Level 2 onward, whatever is in here is what the agents believe. Which makes two questions worth deciding rather than defaulting: what gets in, and where the API contracts come from.

Day 6 — the ingestion pipeline

Discover, load, hash, store, prune. Every interesting decision sits in a module the pipeline calls, and the store sits behind a protocol — so Day 7 puts a vector store underneath upsert without the pipeline changing a line.

flowchart LR
    RULES["CORPUS_RULES<br/>14 globs, in order"] --> DISC["discover()<br/>63 files claimed"]
    APPS["4 FastAPI app objects<br/>app.openapi()"] --> SPECS["4 contracts<br/>generated, not scraped"]
    DISC --> LOAD["load()<br/>title · component · attributes"]
    LOAD --> HASH["normalise CRLF → LF<br/>sha256 content hash"]
    SPECS --> HASH
    HASH --> UP["store.upsert()<br/>added / updated / unchanged"]
    UP --> PRUNE["store.prune()<br/>orphans removed"]
    PRUNE --> CORPUS["corpus.jsonl<br/>67 docs · 550 KiB"]
    CORPUS -.->|"Day 7"| VEC["vector store<br/>behind the same protocol"]
    

The corpus is a manifest, not a crawl

A crawl of "everything except an exclusion list" grows silently — add a directory and the context layer quietly starts grounding agents in whatever landed there. An ordered rule list shows up in a diff, with a test attached. The ADR template is excluded on purpose: it is a form, not a decision, and grounding an agent in it teaches it to answer with blanks.

Source typeDocsWhat it grounds
code33substrate + platform Python — the system as built
plan6how each day was planned, and what the plan missed
config6Compose, Prometheus scrape config, Grafana dashboards
adr6the decisions, their alternatives, and their triggers
devlog6what actually happened, including the defects
openapi4the four service contracts, generated from the apps
readme2the project and the platform directory
runbook · spec · standards · backlog4ops procedure, the design, the rules, the known gaps

Generated, not scraped

The OpenAPI contracts come from importing the FastAPI app objects and calling app.openapi() — never from curling a running service. A corpus that depends on which containers happen to be up cannot be built in CI, cannot be built on a laptop with Docker stopped, and quietly differs between machines. Ingestion opens no socket.

Hashing is what makes Day 7 affordable

Content is normalised to \n before hashing, so a Windows checkout and a Linux CI runner agree on what a document is. Run the pipeline twice and the second run reports 67 unchanged, 0 written. Once Day 7 attaches an embedding cost to every changed document, that is the difference between re-running ingestion freely and not re-running it at all.

Day 7 — cutting the corpus so it can be retrieved

A whole document is the wrong unit. Nobody asks "what is in campaign-service's OpenAPI contract" — they ask what POST /campaigns accepts. So the split points come from the content's own structure: Python at symbol boundaries via the AST, Markdown at headings, OpenAPI one chunk per operation. Every chunk keeps an anchor — the symbol, heading path, or route that names it. A chunk that cannot name itself can be returned but not defended.

StrategyChunksp50 charsAnchoredWhat it cuts on
markdown3311428100%headings, carrying the full path down
python_ast177386100%functions and classes, decorators attached
openapi60851100%one operation or schema at a time
window1719630%the fallback nothing may fail past
all58585497%structure-aware routing by document kind
naive_window34919600%splitting on a character count — the default everyone ships

Four engines, one point, one set of chunks

The engine is a parameter, not a commitment. Qdrant stores one point per chunk with a named vector per engine, so all four are compared on byte-identical inputs — fairness becomes a property of the schema rather than a discipline someone has to maintain (ADR-0008). The spec called for local sentence-transformers; measuring the corpus inverted that, and ADR-0007 says why.

EngineDimsCostWallChunks/sRole
gemini-embedding-0013072$0.022310.1s58quality tier — #1 on MTEB
voyage-3.51024free-tier rate limit, not yet run
potion-retrieval-32M512free2.1s282local, offline, no torch, no GPU
hashing256free0.2s3109the CI engine — no deps, no network

What two cents buys

Asked "how often can the same viewer be shown one advert?" — a question containing none of the words frequency, cap, member, or impression — the three engines that ran disagree in exactly the way their price suggests.

EngineScoreTop hit
gemini0.765Substrate gaps › SG-04 › Multiple frequency-cap windows
potion0.428substrate.ad_decision_service.main.decide
hashing0.328a Day 7 plan section — noise, exactly as designed

Gemini understood the question. potion found the neighbourhood. The lexical baseline matched nothing meaningful, because no word in the query appears in the answer. That gap is the entire argument for paying for embeddings, and it is measured here rather than quoted from a vendor chart.

flowchart TB
    R1["run 1<br/>65 added"] --> C["corpus.jsonl"]
    R2["run 2, nothing edited<br/>65 unchanged · 0 written"] --> C
    R3["run 3, ADR-0006 + day-06<br/>2 added · 65 unchanged"] --> C
    R4["run 4, a devlog deleted<br/>1 pruned"] --> C
    C -.->|"an orphan is worse<br/>than a gap — an agent<br/>will cite it"| P["prune()"]
    

Day 8 — the day the prediction was wrong

Day 7 could find chunks that mean what you asked. It could not reliably find chunks that are named what you asked — embedding budget_micros lands near "budget" and "spending", and no nearer the one chunk that defines the field than twenty that discuss money. So Day 8 added keyword search (BM25) beside the vector side, fused the two, reranked the result, and — the part that matters — measured whether any of it helped.

The design spec pre-registered a prediction in writing, committed before a line of retrieval code existed: hybrid would win, and its win would come almost entirely from identifier queries. The premise was wrong. There is no aggregate hybrid win.

ConfigurationnDCG@10recall@10MRRp50
dense — vector only (gemini)0.6700.7570.774812 ms
hybrid + LLM rerank0.6560.6820.8709,309 ms
hybrid + heuristic rerank0.6390.6890.791783 ms
hybrid — reciprocal rank fusion0.6280.6740.844842 ms
hybrid — weighted fusion0.6240.6780.834802 ms
lexical — BM25 only0.3950.4960.5562.9 ms

Vector search alone beat every hybrid configuration. Adding the lexical side lowered the score.

The category that settles it

The prediction was most specific about identifier questions — where is budget_micros enforced — where keyword search was supposed to dominate. It came last.

Configurationidentifierconceptualcross-document
dense (gemini)0.7110.5980.721
hybrid (RRF)0.6960.5830.589
hybrid + heuristic0.6520.6010.681
hybrid + LLM0.6960.5830.711
hybrid (weighted)0.6610.6200.569
lexical (BM25)0.4260.4520.250

Gemini's embeddings are strong enough on this corpus that they win the category that was supposed to be lexical search's home ground. What I would have shipped without the evaluation is "hybrid retrieval with reranking" — slower, more complex, and worse. It would have looked like progress.

How the answer key was built so it could not flatter anything

A retrieval score is only as honest as the ground truth under it, and the cheap ways to build one are all rigged: label from what the retriever returned and you reward what you built; derive queries from chunk text and BM25 wins by construction; judge only the favoured system's results and nothing else can ever score.

Two rows that agreed too well

The first run had a tell: hybrid-llm scored byte-identical to hybrid — same nDCG, recall and MRR to three decimals, in every category. That is impossible if a reranker is doing anything, so it wasn't. Every Gemini call was failing and the reranker was falling back to the fused order, exactly as designed. Three causes, stacked, and none was visible to a test suite that fakes the network:

A fake transport can validate neither a provider's schema dialect, its model catalogue, nor how long its answers run. This is Day 7's lesson in a new place: anything faked in tests needs one live round-trip before it is trusted. The silent degrade path was the correct behaviour and an observability hole — a completely broken reranker produced entirely plausible numbers, which is why the eval now counts every degrade and prints the total whether or not it is zero.

0.670 is a baseline, not a ceiling

nDCG is not a percentage of right answers: it gives half credit for partially-relevant chunks, penalises a good answer at rank 3, and is measured against a strict hand-written key. The companion numbers are healthier — 76% of relevant chunks reach the top 10, and the first useful answer is usually rank 1 or 2. The real deliverable is that every idea below is now testable against a fixed answer key instead of argued about: a purpose-built reranker, contextual chunk headers, a larger second-opinion gold set, query rewriting for the conceptual shapes that scored lowest.

SEQ 04 · DAY TRACKER

The full manifest

DayLvlDeliverableStatus
010Repo, architecture docs, README, running doc v1, ADR-0001, infra skeleton, CI● SHIPPED
020campaign-service (CRUD, Postgres, tests, OpenAPI)● SHIPPED
030ad-decision-service (targeting, freq capping, brand safety, pacing)● SHIPPED
040event-service + observability stack (Grafana dashboards, JSON logging lib)● SHIPPED
050traffic-simulator + failure injection · Level 0 quality gate● SHIPPED
061Document ingestion pipeline● SHIPPED
071Chunking + embedding strategies (comparison documented)● SHIPPED
081Hybrid retrieval + reranking + pooled-judgement eval● SHIPPED
091AST knowledge graph — what depends on X, and a call graph that audits itself● SHIPPED
101MCP server + retrieval eval suite · Level 1 quality gate○ QUEUED
112Code-generation agent○ QUEUED
122Test-generation agent + mutation testing○ QUEUED
132PR pre-review agent (GitHub Actions)○ QUEUED
142Deployment-validation agent○ QUEUED
152E2E pipeline: ticket → code → test → PR → deploy · Level 2 gate○ QUEUED
163Log intelligence pipeline○ QUEUED
173Incident triage agent○ QUEUED
183RCA agent○ QUEUED
193Guided resolution + runbook generation○ QUEUED
203Self-healing loop E2E demo · Level 3 quality gate○ QUEUED
214Agent communication protocol○ QUEUED
224Parallel agent execution○ QUEUED
234Conflict resolution + escalation○ QUEUED
244Human-in-the-loop gates○ QUEUED
254Actor-Critic process-audit evals · Level 4 quality gate○ QUEUED
265AI-first dev environment (devcontainer, one-command setup)○ QUEUED
275CI/CD AI gates○ QUEUED
285Consolidated eval dashboard○ QUEUED
295Grand demo (feature E2E through the whole platform)○ QUEUED
305Launch: open-source release, outreach kickoff○ QUEUED

SEQ 05 · DECISIONS

Written down, or it didn't happen

The name

bellwether (noun) — a leading indicator; the thing you watch to know where something is heading before the rest of the picture catches up. That is the job twice over: the first AI engineer on a team is a leading indicator of how that team will build a year from now, and the eval scoreboard is a leading indicator of whether the agents are actually making the work better.

ADR-0008 — Qdrant over ChromaDB, for named vectors

Four engines embed the same corpus, and the comparison only means something if every engine sees identical chunks. Qdrant stores one point per chunk carrying a named vector per engine, so switching engines is a query parameter and fairness is structural rather than procedural. Chroma has no clean equivalent; pgvector would have put the AI foundation's storage inside the ads platform's own database. The dangerous part, found only against a running server: PUT /points replaces a point, so writing the second engine's vector silently wiped the first. Every test passed. The chart would have rendered beautifully and been a lie.

ADR-0007 — Hosted embeddings by default, and the spec was backwards

The design spec called for local sentence-transformers to save money. The corpus is 149k tokens — about two cents with the best model available — while PyTorch is 2.5 GB and a resident process competing with Docker on a 16 GB laptop. The quality tier turned out to be the lighter option locally, because the compute happens on someone else's hardware. The free tier is the real constraint: Gemini allows 100 embed requests per day, and one pass needs 585. Local potion stays as the free-iteration tier and a dependency-free hashing engine keeps CI offline.

ADR-0006 — The corpus is a manifest, and contracts are generated

What the context layer holds is what every agent from Level 2 onward will believe and cite, so both halves of it are decided rather than defaulted. What gets in is an ordered list of rules in code — a crawl-minus-exclusions grows silently, and nobody reviews a diff that does not exist. Where the contracts come from is app.openapi() on the imported app objects, never a curl against a running service: a corpus that depends on which containers are up cannot be built in CI. Content is normalised before hashing so Windows and Linux agree, and orphans are pruned — a document whose source has left the repo is worse than a missing one, because an agent will happily cite it.

ADR-0005 — Real failures, not mocked ones

The simulator breaks the platform by changing real configuration through public APIs — a bad config deploy really does PATCH every campaign's targeting, and the decision path then behaves perfectly while fill rate collapses. No service has a branch that knows it is being tested. The reason is Level 3: an ops agent evaluated against a mocked failure can only ever find the mock. Because the change is real, recovery is real too — steady is a genuine rollback. Day one of running it found two real defects in the platform it was pointed at.

ADR-0004 — Idempotent on the primary key, aggregated on read

A delivery report gets retried whenever a network hiccups, so the caller chooses the event_id and the primary key does the deduplication — a check-then-insert races, an insert that collides cannot. Delivery numbers are then a single GROUP BY over the event table rather than a rollup that can drift out of sync with it. The trigger that flips this: Day 5's simulator, when sustained load pushes /delivery past ~200 ms.

ADR-0003 — Redis holds the decision state, Postgres holds the truth

Frequency counters and daily pacing spend are written on every served impression, scoped per member, and worthless after midnight — a counter with a TTL, not a row. They live in Redis under day-scoped keys that expire themselves; campaign-service stays the system of record, read over its HTTP API rather than its tables. Day 4 update — the trigger fired. event-service made impressions durable, and the decision narrowed rather than reversed: Postgres now holds the auditable record of what was served, Redis keeps only the hot counters the serving path reads. The gap between the two is not a bug — it is reporting loss, and it is a panel on the dashboard.

ADR-0002 — create_all over Alembic, for now

The schema has one writer and no data anyone depends on, and it will churn daily through Day 5. Migrations written against a schema that changes every day are noise, not safety. The trigger that flips this: the first second service that reads these tables.

ADR-0001 — Docker Compose over Kubernetes

Four services run by one developer on one machine. Compose gives one-command startup and reproducible demos; Kubernetes adds operational overhead while proving nothing relevant to AI foundation engineering. Choosing infrastructure proportionate to the problem is the signal.

SEQ 06 · EVAL SCOREBOARD

Numbers or it's a demo

Actuals publish as each level's quality gate runs. Empty cells are honest cells. The Level 0 number is what platform/level0_gate.py printed against the running stack — five health endpoints, one scrape check, and one per injectable failure mode.

LvlMetricTargetActual
L0Services healthy under simulator load; failure injection works100%11/11 · 100%
L1Context relevance (50-query synthetic dataset)>85%
L1Retrieval latency<500ms
L2Generated code compiles / passes lint + types100%
L2Generated test pass rate; mutation score improvement>90%
L2PR reviewer seeded-bug catch ratepublished
L3Synthetic incidents flagged5/5
L3RCA correct root cause>80%
L4Complex tasks completed by parallel agents3/3
L4HITL approval acceptance rate>90%
L5Fresh clone → running system<10 min
L5CI/CD gates pass100%