▸ BUILT IN PUBLIC · TARGET ROLE: NETFLIX STAFF AI ENGINEER — AI FOUNDATION & TOOLING (AJRT30201)
An AI-native engineering platform for ad-tech teams. A mini ads platform as the substrate; a context layer, dev agents, ops agents, and an orchestrator on top — every capability shipped with published eval numbers.
SEQ 01 · SYSTEM OVERVIEW
The Substrate is a mini Netflix-style ads platform — four small services producing real traffic, logs, metrics, and stageable incidents. Bellwether is the AI foundation that operates on it. The thesis: AI velocity with provable quality — agents are easy to demo and hard to trust, so every capability here ships with numbers.
flowchart TB
subgraph BELLWETHER["BELLWETHER — AI Foundation"]
CTX["Context Layer<br/>RAG + Knowledge Graph + MCP"]
DEV["Dev Lifecycle Agents"]
OPS["Ops Agents"]
ORCH["Multi-Agent Orchestrator"]
EVAL["Eval Harness + Dashboard"]
LLM["LLM Abstraction + Cost Tracking"]
end
subgraph SUB["SUBSTRATE — Mini Ads Platform"]
CS["campaign-service"]
ADS["ad-decision-service"]
EV["event-service"]
SIM["traffic-simulator + failure injection"]
OBS["Prometheus + Grafana + JSON logs"]
end
CTX --> DEV & OPS
ORCH --> DEV & OPS
DEV & OPS --> LLM
EVAL -.evaluates.-> CTX & DEV & OPS & ORCH
SIM --> ADS
ADS --> CS & EV
SUB --> OBS
OBS --> OPS
SUB -. "code, docs, logs, metrics" .-> CTX
SEQ 02 · LEVEL 0 — THE SUBSTRATE (DAYS 1–5)
Every later AI capability needs real traffic to observe, real logs and metrics to correlate, real incidents to diagnose. Level 0 builds that: a small but production-shaped ads platform. No mock-data theater.
Read the Level 0 deep dive →The whole story, so far →
sequenceDiagram
participant SIM as traffic-simulator
participant ADS as ad-decision-service
participant CS as campaign-service
participant R as Redis
participant EV as event-service
SIM->>ADS: POST /ad-request {member_ctx, slot}
ADS->>CS: eligible campaigns (targeting)
ADS->>R: frequency cap check
ADS->>ADS: brand-safety + budget pacing
ADS-->>SIM: selected ad
SIM->>EV: impression / click events
EV-->>EV: aggregate, emit metrics
flowchart LR
DEVBOX["Developer machine<br/>uv + pytest + ruff + mypy"] -->|docker compose up| NET["bellwether network"]
NET --> PG[("Postgres 16")]
NET --> RD[("Redis 7")]
NET --> PROM["Prometheus :9090"]
NET --> GRAF["Grafana :3000"]
PROM --> GRAF
erDiagram
CAMPAIGN ||--o{ CREATIVE : "carries"
CAMPAIGN {
uuid id PK
string name
string advertiser
string status "draft|active|paused|completed"
int budget_micros
int daily_budget_micros
int frequency_cap_per_day
json targeting "countries, device_types, content_ratings"
json brand_safety_exclusions
datetime starts_at
datetime ends_at
}
CREATIVE {
uuid id PK
uuid campaign_id FK
string name
int duration_seconds
string asset_url
}
One ad request in, one decision out. Every candidate campaign walks the same chain, and every rejection carries a named reason — which is what turns "fill rate dropped at 14:20" into an answerable question three levels from now.
flowchart LR
REQ["POST /ad-request<br/>member + slot"] --> FETCH["active campaigns<br/>(campaign-service HTTP)"]
FETCH --> F1{"in flight?"}
F1 -->|no| X1["not_active /<br/>outside_flight_window"]
F1 -->|yes| F2{"targeting match?"}
F2 -->|no| X2["targeting_mismatch"]
F2 -->|yes| F3{"brand safe?"}
F3 -->|no| X3["brand_safety_excluded"]
F3 -->|yes| F4{"under frequency cap?"}
F4 -->|no| X4["frequency_capped"]
F4 -->|yes| F5{"within pacing allowance?"}
F5 -->|no| X5["pacing_throttled"]
F5 -->|yes| WIN["most daily budget remaining wins"]
WIN --> REC["record impression<br/>(Redis: freq + spend)"]
REC --> RES["AdDecision + full candidate trace"]
X1 --> RES
X2 --> RES
X3 --> RES
X4 --> RES
X5 --> RES
flowchart LR
ADS["ad-decision-service"] -->|"GET /campaigns?status=active"| CS["campaign-service"]
CS --> PG[("Postgres<br/>system of record<br/>campaigns · budgets · creatives")]
ADS -->|"freq: / spend: keys, 48h TTL"| RD[("Redis<br/>per-day decision state")]
ADS --> PROM["/metrics<br/>ad_decisions_total<br/>ad_candidates_filtered_total{reason}"]
Serving an ad is a decision; reporting it is a fact. Events land keyed on an id the caller chose, so a retried report collides with itself instead of inflating the numbers — idempotency enforced by the primary key, not by a check that races. Delivery numbers are then one GROUP BY over that table, never a second copy of the truth.
flowchart LR
SIM["traffic-simulator<br/>(Day 5)"] -->|"POST /events"| EV["event-service"]
EV --> PK{"event_id already stored?"}
PK -->|yes| DUP["200 duplicate<br/>ad_events_duplicate_total"]
PK -->|no| INS["201 recorded<br/>INSERT into ad_events"]
INS --> MET["ad_events_total{event_type}<br/>ad_spend_micros_total"]
INS --> PGE[("Postgres<br/>ad_events — append only")]
PGE -->|"GROUP BY campaign_id"| DEL["GET /delivery<br/>impressions · clicks · CTR · spend"]
Three services, one metric vocabulary, two dashboards provisioned from files in the repo. Nothing here was clicked into existence: a fresh docker compose up lands on working graphs, and a hermetic test fails the build if a panel loses its query or graphs a metric no service emits.
flowchart LR
CS["campaign-service<br/>/metrics"] --> PROM["Prometheus :9090"]
ADS["ad-decision-service<br/>/metrics"] --> PROM
EVS["event-service<br/>/metrics"] --> PROM
PROM --> D1["Grafana · substrate-health<br/>traffic · errors · p95 · targets"]
PROM --> D2["Grafana · ads-delivery<br/>fill rate · why candidates lost<br/>events · CTR · spend"]
The simulator does not fake a symptom. It changes real configuration through campaign-service's public API, so the incident that follows has an actual cause sitting in an actual table — which is the only way Level 3's RCA agent can be honestly evaluated. No service contains a branch that knows it is being tested.
flowchart LR
OP["POST /scenario<br/>{bad_config_deploy}"] --> SIM["traffic-simulator"]
SIM -->|"PATCH /campaigns/{id}<br/>targeting → AQ"| CS["campaign-service"]
SIM -->|"POST /ad-request<br/>at scenario rate"| ADS["ad-decision-service"]
ADS -->|"reads the changed config"| CS
ADS --> NOFILL["targeting_mismatch<br/>fill rate 43% → 2%"]
SIM -->|"impression / click"| EV["event-service"]
NOFILL --> PROM["Prometheus"]
EV --> PROM
PROM --> GRAF["Grafana<br/>the incident, visible"]
GRAF -.->|"Level 3"| RCA["ops agent<br/>finds the real cause"]
The spec's Level 0 metric turned into a script that exits zero or names the check that failed. Its decision logic is pure and unit-tested; the live run only supplies numbers, so it cannot pass by accident with the substrate switched off. On its first run it failed itself — the traffic-surge check was comparing the surged rate against the surged rate — which is exactly what a gate is for.
flowchart TB
G["platform/level0_gate.py"] --> H["5 health endpoints"]
G --> T["Prometheus scrapes<br/>all 4 services"]
G --> S1["steady<br/>serves ads"]
G --> S2["error_burst<br/>produces 422s"]
G --> S3["traffic_surge<br/>raises the rate"]
G --> S4["bad_config_deploy<br/>collapses fill rate"]
G --> S5["budget_runaway<br/>changes config"]
H & T & S1 & S2 & S3 & S4 & S5 --> R["11/11 · exit 0<br/>platform left healthy"]
SEQ 03 · LEVEL 1 — THE CONTEXT LAYER (DAYS 6–10)
Everything Level 0 produced — four services' source, five ADRs, the devlogs, the runbook, the coding standards, the design spec, the plans, the Compose and Prometheus and Grafana config — is now a corpus. From Level 2 onward, whatever is in here is what the agents believe. Which makes two questions worth deciding rather than defaulting: what gets in, and where the API contracts come from.
Discover, load, hash, store, prune. Every interesting decision sits in a module the pipeline calls, and the store sits behind a protocol — so Day 7 puts a vector store underneath upsert without the pipeline changing a line.
flowchart LR
RULES["CORPUS_RULES<br/>14 globs, in order"] --> DISC["discover()<br/>63 files claimed"]
APPS["4 FastAPI app objects<br/>app.openapi()"] --> SPECS["4 contracts<br/>generated, not scraped"]
DISC --> LOAD["load()<br/>title · component · attributes"]
LOAD --> HASH["normalise CRLF → LF<br/>sha256 content hash"]
SPECS --> HASH
HASH --> UP["store.upsert()<br/>added / updated / unchanged"]
UP --> PRUNE["store.prune()<br/>orphans removed"]
PRUNE --> CORPUS["corpus.jsonl<br/>67 docs · 550 KiB"]
CORPUS -.->|"Day 7"| VEC["vector store<br/>behind the same protocol"]
A crawl of "everything except an exclusion list" grows silently — add a directory and the context layer quietly starts grounding agents in whatever landed there. An ordered rule list shows up in a diff, with a test attached. The ADR template is excluded on purpose: it is a form, not a decision, and grounding an agent in it teaches it to answer with blanks.
| Source type | Docs | What it grounds |
|---|---|---|
| code | 33 | substrate + platform Python — the system as built |
| plan | 6 | how each day was planned, and what the plan missed |
| config | 6 | Compose, Prometheus scrape config, Grafana dashboards |
| adr | 6 | the decisions, their alternatives, and their triggers |
| devlog | 6 | what actually happened, including the defects |
| openapi | 4 | the four service contracts, generated from the apps |
| readme | 2 | the project and the platform directory |
| runbook · spec · standards · backlog | 4 | ops procedure, the design, the rules, the known gaps |
The OpenAPI contracts come from importing the FastAPI app objects and calling app.openapi() — never from curling a running service. A corpus that depends on which containers happen to be up cannot be built in CI, cannot be built on a laptop with Docker stopped, and quietly differs between machines. Ingestion opens no socket.
Content is normalised to \n before hashing, so a Windows checkout and a Linux CI runner agree on what a document is. Run the pipeline twice and the second run reports 67 unchanged, 0 written. Once Day 7 attaches an embedding cost to every changed document, that is the difference between re-running ingestion freely and not re-running it at all.
A whole document is the wrong unit. Nobody asks "what is in campaign-service's OpenAPI contract" — they ask what POST /campaigns accepts. So the split points come from the content's own structure: Python at symbol boundaries via the AST, Markdown at headings, OpenAPI one chunk per operation. Every chunk keeps an anchor — the symbol, heading path, or route that names it. A chunk that cannot name itself can be returned but not defended.
| Strategy | Chunks | p50 chars | Anchored | What it cuts on |
|---|---|---|---|---|
| markdown | 331 | 1428 | 100% | headings, carrying the full path down |
| python_ast | 177 | 386 | 100% | functions and classes, decorators attached |
| openapi | 60 | 851 | 100% | one operation or schema at a time |
| window | 17 | 1963 | 0% | the fallback nothing may fail past |
| all | 585 | 854 | 97% | structure-aware routing by document kind |
| naive_window | 349 | 1960 | 0% | splitting on a character count — the default everyone ships |
The engine is a parameter, not a commitment. Qdrant stores one point per chunk with a named vector per engine, so all four are compared on byte-identical inputs — fairness becomes a property of the schema rather than a discipline someone has to maintain (ADR-0008). The spec called for local sentence-transformers; measuring the corpus inverted that, and ADR-0007 says why.
| Engine | Dims | Cost | Wall | Chunks/s | Role |
|---|---|---|---|---|---|
| gemini-embedding-001 | 3072 | $0.0223 | 10.1s | 58 | quality tier — #1 on MTEB |
| voyage-3.5 | 1024 | — | — | — | free-tier rate limit, not yet run |
| potion-retrieval-32M | 512 | free | 2.1s | 282 | local, offline, no torch, no GPU |
| hashing | 256 | free | 0.2s | 3109 | the CI engine — no deps, no network |
Asked "how often can the same viewer be shown one advert?" — a question containing none of the words frequency, cap, member, or impression — the three engines that ran disagree in exactly the way their price suggests.
| Engine | Score | Top hit |
|---|---|---|
| gemini | 0.765 | Substrate gaps › SG-04 › Multiple frequency-cap windows |
| potion | 0.428 | substrate.ad_decision_service.main.decide |
| hashing | 0.328 | a Day 7 plan section — noise, exactly as designed |
Gemini understood the question. potion found the neighbourhood. The lexical baseline matched nothing meaningful, because no word in the query appears in the answer. That gap is the entire argument for paying for embeddings, and it is measured here rather than quoted from a vendor chart.
flowchart TB
R1["run 1<br/>65 added"] --> C["corpus.jsonl"]
R2["run 2, nothing edited<br/>65 unchanged · 0 written"] --> C
R3["run 3, ADR-0006 + day-06<br/>2 added · 65 unchanged"] --> C
R4["run 4, a devlog deleted<br/>1 pruned"] --> C
C -.->|"an orphan is worse<br/>than a gap — an agent<br/>will cite it"| P["prune()"]
Day 7 could find chunks that mean what you asked. It could not reliably find chunks that are named what you asked — embedding budget_micros lands near "budget" and "spending", and no nearer the one chunk that defines the field than twenty that discuss money. So Day 8 added keyword search (BM25) beside the vector side, fused the two, reranked the result, and — the part that matters — measured whether any of it helped.
The design spec pre-registered a prediction in writing, committed before a line of retrieval code existed: hybrid would win, and its win would come almost entirely from identifier queries. The premise was wrong. There is no aggregate hybrid win.
| Configuration | nDCG@10 | recall@10 | MRR | p50 |
|---|---|---|---|---|
| dense — vector only (gemini) | 0.670 | 0.757 | 0.774 | 812 ms |
| hybrid + LLM rerank | 0.656 | 0.682 | 0.870 | 9,309 ms |
| hybrid + heuristic rerank | 0.639 | 0.689 | 0.791 | 783 ms |
| hybrid — reciprocal rank fusion | 0.628 | 0.674 | 0.844 | 842 ms |
| hybrid — weighted fusion | 0.624 | 0.678 | 0.834 | 802 ms |
| lexical — BM25 only | 0.395 | 0.496 | 0.556 | 2.9 ms |
Vector search alone beat every hybrid configuration. Adding the lexical side lowered the score.
The prediction was most specific about identifier questions — where is budget_micros enforced — where keyword search was supposed to dominate. It came last.
| Configuration | identifier | conceptual | cross-document |
|---|---|---|---|
| dense (gemini) | 0.711 | 0.598 | 0.721 |
| hybrid (RRF) | 0.696 | 0.583 | 0.589 |
| hybrid + heuristic | 0.652 | 0.601 | 0.681 |
| hybrid + LLM | 0.696 | 0.583 | 0.711 |
| hybrid (weighted) | 0.661 | 0.620 | 0.569 |
| lexical (BM25) | 0.426 | 0.452 | 0.250 |
Gemini's embeddings are strong enough on this corpus that they win the category that was supposed to be lexical search's home ground. What I would have shipped without the evaluation is "hybrid retrieval with reranking" — slower, more complex, and worse. It would have looked like progress.
A retrieval score is only as honest as the ground truth under it, and the cheap ways to build one are all rigged: label from what the retriever returned and you reward what you built; derive queries from chunk text and BM25 wins by construction; judge only the favoured system's results and nothing else can ever score.
The first run had a tell: hybrid-llm scored byte-identical to hybrid — same nDCG, recall and MRR to three decimals, in every category. That is impossible if a reranker is doing anything, so it wasn't. Every Gemini call was failing and the reranker was falling back to the fused order, exactly as designed. Three causes, stacked, and none was visible to a test suite that fakes the network:
enum. Gemini's responseSchema is a restricted dialect that rejects integer enums outright — 400 on every request.gemini-2.5-flash now returns 404 — "no longer available to new users". A hardcoded model id went stale between writing the plan and running it.chunk_ids past a model that spends output budget thinking runs to ~5,000 tokens, and a reply that starts like JSON and stops mid-value raises exactly like malformed JSON. Raised to 8,192.A fake transport can validate neither a provider's schema dialect, its model catalogue, nor how long its answers run. This is Day 7's lesson in a new place: anything faked in tests needs one live round-trip before it is trusted. The silent degrade path was the correct behaviour and an observability hole — a completely broken reranker produced entirely plausible numbers, which is why the eval now counts every degrade and prints the total whether or not it is zero.
nDCG is not a percentage of right answers: it gives half credit for partially-relevant chunks, penalises a good answer at rank 3, and is measured against a strict hand-written key. The companion numbers are healthier — 76% of relevant chunks reach the top 10, and the first useful answer is usually rank 1 or 2. The real deliverable is that every idea below is now testable against a fixed answer key instead of argued about: a purpose-built reranker, contextual chunk headers, a larger second-opinion gold set, query rewriting for the conceptual shapes that scored lowest.
SEQ 04 · DAY TRACKER
| Day | Lvl | Deliverable | Status |
|---|---|---|---|
| 01 | 0 | Repo, architecture docs, README, running doc v1, ADR-0001, infra skeleton, CI | ● SHIPPED |
| 02 | 0 | campaign-service (CRUD, Postgres, tests, OpenAPI) | ● SHIPPED |
| 03 | 0 | ad-decision-service (targeting, freq capping, brand safety, pacing) | ● SHIPPED |
| 04 | 0 | event-service + observability stack (Grafana dashboards, JSON logging lib) | ● SHIPPED |
| 05 | 0 | traffic-simulator + failure injection · Level 0 quality gate | ● SHIPPED |
| 06 | 1 | Document ingestion pipeline | ● SHIPPED |
| 07 | 1 | Chunking + embedding strategies (comparison documented) | ● SHIPPED |
| 08 | 1 | Hybrid retrieval + reranking + pooled-judgement eval | ● SHIPPED |
| 09 | 1 | AST knowledge graph — what depends on X, and a call graph that audits itself | ● SHIPPED |
| 10 | 1 | MCP server + retrieval eval suite · Level 1 quality gate | ○ QUEUED |
| 11 | 2 | Code-generation agent | ○ QUEUED |
| 12 | 2 | Test-generation agent + mutation testing | ○ QUEUED |
| 13 | 2 | PR pre-review agent (GitHub Actions) | ○ QUEUED |
| 14 | 2 | Deployment-validation agent | ○ QUEUED |
| 15 | 2 | E2E pipeline: ticket → code → test → PR → deploy · Level 2 gate | ○ QUEUED |
| 16 | 3 | Log intelligence pipeline | ○ QUEUED |
| 17 | 3 | Incident triage agent | ○ QUEUED |
| 18 | 3 | RCA agent | ○ QUEUED |
| 19 | 3 | Guided resolution + runbook generation | ○ QUEUED |
| 20 | 3 | Self-healing loop E2E demo · Level 3 quality gate | ○ QUEUED |
| 21 | 4 | Agent communication protocol | ○ QUEUED |
| 22 | 4 | Parallel agent execution | ○ QUEUED |
| 23 | 4 | Conflict resolution + escalation | ○ QUEUED |
| 24 | 4 | Human-in-the-loop gates | ○ QUEUED |
| 25 | 4 | Actor-Critic process-audit evals · Level 4 quality gate | ○ QUEUED |
| 26 | 5 | AI-first dev environment (devcontainer, one-command setup) | ○ QUEUED |
| 27 | 5 | CI/CD AI gates | ○ QUEUED |
| 28 | 5 | Consolidated eval dashboard | ○ QUEUED |
| 29 | 5 | Grand demo (feature E2E through the whole platform) | ○ QUEUED |
| 30 | 5 | Launch: open-source release, outreach kickoff | ○ QUEUED |
SEQ 05 · DECISIONS
bellwether (noun) — a leading indicator; the thing you watch to know where something is heading before the rest of the picture catches up. That is the job twice over: the first AI engineer on a team is a leading indicator of how that team will build a year from now, and the eval scoreboard is a leading indicator of whether the agents are actually making the work better.
Four engines embed the same corpus, and the comparison only means something if every engine sees identical chunks. Qdrant stores one point per chunk carrying a named vector per engine, so switching engines is a query parameter and fairness is structural rather than procedural. Chroma has no clean equivalent; pgvector would have put the AI foundation's storage inside the ads platform's own database. The dangerous part, found only against a running server: PUT /points replaces a point, so writing the second engine's vector silently wiped the first. Every test passed. The chart would have rendered beautifully and been a lie.
The design spec called for local sentence-transformers to save money. The corpus is 149k tokens — about two cents with the best model available — while PyTorch is 2.5 GB and a resident process competing with Docker on a 16 GB laptop. The quality tier turned out to be the lighter option locally, because the compute happens on someone else's hardware. The free tier is the real constraint: Gemini allows 100 embed requests per day, and one pass needs 585. Local potion stays as the free-iteration tier and a dependency-free hashing engine keeps CI offline.
What the context layer holds is what every agent from Level 2 onward will believe and cite, so both halves of it are decided rather than defaulted. What gets in is an ordered list of rules in code — a crawl-minus-exclusions grows silently, and nobody reviews a diff that does not exist. Where the contracts come from is app.openapi() on the imported app objects, never a curl against a running service: a corpus that depends on which containers are up cannot be built in CI. Content is normalised before hashing so Windows and Linux agree, and orphans are pruned — a document whose source has left the repo is worse than a missing one, because an agent will happily cite it.
The simulator breaks the platform by changing real configuration through public APIs — a bad config deploy really does PATCH every campaign's targeting, and the decision path then behaves perfectly while fill rate collapses. No service has a branch that knows it is being tested. The reason is Level 3: an ops agent evaluated against a mocked failure can only ever find the mock. Because the change is real, recovery is real too — steady is a genuine rollback. Day one of running it found two real defects in the platform it was pointed at.
A delivery report gets retried whenever a network hiccups, so the caller chooses the event_id and the primary key does the deduplication — a check-then-insert races, an insert that collides cannot. Delivery numbers are then a single GROUP BY over the event table rather than a rollup that can drift out of sync with it. The trigger that flips this: Day 5's simulator, when sustained load pushes /delivery past ~200 ms.
Frequency counters and daily pacing spend are written on every served impression, scoped per member, and worthless after midnight — a counter with a TTL, not a row. They live in Redis under day-scoped keys that expire themselves; campaign-service stays the system of record, read over its HTTP API rather than its tables. Day 4 update — the trigger fired. event-service made impressions durable, and the decision narrowed rather than reversed: Postgres now holds the auditable record of what was served, Redis keeps only the hot counters the serving path reads. The gap between the two is not a bug — it is reporting loss, and it is a panel on the dashboard.
create_all over Alembic, for nowThe schema has one writer and no data anyone depends on, and it will churn daily through Day 5. Migrations written against a schema that changes every day are noise, not safety. The trigger that flips this: the first second service that reads these tables.
Four services run by one developer on one machine. Compose gives one-command startup and reproducible demos; Kubernetes adds operational overhead while proving nothing relevant to AI foundation engineering. Choosing infrastructure proportionate to the problem is the signal.
SEQ 06 · EVAL SCOREBOARD
Actuals publish as each level's quality gate runs. Empty cells are honest cells. The Level 0 number is what platform/level0_gate.py printed against the running stack — five health endpoints, one scrape check, and one per injectable failure mode.
| Lvl | Metric | Target | Actual |
|---|---|---|---|
| L0 | Services healthy under simulator load; failure injection works | 100% | 11/11 · 100% |
| L1 | Context relevance (50-query synthetic dataset) | >85% | — |
| L1 | Retrieval latency | <500ms | — |
| L2 | Generated code compiles / passes lint + types | 100% | — |
| L2 | Generated test pass rate; mutation score improvement | >90% | — |
| L2 | PR reviewer seeded-bug catch rate | published | — |
| L3 | Synthetic incidents flagged | 5/5 | — |
| L3 | RCA correct root cause | >80% | — |
| L4 | Complex tasks completed by parallel agents | 3/3 | — |
| L4 | HITL approval acceptance rate | >90% | — |
| L5 | Fresh clone → running system | <10 min | — |
| L5 | CI/CD gates pass | 100% | — |