Tests, CI, and Shipping

In Chapter 13 you wrote unit tests next to the router; in Chapter 14 you probably poked the gateway with curl by hand. This chapter makes that checking permanent: turn it into tests, hand it to CI, and end with a binary you can copy around. By the end you can answer:

• what the 0 in TcpListener::bind("127.0.0.1:0") means, and why integration tests can't do without it;
• when CI goes red under cargo test versus cargo clippy -- -D warnings, what you fix in each case;
• where the cargo build --release artifact lands, and what lto and strip in Cargo.toml each trade away.

Where tests live: #[test] up close, tests/ end to end

Unit tests live in the same file as the code under test and check one function's inputs against its outputs; functions marked #[test] get compiled and run in place by cargo test.

But unit tests have limits. jackdaw's value isn't in any single function — it's in the glue: HTTP comes in → parse JSON → the router rewrites the model name → the request goes to the provider → the response goes back with the x-jackdaw-model header. That chain crosses several modules, and no single-link unit test can tell you whether the whole thing works. Hence jackdaw/tests/integration.rs: by Cargo convention, every .rs file under tests/ is a separate integration-test crate that may only use jackdaw's public API — exactly the position of a real caller. cargo test runs both layers; "all green" means the chain ran end to end over real HTTP.

A stub provider on a random port

The first problem an integration test hits: where does the upstream provider come from? Calling OpenAI for real costs money, needs keys, and gives unstable results. The answer is a local mock whose behavior the test controls completely:

async fn spawn_stub(mode: StubMode) -> Stub {
    let received: Arc<Mutex<Vec<Value>>> = Arc::new(Mutex::new(Vec::new()));
    // ...
    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
    let addr = listener.local_addr().unwrap();
    tokio::spawn(async move {
        axum::serve(listener, app).await.unwrap();
    });
    Stub { addr, received }
}

Start at the TcpListener::bind line: the port argument is 0, which means "don't pick one — OS, choose a free port", and the next line reads the chosen address back with local_addr(). tokio::spawn moves the axum server into a background task so the test can keep going. The returned Stub also carries received, a mutex-wrapped vector that records one entry per request body the stub sees — after the test, you read it back to assert what the upstream actually saw.

Why insist on 0? A fixed port collides with itself when tests run in parallel; worse, it can collide with a real service running on your machine, and the test mistakes that service for its stub.

StubMode picks the behavior: Echo answers with a completion echoing the (rewritten) model name, Fail always returns a given status, StreamOk emits a complete SSE stream, StreamThenDie emits one delta and then cuts the connection. The config side stays just as clean: make_config builds a TOML string and parses it, and api_key_env points at HOME, an environment variable that always exists in tests — the stub ignores the Authorization header anyway. The comment at the top of the file says it plainly: nothing leaves 127.0.0.1.

One request through the whole chain

Take the most representative one — routing-group failover:

#[tokio::test]
async fn order_group_fails_over_to_the_next_member() {
    let broken = spawn_stub(StubMode::Fail {
        status: StatusCode::INTERNAL_SERVER_ERROR,
        body: "upstream exploded",
    })
    .await;
    let healthy = spawn_stub(StubMode::Echo).await;
    // ...
    let response = post_chat(
        gateway,
        TOKEN,
        r#"{"model":"group/coder","messages":[{"role":"user","content":"hi"}]}"#,
    )
    .await;

    assert_eq!(response.status(), 200);
    assert_eq!(response.headers()["x-jackdaw-model"], "healthy/m-b");
}

Start with the two spawn_stub calls: one stub fails with 500 forever, the other echoes normally; the make_config call hidden in the ellipsis packs them into group/coder with the order strategy, broken member first. #[tokio::test] gives the test a Tokio runtime so the body can .await (Chapter 7); post_chat plays the agent with reqwest, sending real HTTP at the gateway. Two assertions, but end-to-end meaning: the 200 says the broken member really failed and failover really happened; x-jackdaw-model pointing at healthy/m-b says the response came from the second member — Chapter 14's transparency header doubles nicely as forensic evidence here.

The sneakier test is no_failover_after_the_first_byte_honest_failure_instead: the dying stub answers 200, emits the first delta, then breaks the connection mid-stream. The assertions check that "partial" really arrived and that the failure was reported in-band — and that healthy.received is empty: once the first byte is out, silently switching members is forbidden. That "bytes have already left the process" timing is something a function-level unit test cannot reproduce; only a real TCP connection can.

Two workflows, two kinds of guarding

Chapter 3's four arbiters and Chapter 8's subcommand table meet in this section: CI is those same commands, run automatically on a server. Green locally is a fact about your machine; CI turns it into a fact about the server. The repo has two workflows, guarding different things:

name: Rust CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Install stable Rust toolchain
        uses: dtolnay/rust-toolchain@stable

      - name: Run tests
        run: cargo test --manifest-path jackdaw/Cargo.toml

      - name: Run clippy
        run: cargo clippy --manifest-path jackdaw/Cargo.toml -- -D warnings

Read the on block first — every push to main and every PR trigger it, so during review it is the gate. Four steps: check out, install the stable toolchain, run tests, run clippy. --manifest-path is there because Cargo.toml lives in the jackdaw/ subdirectory; -- -D warnings promotes warnings to errors. Red under cargo test means behavior regressed; red under clippy means the code breaks convention. Two reds, two fixes — but either way you fix the code, not the CI.

deploy.yml ships this book's VitePress site, not jackdaw. The skeleton is the same (checkout → install toolchain → run commands); the differences are all in configuration. It triggers only on push to main (plus manual dispatch), so nothing deploys during review. The work splits into a build job and a deploy job, where needs: build makes sure the site is built and uploaded as an artifact before the second job publishes it to GitHub Pages. And permissions grants pages: write and id-token: write, plus contents: read — and nothing more. Read one workflow and you can read them all: on tells you when it wakes up, steps tell you what it does.

TIP

Run cargo test and cargo clippy locally before you push. CI is a second opinion, not the first place your tests ever run.

Release profile and the artifact

The release profile decides what the artifact looks like. Two defaults: dev compiles fast and keeps debug info; release optimizes hard. jackdaw adds two more switches to release:

# Shipping profile: small binaries beat last-percent compile speed for a
# teaching project you build once and copy around.
[profile.release]
lto = true
strip = true

Read the comment first — this comment and this [profile.release] appeared in Chapters 8 and 9, so the trade isn't re-read: a teaching project gets built once and copied around, and a small binary beats the last bit of compile speed. What's new here is what lands in target/. lto (link-time optimization) lets the compiler inline across crate boundaries, at the cost of a slower link; strip drops the symbol table, which shrinks the binary but makes it useless for debugging. After cargo build --release, the executable sits at target/release/jackdaw (dev builds land in target/debug — Chapter 8 covered the split inside target/), with all its Rust dependencies compiled into that one file; copy it to any machine on the same platform and it runs. Building for a different platform is cross-compilation: rustup target add the target, then cargo build --release --target x86_64-unknown-linux-gnu. Chapter 9 covered the concept; look up the flags when you need them.

Recap

Port 0 asks the OS for a free port, so stubs can play provider in parallel without collisions. Red under cargo test means fix behavior; red under clippy -D warnings means fix convention — never edit the CI. The release artifact is target/release/jackdaw, where lto and strip trade compile time for size. The next chapter is the last one: taking this reading habit into the territory of the three paradigms.

Self-Check

1. You want to copy the "stub on a random port" pattern into a project in another language. What does the 0 in bind("127.0.0.1:0") correspond to in a language you already know? A teammate insists on the fixed port 3425 — which two concrete risks do you point out?
2. A push turns Rust CI red: the log shows cargo test passing and cargo clippy -- -D warnings failing. Your teammate says "tests pass, it's just warnings, merge it." Where does that judgment go wrong — twice?
3. Why can't "no failover after the first byte" be verified by a unit test that just calls functions? What exactly does StubMode::StreamThenDie manufacture?

Answers

1. 0 means "let the OS assign a free port" — the same as HTTPServer(("127.0.0.1", 0), ...) in Python or net.Listen("tcp", "127.0.0.1:0") in Go; afterwards you read the real port back from the listener. Two risks with a fixed port: tests running in parallel (or a leftover process from the last run) make the second bind on the same port fail outright; and a real service may already be listening there (magpie defaults to 3425), so the stub never sees the request and the test mistakes the real service's responses for its own.
2. First: -D warnings makes the contract "zero warnings" — red means the diff doesn't meet the merge bar, not "just warnings". Second: both ways out of the bind are wrong — merging without fixing drifts main away from that zero-warnings contract, and loosening or routing around the CI breaks the repo's rule (writers don't edit the CI file). The one right fix is to make the code (or the AI that wrote it) remove the warning.
3. The premise of the rule is that "bytes have already left the gateway process and reached the client" — a cross-process physical fact. A function-level call has no byte stream and no connection, so it can't reproduce one. StreamThenDie first emits a complete delta (so the first byte really leaves), then errors on the stream 100 ms later, forcing the gateway to choose between "report the failure honestly in-band" and "silently retry on another member". The test asserts it chose the former, and that the healthy stub received zero requests.