Contracts of Behavior: Traits, Generics, and async

Chapter 6's structs and enums answered "what does the data look like". This chapter switches to verbs: "what can the data do". By the end you should be able to:

• open jackdaw/src/provider.rs and name the three methods a type must fulfill to call itself a provider;
• explain why the router's candidate list holds Box<dyn Provider> rather than any one concrete vendor type;
• look at #[async_trait::async_trait], async fn, and .await and say what gets boxed, who yields, and who drives.

The contract: what a type must do to count as a Provider

Start with the world without a contract. jackdaw is a gateway: one request may go to any OpenAI-compatible upstream, and when you run unit tests you want the same path exercised with in-memory fakes instead of real HTTP. If the routing code depended on one concrete type, every new upstream and every mock would drag the router and the tests along with it — "switch to another one" would have no shared meaning.

A trait is that contract: a set of method signatures declaring what something does, with no data and no implementation. A type fulfills the signatures with impl Provider for Xxx, and from then on it counts as a provider. Where chapter 6's structs were nouns, traits are verbs.

/// One upstream LLM API. [`crate::openai`] implements it for any
/// OpenAI-compatible endpoint; unit tests implement it with in-memory fakes.
#[async_trait::async_trait]
pub trait Provider: Send + Sync {
    /// Provider name, as used in `[providers.<name>]` and model references.
    fn name(&self) -> &str;

    /// Send the request and wait for the full JSON response.
    async fn complete(&self, request: ChatRequest) -> Result<ChatResponse, ProviderError>;

    /// Send the request and return the raw SSE byte stream. A successful
    /// return means the upstream already *accepted* the request (the HTTP
    /// status was checked); failures after the first byte are delivered
    /// through the stream itself — at that point failover is impossible.
    async fn complete_stream(&self, request: ChatRequest) -> Result<ByteStream, ProviderError>;
}

How to read it: start at pub trait Provider: Send + Sync — the part after the colon is an extra promise from implementers, explained in the last section of this chapter. All three method signatures end in a semicolon with no body: a contract fixes signatures, nothing else. name reports the provider's name; complete waits for a full JSON response; complete_stream returns a stream. The parameter and error types belong to the contract too: ChatRequest wraps the request body (you traced its ownership in chapter 5), and ProviderError classifies failures (chapter 12). Read the doc comments as well: crate::openai implements the contract for real endpoints, and unit tests implement it with in-memory fakes — one contract, two fulfillments.

impl blocks: inherent methods and fulfilling a contract

Fulfillment happens in impl blocks. The same file shows the other kind of impl — one that fulfills no contract at all and just adds methods to a type you defined yourself:

#[derive(Debug, Clone)]
pub struct ChatRequest(pub Value);

impl ChatRequest {
    // ...
    /// Whether the caller asked for an SSE stream. Absent means `false`,
    /// matching the OpenAI API.
    pub fn wants_stream(&self) -> bool {
        self.0
            .get("stream")
            .and_then(Value::as_bool)
            .unwrap_or(false)
    }
    // ...
}

How to read it: look at pub struct ChatRequest(pub Value) first — it wraps a JSON value, the kind you met in chapter 6. The gateway deliberately does not model fields like tools or temperature; it forwards them untouched, and the comment at the top of the file spells out that trade-off. Then wants_stream: &self is a read-only borrow (chapter 5), self.0 digs out the inner JSON, and get("stream") through unwrap_or is the Option pipeline from chapter 6 — a missing field means false, matching the OpenAI convention. Distinguish the two kinds of impl blocks: impl ChatRequest here is an inherent method; impl Provider for SomeType is the contract-fulfilling kind.

Generics and trait objects: two ways to code against a contract

Given the contract, how do you write code that holds for any Provider? Rust offers two roads.

The first is generics: a signature like fn pick<T: Provider>(p: T, ...) reads as "T is any type that satisfies the Provider contract". The compiler generates a separate copy of the code for each T you actually use — monomorphization. No runtime cost, but every type must be known at compile time.

The second road handles what generics cannot: routing (chapter 13) works from a candidate list that is only decided at runtime, and the first and second entries may be different concrete provider types — while every element of a Vec must share one type. The answer is a trait object: Box<dyn Provider>. Read dyn as "dynamic". Box is the simplest smart pointer: the value lives on the heap, and the box itself stays one fixed size, whatever it holds. So every element of Vec<Box<dyn Provider>> is the same size — a pointer to the provider on the heap, plus a pointer to its method table — and who exactly sits underneath stops mattering. The cost is one extra pointer hop per call; the payoff is polymorphism decided at runtime.

One-line rule: if you know the type at compile time, use generics; if you learn it at runtime, use Box<dyn Trait>. A gateway lives mostly in the second case.

Why async-trait exists

The line above the trait, #[async_trait::async_trait], is an attribute macro — read that as "code that rewrites your code before compilation" (simplified; the real machinery is deeper, but that is the reading you need). The problem it solves: an async fn is really "a plain function returning some anonymous Future type", and each impl's async fn body compiles to a different anonymous type. But dyn Provider needs uniform method signatures to build its method table. With every signature different, no table — so early Rust simply banned async fn in traits.

The fix is boxing: the macro rewrites async fn complete(...) -> Result<...> into a plain function returning Pin<Box<dyn Future<...>>>, boxing each implementer's own Future type into one uniform type. Signatures now match, and dyn Provider works. A handwritten version of the same move sits in this very file — the return type of complete_stream:

/// Byte chunks of a streaming (SSE) response body, in arrival order.
pub type ByteStream = Pin<Box<dyn Stream<Item = Result<Bytes, ProviderError>> + Send>>;

How to read it: this is a type alias — a short name for a long type. Read it inside out: Stream<Item = ...> is a stream yielding Bytes (chunks of bytes) or errors; Box<dyn ...> boxes it into one type; Pin pins the value in memory so it cannot move, which Future and stream internals require (on a first pass, just recognize the Pin<Box<...>> combination and move on); + Send is the next section. One boundary note: newer Rust lets you write async fn in traits natively, but combining that with dyn is still limited, so async-trait remains common across the ecosystem. When you see it, know it is doing the boxing rewrite.

await is a yield point, not a thread

The motivation: a gateway serves several agent requests at once, each waiting on an upstream response. If every wait occupied an OS thread, you would run out of threads first. async exists so that waiting costs no thread. Three facts, at reading level:

1. Calling an async fn does nothing yet. It returns a Future immediately — a description of a value that arrives later. No .await, no execution.
2. .await is a yield point. It drives the Future forward; while that Future waits on the network, the current task hands its execution back, the thread runs other tasks, and when the awaited event fires, work resumes from the break. What yields is a task, not a thread — and nothing spawns a new one.
3. Something has to drive all of this: the runtime. The standard library ships none; jackdaw uses Tokio, the de facto standard — the scheduler that knows when to call which task back (chapters 9 and 14).

Compare it to a fast-food buzzer: order, sit down, come back when it buzzes. Nobody stands at the counter waiting for you; one thread carries dozens or hundreds of tasks at once. One mismatch matters: a Future never buzzes on its own — the runtime is always the one watching. Back to the mechanism: that is the whole reading of async.

Finally, Send / Sync: trait Provider: Send + Sync means "the implementer promises this value can safely move across threads and be shared". A gateway is a concurrent service, and several tasks may use the same provider at once, so the promise goes into the contract itself. When you hit a Send/Sync compiler error, your first thought should be "a data race just got stopped" — chapter 3 put concurrency on the must-read-it-yourself list.

Recap

• A trait is a contract of behavior: trait Provider lets the router depend on the contract, not the vendor — any upstream plugs in, and tests run on fakes.
• impl Trait for Type fulfills a contract. Generics specialize at compile time; Box<dyn Provider> unifies types at runtime. A gateway mostly wants the latter.
• #[async_trait::async_trait] boxes each async fn's return type so dyn works; ByteStream is the same move written by hand.
• .await is a yield point; Tokio drives. Recognize Pin<Box<...>> and move on.
• That closes the language part. Next: jackdaw's Cargo.toml and directory tree — how all this code gets organized, built, and tested (chapter 8).

Self-Check

1. You arrive with interface / Protocol experience. Name two ways trait Provider differs from them.
2. Someone changes the failover candidate list from Vec<Box<dyn Provider>> to Vec<OpenAiProvider> (a stand-in name for the concrete type implementing Provider in openai.rs). Predict what happens when a test fake provider joins the list, and why.
3. Why does complete_stream return Pin<Box<dyn Stream<...> + Send>> instead of making the stream a generic parameter?

Answers

1. Any two: a trait can be implemented for types you don't own; trait methods can ship default implementations that implementers inherit unless they override them; and Rust offers both static dispatch (generics) and dynamic dispatch (dyn Trait), where most languages give interfaces only the dynamic kind.
2. A compile error. Vec<OpenAiProvider> fixes the element type; the fake is a different type, so it doesn't fit. To mix implementations, the list must unify on the shared contract: Box<dyn Provider>.
3. The concrete stream type depends on implementation details (which HTTP client, which response-body stream), and callers neither need nor should know them. Boxing into one type fixes the trait's method signature so dyn Provider stays possible; + Send keeps it safe across tasks — the gateway is concurrent.