Routing and Model Groups
At the end of Chapter 12 you finished trait Provider: jackdaw can call one concrete model. But agents usually ask for group/coder — a group name, not a single model. This chapter reads the layer in between: routing. By the end you can answer:
- Given
resolveinrouter.rs: which steps take the string"group/coder"to a candidate list? Under rotate, which member starts request one, and which starts request two? - An upstream timeout and an upstream 404 — how does the retry loop treat each one? When does each of the three
RouteErrorvariants appear? - A streaming answer breaks halfway through. Why does jackdaw fail honestly instead of switching to the next model?
One Name, Two Meanings
The model field in a request body is just a string: it may name one concrete model, or a group. Until that string becomes a type, every call site has to guess for itself. Step one is parsing it into an enum:
/// A parsed model reference.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ModelRef {
/// Direct reference: name of a `[providers.<name>]` table plus the
/// upstream model id.
Provider { provider: String, model_id: String },
/// Routing-group reference: `group/<id>`.
Group { id: String },
}
impl ModelRef {
pub fn parse(raw: &str) -> Result<Self, String> {
let raw = raw.trim();
let Some((head, rest)) = raw.split_once('/') else {
return Err(format!(
"invalid model reference {raw:?}: no `/` found. \
Expected `<provider>/<model-id>` or `group/<id>`"
));
};
// ...
if head == RESERVED_PROVIDER_NAME {
// The reserved name routes to groups, so `group/coder` means
// "group with id `coder`", never "provider named group".
Ok(ModelRef::Group {
id: rest.to_string(),
})
} else {
Ok(ModelRef::Provider {
provider: head.to_string(),
model_id: rest.to_string(),
})
}
}
}How to read it: start with the two variants — they map exactly onto the two legal spellings. parse splits the string at the first / into head and rest. If head equals the reserved word group (the constant RESERVED_PROVIDER_NAME), you get Group; otherwise head is the provider name and rest becomes model_id verbatim — so openrouter/deepseek/deepseek-chat yields the model id deepseek/deepseek-chat; multi-segment ids stay intact. The Err payload is user-facing error text; callers print it as-is. Config validation (Chapter 11) builds on this: no provider may be named group.
From Reference to Candidates: order and rotate
Parsing recognizes the name; routing answers "in what order do we try the members for this request." resolve turns a ModelRef into Vec<Candidate>:
/// One concrete upstream to try: a provider handle plus the model id that
/// must be written into the request body.
#[derive(Clone)]
pub struct Candidate {
pub provider: Arc<dyn Provider>,
pub model_id: String,
}How to read it: a Candidate is a provider handle (Arc<dyn Provider>, the trait object from Chapter 7) plus the model id to write into the request body. A single-model reference resolves to a list of length 1; every member string of a routing group goes through ModelRef::parse again and a lookup in the providers table. Members must be provider references — no nested groups. validate rejects nesting once at load time, and resolve rejects it again: two layers of defense keep resolve total — Ok or Err for any input, never a panic.
With the members in place, one question remains: who starts.
match group.strategy {
Strategy::Order => {}
Strategy::Rotate => {
if !members.is_empty() {
let start =
self.rotate_counter.fetch_add(1, Ordering::Relaxed) % members.len();
members.rotate_left(start);
}
}
}How to read it: start at match group.strategy. The Order arm is empty — doing nothing is the strategy: keep the configured order. The Rotate arm takes three steps: fetch_add(1, …) atomically increments and returns the old value (0 on the first call, 1 on the second); the modulo by member count gives the start index; rotate_left rotates the list to put that member first. A two-member group: request 1 starts at a, request 2 at b. That is rotate / round-robin: load spreads out, and the entire history the scheme keeps is one integer — no success or failure statistics. Ordering::Relaxed is the memory-ordering argument of the atomic operation; on a first read, all you need is that it lets concurrent requests pick up different starts — the deeper concurrency story stays out of this chapter.
The Retry Loop: Where Failover Happens
The candidate list is a plan; the loop decides behavior. First, pin down the three outcome shapes, RouteError: Unresolved (the reference didn't parse, or names no provider or group in the config), Fatal (the upstream refused the request; status and body pass through untouched), AllFailed (every member failed with a retryable error; one detail line per member). The classification comes from Chapter 12: network errors / timeouts / 429 / 5xx → Retryable; every other 4xx → Fatal.
for candidate in &candidates {
let attempt = candidate
.provider
.complete(request.with_model(&candidate.model_id))
.await;
match attempt {
Ok(response) => return Ok((candidate.display(), response)),
Err(ProviderError::Retryable(why)) => {
tracing::warn!(
candidate = candidate.display(),
error = %why,
"retryable failure, trying next model"
);
details.push(format!("{}: {}", candidate.display(), why));
}
Err(ProviderError::Fatal { status, body }) => {
// The request itself is the problem — the next model
// would hit the same wall. Stop and pass it through.
// ...
return Err(RouteError::Fatal { status, body });
}
}
}
Err(RouteError::AllFailed { details })How to read it: data flows from request.with_model(&candidate.model_id) into the current candidate's provider. The Ok arm returns with the winner; candidate.display() is provider-name/model-id — the group name ends here, resolution always names a winner (that string later becomes the x-jackdaw-model response header; Chapter 14). Retryable: log it, push one line into details, try the next. Fatal: return immediately, no failover — the comment says it plainly: the request itself is the problem, and the next model hits the same wall (a mistyped model id returns 404 no matter which candidate answers). Only when the loop ends without success do you get AllFailed { details }, exactly one line per member. That loop is the whole runtime story of failover; a single-model reference rides the same pipe — a list of length 1, failing into AllFailed with one detail line.
After the First Byte, No More Failover
Why can't the streaming path "switch and resend from the top" when the stream breaks? The short answer first: bytes already sent to the agent cannot come back. The answer sits verbatim in the doc comment on stream:
/// Streaming path. Failover is only possible *before* the first byte
/// reaches the client, so this loop stops at the first candidate whose
/// upstream accepts the request (status already checked by the
/// provider). Everything after that point — including a stream that
/// breaks mid-answer — is delivered in-band through the stream, because
/// rewinding bytes the client already received is impossible.
pub async fn stream(&self, request: ChatRequest) -> Result<(String, ByteStream), RouteError> {
// ...
}How to read it: the loop can fail over only before the first byte reaches the client; once the upstream accepts the request and the status check passes, the winner locks in. Everything after that — including a stream that breaks mid-answer — travels through the stream itself. The mechanical reason: an HTTP response is a one-way pipe, and SSE has no "recall the last N events" operation; the agent already holds half an answer, and restarting with another model either repeats content or splices two models' text into one. Compare non-streaming complete: the body is assembled before anything returns, the client has zero bytes, so switching costs nothing and the loop is safe.
WARNING
"Auto-switch and resend when the stream breaks" sounds helpful; in practice it stitches two half-answers together. A plausible-looking but incoherent answer does far more damage than an honest error in the stream.
Testing the Loop with a Mock
Failover depends on a real upstream failing — you can't run tests against a live API, and jackdaw's spec demands fully offline tests with no API keys. So the tests use a mock: an in-memory FakeProvider whose behavior is scripted by the three-variant enum Behavior (Ok/Retryable/Fatal):
struct FakeProvider {
name: String,
behavior: Behavior,
calls: AtomicUsize,
}
#[async_trait::async_trait]
impl Provider for FakeProvider {
fn name(&self) -> &str {
&self.name
}
async fn complete(&self, _request: ChatRequest) -> Result<ChatResponse, ProviderError> {
self.calls.fetch_add(1, Ordering::Relaxed);
match self.behavior {
Behavior::Ok => Ok(ChatResponse(json!({"role": "assistant"}))),
Behavior::Retryable => Err(ProviderError::Retryable("stub 500".into())),
Behavior::Fatal => Err(ProviderError::Fatal {
status: 404,
body: "{\"error\":{}}".into(),
}),
}
}
// ...
}How to read it: the stand-in implements the full Provider trait, but complete contains no HTTP — just a pattern match on the script. calls increments on every invocation; it is the observation point for asserting how many times each member was tried. router_with(&group_config("order", …), …) builds the Router: config, providers, and group all assembled from one TOML string on the spot.
#[tokio::test]
async fn order_fails_over_to_the_next_member() {
let a = FakeProvider::shared("a", Behavior::Retryable);
let b = FakeProvider::shared("b", Behavior::Ok);
let router = router_with(
&group_config("order", "\"a/m-a\", \"b/m-b\""),
vec![a.clone(), b.clone()],
);
let (used, _) = router
.complete(ask_model("group/g"))
.await
.expect("b answers");
assert_eq!(used, "b/m-b");
assert_eq!(a.calls(), 1, "the failing member is tried exactly once");
assert_eq!(b.calls(), 1);
}How to read it: a must fail, b must succeed. The test asserts three things: the winner is b/m-b; the failing member a was tried exactly once (no hidden retries); b was called exactly once too. The same file covers: a Fatal failure stops the loop (b.calls() == 0); all-retryable failures produce one details line per member; rotate makes two requests start at a, then b; the streaming loop fails over before the first byte as usual.
TIP
Run just this module: cargo test --manifest-path jackdaw/Cargo.toml router (filter by module name). Fully offline, done in seconds.
Against magpie: Why Not smart, usage, intent
magpie's routing groups offer four strategies — smart / order / rotate / usage — plus rule-based routing (tokens ≥ N, contains images, reasoning, agent, and the built-in intent classifier), and groups may nest (up to 8 levels, acyclic). jackdaw deliberately implements only order/rotate + failover, no nesting. That is a teaching choice, not a missing feature:
- smart and usage both need extra state: statistics on usage, latency, and success rates — collection, storage, eviction; each is an order of magnitude heavier than "one atomic counter," and behavior drifts with the statistics: the same request may route differently tomorrow than it does today, which makes incidents hard to reason about.
- intent rules need one extra model call to classify the request — a new failure point guarding an existing one.
- Nested groups turn resolution into a graph problem (cycle checks); jackdaw's
resolveis one flat loop you can read in a dozen lines.
This is exactly where pi's philosophy (minimal core, transparent, no presets) lands in jackdaw: the simpler the strategy, the more directly you can predict behavior from the config. Not that it can't be done — Chapter 16 lists intent rule routing and usage accounting as extension exercises.
INFO
The magpie capability list above comes from this tutorial's blueprint fact sheet; how smart scores or what usage measures isn't recorded there, and this chapter won't guess.
Summary
- String →
ModelRef→Vec<Candidate>; the reserved wordgroupand the no-nesting rule are enforced at both the config and the parsing layer. orderkeeps the configured order;rotateadvances the start with an atomic counter and keeps no statistics.- The loop continues only on
Retryable;Fatalpasses through immediately; total failure yields onedetailsline per member. - Streaming locks the winner after the first byte: sent bytes can't come back, and an honest failure beats an incoherent splice.
FakeProvidermakes the whole loop testable offline; the next chapter (14) wires this pipe into the HTTP gateway.
Self-Check
- You want a third strategy,
random(a fresh random start each request). Besides thematch group.strategyinrouter.rs, what else changes? What test do you add? - An order group with members
a/m-a, thenb/m-b; a is scripted Fatal (404), b is Ok. What doesrouter.complete(...)return, and what isb.calls()? What changes if a is Retryable instead? - Someone calls
streamlazy for not switching after the winner locks in — "it should resend from the top with the next model." Refute them with this chapter's mechanics.
Answers
- Add a variant to the
Strategyenum inconfig.rsand let serde keep accepting lowercase strings (see the teststrategy_parses_lowercase_from_toml); add the arm inrouter.rs's match, watching the empty-member case; add at least one test that the start varies across requests, modeled onrotate_advances_the_starting_member(jackdaw's dependency list has norand— hash the counter if you don't want a new dependency). - Fatal: returns
RouteError::Fatal { status: 404, .. }immediately,b.calls() == 0— fatal never fails over. Retryable: the loop continues and returnsOk(("b/m-b", _)), witha.calls() == 1andb.calls() == 1. - Sent bytes can't come back: SSE has no recall, and the agent already holds half an answer; resending with another model repeats content or splices two models' text. An error carried honestly through the stream beats an answer that looks complete but is incoherent.