Skip to main content
The routing engine guide covers what routing does. This page covers how each stage behaves under failure, where the pipeline runs, and the flow guard.

​
How each stage degrades

Agent selection runs in three stages, and every stage fails forward. The worst case is a slightly worse agent pick, never a dropped query. Stage 1 — shortlist. Fetches every accessible agent and the session’s recent history in parallel, then narrows candidates by semantic similarity between the query and each agent’s description. Agent embeddings are cached per agent. Skipped entirely — no embedding calls — when the accessible fleet is smaller than ROUTER_SHORTLIST_THRESHOLD (default 15); below that, every agent goes straight to Stage 2. ROUTER_SHORTLIST_SIZE (default 10) caps surviving candidates. Stage 2 — rerank. Re-scores candidates against the conversation’s running history plus the current query, so a follow-up is more likely to stay with the agent already handling the session. Two conditions return the shortlist unchanged: empty conversation history (no embedding call at all) and an embedding-provider failure, which is caught and logged rather than propagated. Stage 3 — select. An LLM (ROUTER_MODEL, default gpt-4o) makes the final call. If it errors, the engine picks the highest-ranked remaining candidate and marks the decision fallback_used: true.

​
Routing decisions are logged off the response path

Decisions are recorded asynchronously — the response is already back by the time the log write happens. Each captures the agents considered, the reasoning, whether a fallback was used, per-stage candidate counts and latencies, and the selection call’s token cost.
The LLM call that picks an agent is metered like any other, not folded into the chosen agent — so routing cost is its own line item in TokenOps.
Aggregated stats come from GET /api/orchestrator/stats. They’re a periodic rollup, so they lag recent activity — for a specific query, open its session trace.

​
Where the pipeline actually runs

The three-stage pipeline is not what picks the agent on every chat message. POST /api/orchestrator/a2a dispatches on agent_id metadata: The three-stage pipeline is invoked in exactly one place today: auto-assigning an agent to a MAF workflow step when the step doesn’t name one. If it errors, workflow creation falls back to matching the step’s task description against the caller’s registered agents by name and description. Both mechanisms answer “which agent should handle this,” but they’re different code paths. A workflow step’s auto-assignment doesn’t run a ReAct loop; a chat message doesn’t run Stage 1/2/3.

​
The flow guard

Every agent-to-agent call the control plane makes is wrapped by a flow guard, regardless of how the destination was picked. It enforces five limits scoped to a single flow, identified by the W3C traceparent trace ID: Checks run before an agent is invoked, so a rejected call is never paid for. Counters are re-checked after incrementing in case a concurrent hop crossed a limit in between.
If the guard’s backing store is unreachable, it fails closed — calls are rejected rather than allowed through unchecked. The threat model is unvetted, user-authored agent and MCP code: silently disabling cascade limits during an outage would open an unbounded recursion and fan-out window.
Live flow status is broadcast to the dashboard’s Flows view (/api/flows/*) independently of the guard’s limit tracking.

Routing engine guide

Tuning knobs, inspecting decisions, reusable LLM configs.

LLM router dashboard

Configure providers and per-level models.

MAF

Multi-agent workflows and step assignment.

Observability

Sessions, traces, and how a flow’s hops tie together.