Routing engine

The routing engine is a three-stage pipeline that picks one agent from a candidate set. It makes one decision, then hands off — it’s not an agentic loop. For several agents collaborating on one task, see multi-agent workflows.

When routing runs

This pipeline does not run on every chat message that omits an agent. Read this section before assuming what powers your unrouted chat traffic.

Chat requests go to POST /api/orchestrator/a2a. What happens depends on agent_id:

Request	What runs
Explicit `agent_id`	Proxies straight to that container. No selection.
No `agent_id`, or `"orchestrator"`	A multi-turn orchestrator — every accessible agent becomes a callable tool and an LLM decides across up to ten turns which to call. No shortlist or rerank. See how Nasiko dispatches to agents.

The pipeline below is invoked in one place today: auto-assigning an agent to a MAF workflow step that doesn’t name one. Tuning it shapes MAF step assignment, not live chat routing. Full detail: where the pipeline actually runs.

The three stages

Shortlist — ranks agents by semantic similarity between the query and each agent’s description, keeping the top candidates. Skipped for small fleets (ROUTER_SHORTLIST_THRESHOLD) or if the embedding provider is unavailable; either way every agent advances.
Rerank — re-scores candidates using conversation context, so a follow-up is more likely to stay with the agent already handling the session. On failure, the shortlist order is kept.
Select — an LLM makes the final call. On failure, the top-ranked candidate is chosen and the decision is marked fallback_used: true.

Every stage degrades gracefully — a query is never rejected because a routing sub-step had trouble. Decisions are logged asynchronously, off the response path.

Tuning

Environment variables on the control plane:

Env var	Default	What it controls
`ROUTER_SHORTLIST_THRESHOLD`	`15`	Fleet size above which shortlisting runs at all
`ROUTER_SHORTLIST_SIZE`	`10`	Max candidates kept for reranking
`ROUTER_MODEL`	`gpt-4o`	Model for the selection stage
`EMBEDDING_MODEL`	provider default	Embedding model for the shortlist stage
`ROUTER_AGENT_TIMEOUT_SECS`	`60`	How long to wait on the chosen agent’s response

These tune the engine’s own decision-making, separate from which model an agent calls once selected — see reusable LLM configs.

Inspecting decisions

GET /api/orchestrator/stats returns aggregated stats from this pipeline’s decisions — today, MAF step auto-assignments. One row per agent per day, newest and most-selected first, capped at 200 rows.

curl -H "Authorization: Bearer $TOKEN" https://<control-plane>/api/orchestrator/stats

{
  "data": [
    {
      "agent_name": "weather-bot",
      "selection_count": 142,
      "successful_calls": 138,
      "failed_calls": 4,
      "avg_agent_latency_ms": "812.4",
      "avg_selection_latency_ms": "310.2",
      "avg_stage1_candidates": "6.0",
      "avg_stage2_candidates": "3.0",
      "date": "2026-07-30"
    }
  ],
  "total": 1
}

There’s no CLI command for this yet — call the route directly. To see which agent handled a live chat query, look at its session and trace instead: GET /api/observability/session/{id} and GET /api/observability/trace/{id}.

Reusable LLM configs

Separate from engine tuning, you can control which LLM an individual agent calls, through a per-user library of named configs:

# available providers, models, price per 1M tokens
nasiko llm-config providers

# create a named config
nasiko llm-config create --name support-cheap --provider openai --model gpt-4o-mini

# attach it to an agent you own
nasiko llm-config attach support-bot support-cheap

# check what an agent will use, and where that resolved from
nasiko llm-config get support-bot

A config carries a provider, model, optional fallback models, tuning (temperature, max tokens), and optionally your own API key. --pin locks an agent to one model — the engine still picks which agent answers, but that agent’s LLM calls stay fixed. nasiko model-registry ls / set manage a platform-wide strength-level→model table (level 1 strongest, level 3 smallest) that a smart-routing config can draw from. Changing it requires superuser. Full command set: nasiko llm-config --help.

LLM router dashboard

Configure providers and per-level models in the web app.

TokenOps

What your routing and model choices cost.

A2A agents and frameworks

What makes an agent discoverable and routable.

Orchestrator reference

Failure behavior and the flow guard.

Getting Started

CLI

ADLC

Dashboard

Platform

MCP Gateway

MCP Hub

Agents in Artifact Registry

When routing runs

The three stages

Tuning

Inspecting decisions

Reusable LLM configs

LLM router dashboard

TokenOps

A2A agents and frameworks

Orchestrator reference

​When routing runs

​The three stages

​Tuning

​Inspecting decisions

​Reusable LLM configs

LLM router dashboard

TokenOps

A2A agents and frameworks

Orchestrator reference

When routing runs

The three stages

Tuning

Inspecting decisions

Reusable LLM configs