Multi-Provider AI Routing: Handling Provider Failures Without Losing the Session

A local coding agent should not be hard-wired to one provider failure mode. If the active API rate limits, stalls, or returns a retryable error, the user should see a controlled fallback path instead of a half-finished session.

That is the point of Mythos Router's provider orchestration layer.

The Architecture

The orchestrator sits between the CLI session and the configured model providers. Instead of assuming one API is always available, it can evaluate a pool of providers and route requests through the best available candidate.

Current provider families are:

The exact model is resolved by the provider adapter and effort setting. The important part is that routing is provider-aware while file writes remain SWD-verified.

What Mythos Tracks

Provider selection can account for:

- startup priority - recent success and failure state - retry behavior - latency - estimated token cost - whether the request requires tool-capable providers - whether fallback is enabled for the current run

That gives Mythos a more realistic failure model than a single hard-coded API call.

Retry and Fallback

If a provider fails with a retryable error, Mythos can retry with backoff. If the provider remains unhealthy, the orchestrator can move to another configured provider when fallback is allowed.

Primary provider -> retryable failure -> retry/backoff -> fallback candidate

The goal is not to promise magic uptime. The goal is to avoid making one provider failure equivalent to total session failure.

Stream Watchdog

Some failures are quiet. A stream can stay open while no useful chunks arrive. Mythos uses a watchdog around streaming responses so a stalled provider can be treated as a failure and handled through the same retry/fallback logic.

Deterministic Runs

Some users want a fixed provider for repeatability. Mythos supports provider forcing through the CLI and skill metadata, so a session can choose stability over fallback when needed.

mythos run --provider anthropic --file TASK.md
mythos run --provider deepseek --file TASK.md
mythos run --provider openai --file TASK.md

Setup

Configure whichever providers you want Mythos to use:

export ANTHROPIC_API_KEY="sk-ant-..."
export DEEPSEEK_API_KEY="sk-..."
export OPENAI_API_KEY="sk-..."

Then run Mythos as usual:

npx mythos-router chat

The routing layer improves provider resilience. SWD remains the execution boundary that verifies real filesystem state before any file edit is treated as complete.