Last updated: August 2026
A CLI in front of several LLM providers. Every prompt gets classified by task, routed to the cheapest backend that's allowed to serve it, and — for coding, review, and design work — handed off to a persistent Claude Code session instead of a one-shot API call.
Tasks: chat, search, ideation, feedback, review, design, coding Free tier: Groq → Gemini → OpenRouter, fixed order, sticky-first + cooldowns Protected: review / design / coding never touch the free tier Deep backend: Claude Code CLI, resumable session, approval-gated edits
This project was inspired after reading Glean's Waldo launch post. This got me thinking about how I could write my own model router that selects models based on the task I'm working on. I primarily use Claude Code from the CLI. My searches range from coding-specific questions to queries like how to best prepare for my interview at company X. That is why I came up with some predefined categories for my prompts and decided to use free models for non-coding tasks.
OpenRouter was the starting point, for model choice — one API in front of a wide catalog, so classification could eventually route to whichever model actually fit a task instead of being locked to one provider's lineup.
When testing with OpenRouter I kept getting 402s — insufficient credits, a consequence of running on a free account. I was looking for popular models that have sufficient tokens/request limits, which is how Groq and Gemini ended up in the mix. Groq is tried first on every call, and stays first as long as it keeps succeeding — in practice that means Groq ends up serving nearly everything, with Gemini and OpenRouter sitting as pure failover for whenever it actually fails or hits a quota. Details below.
The [routed -> ...] line is the router printing its decision — task, tier, classifier confidence,
and source — before the call is made. It deliberately carries no model id: at that point the free-tier chain
hasn't picked a provider yet, so a model name there would just be a placeholder. The
[served by model=...] line after it is the real answer, printed once complete()
returns:
python main.py
Model Router — type a prompt (or 'exit' / 'quit' to stop).
> gm my name is jp
[routed -> task=chat tier=free confidence=95% source=llm]
[served by model=openai/gpt-oss-20b]
Good morning, JP! How can I help you today?
> what is best restaurant in sf
[routed -> task=search tier=free confidence=95% source=llm]
[served by model=openai/gpt-oss-20b]
A table of SF restaurant picks by cuisine (French, Asian, Mediterranean, American,
Greek, bakery) with signature dishes, price ranges, and reservation tips. [...]
> where is delhi
[routed -> task=search tier=free confidence=95% source=llm]
[served by model=openai/gpt-oss-20b]
Delhi is the capital territory of India, on the banks of the Yamuna River in the
north. The National Capital Territory (NCT) includes New Delhi (seat of government)
and surrounding districts.
> write a code for fizbuzz ?
[routed -> task=coding tier=standard confidence=99% source=llm]
[claude code wants to run: Bash {'command': 'cat -A fizzbuzz.java 2>/dev/null | head -20; ...'}]
Approve? [y/N]: n
[served by model=claude-sonnet-5]
There's already a working FizzBuzz implementation at `fizzbuzz.java`:
```java
public class fizzbuzz {
public static void main(String[] args) {
for (int i = 1; i <= 100; i++) {
if (i % 15 == 0) {
System.out.println("FizzBuzz");
} else if (i % 3 == 0) {
System.out.println("Fizz");
} else if (i % 5 == 0) {
System.out.println("Buzz");
} else {
System.out.println(i);
}
}
}
}
```
It prints 1-100, "Fizz" for multiples of 3, "Buzz" for multiples of 5, "FizzBuzz"
for multiples of 15. Want me to run it or make changes?
Every prompt is classified into one of seven tasks by an LLM call (Claude Haiku by default), which also judges whether the prompt continues an already-open context — more on that below. If the classifier itself fails or comes back malformed, a whole-word keyword matcher takes over as a fallback so the router degrades instead of breaking. The (task, tier) pair resolves to a concrete model id; tier defaults per-task rather than globally, so a bare "chat" prompt defaults to free while "coding" defaults to standard.
| Task | Free tier | Default |
|---|---|---|
| chat, search, ideation, feedback | allowed | free |
| review, design, coding | blocked | standard |
review, design, and coding are hard-excluded from the free tier — not a soft preference, a rule enforced in tier resolution itself, so no config typo or keyword collision can accidentally route a refactor to a free model. Those three always dispatch to Claude Code instead, drawing on a Claude Code subscription's usage rather than pay-as-you-go token billing.
Groq is the primary LLM and in case of any errors other LLMs are called into action. Gemini and OpenRouter are fallback options.
Providers are tried in the same declared order every single call — Groq, then Gemini, then OpenRouter — no rotation, no counter. Groq stays first as long as it keeps succeeding; only cooldown (layer 2) moves a call off it, and the instant its cooldown clears, the very next call tries Groq first again. In practice this means Groq serves nearly everything while it's healthy, with Gemini and OpenRouter sitting as pure failover capacity — confirmed live: a whole session's worth of turns landed on Groq with zero calls ever reaching the other two.
A provider that fails is put in cooldown and skipped entirely by subsequent calls instead of being retried
immediately — not tried as a last resort, filtered out before the fixed-order loop even runs. Cooldown length
depends on what actually went wrong, not one flat guess for everything. A rate limit honors the provider's own
Retry-After value when it sends one (Gemini instead nests its own retryDelay field in
the error body, since it doesn't send the header at all), plus a small jitter so concurrent calls don't all
retry in lockstep. A 402 (insufficient credits) gets a coarse one-hour cooldown since there's no time-based
signal for that at all — it needs a balance top-up, not a wait. A bare network timeout gets the same
default-plus-jitter as an unsignaled rate limit, so a hung provider isn't paying its full timeout again on
every single call. Cooldown state is shared between the free-tier and classifier chains — they hit the same
API keys — and persisted to disk, so a process restart doesn't rediscover OpenRouter's hour-long 402 cooldown
via another wasted real call.
Escalation to Claude Code (on a cheap model, Haiku) happens two ways: every provider in the chain is tried and
fails, or — a distinct, zero-attempt case — every provider is already known to be in cooldown before the call
starts, and the chain doesn't bother attempting a doomed call at all. Either way the caller sees the same
outcome, but the log line differs (all providers cooling down vs. an actual per-provider failure
trail).
Shape of it, simplified (real thing has jitter and persistence on top):
class FallbackChain:
def __init__(self, providers):
self.providers = providers # [(backend, model_id), ...], fixed order -> layer 1
def complete(self, messages):
for backend, model_id in self.providers:
if backend.in_cooldown():
continue
try:
return backend.call(model_id, messages) # success -> return, stays first next call
except ProviderError as exc: # fall through to the next provider
backend.cooldown_for(seconds_from(exc)) # layer 2: real signal or a guess
raise AllProvidersExhausted()
review and design run the Claude Code CLI in read-only plan mode — nothing to approve, no tool loop that can
touch disk. coding runs in a mode where every Edit/Write/Bash call is gated behind an approval callback; the
CLI's own permission hooks surface pending denials, which the router loops through one at a time until the
caller approves or declines. A single coding session persists across turns via the CLI's --resume,
so "now add tests for that" on turn three still has turns one and two in scope — the router never duplicates
that history locally, Claude Code owns it.
A per-call spend cap (--max-budget-usd) sits on every Claude Code call regardless of mode, since
those calls draw on a subscription with its own usage limits — a single runaway agentic loop shouldn't be able
to silently eat an unbounded share of it.
Light and deep tasks keep separate context, but a conversation routinely crosses between them — you ideate, then ask for the design, then ask it to implement. Two summaries carry state across that boundary rather than dropping it:
The classifier also flags whether a prompt continues an already-open context, so "did you write ratelimiter.py?" — which reads as a plain chat message in isolation — can still get routed back into a live coding session instead of answered blind. That override only ever promotes a light task into a protected one, never the reverse, and only above a confidence threshold raised after a live run showed a stale session getting reattached to an unrelated question at the original, lower bar.
Everything written to the log file or to on-disk session state passes through a scrub filter first — patterns for OpenRouter, Anthropic, GitHub, and AWS key shapes, plus generic bearer tokens. It's attached at the log handler level rather than the root logger, since a logger-level filter only sees records logged directly on that logger and misses ones from child loggers that merely propagate up to it — confirmed live, the logger-level version let a secret through. The rule exists because of a real incident earlier in this project: an API key printed in plaintext via a careless shell command.
To understand if my approach is saving any money or not, I leveraged the responses from API calls and logged them to a log. Every provider's API response already includes token usage and — for Claude Code — real per-call cost, computed by the provider itself. Wrote a simple script that parses the logs to generate a small summary of usage.
What's actually in the response, confirmed live rather than assumed:
{
"choices": [{"message": {"content": "OK"}, "finish_reason": "stop"}],
"usage": {
"prompt_tokens": 76,
"completion_tokens": 10,
"total_tokens": 86
}
}
// Groq/Gemini/OpenRouter (OpenAI-compatible shape) -- backends/_openai_compat.py
// only ever extracted choices[0].message.content, discarding "usage" entirely.
{
"result": "OK",
"session_id": "2565ab84-...",
"total_cost_usd": 0.0091797,
"duration_ms": 1282,
"usage": { "input_tokens": 10, "output_tokens": 41, "...": "..." }
}
// Claude Code CLI's --output-format json -- backends/claude_code.py only
// ever kept "result" and "session_id", discarding cost and duration.
Both now get logged as a normal logger.info(...) line next to the success/latency line that was
already there — a provider-and-token-count line from _openai_compat.py, and a
cost_usd=... field appended to claude_code.py's existing success log.
$ python tools/metrics_report.py Provider Requests Failures p95 latency Tokens Cost ($) ---------------------------------------------------------------------- groq 148 33 3.15s 7873 0.0000 gemini 60 52 7.99s 0 0.0000 openrouter 45 37 311.32s 0 0.0000 claude 2 0 8.25s 0 0.3560 ---------------------------------------------------------------------- TOTAL 255 122 7873 0.3560
Character-count compaction, not real tokenization: rolling chat history compacts once it crosses a character threshold, a cheap proxy rather than an actual token count. Good enough to keep a session from growing unbounded, not a precise budget.
No scoped approval: every mutating Claude Code action is confirmed individually — there's no "trust this session" or remembered approval yet, so a long coding turn with several edits means several prompts.