Arbiter is a self-hosted, production-ready gateway that aggregates 11+ LLM providers behind a single OpenAI-compatible API. It routes each request to the best model using complexity analysis, rotates keys across accounts, falls back automatically, caches responses, and gives you a real-time observability dashboard — so you maximise free quota and never hit a wall.
# Point any OpenAI SDK at Arbiter from openai import OpenAI client = OpenAI( base_url="https://your-host/v1", api_key="gwtk_your_gateway_token", ) resp = client.chat.completions.create( model="auto", # Arbiter picks the best model messages=[{"role":"user", "content":"Explain quantum tunnelling"}], ) print(resp.choices[0].message.content) # → routed to gemini / groq / cerebras / … automatically
Built for multi-agent workloads that burst concurrent requests. Arbiter squeezes the most out of free tiers while staying observable and resilient.
Every request is scored TRIVIAL→EXPERT. Hard prompts go to 120B–671B flagships; trivial ones to fast 7B models — matching capability to cost automatically.
Multiple accounts per provider, scored in real time by remaining RPM / TPM / daily / monthly quota. Load spreads to the key with the most headroom.
Key → model hierarchy → cross-vendor. A failure on one account or model transparently cascades to the next until the request succeeds.
A provider/model that fails repeatedly is skipped for 5 minutes — so a reliably-broken upstream never burns latency on every request. Rate-limits never trip it.
Keys are skipped at ~85–95% of quota so the router routes around exhaustion before a 429 ever reaches you. Retry-after-aware cooldowns.
Deterministic requests (temp ≤ 0.3) are cached & gzip-compressed. Cache key includes tools / response_format / seed so tool & JSON-mode calls never collide.
Opt-in X-Arbiter-Realtime header grounds answers in live Tavily results with numbered citations + Gemini Google-Search grounding.
Pass an OpenAI tools array — Arbiter auto-routes to tool-capable providers and forwards function definitions verbatim.
Dashboards for per-provider health, per-key quota gauges, latency percentiles, token spend, anomaly detection, and a 180-day audit log.
Google SSO with admin approval for the UI; per-token Bearer auth for the API with named tokens, routing policies, model allow/block lists, and rate caps.
Built-in /v1/images/generations via Pollinations — no key required — plus a UI gallery.
Add any OpenAI- or Anthropic-compatible endpoint from the UI with a DNS-aware SSRF guard. Live model discovery included.
From your OpenAI SDK call to the chosen upstream model — with caching, scoring, and fallback at every hop.
Bearer gateway-token check (constant-time) + per-token sliding-window rate limit.
Redis cache lookup, then complexity + intent classification of the prompt.
Candidate (provider, model) list ordered by capability, health & quota — circuit-broken pairs skipped.
Best key chosen by weighted availability; request sent over a pooled HTTP client.
On failure, rotate key → next model → next vendor. On success, cache & record stats.
gateway + redis), one .env, port 8080. Background schedulers handle backups (S3), weekly model-health probes, daily email reports, and 180-day log retention.Mix and match unlimited accounts per provider. Overlapping models sum their rate limits for additive capacity.
| Provider | Highlights | Free tier | Context |
|---|---|---|---|
| NVIDIA NIM | Nemotron-3-Super 120B, Llama-3.3-70B, Mistral-Medium-3.5 | 40 RPM · 1k RPD | 131K |
| Google Gemini | 3.1 Flash-Lite / Flash, Google-Search grounding, #paid tier tagging | 15 RPM · 1k RPD | 1M |
| Groq | Llama 3.1/3.3, Llama 4 Scout, Kimi K2, GPT-OSS — fastest tokens/sec | 30 RPM · 14.4k RPD | 131K |
| Cerebras | Llama 3.1/3.3, Qwen-3 235B, GLM-4.7 — ultra-fast inference | 5 RPM · 1M TPD | 8K+ |
| Cloudflare Workers AI | Llama 4 Scout, GPT-OSS, Qwen, DeepSeek-R1, Gemma — 300 RPM | 300 RPM | 256K |
| OpenRouter | Curated :free models across many upstreams, :online opt-in | 20 RPM | 131K |
| Cohere | Command-R / Command-A family, monthly-quota aware | trial | 256K |
| Ollama Cloud | Free :cloud models — gpt-oss, deepseek, glm, qwen3-coder, kimi | 60 RPM · 5k RPD | 131K |
| Routeway | Unified gateway, 15 :free models seeded; paid opt-in | free + paid | 256K |
| HuggingFace | Inference router — open-weight models | free credits | 32K |
| Pollinations | Text + free image generation (anonymous tier works) | free | 131K |
| Z.ai / Zhipu | GLM-4.7 / 4.5 Flash (optional, configurable) | free | 128K |
One command to clone, one to configure, one to launch. Docker does the rest.
# 1. Clone git clone https://github.com/koushikch7/arbiter.git && cd arbiter # 2. Configure (add at least one provider key) cp .env.example .env nano .env # 3. Launch (gateway + redis) docker compose up -d --build # Open the dashboard open http://localhost:8080/dashboard
# Minimum: one provider key + a gateway token. # Add one or more keys per provider, comma-separated. GEMINI_API_KEYS=AIzaSy...your-key GROQ_API_KEYS=gsk_...your-key CEREBRAS_API_KEYS=csk-...your-key # Protect the API (comma-separated list of valid Bearer tokens) GATEWAY_API_KEYS=gwtk_choose_a_strong_token # Optional: Google SSO for the admin UI, SMTP, S3 backups… # See .env.example for the full list.
# Health check curl http://localhost:8080/health # → {"status":"ok","providers":[...],"version":"1.21.0"} # First completion curl http://localhost:8080/v1/chat/completions \ -H "Authorization: Bearer gwtk_your_token" \ -H "Content-Type: application/json" \ -d '{"model":"auto","messages":[{"role":"user","content":"Hello!"}]}'
A drop-in OpenAI surface plus Arbiter extensions. Authenticate every /v1/* call with Authorization: Bearer <gateway-token>.
| Method | Endpoint | Description |
|---|---|---|
| POST | /v1/chat/completions | Chat completion (streaming & non-streaming). Smart auto-routing. |
| GET | /v1/models | List all available models across providers. |
| POST | /v1/images/generations | Generate images (free via Pollinations). |
| GET | /v1/images/models | List image-generation models & sizes. |
| GET | /health | Health + provider list + version (no auth). |
| Header | Body field | Effect |
|---|---|---|
X-Arbiter-Priority | metadata.priority | speed | quality | balanced — scoring bias |
X-Arbiter-Prefer-Provider | metadata.prefer_provider | Boost a provider in auto routing |
X-Arbiter-Fallback | fallback | none | same_provider | chain |
X-Arbiter-Realtime | metadata.realtime | true → Tavily web search + OpenRouter :online |
?vendor= / ?force_model= (query) | Pin a provider or exact model, bypassing auto-routing | |
x_arbiter block| Header | Body field | Meaning |
|---|---|---|
X-Arbiter-Model-Used | x_arbiter.provider/model | Provider & model that actually served the request |
X-Arbiter-Complexity | x_arbiter.complexity | TRIVIAL | SIMPLE | MODERATE | COMPLEX | EXPERT |
X-Arbiter-Realtime-Sources | x_arbiter.realtime_sources | Source URLs when web search is on |
X-RateLimit-Limit/Remaining | — | Per-token rate-limit state |
Retry-After | — | Seconds until quota reset (on HTTP 429) |
curl https://your-host/v1/chat/completions \ -H "Authorization: Bearer gwtk_your_token" \ -H "Content-Type: application/json" \ -H "X-Arbiter-Priority: quality" \ -d '{ "model": "auto", "messages": [{"role":"user","content":"Write a haiku about routing"}], "temperature": 0.7 }'
from openai import OpenAI client = OpenAI(base_url="https://your-host/v1", api_key="gwtk_your_token") resp = client.chat.completions.create( model="auto", messages=[{"role":"user","content":"Summarise the CAP theorem"}], extra_headers={"X-Arbiter-Priority":"balanced"}, ) print(resp.model) # actual model used print(resp.choices[0].message.content)
import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://your-host/v1", apiKey: "gwtk_your_token", }); const r = await client.chat.completions.create({ model: "auto", messages: [{ role: "user", content: "Explain event loops" }], }); console.log(r.choices[0].message.content);
curl -N https://your-host/v1/chat/completions \ -H "Authorization: Bearer gwtk_your_token" \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "stream": true, "messages": [{"role":"user","content":"Count to five"}] }' # → Server-Sent Events: chat.completion.chunk … data: [DONE]
curl https://your-host/v1/chat/completions \ -H "Authorization: Bearer gwtk_your_token" \ -H "Content-Type: application/json" \ -H "X-Arbiter-Realtime: true" \ -d '{ "model": "auto", "messages": [{"role":"user","content":"Latest SpaceX launch result?"}] }' # Response grounded in live web data with [1]-style citations. # Sources in X-Arbiter-Realtime-Sources header.
base_url and gateway_token variables and start firing requests.Beyond the public API, Arbiter exposes 78 endpoints total — provider/key management, gateway tokens, analytics data, backups, users, announcements, Cloudflare Worker management and more. Explore them live at /docs (Swagger) on your instance.
Manage everything from the web UI — no restarts, no YAML surgery.
.env, auto-tested#paid to reserve frontier models for billing-enabled accountsauto · restricted · preferredClone it, point your OpenAI SDK at it, and let Arbiter handle the providers, quotas, fallback, and observability.