OpenAI-compatible ⚡ 11+ providers 🧠 Complexity-aware routing 🔒 Self-hosted

One endpoint for every free-tier LLM.

Arbiter is a self-hosted, production-ready gateway that aggregates 11+ LLM providers behind a single OpenAI-compatible API. It routes each request to the best model using complexity analysis, rotates keys across accounts, falls back automatically, caches responses, and gives you a real-time observability dashboard — so you maximise free quota and never hit a wall.

drop-in replacement
# Point any OpenAI SDK at Arbiter
from openai import OpenAI

client = OpenAI(
    base_url="https://your-host/v1",
    api_key="gwtk_your_gateway_token",
)

resp = client.chat.completions.create(
    model="auto",   # Arbiter picks the best model
    messages=[{"role":"user",
               "content":"Explain quantum tunnelling"}],
)
print(resp.choices[0].message.content)
# → routed to gemini / groq / cerebras / … automatically
11+LLM providers aggregated
1OpenAI-compatible endpoint
5-levelfallback & key rotation
100%self-hosted & free-tier first
Capabilities

Everything a gateway should do — and more

Built for multi-agent workloads that burst concurrent requests. Arbiter squeezes the most out of free tiers while staying observable and resilient.

🧠

Complexity-aware routing

Every request is scored TRIVIAL→EXPERT. Hard prompts go to 120B–671B flagships; trivial ones to fast 7B models — matching capability to cost automatically.

⚖️

Weighted key rotation

Multiple accounts per provider, scored in real time by remaining RPM / TPM / daily / monthly quota. Load spreads to the key with the most headroom.

🔁

Two-level fallback

Key → model hierarchy → cross-vendor. A failure on one account or model transparently cascades to the next until the request succeeds.

🚦

Circuit breaker

A provider/model that fails repeatedly is skipped for 5 minutes — so a reliably-broken upstream never burns latency on every request. Rate-limits never trip it.

⚡

Predictive rate-limiting

Keys are skipped at ~85–95% of quota so the router routes around exhaustion before a 429 ever reaches you. Retry-after-aware cooldowns.

🗜️

Redis response cache

Deterministic requests (temp ≤ 0.3) are cached & gzip-compressed. Cache key includes tools / response_format / seed so tool & JSON-mode calls never collide.

🌐

Real-time web search

Opt-in X-Arbiter-Realtime header grounds answers in live Tavily results with numbered citations + Gemini Google-Search grounding.

🛠️

Native tool calling

Pass an OpenAI tools array — Arbiter auto-routes to tool-capable providers and forwards function definitions verbatim.

📊

Real-time observability

Dashboards for per-provider health, per-key quota gauges, latency percentiles, token spend, anomaly detection, and a 180-day audit log.

🔐

Enterprise auth

Google SSO with admin approval for the UI; per-token Bearer auth for the API with named tokens, routing policies, model allow/block lists, and rate caps.

🎨

Free image generation

Built-in /v1/images/generations via Pollinations — no key required — plus a UI gallery.

🧩

Bring your own provider

Add any OpenAI- or Anthropic-compatible endpoint from the UI with a DNS-aware SSRF guard. Live model discovery included.

Architecture

How a request flows

From your OpenAI SDK call to the chosen upstream model — with caching, scoring, and fallback at every hop.

STEP 1

Authenticate

Bearer gateway-token check (constant-time) + per-token sliding-window rate limit.

STEP 2

Cache & classify

Redis cache lookup, then complexity + intent classification of the prompt.

STEP 3

Build chain

Candidate (provider, model) list ordered by capability, health & quota — circuit-broken pairs skipped.

STEP 4

Score & route

Best key chosen by weighted availability; request sent over a pooled HTTP client.

STEP 5

Fallback or return

On failure, rotate key → next model → next vendor. On success, cache & record stats.

⚙️ Stack: Python 3.12 · FastAPI · Redis 7 · Docker Compose. Two containers (gateway + redis), one .env, port 8080. Background schedulers handle backups (S3), weekly model-health probes, daily email reports, and 180-day log retention.
Coverage

11+ providers, free-tier first

Mix and match unlimited accounts per provider. Overlapping models sum their rate limits for additive capacity.

ProviderHighlightsFree tierContext
NVIDIA NIMNemotron-3-Super 120B, Llama-3.3-70B, Mistral-Medium-3.540 RPM · 1k RPD131K
Google Gemini3.1 Flash-Lite / Flash, Google-Search grounding, #paid tier tagging15 RPM · 1k RPD1M
GroqLlama 3.1/3.3, Llama 4 Scout, Kimi K2, GPT-OSS — fastest tokens/sec30 RPM · 14.4k RPD131K
CerebrasLlama 3.1/3.3, Qwen-3 235B, GLM-4.7 — ultra-fast inference5 RPM · 1M TPD8K+
Cloudflare Workers AILlama 4 Scout, GPT-OSS, Qwen, DeepSeek-R1, Gemma — 300 RPM300 RPM256K
OpenRouterCurated :free models across many upstreams, :online opt-in20 RPM131K
CohereCommand-R / Command-A family, monthly-quota awaretrial256K
Ollama CloudFree :cloud models — gpt-oss, deepseek, glm, qwen3-coder, kimi60 RPM · 5k RPD131K
RoutewayUnified gateway, 15 :free models seeded; paid opt-infree + paid256K
HuggingFaceInference router — open-weight modelsfree credits32K
PollinationsText + free image generation (anonymous tier works)free131K
Z.ai / ZhipuGLM-4.7 / 4.5 Flash (optional, configurable)free128K
Get running in 3 minutes

Quick start

One command to clone, one to configure, one to launch. Docker does the rest.

🐳 Docker (recommended)
⚙️ Configure .env
✅ Verify
# 1. Clone
git clone https://github.com/koushikch7/arbiter.git && cd arbiter

# 2. Configure (add at least one provider key)
cp .env.example .env
nano .env

# 3. Launch (gateway + redis)
docker compose up -d --build

# Open the dashboard
open http://localhost:8080/dashboard
# Minimum: one provider key + a gateway token.
# Add one or more keys per provider, comma-separated.

GEMINI_API_KEYS=AIzaSy...your-key
GROQ_API_KEYS=gsk_...your-key
CEREBRAS_API_KEYS=csk-...your-key

# Protect the API (comma-separated list of valid Bearer tokens)
GATEWAY_API_KEYS=gwtk_choose_a_strong_token

# Optional: Google SSO for the admin UI, SMTP, S3 backups…
# See .env.example for the full list.
# Health check
curl http://localhost:8080/health
# → {"status":"ok","providers":[...],"version":"1.21.0"}

# First completion
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer gwtk_your_token" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello!"}]}'
Developer reference

API endpoints

A drop-in OpenAI surface plus Arbiter extensions. Authenticate every /v1/* call with Authorization: Bearer <gateway-token>.

Core OpenAI-compatible API

MethodEndpointDescription
POST/v1/chat/completionsChat completion (streaming & non-streaming). Smart auto-routing.
GET/v1/modelsList all available models across providers.
POST/v1/images/generationsGenerate images (free via Pollinations).
GET/v1/images/modelsList image-generation models & sizes.
GET/healthHealth + provider list + version (no auth).

Arbiter request controls

HeaderBody fieldEffect
X-Arbiter-Prioritymetadata.priorityspeed | quality | balanced — scoring bias
X-Arbiter-Prefer-Providermetadata.prefer_providerBoost a provider in auto routing
X-Arbiter-Fallbackfallbacknone | same_provider | chain
X-Arbiter-Realtimemetadata.realtimetrue → Tavily web search + OpenRouter :online
?vendor=  /  ?force_model= (query)Pin a provider or exact model, bypassing auto-routing

Response headers & x_arbiter block

HeaderBody fieldMeaning
X-Arbiter-Model-Usedx_arbiter.provider/modelProvider & model that actually served the request
X-Arbiter-Complexityx_arbiter.complexityTRIVIAL | SIMPLE | MODERATE | COMPLEX | EXPERT
X-Arbiter-Realtime-Sourcesx_arbiter.realtime_sourcesSource URLs when web search is on
X-RateLimit-Limit/Remaining—Per-token rate-limit state
Retry-After—Seconds until quota reset (on HTTP 429)

Examples

cURL
Python (OpenAI SDK)
Node.js
Streaming
Web search
curl https://your-host/v1/chat/completions \
  -H "Authorization: Bearer gwtk_your_token" \
  -H "Content-Type: application/json" \
  -H "X-Arbiter-Priority: quality" \
  -d '{
    "model": "auto",
    "messages": [{"role":"user","content":"Write a haiku about routing"}],
    "temperature": 0.7
  }'
from openai import OpenAI
client = OpenAI(base_url="https://your-host/v1", api_key="gwtk_your_token")

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role":"user","content":"Summarise the CAP theorem"}],
    extra_headers={"X-Arbiter-Priority":"balanced"},
)
print(resp.model)            # actual model used
print(resp.choices[0].message.content)
import OpenAI from "openai";
const client = new OpenAI({
  baseURL: "https://your-host/v1",
  apiKey:  "gwtk_your_token",
});

const r = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Explain event loops" }],
});
console.log(r.choices[0].message.content);
curl -N https://your-host/v1/chat/completions \
  -H "Authorization: Bearer gwtk_your_token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "stream": true,
    "messages": [{"role":"user","content":"Count to five"}]
  }'
# → Server-Sent Events: chat.completion.chunk … data: [DONE]
📮 Prefer Postman? Download the ready-made Postman collection — set the base_url and gateway_token variables and start firing requests.

Beyond the public API, Arbiter exposes 78 endpoints total — provider/key management, gateway tokens, analytics data, backups, users, announcements, Cloudflare Worker management and more. Explore them live at /docs (Swagger) on your instance.

Operating Arbiter

User guide highlights

Manage everything from the web UI — no restarts, no YAML surgery.

🎛️ Admin dashboards

  • /dashboard — live provider & per-key health, success rates, quota bars
  • /analytics — latency percentiles, token spend, anomaly detection, CSV/JSON export
  • /settings — add/remove/test keys, toggle providers & models, routing policy, cache
  • /playground — chat UI with model picker & smart-route option
  • /logs & /logs/persistent — live buffer + 180-day audit viewer
  • /backup — S3 backup & restore management

🔑 Runtime management

  • Add & rotate provider keys at runtime — written to .env, auto-tested
  • Issue named gateway tokens with per-token rate limits & model allow/block lists
  • Tag keys #paid to reserve frontier models for billing-enabled accounts
  • Per-token routing policies: auto · restricted · preferred
  • Google SSO with admin-approval workflow for UI access
  • Weekly AI-analysis email reports & major-change dashboard banners
📚 Full docs in the repo: USERGUIDE.md · README.md · REQUIREMENTS.md · CHANGELOG.md
Open source · self-hosted

Stop hitting rate limits. Start routing intelligently.

Clone it, point your OpenAI SDK at it, and let Arbiter handle the providers, quotas, fallback, and observability.