Aryan SinghandClaude Opus 5
cb985dcf9c
One endpoint in front of every free LLM tier, with automatic failover and no provider key ever leaving the box. Providers are data, not code: adding one is a JSON file plus an env var, never a TypeScript change. Ships 12 providers pre-written; `verify-provider --apply` derives the live model list from your own key. - core: registry, bucketed quota meter (rolling + UTC-calendar windows), key-pool store, 6-step model router with diagnostic rejection reasons - gateway: node:http, failover that commits to a stream only after 200 headers, SSE passthrough with usage sniffing, virtual keys with admin/inference roles - context engine: sqlite sessions and a 5-rung refit ladder, so a 128k thread keeps working when it fails over to an 8k model - cache: exact (sha256, safe-by-default) + opt-in semantic (flat Float32Array) - guard: inbound limits, quota forecasting, limit calibration from upstream headers, key health, secret/PII/injection scanning - dashboard: one self-contained HTML file, live SSE request log, token usage - sdk: zero-dep typed client, drop-in for the OpenAI SDK Zero runtime dependencies. 326 tests, 27-check smoke suite against a scripted mock upstream so routing is testable without spending quota. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
86 lines
4.0 KiB
Markdown
86 lines
4.0 KiB
Markdown
# Freeway — notes for whoever works on this next
|
|
|
|
Self-hosted, OpenAI-compatible gateway in front of free-tier LLM providers.
|
|
One endpoint, automatic failover, live quota visibility, and no real provider
|
|
key ever leaving the box.
|
|
|
|
## The one rule
|
|
|
|
**Providers are data, not code.** Adding a provider is one JSON file in
|
|
`providers/` plus one env var. It must never require a TypeScript edit.
|
|
|
|
If you are adding a provider, read **[docs/ADDING-A-PROVIDER.md](docs/ADDING-A-PROVIDER.md)**
|
|
and use the tools — do not hand-write the file:
|
|
|
|
```bash
|
|
node scripts/new-provider.mjs <id> --url <baseUrl> --env <ENV_VAR>
|
|
node scripts/verify-provider.mjs <id> # your feedback loop; --json for machine output
|
|
```
|
|
|
|
The schema is the contract: [`providers/_schema.json`](providers/_schema.json).
|
|
|
|
## Layout
|
|
|
|
```
|
|
packages/core/ registry, meter, store, router, guard, context, cache, probe
|
|
packages/gateway/ node:http server + public/index.html dashboard
|
|
packages/sdk/ @freeway/sdk client
|
|
providers/ the registry — one JSON file per provider, _-prefixed files skipped
|
|
scripts/ mock-upstream.js, verify-provider.mjs, new-provider.mjs
|
|
```
|
|
|
|
## Hard constraints
|
|
|
|
- **Node 22+, TypeScript, ESM, npm workspaces. Zero runtime dependencies.**
|
|
Dev deps are `typescript` and `@types/node` only. If you think you need a
|
|
runtime dep, stop and ask.
|
|
- **No TypeScript parameter properties.** `constructor(private readonly x: T)`
|
|
throws `ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX` under Node's strip-only type
|
|
stripping, which is how the tests and `verify-provider.mjs` run `.ts` sources
|
|
directly. Declare fields, assign in the constructor.
|
|
- `strict` + `noUncheckedIndexedAccess`: indexing returns `T | undefined`. Guard
|
|
or `?? fallback`; never `!`. No `any` without a comment justifying it.
|
|
- Tests are `node:test` only. Run them against `scripts/mock-upstream.js`, never
|
|
a real provider.
|
|
- **Never log a key.** Everything goes through `mask()` in `packages/core/src/util.ts`.
|
|
There are tests asserting raw keys cannot appear in errors or `/api/*`.
|
|
- Comments explain *why*. Skip comments that restate the code.
|
|
|
|
## Things that look wrong but are deliberate
|
|
|
|
- **`null` limits.** `null` means "unknown or unlimited" and beats a guessed
|
|
number, because the guard layer overwrites nulls with values observed from
|
|
upstream rate-limit headers. A wrong number never self-corrects. Mistral ships
|
|
with null `rps`/`tpm` because Mistral stopped publishing them.
|
|
- **Day and month limits are calendar windows, not rolling.** Free tiers reset
|
|
at 00:00 UTC. A rolling 24h window would block for hours after quota refilled.
|
|
- **Model ids are split on the *first* slash only.** `cloudflare/@cf/meta/llama-3.1-8b-instruct`
|
|
is provider `cloudflare`, model `@cf/meta/llama-3.1-8b-instruct`.
|
|
- **Failover commits after 200 headers, before piping the body.** Once a byte of
|
|
a stream reaches the client you cannot fail over, so the commit point sits
|
|
exactly there.
|
|
- **A 400 is never retried and benches nothing.** It fails identically everywhere.
|
|
A 401/403/429 benches the *key*; a 5xx or network error benches the *provider*.
|
|
- **Per-model credit rates, not one per provider.** Output tokens cost ~8x input
|
|
and a 70B ~10x an 8B, so a single flat number is wrong by an order of magnitude.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
npm run build # tsc --build, must be clean
|
|
npm test # node:test across all workspaces
|
|
npm run smoke # gateway against the mock upstream, every route
|
|
node scripts/mock-upstream.js --port 9101 --name alpha # a fake provider you control
|
|
```
|
|
|
|
## Where things live
|
|
|
|
| Want to change… | File |
|
|
|---|---|
|
|
| provider file format | `packages/core/src/spec.ts` + `providers/_schema.json` |
|
|
| quota accounting | `packages/core/src/meter.ts` |
|
|
| which provider serves a request | `packages/core/src/router.ts` |
|
|
| failover / streaming behaviour | `packages/gateway/src/proxy.ts` |
|
|
| routes | `packages/gateway/src/server.ts` |
|
|
| the dashboard | `packages/gateway/public/index.html` (one file, no build step, no CDN) |
|