00 · ABSTRACT
Krindel — one base_url,
every provider, governed.
This brief presents Krindel with the structure of a systems paper: architecture, method, measured results, a live demonstration, and a named list of what isn't done yet. Every factual claim below cites the repository file or ADR it comes from (ADR-025) — nothing here is aspirational copy.
What this is
Krindel is a commercial, OpenAI-compatible LLM gateway: one
base_url in front of many providers, reached with an
unmodified OpenAI SDK — no client-side rewrite, just a changed
endpoint and a Krindel-minted key.
Thesis under test: provider-agnostic routing and failover, one-way PII redaction, budget reservation, and tool-calling/multimodal passthrough can sit in front of every completion at microsecond-scale added latency — not millisecond-scale — without becoming the bottleneck they're meant to govern.
Result, in one line
Added gateway latency lands in the hundreds of microseconds: Δp50 149µs for a synchronous completion at a sustained 1,000 RPS, zero errors (§03).
Caveat: measured on a shared 2-vCPU sandbox VM with an in-process open-loop harness — not reference hardware, not a claim for commercial or external publication yet. Full environment, methodology, and a pending reference-hardware slot are in §03.
01 · ARCHITECTURE
A transport-agnostic router core, wrapped by thin shells
Three strategic bets shape the codebase (docs/ARCHITECTURE.md,
“Positioning”): the synchronous /v1/chat/completions
proxy is the product, not an add-on (ADR-002); the core is Go standard
library only, zero third-party dependencies (ADR-001); and
router.Execute is a plain function shared by the sync HTTP
handler and the async jobs worker, so POST /v1/jobs is an
additive queue over the same core, not a fork of the routing logic.
The package map keeps that split legible: internal/auth,
internal/pii, internal/ratelimit, and
internal/router each own one gate in the pipeline;
internal/providers holds the canonical contract and its
three adapters (openai-compat, anthropic, mock);
internal/store and internal/jobs hold state,
swappable between an in-memory/JSONL backend and Postgres/Redis without
changing callers.
router.Execute, response translation, usage record.
Source: docs/ARCHITECTURE.md § “Request lifecycle (sync)”.
The async path (POST /v1/jobs) runs the identical gate
before persisting and enqueuing the job — same pipeline, different
shell (ADR-002).
router.Execute, which orders them by one
strategy (§02) and returns one response shape to the caller
regardless of which provider answered. One OpenAI-compatible adapter
covers OpenAI, Groq, self-hosted/vLLM-class endpoints, OpenRouter,
Together, and Fireworks; Anthropic has a separate native adapter
(internal/providers, docs/ARCHITECTURE.md).
02 · METHOD
The four mechanisms this brief is making claims about
Routing strategies
internal/router orders a route's configured targets by
one of four strategies (route.strategy); the same
failover, backoff, and per-(provider, model) circuit breaker
machinery runs underneath all four.
len(targets) + retries attempts.length, a minimum character count, or a refusal-phrase match) and a weak answer escalates to the next, stronger rung; the last good answer is served if every stronger rung also fails. Streaming cascade routes degrade to priority order — a quality check can't run on bytes already sent to the client.One-way PII redaction (ADR-010)
A single-pass span scan (internal/pii) locates emails,
phone numbers, SSNs, credit-card numbers, and more; Scan
and Redact share the same span resolution against the
original text.
off / log / redact / block, set per deployment (pii.mode).off is configured. In redact mode a detected span is replaced with [REDACTED_<TYPE>] before the provider ever sees the message.pii.scan_responses is enabled and pii.mode == redact. Also covers tool_calls[].function.arguments and role:"tool" content, but never image_url values or tools schema definitions.Budget reservation
ReserveSpend (ADR-011) sits at the gate, before a
request is admitted to the router.
max_tokens).Tool-calling + multimodal passthrough (ADR-028)
json.RawMessage end to end, from the request body to the OpenAI-compatible adapter's wire request — Krindel never parses their internal shape.ToolCall{Index, ID, Type, Function{Name, Arguments}}) shared by oai/providers/the OpenAI adapter/the mock. Streaming deltas carry a non-nil Index and are reassembled by index in the adapter.image_url); a plain string body becomes a single text part. image_url values pass through unscanned — a regex engine cannot safely redact binary/URL image data without corrupting it.tools, tool_choice, an image_url part, or any tool-calling message shape. Translating OpenAI's tool-calling/multimodal semantics into Anthropic's block-based Messages API is real, separate work, deferred until a real requirement exists.03 · RESULTS
Numbers this repository measured
Measured with krindlebench (ADR-016), an open-loop
harness that boots the full gateway in-process against an
identical-stack baseline, with no coordinated omission and a <1%
error acceptance threshold. Shared 2-vCPU sandbox, go1.24.4,
2026-07-02. Zero errors at a sustained 1,000 RPS on 2 vCPUs.
| Path | 100 RPS | 1,000 RPS | 5,000 RPS · reference hardware | ||||
|---|---|---|---|---|---|---|---|
| Δp50 | Δp95 | Δp99 | Δp50 | Δp95 | Δp99 | status | |
| sync | 355µs | 580µs | 536µs | 149µs | 145µs | 180µs | pending — see docs/BENCHMARK_RUNBOOK.md |
| stream (TTFB) | 291µs | 440µs | 621µs | 46µs | 134µs | 248µs | pending — see docs/BENCHMARK_RUNBOOK.md |
| sync+pii(redact) | 583µs | 743µs | 875µs | 151µs | 123µs | 181µs | pending — see docs/BENCHMARK_RUNBOOK.md |
5,000 RPS + reference-hardware column: pending run per docs/BENCHMARK_RUNBOOK.md — that runbook requires ≥4 vCPU reference hardware (e.g. a c6i.xlarge/c3-standard-4-class instance) plus specific OS/kernel tuning before a 5,000+ RPS run is valid. This column is left empty deliberately rather than filled with a number the repository hasn't produced.
Caveats on the numbers above (docs/ARCHITECTURE.md): the baseline serves precomputed bytes, so these deltas are a conservative upper bound of gateway cost, not a floor; percentile deltas compare distributions, not per-request differences; client and servers share CPU in-process. These sandbox numbers gate internal claims only.
Caveat: benchmark numbers were measured on a shared 2-vCPU VM. A reference-hardware re-run is on the roadmap before these numbers appear anywhere money changes hands.
04 · LIVE DEMONSTRATION
See it run — against a deterministic mock provider
When the hosted demo is provisioned,
web/playground.html mints a real,
budget-capped virtual key from a running Krindel instance behind a
Turnstile check, then sends that key a real chat completion through
the actual gateway pipeline (§01): auth, PII policy, rate
limiting, budget reservation, cache probe, routing, streaming, and the
returned krindle cost/usage extension are the real,
shipping engine — not a simulation of it. Provisioning gates on
owner steps (deploy/demo/RUNBOOK.md); until they run, the
playground detects this and says so plainly rather than faking a
response. Either way, the offline homepage demo runs the same engine,
compiled to WebAssembly, entirely in your browser.
What is not live: the model behind
demo-model is Krindel's deterministic mock provider, not
a hosted LLM. Real-model answers are gated on further hardening (key
expiry/revoker cron, mint rate-limiting, origin lockdown, a budget
ceiling, and rotating a previously-exposed key) that has not shipped
yet — stated plainly here rather than implied away.
Here's your curl (drop-in OpenAI SDK story)
curl "$KRINDLE_BASE_URL/v1/chat/completions" \
-H "Authorization: Bearer $KRINDLE_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"demo-model","messages":[{"role":"user","content":"hello"}]}'
# equivalently, with the official OpenAI SDK unmodified:
client = OpenAI(base_url=KRINDLE_BASE_URL, api_key=KRINDLE_KEY)
client.chat.completions.create(model="demo-model", messages=[...])
KRINDLE_BASE_URL and KRINDLE_KEY are filled
in for you after you mint a key on the playground — nothing
here is a placeholder domain dressed up as a real one. The SDK
compatibility claim itself is tested, not assumed: the official,
unmodified OpenAI SDKs (openai-python, openai-node, versions pinned
in scripts/sdk_smoke.sh) pass non-streaming calls, SSE
streaming, and typed error handling against a live gateway.
05 · LIMITATIONS & HONESTY
What this brief does not claim
Krindel's constitution (PROJECT.md) and ADR-025 forbid stating a capability the repository can't back up to a specific line or file. That policy is treated as a property of this page, not a disclaimer bolted on afterward: every number above links to its source, and every gap below is named rather than smoothed over.
- Benchmark hardware
- All §03 numbers are from a shared 2-vCPU sandbox VM, not the ≥4 vCPU reference hardware docs/BENCHMARK_RUNBOOK.md requires before any of these numbers may appear in an external or commercial claim. The 5,000 RPS + reference-hardware run has not happened yet.
- Anthropic tool-calling / multimodal
- Deferred (ADR-028). The Anthropic adapter rejects
tools/tool_choice/image_url/tool-calling message shapes with an honest 400 rather than attempting a lossy translation into its block-based Messages API. - Streaming-response PII redaction
- Not implemented. Sync responses can be redacted (opt-in,
pii.scan_responses); streamed text and streamed tool-call argument fragments are not, because per-chunk regex scanning cannot safely see a whole PII pattern split across chunk boundaries without buffering and reassembling first. - Real-model demo
- §04's live demonstration runs against Krindel's deterministic mock provider. Enabling a real upstream model needs further hardening (key expiry/revoker cron, mint rate-limiting, origin lockdown, a budget ceiling, rotating a previously-exposed key) that has not shipped.
- In-memory demo store
- The hosted demo runs Krindel's in-memory, event-sourced store
(
data/events.jsonlreplay), not the clustered Postgres/Redis backend docs/ARCHITECTURE.md describes for cluster mode.
REFERENCES
Each claim above traces to one of the repository's own design records, cited below by name. Krindel's implementation is distributed as a commercial product; whether the source repository is published publicly is a separate, pending decision, so these are listed as citations rather than public links.