Langfuse Launches an Open Source AI Gateway in Public Beta: Built in Rust as a 22MB Image, It Tracks Claude Code and Codex Spend per User, Key, and Model After One Base URL Change

Core Background and Industry Pain Points

AI use inside companies is no longer one team building one chatbot: every engineer runs Claude Code or Codex, and teams adopt their own assistants and scripts. Platform teams are asked who uses which model, for what, and at what cost, yet most of these tools are not instrumented, and provider keys are often handed to individuals and are hard to reclaim when people leave. On November 9 (the changelog entry is dated November 4), Langfuse announced its AI Gateway in public beta to route every model call through one entry point.

Architecture Highlights and Underlying Mechanisms

An admin stores OpenAI and Anthropic provider credentials in Langfuse, picks which models may be called, and issues gateway keys to teammates. User keys stop working when the person leaves the organization, service account keys cover CI, and any key can have an expiry, be revoked instantly, and carry metadata such as team or cost_center that the client cannot change. The gateway forwards requests in their native format and records each call as a generation, in either Full mode (with inputs and outputs) or Usage mode (usage and cost only). Unlike standalone gateways that only count tokens, these records are regular Langfuse traces that can be searched, evaluated, and annotated.

Authoritative Benchmarks and Measured Results

Langfuse's own performance figures (vendor figures): under 3 ms median latency with tracing on, more than 1,000 concurrent streams on a single 2 vCPU node, and memory that scales with load. No third-party load tests or comparisons with gateways such as LiteLLM or Portkey have been published. The beta supports only OpenAI and Anthropic, and routes and headers may still change.

Developer Playbook and Getting Started

After an admin finishes setup under Organization settings, AI Gateway, teammates point the OpenAI SDK base_url at their region's gateway host (for example gateway.cloud.langfuse.com/openai/) and swap the provider key for a gateway key. Setup guides exist for Claude Code, Codex, Copilot CLI, OpenCode, and pi. Regulated teams can self-host the 22MB image on their own network; by default only metadata and cost are recorded, so turn on full logging only when needed. The roadmap includes spend limits per team, failover between providers, PII redaction, and automatic model routing.