Self-Hosted LLM API Gateway: Architecture and Operations

A self-hosted LLM API gateway is software you run in your own network or cloud account that accepts OpenAI-compatible requests, authenticates callers, routes traffic to one or more upstream model providers, and records usage for operations and cost control. Unlike a pure reverse proxy, it is a control plane: keys, model aliases, channel credentials, quotas, and request history live next to the request path. This article explains when self-hosting is justified, what architecture to expect from a Go-based gateway such as this repository, and how to operate it without turning model access into an unowned black box.

If you are still choosing between a thin hop and a control plane, read LLM proxy vs AI gateway first. This page assumes you already want gateway responsibilities and need a deployable baseline.

When self-hosting is the right default

Self-host a gateway when at least two of these are true:

• provider API keys must not be distributed to every application environment
• multiple products or teams share model access under different budgets
• you need request-level logs inside your own retention boundary
• data-path or procurement rules favor infrastructure you operate
• you want multi-provider routing without rewriting clients
• SaaS gateway constraints (data residency, network egress, change control) do not fit

Self-hosting is the wrong default when a single prototype uses one key, one model, and has no audit requirement. In that case, call the provider directly or use a thin proxy until the control-plane needs appear.

Reference architecture

Clients (apps, agents, eval jobs)
        |
        |  HTTPS + Bearer gateway key
        v
 Self-hosted AI gateway
   - OpenAI-compatible API (/v1/chat/completions, /v1/responses, ...)
   - Admin API for users, tokens, channels
   - Usage logs, dashboard, pricing context
        |
        +--> Channel: OpenAI
        +--> Channel: Anthropic / Claude (native Messages adaptation)
        +--> Channel: Gemini
        +--> Channel: Azure / Bedrock / others
        |
        v
 Your datastore (SQLite / MySQL / PostgreSQL)
 optional Redis for cache and rate limiting

Clients keep using the OpenAI SDK or raw HTTP. Upstream diversity is configuration, not a new SDK per provider. That is the same value proposition as an OpenAI-compatible API gateway, with the added requirement that you own uptime and backups.

What this repository-shaped gateway typically includes

This project is a self-hostable Go gateway and AI asset management system. Repository-visible capabilities that matter for planning:

• OpenAI-compatible relay endpoints for chat and related surfaces
• multi-provider channel configuration
• user/token management for scoped access
• usage logging and operational dashboards
• web admin UI embedded with the service
• SQLite for simple deployments; MySQL or PostgreSQL for durable multi-user production
• Docker-based deployment paths using the published image calciumion/new-api

Do not invent compliance certifications or unstated SLA claims. Validate the exact feature set you need against the running version's UI and API before promising them to stakeholders.

Verified minimal Docker path

The repository ships a real docker-compose.yml that runs the gateway with PostgreSQL and Redis. That is the preferred starting point for multi-user staging. Quick path from the project README:

docker pull calciumion/new-api:latest
# from a checkout that includes docker-compose.yml
docker compose up -d
# gateway listens on http://localhost:3000 by default

Core service shape (abridged from the repo compose file; keep the full file for Redis/Postgres services, healthchecks, and comments):

services:
  new-api:
    image: calciumion/new-api:latest
    container_name: new-api
    restart: always
    command: --log-dir /app/logs
    ports:
      - "3000:3000"
    volumes:
      - ./data:/data
      - ./logs:/app/logs
    environment:
      # Change passwords before any shared or production use.
      - SQL_DSN=postgresql://root:123456@postgres:5432/new-api
      - REDIS_CONN_STRING=redis://:123456@redis:6379
      - TZ=Asia/Shanghai
      - ERROR_LOG_ENABLED=true
      - BATCH_UPDATE_ENABLED=true
      - NODE_NAME=new-api-node-1
    depends_on:
      - redis
      - postgres

Operational notes that keep this example honest:

• Default compose passwords (123456) are for local bring-up only. Rotate them before any shared environment.
• Prefer pinned image digests in regulated environments instead of the floating :latest tag used in docs.
• For a single-node SQLite experiment, the README also documents running the container with a mounted /data volume and without SQL_DSN; treat that as a small-footprint path, not multi-writer production.
• Place TLS termination at your ingress (nginx, Caddy, cloud load balancer, Traefik). Do not expose admin UI paths to the public internet without additional access control.
• Provider API keys belong in channel configuration or a secret manager after first login, not in git.

If you cannot use Docker Compose as-is (air-gapped registries, custom orchestration), treat the snippet as a verified baseline to translate, not as optional pseudocode. The image name, port, volume, and SQL_DSN variables above match this repository's published deployment docs.

First-week configuration checklist

1. Identity and keys

• create admin users with strong auth
• issue per-service tokens with least model scope
• separate staging and production keys
• document who owns rotation

2. Channels

Under channel configuration in the admin UI:

• one channel per upstream credential boundary
• explicit base URLs for Azure or regional endpoints
• health checks or manual smoke tests after credential changes
• never share a single unrestricted upstream key across hostile tenants

3. Catalog and pricing context

• publish stable model aliases (claude-sonnet, gemini-flash, gpt-...) that apps depend on
• map aliases to current upstream IDs (for example Claude Sonnet 5 / Opus 4.8 and Gemini 3.5 Flash class models as listed by vendors at cutover time)
• review pricing context when default aliases change

4. Observability

• enable usage logs for billable routes
• confirm log retention and export paths
• alert on error rate and spend velocity, not only CPU

5. Backups

• automate database backups (PostgreSQL/MySQL dumps or volume snapshots)
• test restore on a blank host before the first real incident
• store backup encryption keys outside the gateway container

Security model for a self-hosted control plane

Threats to design against

• stolen gateway keys that can burn provider budget
• admin UI exposure on the public internet
• prompt and tool logs containing customer secrets
• SSRF-style channel base URLs if operators can point channels at internal metadata endpoints
• multi-node session fixation if SESSION_SECRET is left default in a cluster

Controls that actually help

• TLS at the edge; private admin access (VPN, SSO, IP allow lists)
• unique keys per application; model allow lists on batch keys
• provider secrets only on the gateway
• quotas on any key that can trigger tools or long contexts
• redaction policies for usage log exports
• SESSION_SECRET set for multi-node deploys (see compose comments in-repo)

Review OWASP Top 10 for LLM Applications when agents can call tools automatically. Self-hosting does not remove excessive-agency risk; it only moves where you enforce it.

Routing policy as configuration

Write routing policy as configuration:

1. default model alias per product
2. backup alias for transient upstream errors
3. pinned alias for regulated or eval-sensitive flows
4. explicit denial for models a key must never touch

Pair routing with LLM API failover strategy so mid-stream failures are not naively retried. For tool agents, keep schemas portable as in portable tool calling across providers.

Capacity, databases, and Redis

Footprint	Database	Notes
Solo / small team	SQLite	Simple, mount persistent volume
Multi-team production	MySQL ≥ 5.7.8 or PostgreSQL ≥ 9.6	Better concurrency and ops tooling
Hot rate-limit/cache paths	Redis optional	Use when process-local limits are insufficient

Size CPU and memory from concurrent streams, not from idle admin UI usage. Streaming responses hold connections open; load tests should include SSE-style traffic if your clients stream.

Upgrade and change management

1. Read release notes for channel and billing-related changes.
2. Deploy to staging with production-like channel config.
3. Run curl smoke tests for /v1/chat/completions and any /v1/responses usage.
4. Verify log ingestion and wallet/quota behavior on a canary key.
5. Roll forward with a known previous image for rollback.
6. After upgrade, re-check model aliases against upstream renames.

Never combine database migrations, provider credential rotation, and routing policy changes in the same unattended window.

Smoke tests after every deploy

curl -sS https://gateway.example.com/v1/models \
  -H "Authorization: Bearer $GATEWAY_API_KEY" | head

curl -sS https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-mini",
    "messages": [{"role": "user", "content": "ping"}],
    "max_tokens": 16
  }'

Add one Claude alias (for example claude-sonnet → current Claude ID) and one Gemini alias (gemini-flash → gemini-3.5-flash or successor) if those channels are in scope. Confirm entries in usage logs before declaring the deploy healthy.

Cost control without giving up self-hosting

Self-hosted does not mean unlimited. Configure:

• per-key and per-group quotas
• model allow lists that block flagship models on batch keys
• alerts on spend velocity using log exports or dashboard checks
• eval jobs on cheaper models by default

Review public pricing context when default aliases change. A self-hosted gateway that cannot answer cost questions becomes a shared liability.

When to choose managed hosting instead

Stay managed or hybrid when:

• you lack on-call capacity for a tier-0 dependency
• you cannot meet backup/restore obligations
• procurement prefers a vendor-operated control plane
• your team only needs a single-provider secret hop

You can still keep application clients OpenAI-compatible in either case. Hosting mode should not force a client rewrite.

Official and project sources

• Project README deployment section (docker pull calciumion/new-api:latest, docker-compose.yml)
• OWASP Top 10 for LLM Applications
• OpenAI API reference
• Docker documentation

Last checked: 2026-07-15. Validate image tags, env vars, and UI paths against the version you actually deploy.

Common questions

Is a self-hosted LLM gateway the same as hosting the models yourself?

No. Most self-hosted gateways still call hosted provider APIs. You host the control plane—auth, routing, logs—not necessarily the weights. Self-hosting open-weight models is an additional GPU stack.

Can I start on SQLite and move to PostgreSQL later?

Yes for many deployments, but plan the migration instead of discovering lock contention during an incident. Use SQLite for small footprints and move when concurrent writers and backup tooling demand it.

What is the minimum viable secure setup?

TLS at the edge, private admin access, unique keys per application, provider secrets only on the gateway, usage logs enabled, and quotas on any key that can trigger tools or long contexts.

How do I avoid becoming the single point of failure?

Run at least two app instances behind a load balancer when uptime matters, use an external database, test restores, and document channel failover. The gateway should reduce provider outages' blast radius, not create a worse one.

Which endpoint should new clients target?

Prefer the gateway's OpenAI-compatible /v1 root with /v1/chat/completions for broad client support. Adopt /v1/responses only when your product intentionally depends on that surface and your gateway version supports the workflows you need.

Staging cutover

Deploy a staging gateway with the repository compose file (or the equivalent image and SQL_DSN), issue a scoped key to a single non-critical service, and verify chat completions, usage logs, and backup restore before onboarding a second provider or production traffic. Use channel config, the model catalog, and usage history as the daily operator loop once the first canary is green.