Multi-Tenant LLM Gateway Architecture for Teams

A multi-tenant LLM gateway is a control plane that lets many teams, projects, and applications share one OpenAI-compatible ingress while keeping credentials, quotas, usage, and media ownership separated. The architecture problem is not "add a tenant_id column." It is making every request answer four questions consistently: who is calling, who pays, which models they may use, and which artifacts they may read afterward. This article maps those questions onto gateway patterns you can implement, including team workspaces, project-scoped keys, rate-limit scopes, and media isolation for image and video jobs.

Why multi-tenancy appears after the first production win

Early deployments often start with one admin key and one provider channel. That works until:

• product, research, and support share the same spend pool
• contractors need temporary model access without seeing org secrets
• finance asks which project drove last month's invoice
• a generated video URL must not be downloadable by a sibling team
• one noisy agent exhausts rate limits for everyone

At that point you need tenancy boundaries in the gateway, not only in application code. Application-side soft limits help, but they cannot stop a leaked key or a misconfigured worker from charging the wrong owner. A multi-provider LLM gateway without tenant isolation eventually becomes shared infrastructure with shared blast radius. Contrast that with a thin reverse proxy in LLM proxy vs AI gateway: proxies forward; multi-tenant gateways authorize and account.

Core tenancy model: owner, actor, and scope

Keep three identities separate on every request:

1. Actor — the human or service principal that initiated the call.
2. Billing owner — the wallet that loses quota when the call settles.
3. Access scope — the team/project (or personal workspace) that bounds keys, models, and artifacts.

In repository-shaped systems, personal tokens typically bill a user owner, while team-scoped tokens bind both a team and a project and resolve billing to the team wallet. Middleware should reject team-scoped tokens when team workspaces are disabled, when the project is inactive, or when the actor lacks the action needed for that project. That check belongs on the hot path before channel distribution.

Client / agent
   |  Bearer sk-...
   v
Auth middleware
   - resolve token -> user, team_id, project_id, owner_type
   - validate membership + project action grants
   v
Policy + rate-limit scope
   - team:project:user (or personal user scope)
   - model allowlists / group grants
   v
Relay / task submit
   - upstream channel selection
   - pre-consume quota against billing owner
   v
Usage log + optional media persistence
   - store actor, team, project, billing owner
   - authorize later content proxy by same scope

If actor and billing owner collapse into one field, you cannot answer "who spent team budget" versus "who pressed generate." If scope is only a label in logs, authorization on /v1/videos/{id}/content will leak across tenants.

Tenancy surfaces that must stay aligned

1. API keys and project binding

Issue keys as first-class tenancy objects:

• personal keys for individual sandboxes
• team keys bound to team_id + project_id
• optional group or model grants that limit which stable gateway aliases the key may call

Stable aliases (for example gpt-image-2 or a product-facing video-default) should map to upstream model IDs in channel configuration. Clients keep the alias; operators rotate upstream raw IDs without rewriting every service. Never embed upstream provider secrets in client environments. Clients authenticate only to the gateway; channels hold provider credentials. Apply the same isolation discipline to RBAC for AI API access across multi-team products.

2. Quotas and pre-consume safety

Billing safety is a tenancy concern. Pre-consume against the billing owner before expensive work; settle or refund after upstream completion. For media multipliers such as image n or video duration, enforce hard bounds at validation time so a hostile or buggy client cannot overflow quota math into a negative charge. In this codebase, image count is capped (MaxImageN = 128) and video duration is capped (MaxTaskDurationSeconds). Multi-tenant systems should keep those guards global even when team policies add tighter project budgets.

3. Rate limits with composite scope keys

A useful scope key looks like team:{id}:project:{id}:user:{id} for team traffic and a user-only key for personal traffic. That prevents one project's agent storm from starving another project's chat traffic. Keep limit decisions independent from channel selection: a team can be allowed a model and still be throttled.

4. Usage logs as the audit surface

Every settled call should retain actor, billing owner, team/project (when present), model alias, request id, and media multipliers (n, seconds, size). Finance and security reviewers both need those fields. Product-facing usage views are authenticated console surfaces; public documentation should explain the audit model instead of repeatedly linking readers into private console routes.

5. Media ownership is not optional metadata

Image and video artifacts outlive the HTTP request. Treat object keys and task IDs as tenant resources:

Surface	Scope in this repository (verify before you claim it)	Ownership rule
Image workbench UI/API	Personal/session/token under /api/user/image-workbench and API-key variants; object keys are user-owned	No team/project image-workbench routes today
Video workbench UI/API	Personal under /api/user/video-workbench and team/project under /api/teams/:team_id/video-workbench	Team runs and assets authorize by team + project grants
OpenAI-compatible image ingress	POST /v1/images/generations, POST /v1/images/edits	Bill the token's billing owner; do not invent team history unless you persist it yourself
OpenAI-compatible video ingress	POST /v1/videos, retrieve, content; remix still registered	Content GET must re-check the same owner/scope that created the task

Upstream provider URLs often expire. Gateways that persist objects to private storage and re-serve through an authorized proxy reduce broken links and accidental public exposure. When object storage is required for a workbench, fail closed if storage is misconfigured rather than returning an unscoped temporary URL.

Isolation checklist before you onboard a second team

Use this as a release gate, not a slide:

Boundary	Pass criteria
Auth	Team-scoped token fails for removed members and disabled projects
Billing	Team job never debits a personal wallet (and reverse)
Models	Project without grant cannot call restricted aliases
Logs	Cross-team log query returns empty or denied, not partial rows
Media	Video content proxy denies foreign team task IDs as not found/forbidden
Rate limit	Project A load does not consume Project B's token bucket
Admin	Super-admin actions are audited separately from tenant admins

Automated tests should hit the deny path, not only the happy path. Multi-tenant bugs hide in "soft 404" decisions: returning 404 for foreign resources is often better than 403 if resource existence itself is sensitive.

Reference request path for team-scoped chat

curl https://your-gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $TEAM_PROJECT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_STABLE_CHAT_ALIAS",
    "messages": [
      {"role": "user", "content": "Summarize yesterday deploy risk for project alpha."}
    ],
    "max_tokens": 400
  }'

What the gateway should do internally:

1. Authenticate the key and resolve team_id / project_id.
2. Confirm the actor may use a project token for that project.
3. Resolve the model alias through the allowed group/channel set.
4. Pre-consume estimated quota from the team billing owner.
5. Relay upstream and write a consume log with actor + owner fields.
6. Expose the same identifiers in authenticated usage views for finance and support.

Media under the same token rules, different workbench scopes

Media endpoints should not invent a second auth system for billing. Prefer the same token and session rules for OpenAI-compatible ingress:

# Image generation through OpenAI-compatible ingress (GPT Image returns b64_json)
curl https://your-gateway.example.com/v1/images/generations \
  -H "Authorization: Bearer $TEAM_PROJECT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-image-2",
    "prompt": "Product hero shot on matte black pedestal, studio light",
    "size": "1024x1024",
    "n": 1,
    "output_format": "png"
  }'

# Async video job (evaluate Sora 2 only with the official deprecation timeline in mind)
curl https://your-gateway.example.com/v1/videos \
  -H "Authorization: Bearer $TEAM_PROJECT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_STABLE_VIDEO_ALIAS",
    "prompt": "Slow orbit around a ceramic mug on a wood table",
    "seconds": "4",
    "size": "1280x720"
  }'

Workbench UIs differ:

• Image workbench is a personal operator surface: presets, runs, and assets are user-scoped. Do not claim team image history or /api/teams/:team_id/image-workbench routes unless you have implemented them.
• Video workbench supports personal and team/project scopes. Map generate versus manage-asset capabilities to team actions, and keep run history project-scoped when the session is team-bound.

Retry-persistence endpoints must not re-home an asset into another owner. For video greenfield work, prefer vendors and gateway adapters you can still support after OpenAI's Videos API / Sora 2 shutdown on 2026-09-24 (OpenAI deprecations); if you still hold Sora traffic, plan migration of task storage and client code paths now.

Policy layers that scale without rewriting clients

Stack policies from coarse to fine:

1. Platform defaults — max image n, max video seconds, blocked models.
2. Team policy — allowed groups, budget alerts, feature flags.
3. Project grants — which members may generate media or manage video assets.
4. Key constraints — IP allow lists, expiry, model subset.
5. Runtime limits — RPM/TPM and concurrent task caps.

Clients keep calling /v1/*. Operators change policy in admin configuration and channel model mappings. That is the operational payoff of an OpenAI-compatible API gateway.

Failure modes unique to multi-tenant gateways

• Silent owner fallback: if team resolution fails, do not bill the actor's personal wallet "to be helpful." Fail the request.
• Shared channel affinity without isolation labels: cache keys must include tenant scope or you will mix sticky routing state.
• Assuming image workbench is team-aware: personal image assets under user ownership are not the same as team video projects.
• Log retention that ignores legal boundaries: team offboarding may require selective deletion; design retention jobs with owner filters.
• Content proxy confusion: dashboard session auth and API token auth both need the same ownership checks.
• Budget alerts without action grants: alerting finance is useless if any member can still mint keys.

Security guidance from OWASP LLM Top 10 applies: excessive agency, sensitive information disclosure, and unbounded consumption become tenancy incidents when one tenant can affect another.

Minimal implementation sequence

1. Model teams, projects, members, and action grants.
2. Add owner fields to tokens and consume logs.
3. Enforce team token validation in auth middleware.
4. Scope rate limits and usage queries.
5. Port video task authorization and content proxy to the same scope helper; keep image workbench user-owned unless you deliberately add team image persistence.
6. Add e2e tests for cross-tenant denial on chat, image bytes you store, and video content.
7. Only then migrate real product traffic onto team keys.

Skipping the deny-path tests is how "we have multi-tenant UI" ships without multi-tenant safety.

Related reading

• Video generation API integration

Sources and further reading

• OWASP Top 10 for LLM Applications
• OpenAI production best practices
• OpenAI deprecations (includes Videos API / Sora 2 shutdown 2026-09-24)
• Google Cloud multi-tenancy guidance for enterprise organizations

Decision checks

Is a separate database per tenant required?

No for most AI gateways. Row-level isolation with disciplined query filters is common. Separate databases help regulated isolation or noisy-neighbor storage limits, but they multiply migrations and channel config. Choose based on compliance, not fashion.

Should every microservice mint its own provider key?

No. Mint gateway keys per service identity and keep provider keys in channels. That preserves rotation, spend visibility, and failover.

How do personal workspaces coexist with teams?

Keep personal owner type as a first-class path. Users may prototype on personal keys and the personal image workbench, then switch to team project keys for shared chat/video spend without changing request shapes—only the Bearer token and policy change.

What is the biggest multi-tenant media mistake?

Treating upstream result URLs as durable public links, or assuming every workbench is team-scoped. Persist when the product requires durability, proxy through authorized routes, document expiry when you cannot persist, and state personal versus team ownership honestly.

Migration drill

Pick one production application and migrate it from a shared admin key to a team project key. Verify that consume logs show the correct billing owner, that foreign video task IDs are denied on /v1/videos/{id}/content, that image workbench assets remain user-owned unless you built additional sharing, and that project rate limits no longer collide with a sibling team's load tests.