Area 05 · AI System Design · Concept 17 of 20

Multi-tenant Design

Many customers on one system, each with separate data, limits and costs.

11–20 · Production levelPhase 6 · Weeks 21–23≈ 1.5 hours

The idea

B2B AI products serve many customers (tenants) from shared infrastructure. Every layer must keep them apart: documents and vectors filtered by tenant, caches keyed by tenant, memories and logs scoped by tenant, and prompts that never mix data from two customers.

Fairness matters too: one tenant’s huge batch job must not exhaust the provider rate limit for everyone. Use per-tenant quotas, queues and concurrency limits. Track cost per tenant for billing and margin.

Some customers will require stronger isolation: dedicated indexes, their own encryption keys, a specific region, or their own model deployment. Design the tenant ID into everything from day one so these options stay possible.

Why it matters

Data leakage between tenants is a company-ending bug, and noisy neighbours are a constant production issue.

Words to know

  • TenantOne customer organisation.
  • Noisy neighbourA tenant whose load hurts others.

Try it without code

Trace a tenant ID

  1. Follow one request through your system on paper.
  2. At every store and cache, write how tenant isolation is enforced.

Use it in code

Enforce tenant isolation in Postgres with row-level security.

sql
ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON chunks
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- In each request, inside a transaction:
SET LOCAL app.tenant_id = '6f1c...';
SELECT body FROM chunks ORDER BY embedding <=> $1 LIMIT 5;  -- only this tenant's rows
PRACTICE · TICK IT AFTER THIS

Add tenant_id and RLS to your RAG tables, then write a test that proves tenant A cannot retrieve tenant B’s chunks.

Common mistakes

  • Filtering by tenant in application code only.
  • Shared semantic cache across tenants.

Go deeper