Design a Multi-Tenant LLM Inference Platform

1. Problem Statement

2. Architecture Discussion Map

Use this as one discussion aid, not a single correct answer. Your design should follow from the requirements, scale, and trade-offs you establish.

graph LR
    A["Design a Multi-Tenant LLM Inference Platform"]
    A --> F1["Workload classes and SLO admission control"]
    A --> F2["Prefill/decode scheduling and continuous batching"]
    A --> F3["KV-cache placement, paging, and prefix reuse"]
    A --> F4["Fleet routing, autoscaling, and tenant isolation"]
    A --> F5["Correctness gates, observability, and failure handling"]

3. Key Focus Areas

• 1
  Workload classes and SLO admission control
• 2
  Prefill/decode scheduling and continuous batching
• 3
  KV-cache placement, paging, and prefix reuse
• 4
  Fleet routing, autoscaling, and tenant isolation
• 5
  Correctness gates, observability, and failure handling

4. What the Interviewer Looks For

The interviewer probes these areas, and the report grades the interview against them.

• SLOs, workload classes, and capacity
• Scheduling, batching, and admission control
• KV cache and memory management
• Fleet routing, autoscaling, and tenancy
• Correctness, rollout, and reliability

5. What Strong Candidates Should Demonstrate

• Reason separately about prefill and decode bottlenecks.
• Balance p99 latency, throughput, cost per token, fairness, and output correctness.
• Design safe rollouts across model, runtime, and accelerator changes.