Design a Multi-Tenant LLM Inference Platform System Design Interview

1. Problem Statement

2. Architecture Discussion Map

Use this as one discussion aid, not a single correct answer. Your design should follow from the requirements, scale, and trade-offs you establish.

graph LR
    A["Design a Multi-Tenant LLM Inference Platform"]
    A --> F1["Workload classes and SLO admission control"]
    A --> F2["Prefill/decode scheduling and continuous batching"]
    A --> F3["KV-cache placement, paging, and prefix reuse"]
    A --> F4["Heterogeneous accelerator routing and autoscaling"]
    A --> F5["Correctness gates, observability, and multi-region failure handling"]

3. Key Focus Areas

• 1
  Workload classes and SLO admission control
• 2
  Prefill/decode scheduling and continuous batching
• 3
  KV-cache placement, paging, and prefix reuse
• 4
  Heterogeneous accelerator routing and autoscaling
• 5
  Correctness gates, observability, and multi-region failure handling

4. What Strong Candidates Should Demonstrate

• ✓Reason separately about prefill and decode bottlenecks.
• ✓Balance p99 latency, throughput, cost per token, fairness, and output correctness.
• ✓Design safe rollouts across model, runtime, and accelerator changes.

Evaluation Guide

This is an evaluation framework, not a single model answer. Strong designs may make different choices when their assumptions and trade-offs are explicit.