Design an LLM Inference Performance Optimization Program System Design Interview

1. Problem Statement

2. Architecture Discussion Map

Use this as one discussion aid, not a single correct answer. Your design should follow from the requirements, scale, and trade-offs you establish.

graph LR
    A["Design an LLM Inference Performance Optimization Program"]
    A --> F1["Cross-layer latency and throughput attribution"]
    A --> F2["Roofline analysis and representative benchmarks"]
    A --> F3["Kernel fusion, memory bandwidth, and variable sequence lengths"]
    A --> F4["Quantization and numerical correctness"]
    A --> F5["Canary rollout, regression detection, and rollback"]

3. Key Focus Areas

• 1
  Cross-layer latency and throughput attribution
• 2
  Roofline analysis and representative benchmarks
• 3
  Kernel fusion, memory bandwidth, and variable sequence lengths
• 4
  Quantization and numerical correctness
• 5
  Canary rollout, regression detection, and rollback

4. What Strong Candidates Should Demonstrate

• ✓Trace latency across routing, batching, runtime, kernels, memory, and hardware.
• ✓Use roofline and FLOPs funnels rather than isolated microbenchmarks.
• ✓Treat output correctness as a release gate for optimization.