Design a Web Crawler

1. Problem Statement

2. Architecture Discussion Map

Use this as one discussion aid, not a single correct answer. Your design should follow from the requirements, scale, and trade-offs you establish.

graph LR
    A["Design a Web Crawler"]
    A --> F1["URL Frontier Prioritization & Politeness"]
    A --> F2["Distributed Traversal Avoiding cycles & duplicates"]
    A --> F3["Content Deduplication SimHash/Checksums"]
    A --> F4["DNS Resolution Caching to prevent bottlenecks"]
    A --> F5["Storage Raw Pages in Blob Storage, Metadata in a Scalable Table"]

3. Key Focus Areas

• 1
  URL Frontier (Prioritization & Politeness)
• 2
  Distributed Traversal (Avoiding cycles & duplicates)
• 3
  Content Deduplication (SimHash/Checksums)
• 4
  DNS Resolution (Caching to prevent bottlenecks)
• 5
  Storage (Raw Pages in Blob Storage, Metadata in a Scalable Table)

4. What the Interviewer Looks For

The interviewer probes these areas, and the report grades the interview against them.

• Requirements and capacity
• URL frontier and prioritization
• Politeness, robots, and host scheduling
• Deduplication and crawl-trap defense
• Fetching, DNS, storage, and fault tolerance