Back to interview questions

100+ System Design Interview Questions and Answers

100+ System Design Interview Questions and Answers

Explore 120 system design interview questions and answers on capacity, APIs, storage, caching, consistency, messaging, reliability, and architectural trade-offs.

Explore 120 system design interview questions and answers on capacity, APIs, storage, caching, consistency, messaging, reliability, and architectural trade-offs.

By

Team Sherlock.sh

Add Sherlock as a preferred source on Google

A system design interview tests how a developer turns an ambiguous product requirement into a defensible technical proposal. Naming a database, queue, or load balancer is not enough. A useful answer explains the workload, follows a request or event through the system, identifies failure cases, and shows which trade-offs the design accepts.

This guide to 100+ System Design Interview Questions and Answers contains 120 questions across beginner, intermediate, and experienced levels. It focuses on software services and distributed-system architecture, with practical capacity estimates, data-flow reasoning, and reliability scenarios. It is not an IT systems-administration or infrastructure-maintenance question bank.

How can recruiters detect AI-assisted cheating, deepfakes, and impersonation during system design interviews?

System design interviews often require candidates to build a proposal live, estimate capacity, and adapt when a requirement changes. A polished diagram alone cannot show whether the candidate understands its consistency, latency, or failure assumptions. Unauthorized real-time assistance can make that independent reasoning harder to assess.

Sherlock AI helps recruiters detect potential AI-assisted cheating, deepfakes, candidate impersonation, and unauthorized real-time assistance during system design interviews.

Pair interview-integrity checks with a concrete follow-up: increase the write rate, make one dependency unavailable, or require a stricter data guarantee. Ask the candidate to revise the design and explain the consequences. Review suspicious activity in context rather than treating a signal as proof of misconduct.

System Design Interview Questions

Choose questions that match the role’s architectural scope and expected decision-making responsibility. Establish the product behavior, workload, data guarantees, and failure model before comparing technologies. Evaluate the reasoning behind a design rather than whether it resembles a memorized diagram.

System Design Interview Questions for Beginners

These beginner questions establish requirements, latency and throughput, capacity estimates, service boundaries, storage choices, caching, and basic reliability concepts. They test whether a candidate can explain a small system clearly before adding distributed complexity.

  1. What is system design, and how does it differ from low-level design?

  2. Which functional and nonfunctional requirements should you clarify before proposing an architecture?

  3. How do you turn a vague reliability requirement into a measurable SLO?

  4. What is the difference between latency and throughput?

  5. Why do p95 and p99 latency matter more than an average alone?

  6. How do you estimate average and peak requests per second?

  7. How do you estimate storage and network bandwidth for a proposed service?

  8. When should you scale vertically rather than horizontally?

  9. Why are stateless application servers easier to scale?

  10. What does a load balancer do, and how do layer 4 and layer 7 balancing differ?

  11. How are a reverse proxy, an API gateway, and a load balancer different?

  12. What role do DNS and a CDN play in request delivery?

  13. What is caching, and when can it make a system worse?

  14. How does cache-aside work?

  15. How do you choose a cache key and an expiration policy?

  16. How do you choose between relational, document, key-value, and object storage?

  17. What does a database index improve, and what does it cost?

  18. What is the difference between normalization and denormalization?

  19. How do replication and sharding solve different problems?

  20. What changes when replication is synchronous rather than asynchronous?

  21. What is a partition key, and why can a poor choice create a hot shard?

  22. What do consistency guarantees tell a client about reads and writes?

  23. What does the CAP theorem actually say during a network partition?

  24. What do ACID properties guarantee, and what do they not guarantee?

  25. What is the difference between a message queue and a retained event stream?

  26. When should an API do work synchronously versus asynchronously?

  27. What is idempotency, and why is it important for retryable APIs?

  28. How do timeouts, retries, exponential backoff, and jitter work together?

  29. What is backpressure, and why is an unbounded queue dangerous?

  30. What does a rate limiter protect, and how is it different from a circuit breaker?

  31. What is the difference between authentication and authorization in a system design?

  32. Why do systems use metrics, logs, and distributed traces together?

  33. How are high availability, durability, and disaster recovery different?

  34. Why are replicas not a substitute for backups?

  35. When is a modular monolith a better starting point than microservices?

  36. How should you design an API contract before choosing infrastructure?

  37. How do offset and cursor pagination differ?

  38. How do polling, server-sent events, and WebSockets support real-time updates?

  39. What belongs in a useful high-level architecture diagram?

  40. How do you identify the first bottleneck and decide what to optimize?

System Design Interview Questions for Intermediate

These intermediate questions explore data partitioning, replication, consistency, asynchronous workflows, overload handling, and practical service designs. They ask candidates to explain how components interact when data, traffic, or failures cross a single-process boundary.

  1. How would you design a URL shortener for a read-heavy workload?

  2. How would you generate short-link identifiers without assuming hashes never collide?

  3. How would you design durable one-to-one chat messaging?

  4. How would you keep chat presence and typing indicators separate from durable messages?

  5. How would you design a social news feed using fan-out on write, fan-out on read, or a hybrid?

  6. How would you design a secure resumable file-upload service?

  7. How would you design search for a product catalog?

  8. How would you design a payment-creation API that survives client retries?

  9. How would you design an order workflow across inventory, payment, and shipping?

  10. How would you design a notification service for email, SMS, and push?

  11. How would you design a distributed token-bucket rate limiter?

  12. How would you design a background job queue with retries and dead-letter handling?

  13. How would you design a delayed-task scheduler without scanning every future task?

  14. How would you design cache invalidation when data can be updated concurrently?

  15. How would you prevent a cache stampede after a popular key expires?

  16. How would you estimate the cache capacity and origin load for a read-heavy service?

  17. How would you choose a composite database index for a filtered and ordered query?

  18. How would you preserve read-your-writes when reads normally use replicas?

  19. How would you handle a very large tenant without making it a permanent hot shard?

  20. How would you add shards and move data while requests continue?

  21. How would you make a database update and event publication reliable without a dual-write gap?

  22. How would you build an idempotent consumer when messages can be delivered more than once?

  23. How would you preserve ordering where it matters without imposing a global order?

  24. How would you choose and implement optimistic concurrency control?

  25. How would you prevent overselling the final item in inventory?

  26. How would you keep a search index synchronized and recover from indexing failures?

  27. How would you choose between CQRS and a single read-write model?

  28. How would you design an append-only audit trail without leaking sensitive data?

  29. How would you design a multi-tenant service with strong data isolation?

  30. How would you size workers to meet a queue-processing deadline?

  31. How would you distribute a request deadline across dependent service calls?

  32. How would you handle slow consumers in a real-time streaming service?

  33. How would you keep API pagination stable while new records arrive?

  34. How would you upload metadata and an object without pretending they share one transaction?

  35. How would you verify and process third-party webhooks reliably?

  36. How would you design per-user unread counts or view counters under high write volume?

  37. How would you evolve an event or API schema without breaking older clients?

  38. How would you choose health checks that avoid removing every server during an overload?

  39. How would you design an observability dashboard for a newly proposed service?

  40. How would you reduce a design's cost without silently weakening its guarantees?

System Design Interview Questions for Experienced

These experienced-level questions examine difficult consistency and coordination choices, large-scale reliability, multi-region behavior, architecture evolution, and end-to-end design decisions. They assess whether a candidate can challenge assumptions and justify a design under competing constraints.

  1. How would you design active-passive regional failover with explicit RPO and RTO?

  2. How would you design active-active writes without ignoring conflicting updates?

  3. How would you place a consensus quorum across three regions?

  4. What does Raft guarantee, and under which failure assumptions can it make progress?

  5. Why is quorum intersection not by itself a proof of linearizability?

  6. How would you prevent an expired distributed-lock holder from corrupting shared state?

  7. How do clock skew and clock uncertainty affect IDs, leases, and transaction ordering?

  8. What does exactly-once processing mean at a specified system boundary?

  9. How would you recover a payment operation whose outcome is unknown after a timeout?

  10. How would you design reconciliation to detect and repair missing or duplicated financial events?

  11. How would you handle a saga whose compensation also fails?

  12. How would you choose between two-phase commit and a saga?

  13. How would you analyze tail latency in a service that fans out to many dependencies?

  14. When are hedged requests helpful, and when can they worsen an outage?

  15. How would you keep retry storms from causing cascading failure across services?

  16. How would you test whether a capacity plan survives losing an availability zone?

  17. How would you design graceful degradation while preserving core correctness?

  18. How would you design a hot-key strategy when hashing cannot spread one key's traffic?

  19. How would you manage tombstones, deletions, and stale replicas so deleted data does not return?

  20. How would you rebuild a large derived read model without stopping writes?

  21. How would you make a large backfill safe for live serving traffic?

  22. How would you perform a zero-downtime schema migration across independently deployed services?

  23. How would you design disaster-recovery tests that establish recovery rather than merely checking backups?

  24. How would you avoid a shared control plane becoming a data-plane availability dependency?

  25. How would you make authentication and authorization resilient during identity-service outages?

  26. How would you design data residency and deletion across replicas, caches, logs, and backups?

  27. How would you design end-to-end message ordering across regions?

  28. How would you use CRDTs without assuming every business invariant is mergeable?

  29. How would you design an event-time analytics pipeline for late and out-of-order events?

  30. How would you design an event-sourced service and evolve its historical events?

  31. How would you scale a chat system while keeping history, reconnect, and delivery semantics coherent?

  32. How would you evolve a feed design when celebrity fan-out and privacy changes dominate cost?

  33. How would you design a search service that tolerates partial shard failures?

  34. How would you design a global file service with replication, deduplication, and resumable uploads?

  35. How would you separate a payment ledger's correctness path from asynchronous analytics?

  36. How would you choose a failure domain and blast-radius strategy for a large service?

  37. How would you evaluate a new storage engine without relying on a vendor benchmark?

  38. How would you test a distributed design's invariants under duplicated, delayed, and reordered messages?

  39. How would you recognize when a distributed architecture is more complex than the workload requires?

  40. How would you defend a final architecture when latency, consistency, availability, and cost conflict?

System Design Interview Questions and Answers

The answers follow the question bank in order and make the relevant constraints and trade-offs explicit. Numerical examples are illustrative estimates under stated assumptions, not measured production capacity. Text diagrams and pseudocode clarify responsibilities and sequencing; they are not complete deployable implementations.

System Design Interview Questions and Answers for Beginners

The beginner answers connect core terms to concrete request paths and simple calculations. They distinguish a desired property from the mechanism that might provide it, and state the assumptions behind comparisons.

1. What is system design, and how does it differ from low-level design?

System design defines the components, data ownership, interfaces, and operating tradeoffs needed to meet a service's requirements. It connects user-visible behavior to capacity, failure handling, security, and cost.

For a photo-sharing application, system design decides how uploads reach object storage, where metadata lives, how thumbnails are produced, and what happens when processing fails. Low-level design specifies details inside those components: classes, validation functions, algorithms, and method contracts.

The boundary is flexible. A deduplication algorithm can become an architectural concern if it determines whether retries create duplicate work. The goal is a coherent distributed application, not a list of products. Configuring employee devices or administering an organization's network belongs primarily to IT systems engineering rather than this application-design scope.

2. Which functional and nonfunctional requirements should you clarify before proposing an architecture?

Separate what the service must do from the conditions under which it must work. For an assumed team file-sharing service, clarify:

Category

Example requirement to establish

Functional

Upload, download, share, revoke access, delete

Workload

Active users, peak requests/s, file-size distribution

Performance

Download-start latency, measured at a defined boundary

Reliability

Availability target and acceptable lost updates

Security

Tenant isolation, authorization, retention constraints

Scope

Whether editing, search, and public links are included

Prioritize conflicting requirements before selecting infrastructure. Immediate access revocation, for example, changes whether downloads can rely on long-lived cached authorization. Record assumptions when numbers are unavailable, and identify which architectural choices would change if those assumptions are wrong.

3. How do you turn a vague reliability requirement into a measurable SLO?

Replace “highly reliable” with a defined measurement and target. An assumed availability SLO could be: over each rolling 30-day window, at least 99.9% of eligible API requests return a valid successful result within one second, measured at the service edge.

Define eligibility explicitly: valid authenticated requests are included; malformed requests are excluded; service-generated overload responses count as failures. The SLI is good eligible requests divided by all eligible requests. At 10 million eligible requests, the 0.1% error budget permits 10,000 bad requests under this request-based objective, not a fixed outage duration.

An SLO is an operating target. An SLA is a contractual commitment, potentially with consequences for missing it. Separate latency and correctness objectives may be useful when a combined success measure hides their causes.

4. What is the difference between latency and throughput?

Latency is elapsed time for an operation, commonly measured in milliseconds per request. Throughput is completed work per unit time, such as requests per second or megabytes per second. Specify the measurement boundary: server processing time excludes network and client delays.

A service can complete 5,000 requests/s while some requests still wait several seconds. Batching may increase throughput by amortizing overhead while increasing individual waiting time. Adding workers helps only until another resource, such as the database or a lock, becomes the bottleneck.

Measure offered load, completed throughput, latency, and errors together. Near saturation, a queue can grow rapidly even when completed throughput barely changes. A throughput claim without its workload mix and latency target says little about usable capacity.

5. Why do p95 and p99 latency matter more than an average alone?

A p99 latency of 800 ms means approximately 99% of measured requests complete within 800 ms; it does not describe the slowest 1%. An average can hide that tail, so report percentiles alongside errors, sample count, workload, and measurement window.

An illustrative sample with 990 requests at 20 ms and 10 requests at 2,000 ms has a mean of only 39.8 ms, despite ten users waiting two seconds. Fanout can amplify this problem when a parent request waits for every child: one slow dependency delays the whole response.

Do not add component p99 values to claim an end-to-end p99, or average server percentiles to obtain a fleet percentile. Measure the combined request distribution using suitable histograms or raw observations.

6. How do you estimate average and peak requests per second?

Start with an explicit workload, then distinguish average demand from a planning peak. Assume the service receives 10 million API requests per day, including reads and writes.

Capacity calculation

Seconds/day = 24 x 60 x 60 = 86,400
Average rate = 10,000,000 / 86,400 = 115.74 requests/s
Assumed peak factor = 8
Planning peak = (10,000,000 / 86,400) x 8 = 925.93 requests/s
Seconds/day = 24 x 60 x 60 = 86,400
Average rate = 10,000,000 / 86,400 = 115.74 requests/s
Assumed peak factor = 8
Planning peak = (10,000,000 / 86,400) x 8 = 925.93 requests/s
Seconds/day = 24 x 60 x 60 = 86,400
Average rate = 10,000,000 / 86,400 = 115.74 requests/s
Assumed peak factor = 8
Planning peak = (10,000,000 / 86,400) x 8 = 925.93 requests/s

The factor of eight is a scenario assumption, not a universal property of traffic. Validate it against regional activity, promotions, batch jobs, and retry bursts. Split expensive writes from cheap cached reads, and account separately for internal fanout: one external request may generate several database operations. Capacity must also cover the chosen failure scenario, such as losing one application instance, rather than only the healthy steady state.

7. How do you estimate storage and network bandwidth for a proposed service?

Estimate retained bytes and transferred bytes separately. Assume 1 million new records/day, 2,000 payload bytes plus 500 metadata bytes per record, 30-day retention, three full copies, and no compression. Use decimal units: 1 GB is one billion bytes.

Capacity calculation

Logical data = 1,000,000 x 2,500 x 30 = 75 GB
Three copies = 75 x 3 = 225 GB
Peak response traffic = 2,000 requests/s x 10,000 bytes
                      = 20 MB/s = 160 Mb/s
Logical data = 1,000,000 x 2,500 x 30 = 75 GB
Three copies = 75 x 3 = 225 GB
Peak response traffic = 2,000 requests/s x 10,000 bytes
                      = 20 MB/s = 160 Mb/s
Logical data = 1,000,000 x 2,500 x 30 = 75 GB
Three copies = 75 x 3 = 225 GB
Peak response traffic = 2,000 requests/s x 10,000 bytes
                      = 20 MB/s = 160 Mb/s

The storage estimate excludes indexes, logs, backups, temporary space, and free-capacity headroom. Network traffic excludes request bodies, protocol overhead, retries, and replication. Add those explicitly rather than hiding them in the replication multiplier. Average payload sizes support baseline planning; large-object percentiles and burst duration matter for peak bandwidth and admission limits.

8. When should you scale vertically rather than horizontally?

Vertical scaling gives an existing instance more CPU, memory, or I/O capacity. It is often the simplest choice when a database fits on one machine, the immediate bottleneck is memory or CPU, and the next hardware tier has acceptable cost and upgrade disruption.

Horizontal scaling adds instances. It suits independently executable requests and can improve failure isolation, but requires load distribution and a plan for shared state. Adding application instances will not fix a saturated single database writer.

Choose from evidence about the bottleneck and growth horizon. Vertical scaling has hardware ceilings and leaves a large failure domain; horizontal scaling adds coordination, operational complexity, and sometimes cross-node traffic. A common incremental path is to scale a well-structured application tier horizontally while retaining a vertically scaled database until its workload justifies partitioning.

9. Why are stateless application servers easier to scale?

A stateless application server does not require a particular client's durable session state to remain in that server's memory between requests. Any healthy instance can handle the next request when identity and necessary state come from the request or shared stores.

This simplifies load balancing, replacement, and scaling: losing one instance need not lose a user's cart or login session. Durable cart data might live in a database, while an expiring session lives in a shared session store. Those dependencies still need capacity and failure planning.

Stateless does not mean “no memory.” Instances can cache data and hold active connections. A WebSocket connection remains attached to one instance until reconnecting, so connection draining and recovery are still required. Signed client tokens also require expiration and revocation decisions.

10. What does a load balancer do, and how do layer 4 and layer 7 balancing differ?

A load balancer distributes incoming traffic across eligible backends, applying routing policy and health information.

Layer

Typical decision inputs

Main implication

Layer 4

Addresses, ports, transport connections

No need to inspect HTTP paths

Layer 7

HTTP host, path, headers

Supports application-aware routing

Layer 7 routing over HTTPS requires access to decrypted HTTP, commonly through TLS termination; traffic to backends can be encrypted again. Layer 4 balancing can pass encrypted connections through without understanding their contents.

Health checks reduce traffic to unhealthy instances but can miss partial failures or dependency problems. Balancing also needs connection draining and sensible timeout behavior. A load balancer alone does not create end-to-end high availability: its own redundancy, backend capacity, shared databases, and failover paths still matter.

11. How are a reverse proxy, an API gateway, and a load balancer different?

These names describe overlapping responsibilities, not three mandatory network hops.

Component

Primary responsibility

Reverse proxy

Accept requests on behalf of upstream servers

Load balancer

Distribute traffic across multiple eligible targets

API gateway

Apply API-facing routing and policy

One reverse proxy can also balance load and act as a gateway. Gateway policies may include authentication, request-size limits, quotas, and routing to API versions. Keep business authorization close to the resource-owning service as well: an authenticated request is not automatically allowed to read a particular customer's record.

Introduce separate components only when their ownership, scaling, or security boundaries justify them. Every extra hop adds latency, configuration, and a possible failure point, so document which component owns each policy and how bypass paths are prevented.

12. What role do DNS and a CDN play in request delivery?

DNS resolves service names to addresses or other records, helping clients discover a destination. Resolvers cache answers, so changing a DNS record does not instantly move every client; record lifetime and existing connections affect the transition.

A CDN places edge servers near users. It can serve cached responses without contacting the origin, reducing origin traffic and network distance for reusable content. Cache keys, freshness, validation, and permission rules determine what can safely be reused.

Versioned images and scripts are good candidates for long-lived caching. Personalized responses require careful isolation and may be unsuitable for shared caching. A CDN can also proxy dynamic traffic, but that does not make database-dependent responses cache hits. Plan origin capacity for misses, invalidations, and cache failures rather than assuming every request ends at the edge.

13. What is caching, and when can it make a system worse?

Caching stores a reusable result closer to the caller or in a cheaper-to-access system. It can reduce latency and origin work when requests repeatedly access the same data and an appropriate freshness policy exists.

Assume 10,000 reads/s and a 95% hit rate. The baseline miss traffic is 500 reads/s, before refreshes and retries. If the cache becomes unavailable and every request falls through, the origin suddenly receives 10,000 reads/s, twenty times that baseline.

Caching can make a system worse through stale decisions, incorrect tenant scoping, memory cost, or low hit rates that add an extra lookup. Simultaneous expiration can cause a stampede. Use bounded fallback, request coalescing where suitable, and freshness-aware keys. Do not treat a cached authorization decision as permanently valid after access is revoked.

14. How does cache-aside work?

In cache-aside, the application manages the cache rather than treating it as the system of record. On a miss, it fetches authoritative data and populates the cache. After a successful database update, it invalidates the corresponding entry.

Pseudocode example

read(key):
    cached = cache.get(key)
    if cached is present: return cached
    value = database.read(key)
    cache.set(key, value, ttl)
    return value

write(key, value):
    database.commit_update(key, value)
    cache.delete(key)
read(key):
    cached = cache.get(key)
    if cached is present: return cached
    value = database.read(key)
    cache.set(key, value, ttl)
    return value

write(key, value):
    database.commit_update(key, value)
    cache.delete(key)
read(key):
    cached = cache.get(key)
    if cached is present: return cached
    value = database.read(key)
    cache.set(key, value, ttl)
    return value

write(key, value):
    database.commit_update(key, value)
    cache.delete(key)

This minimal flow is not strongly consistent. A concurrent reader can fetch an old value before the update and repopulate it after invalidation. Failed invalidation also leaves stale data. A TTL limits an entry's residence, not every possible source of staleness. Version checks, stronger coordination, or bypassing the cache may be needed for correctness-critical reads.

15. How do you choose a cache key and an expiration policy?

A cache key must include every dimension that changes the reusable result. For an assumed localized product response, that might be API version, tenant, product ID, language, and price-list version. User-specific fields need user scope or a separate uncached response. Normalize equivalent inputs to avoid unnecessary duplicate entries.

Choose expiration from acceptable staleness and refresh cost, not an arbitrary global duration. An assumed product-description policy might use a five-minute TTL plus invalidation on edits, while inventory confirmation reads the authoritative store.

A TTL alone does not guarantee a five-minute freshness bound: the value might already be old when cached, or a stale replica could refill it. Versioned keys, reliable invalidation, and bounded source lag strengthen the policy. Small randomized expiration offsets can reduce synchronized refresh bursts.

16. How do you choose between relational, document, key-value, and object storage?

Choose storage from access patterns and correctness requirements, not from a blanket SQL-versus-NoSQL rule.

Model

Useful starting fit

Important question

Relational

Linked entities and transactional invariants

Which constraints and joins are needed?

Document

Aggregate-shaped records

How are cross-document updates handled?

Key-value

Direct lookup by known key

Are scans or secondary queries required?

Object

Large immutable or replaceable blobs

Where do searchable metadata and permissions live?

For a file-sharing service, object storage can hold bytes while a relational database owns metadata and sharing rules. Upload completion and metadata publication need a defined recovery flow. Product capabilities overlap: some document stores support transactions, and relational databases can store JSON. Evaluate the actual consistency, indexing, size limits, and operational behavior of the selected implementation.

17. What does a database index improve, and what does it cost?

An index gives the database an additional access path, often avoiding a full-table scan for selective lookups and supporting useful ordering. It consumes storage and adds maintenance work to writes; additional indexes are not free read acceleration.

Assume an orders table commonly queried by customer and recent creation time:

SQL example

CREATE INDEX orders_customer_created_idx
ON orders (customer_id, created_at DESC, id DESC)

CREATE INDEX orders_customer_created_idx
ON orders (customer_id, created_at DESC, id DESC)

CREATE INDEX orders_customer_created_idx
ON orders (customer_id, created_at DESC, id DESC)

This B-tree ordering is a candidate for filtering one customer and retrieving recent orders with a deterministic ID tie-breaker. It does not guarantee every query will use the index. Selectivity, table size, statistics, requested columns, and database behavior affect the plan. Inspect the query plan and representative workload before keeping it. Avoid claiming that a multicolumn index is universally useless without its first column; some engines can use other strategies.

18. What is the difference between normalization and denormalization?

Normalization organizes data around dependencies so that a fact has a clear authoritative representation. For example, an order references a customer instead of copying the customer's current contact information into every order. Keys and constraints help preserve relationships.

Denormalization stores selected duplicates or precomputed results to make reads cheaper. A customer-order summary might avoid repeated joins or aggregation, but now updates must keep that summary consistent or tolerate lag.

Separate intentional historical facts from accidental duplication: an order's shipping address at purchase time may be an immutable business record, not a cache of the customer's current address. For a derived read model, specify its source of truth, update mechanism, acceptable staleness, and rebuild procedure. Denormalization trades simpler or faster reads for additional write and repair complexity; it is not automatically required at large scale.

19. How do replication and sharding solve different problems?

Replication keeps copies of the same logical data on multiple nodes. It can support failover and, where the product permits, distribute reads. It does not automatically multiply write capacity because copies must receive the writes.

Sharding divides data into subsets so different nodes own different records. It can spread storage and write load, but introduces routing, rebalancing, and cross-shard query or transaction costs.

Architecture diagram

Shard A: users 1-500     -> primary A + replica A
Shard B: users 501-1000  -> primary B + replica B
Shard A: users 1-500     -> primary A + replica A
Shard B: users 501-1000  -> primary B + replica B
Shard A: users 1-500     -> primary A + replica A
Shard B: users 501-1000  -> primary B + replica B

The ranges are an illustrative partitioning choice, not a recommendation. Each shard can have replicas, combining both techniques. A skewed workload can still overload one shard, while replication lag can make replica reads stale. Plan failure handling independently for each shard and its copies.

20. What changes when replication is synchronous rather than asynchronous?

Synchronous replication delays a successful write acknowledgment until the required replicas have reached a specified stage. That stage might be receipt, durable log flush, or application of the change; the exact configuration matters. Acknowledgment by every replica is not universally required.

Asynchronous replication lets the primary acknowledge before replicas catch up. It generally reduces coordination delay, but a failover can lose acknowledged writes that never reached the promoted replica. Replica reads may also lag.

For synchronous replication, network distance and slow required replicas affect write latency and availability. Its protection applies only to the stated failure model and correctly selected failover replicas. Durable receipt does not necessarily make the change immediately visible on every reader. State the acknowledgment rule, eligible failover targets, and behavior when required replicas are unavailable.

21. What is a partition key, and why can a poor choice create a hot shard?

A partition key determines how records are grouped or routed. A good choice balances load while keeping common operations local. Tenant ID can make tenant-scoped queries efficient, but one very large tenant can become a hot shard even when most tenants are small.

Hashing can spread many distinct keys, yet it cannot divide repeated traffic to one hot key by itself. Time-range partitioning helps retention and range scans, but sending all new writes to the newest range can concentrate load. Bucketing or splitting a large tenant can help, at the cost of querying several buckets.

Distinguish logical partitions from physical placement. Multiple logical partitions can share one node, and table partitioning inside one database does not automatically distribute data across servers. Evaluate both data volume and request-rate skew.

22. What do consistency guarantees tell a client about reads and writes?

Consistency guarantees describe which results clients are allowed to observe when operations overlap or move between replicas.

Guarantee

Client-visible meaning

Linearizability

Operations appear to occur atomically in an order respecting real time

Read-your-writes

A session does not read a version older than its own completed write

Eventual consistency

Replicas converge after updates stop, given successful propagation and reconciliation

For a single linearizable register, a read starting after a completed write sees that value or a later write. Eventual consistency alone gives no fixed convergence deadline and does not promise session guarantees. Read-your-writes does not impose one global order across all clients.

Specify the affected object, transaction, and session boundaries. The word “strong” is insufficient by itself, and database replication does not tell you which consistency contract the API exposes.

23. What does the CAP theorem actually say during a network partition?

CAP concerns a distributed read/write system when network partitions prevent some nodes from communicating. It says such a system cannot guarantee both linearizability and availability for every permitted execution under that failure model.

Here, consistency means operations behave like one atomic object in a real-time-respecting order. Availability means every request to a nonfailed node eventually receives a valid operation response, not merely an error or a high uptime percentage.

If two isolated sides accept conflicting writes and serve local reads, their responses can violate linearizability. Preserving that guarantee may require one side to reject or delay operations, sacrificing availability for those requests. The theorem is not a general “choose any two features” shopping rule. Outside partitions, latency, coordination cost, and weaker consistency guarantees still require separate design decisions.

24. What do ACID properties guarantee, and what do they not guarantee?

ACID describes transaction properties within a defined system boundary:

  • Atomicity: the transaction's changes commit together or do not commit.

  • Consistency: declared invariants remain satisfied when correct transactions move between valid states.

  • Isolation: concurrent transactions have the visibility and interference behavior promised by the selected isolation level.

  • Durability: acknowledged commits survive the failures covered by the configured persistence guarantees.

ACID consistency is not the same concept as CAP's linearizability. “ACID database” also does not mean every transaction runs at serializable isolation; weaker levels permit additional anomalies.

A database transaction cannot automatically make an external email or payment atomic with its rows. Write-ahead logging supports recovery, but durability still depends on storage and configuration. ACID alone promises neither service availability nor immunity from application bugs, catastrophic loss, or faulty business rules.

25. What is the difference between a message queue and a retained event stream?

A work queue primarily distributes jobs among competing consumers. A retained event stream primarily preserves an ordered history within defined partitions so consumers can track positions and replay data while it remains retained.

Concern

Work-queue emphasis

Retained-stream emphasis

Consumption

One worker handles an available job

Consumer groups track independent progress

Replay

Product and retention dependent

Read again from retained positions

Ordering

Depends on queue and concurrency

Commonly partition-local

These are usage patterns, not rigid product categories; a stream can distribute work within a consumer group. Neither label alone guarantees exactly-once business effects. A consumer can finish an external side effect and crash before recording progress, leading to repetition. Define acknowledgment, retry, deduplication, and retention behavior for the actual system.

26. When should an API do work synchronously versus asynchronously?

Use synchronous processing when the caller needs the result immediately and the work reliably fits the response-time budget. Use asynchronous processing for long, bursty, or retryable background work when the caller can accept a pending result.

For an assumed report export, validate the request, durably record a job, and return HTTP 202 with an operation identifier and a status location. The client later observes pending, running, succeeded, or failed. A 202 response means accepted for processing, not completed or guaranteed to succeed.

Avoid acknowledging a job that exists only in process memory. If job creation and message publication touch different systems, a transactional outbox can close the database-to-broker gap, while consumers still handle duplicates. Define job retention, authorization for status/results, cancellation semantics, and what happens after worker failure.

27. What is idempotency, and why is it important for retryable APIs?

An operation is idempotent when repeating it has the same intended effect as performing it once. Responses need not be identical: deleting an already deleted resource may return a different status while leaving the same resource state.

For a retryable create-order request, accept a client-generated idempotency key scoped to the caller and operation. Persist its request fingerprint and outcome, reject reuse with different parameters, and coordinate concurrent requests using the same key. Record the business change and deduplication state atomically when they share a database.

A timeout does not prove the original request failed. A retry with the same key should recover the prior result or continue its tracked operation. Define retention: once the deduplication record expires, the same key may no longer prevent repetition. External side effects require their own coordinated idempotency support.

28. How do timeouts, retries, exponential backoff, and jitter work together?

A timeout bounds how long a caller waits; it does not prove that the remote operation stopped. A retry attempts the operation again, so it should be limited to appropriate transient failures and operations that are safe or protected by idempotency.

Exponential backoff increases delay between attempts. Jitter randomizes the delay so many callers do not retry in lockstep. One illustrative full-jitter policy is:

Pseudocode example

For retry index k, starting at 0:
    cap = min(2 seconds, 0.1 seconds * 2^k)
    delay = uniform_random(0, cap)
    retry only if attempts and overall deadline allow
For retry index k, starting at 0:
    cap = min(2 seconds, 0.1 seconds * 2^k)
    delay = uniform_random(0, cap)
    retry only if attempts and overall deadline allow
For retry index k, starting at 0:
    cap = min(2 seconds, 0.1 seconds * 2^k)
    delay = uniform_random(0, cap)
    retry only if attempts and overall deadline allow

The numbers are assumptions, not defaults. Include connection establishment and downstream work in the deadline budget, honor relevant server guidance, and avoid retrying independently at every layer. Otherwise one failing operation can multiply into many downstream requests precisely when capacity is weakest.

29. What is backpressure, and why is an unbounded queue dangerous?

Backpressure signals that downstream capacity is constrained so upstream producers slow down, pause, or reject work instead of accumulating unlimited demand. It can use bounded queues, credits, concurrency limits, or explicit overload responses.

Assume arrivals stay at 1,200 jobs/s while workers complete 1,000 jobs/s. With no rejection, the backlog grows by 200 jobs/s, or 12,000 jobs per minute. At an assumed 10,000 bytes per queued job, that adds 120 MB of stored payload each minute before overhead.

An unbounded queue postpones failure while increasing waiting time and exhausting memory or storage. Monitor oldest-job age as well as queue length. Set admission limits from acceptable waiting time, prioritize useful work, and define what can be dropped or delayed. If average demand persistently exceeds service capacity, buffering alone cannot solve the mismatch.

30. What does a rate limiter protect, and how is it different from a circuit breaker?

A rate limiter controls admitted demand, such as requests per second per tenant, to protect capacity, fairness, or cost. A circuit breaker protects a caller from repeatedly using an unhealthy dependency: it opens after a failure condition, rejects calls temporarily, and permits controlled probes before recovering.

A healthy API may still rate-limit one abusive tenant. A low-volume request may still be blocked because its downstream dependency is failing. The mechanisms solve different problems and can coexist.

Specify whether a limit is local to one instance or shared across the fleet; five independent limits of 100 requests/s do not enforce a global 100 requests/s cap. Circuit-breaker thresholds also need a window, minimum sample policy, and recovery behavior. Neither mechanism replaces per-call deadlines, bounded concurrency, or dependency capacity planning.

31. What is the difference between authentication and authorization in a system design?

Authentication establishes who or what is making a request. Authorization determines whether that identity may perform the requested action on the particular resource.

A valid token for user A does not authorize reading user B's invoice. The resource-owning service should check tenant membership, ownership or role, operation, and relevant resource state. Apply deny-by-default rules and least privilege to users and service identities. Unpredictable IDs do not replace these checks.

Use TLS to protect credentials and data in transit, and validate token authenticity, audience, and expiry where tokens are used. Avoid embedding long-lived authorization decisions in caches without a revocation plan. Test negative cases such as cross-tenant identifiers, expired credentials, and direct calls that bypass an edge gateway. Keep sensitive tokens out of logs.

32. Why do systems use metrics, logs, and distributed traces together?

Metrics summarize behavior over time, logs capture discrete events, and distributed traces connect work across service boundaries. Together they answer different questions.

Signal

Example use

Metrics

Detect rising error rate, p99 latency, or queue age

Logs

Explain a specific validation or dependency failure

Traces

Locate time spent across a request's downstream calls

For an assumed checkout slowdown, metrics identify when the problem began, traces show that inventory calls dominate latency, and correlated logs expose the relevant timeout condition. No signal automatically establishes causation.

Propagate request or trace identifiers, but control high-cardinality metric labels and instrumentation overhead. Sampling means traces may not represent every request. Redact credentials and personal data, limit access, and define retention. Alert on user impact rather than collecting telemetry without an operational purpose.

33. How are high availability, durability, and disaster recovery different?

High availability concerns whether the service can perform useful work despite failures. Durability concerns whether acknowledged data survives the stated failure conditions. Disaster recovery concerns restoring service and data after a larger disruption, such as losing a region.

Recovery time objective, or RTO, is the targeted time to restore the required service. Recovery point objective, or RPO, is the targeted maximum data-loss interval. For an assumed regional-disaster plan, an RTO of one hour and RPO of five minutes mean restoring within an hour while aiming to lose no more than five minutes of changes. They are objectives, not proof of achieved recovery.

A durable database can still be unavailable, and a reachable service can serve stale or incomplete data. Define failure scope and test the complete recovery path, including application configuration and access to encryption keys.

34. Why are replicas not a substitute for backups?

Replicas help continue service after some node failures, but they usually reproduce valid writes, including an accidental deletion or application-generated corruption. A bad update can therefore reach every replica quickly. Replication also may not protect against shared credentials, regional loss, or a defect affecting the whole cluster.

Backups preserve independently recoverable historical states. A recovery design might combine periodic snapshots with archived logs for point-in-time restore, protected by separate access controls and an appropriate retention policy. The exact mechanism depends on the database.

Test restoration rather than merely checking that backup jobs succeeded. Verify data integrity, keys, application compatibility, and achievable recovery time. Keep copies isolated enough to survive the threats in scope. Replicas and backups are complementary: one supports continuity, while the other provides a path back to an earlier trustworthy state.

35. When is a modular monolith a better starting point than microservices?

A modular monolith is one deployable application with deliberate internal boundaries, such as separate catalog, ordering, and billing modules. It is often a better starting point when one team owns the product, traffic is manageable, and transactions frequently span those modules.

It avoids network calls between every business boundary, simplifies deployment, and makes local transactions easier. The cost is coupled deployment and less freedom to scale or isolate each module independently. Internal interfaces and data ownership still matter; one deployment does not justify unrestricted coupling.

Microservices become more attractive when independent ownership, release cadence, failure isolation, or sharply different scaling needs justify distributed coordination. They add service communication, observability, consistency, and operational costs. Start with the simplest architecture meeting the requirements and preserve boundaries that make later extraction feasible.

36. How should you design an API contract before choosing infrastructure?

Define externally observable behavior before selecting servers or databases. Specify resources and operations, request and response schemas, authorization, error categories, pagination, and which changes must be atomic. Then document retry behavior, idempotency, concurrency controls, and compatibility expectations.

For an assumed order API, POST /orders might accept an idempotency key and return an order identifier; GET /orders/{id} exposes only orders the caller may access. Decide whether creation means fully confirmed or merely accepted for processing, because that choice affects the response and workflow.

Set size limits and validation rules, and avoid exposing internal database shapes unnecessarily. Distinguish a missing resource from an unauthorized one according to the security policy. A clear contract helps infrastructure follow the required semantics instead of allowing a chosen database or queue to dictate accidental product behavior.

37. How do offset and cursor pagination differ?

Offset pagination requests a position, such as skipping 1,000 rows and returning 50. It supports numbered pages but can become expensive at large offsets, and concurrent inserts or deletions can shift positions between requests.

Cursor pagination continues after a known ordering value. For newest-first orders, use a stable pair such as creation time plus unique ID, with a matching index and fixed filters. The cursor can encode that pair without exposing it as editable client state.

A cursor is not automatically a snapshot. New records, updated sort keys, or deletions can still affect traversal. Use immutable ordering keys where practical, or a snapshot mechanism when a consistent export is required. Validate cursor scope against the tenant, filters, and sort order. Choose offset for simple bounded result sets and cursors for predictable deep traversal.

38. How do polling, server-sent events, and WebSockets support real-time updates?

Polling repeatedly requests the latest state. It is simple and works well for infrequent updates, but consumes requests even when nothing changes; a polling interval also introduces detection delay.

Server-sent events, or SSE, deliver a server-to-client text event stream over HTTP. Browser EventSource supports reconnection and event identifiers. WebSockets provide a persistent bidirectional message channel, fitting interactive chat or collaborative actions that flow frequently both ways.

Neither transport by itself provides durable delivery or replay. Define event IDs, retention, duplicate handling, and how clients recover missed updates. SSE's last-event identifier is useful only if the server can honor it. Persistent connections also need authentication renewal, heartbeats where appropriate, connection limits, and graceful draining. Choose from directionality, update frequency, payload needs, and recovery requirements rather than treating WebSockets as universally superior.

39. What belongs in a useful high-level architecture diagram?

A useful diagram shows the main request and data paths, who owns durable state, and where synchronous and asynchronous boundaries occur. Include trust boundaries and the failure assumptions that drive the design; avoid filling it with products that do not explain behavior.

For an assumed single-region photo service:

Architecture diagram

Untrusted client --HTTPS--> API replicas
                              |-- metadata transaction --> Metadata DB
                              |-- image bytes -----------> Object store
                              `-- durable job -----------> Queue
                                                            |
                                                         Workers
                                                            |
                                                       Thumbnails
Untrusted client --HTTPS--> API replicas
                              |-- metadata transaction --> Metadata DB
                              |-- image bytes -----------> Object store
                              `-- durable job -----------> Queue
                                                            |
                                                         Workers
                                                            |
                                                       Thumbnails
Untrusted client --HTTPS--> API replicas
                              |-- metadata transaction --> Metadata DB
                              |-- image bytes -----------> Object store
                              `-- durable job -----------> Queue
                                                            |
                                                         Workers
                                                            |
                                                       Thumbnails

The API owns metadata changes; workers retry thumbnail generation. This sketch omits detailed upload optimization and must still explain how metadata and job publication remain recoverable across failures. Annotate workload rates and redundancy where they matter. A diagram is a communication tool, not evidence that every drawn arrow is reliable or that the system meets its SLO.

40. How do you identify the first bottleneck and decide what to optimize?

Find the resource limiting the required user outcome before changing architecture. Measure latency distribution, completed throughput, errors, and saturation at representative load. Break down database time, dependency calls, queueing, CPU, memory, and I/O instead of assuming the slowest component from its name.

For an assumed read-heavy API, a trace may show database calls dominate elapsed time. Query plans and workload measurements can distinguish missing indexes, lock waits, connection-pool queueing, or a genuinely saturated database. Each suggests a different fix.

Run a bounded test with stated request mix, data size, cache state, concurrency, and failure assumptions. Compare results against the original SLO and cost target, including tail latency and errors. After an optimization, repeat measurement: the bottleneck may move elsewhere. Do not label an architecture diagram or an arithmetic estimate a production benchmark.

System Design Interview Questions and Answers for Intermediate

The intermediate answers trace ownership, state changes, delivery behavior, and recovery paths. They compare alternatives through their correctness and operational costs instead of treating a cache, queue, or replica as an automatic solution.

41. How would you design a URL shortener for a read-heavy workload?

Assume 100 million redirects/day and 1 million creations/day: about 1,157 reads/s and 12 writes/s on average, with a separately provisioned 10× peak. Store code, destination, owner, expiration, and status in a durable table with unique codes. Creation authenticates the owner, validates the destination scheme, and atomically reserves custom aliases.

Data-flow diagram

GET /code -> redirect service -> cache -> mapping database
POST /links -> validation -> unique insert -> creation response
GET /code -> redirect service -> cache -> mapping database
POST /links -> validation -> unique insert -> creation response
GET /code -> redirect service -> cache -> mapping database
POST /links -> validation -> unique insert -> creation response

Cache only active mappings, with TTL no longer than their remaining lifetime. Use a temporary redirect and explicit cache policy when owners can edit or disable destinations; permanent redirects complicate revocation. Replicas can serve reads only within the accepted staleness window. Abuse takedowns need invalidation and a documented propagation target, not a promise of instant global removal. Rate-limit creation and isolate any link-preview fetcher against server-side request forgery. Cache loss must trigger bounded database access rather than unrestricted fallback.

42. How would you generate short-link identifiers without assuming hashes never collide?

Eight Base62 characters provide 62^8 = 218,340,105,584,896 possible codes. For 10 million independently uniform random allocations, the birthday approximation gives about n(n−1)/(2N) = 0.229 expected colliding pairs; the chance of at least one collision is approximately 1−e^−0.229 = 20.5%. A large namespace does not eliminate collisions.

Generate with a suitable random source, attempt a database insert protected by a unique constraint, and retry a bounded number of times on collision. Never use a separate existence check as the uniqueness guarantee. Custom aliases use the same constraint but return a conflict rather than silently changing the requested alias.

A counter encoded in Base62 avoids random collisions if allocation is coordinated; range allocation reduces coordinator traffic but introduces gaps and guessable identifiers. Neither randomness nor unguessability replaces authorization. Define whether expired codes can ever be reused: reuse can redirect old bookmarks to an unrelated owner.

43. How would you design durable one-to-one chat messaging?

Assume each conversation needs ordered history, but unrelated conversations do not share a global order. Authenticate the connection and authorize conversation membership on send and history access. Route writes by conversation ID to a durable owner that assigns a monotonically increasing conversation sequence.

Sequence diagram

sender -> message service -> durable message + delivery outbox
sender <- accepted(message_id, sequence) after commit
outbox -> recipient gateway -> recipient device
sender -> message service -> durable message + delivery outbox
sender <- accepted(message_id, sequence) after commit
outbox -> recipient gateway -> recipient device
sender -> message service -> durable message + delivery outbox
sender <- accepted(message_id, sequence) after commit
outbox -> recipient gateway -> recipient device

A client-generated message ID, unique within sender scope, lets retries return the original acceptance instead of creating another message. WebSocket transport provides a live channel, not durable delivery. If a gateway fails, the recipient reconnects with its highest contiguous acknowledged sequence and fetches missing history. Deduplicate overlapping push and catch-up results.

Distinguish server acceptance, device delivery, and user read receipts; persist the latter only if the product requires them. A busy conversation remains a serialization bottleneck. Large attachments belong in object storage with authorized references, not the message log.

44. How would you keep chat presence and typing indicators separate from durable messages?

Keep presence in a short-lived session registry, separate from the message history database. For an assumed 100,000 connected sessions sending a heartbeat every 30 seconds, the steady heartbeat load is about 3,333 updates/s before reconnect bursts. Jitter heartbeats to avoid synchronized spikes.

Store presence per user and device; a user is online when at least one unexpired authenticated session remains. An illustrative 90-second expiry tolerates two missed heartbeats, but it also means an abruptly disconnected user can look online for roughly that period. A disconnect signal speeds cleanup without replacing expiry.

Typing events expire after a few seconds and can be coalesced or dropped when gateways are overloaded. Publish only to authorized conversation participants, with rate limits and privacy preferences enforced. After failover, rebuild ephemeral state from current sessions rather than replaying old typing events. A lost presence update must never cause a durable message to disappear or be rejected.

45. How would you design a social news feed using fan-out on write, fan-out on read, or a hybrid?

Treat posts and follow relationships as authoritative; each user's feed is a derived list of post IDs. With an assumed 100 posts/s and 300 eligible followers/post, fan-out on write creates about 30,000 feed-entry writes/s before retries. A celebrity with 10 million followers breaks that average.

Strategy

Main benefit

Main cost

Fan-out on write

Cheap feed reads

Write amplification and lag

Fan-out on read

Cheap publishing

Expensive merge on every read

Hybrid

Handles audience skew

Two retrieval paths

A hybrid asynchronously pushes ordinary posts to active followers, then merges celebrity posts at read time using a stable ranking key. Deduplicate by post ID. Partition fan-out jobs and checkpoint progress so workers can resume without rebuilding entire audiences. Check current visibility when serving results: stale feed entries must not bypass blocking or private-account rules. Deletions invalidate projections, while read-time filtering protects users during propagation.

46. How would you design a secure resumable file-upload service?

Separate the control plane from the byte path. The API authenticates the user, reserves quota, and creates an upload session with an unpredictable object key. It issues short-lived, narrowly scoped part-upload URLs; treat these bearer URLs as secrets rather than public links.

Data-flow diagram

client -> upload API -> session/part manifest
client -> object-store multipart upload
finalizer -> verify object -> quarantine scan -> READY metadata
client -> upload API -> session/part manifest
client -> object-store multipart upload
finalizer -> verify object -> quarantine scan -> READY metadata
client -> upload API -> session/part manifest
client -> object-store multipart upload
finalizer -> verify object -> quarantine scan -> READY metadata

The manifest records upload ID, expected sizes, part numbers, and appropriate checksums. Resume by reconciling uploaded parts, not by restarting the whole file. Finalization verifies ownership and the completed object's size/checksum; an S3 multipart ETag is not generally a full-object MD5 checksum.

Keep objects private and unavailable to consumers until content validation and malware scanning finish. Validate allowed file types independently of the supplied filename or content type. Make completion repeatable using session state. Abort abandoned multipart uploads and garbage-collect orphan objects after a grace period, while releasing their reserved quota.

47. How would you design search for a product catalog?

Keep product records in a transactional catalog database and build a derived search index from committed changes. The index owns search-oriented structures—tokenized text, filters, and sortable fields—not authoritative inventory or payment data.

Data-flow diagram

catalog transaction -> product + outbox event
index worker -> search index -> ranked product IDs
search API -> authorized results + authoritative checkout validation
catalog transaction -> product + outbox event
index worker -> search index -> ranked product IDs
search API -> authorized results + authoritative checkout validation
catalog transaction -> product + outbox event
index worker -> search index -> ranked product IDs
search API -> authorized results + authoritative checkout validation

Assume catalog edits may take five seconds to become searchable. Track that freshness target from commit time to query visibility; indexing acknowledgment alone does not mean a refresh has exposed the document. Include product version in updates so delayed retries cannot overwrite newer content, and propagate deletions.

Bound expensive wildcard queries, page sizes, and aggregation cardinality. Apply tenant and visibility restrictions within retrieval, and recheck sensitive access before returning protected fields. Search-time price and stock can be stale; checkout validates authoritative values. Rebuild an index from a consistent baseline plus subsequent changes rather than treating it as irreplaceable state.

48. How would you design a payment-creation API that survives client retries?

Create a durable payment operation keyed by merchant and client idempotency key, with a fingerprint of amount, currency, and order. A unique constraint arbitrates concurrent submissions; identical retries return the existing operation, while changed parameters are rejected. Never store raw card details in this record.

A worker calls the payment provider with a stable provider idempotency key derived from that operation. Persist pending, provider reference, and eventual outcome. A timeout means the outcome is unknown, not that payment failed: reconcile using the provider's status API and verified webhooks rather than sending a new charge under a new key.

For Stripe API v1, saved idempotent responses include failures after execution begins, and keys can be pruned after at least 24 hours. The application's retry/reconciliation policy must respect that window; an old unresolved operation needs investigation rather than blind replay. Commit local state transitions conditionally and deduplicate webhook events. This controls duplicate effects within explicit contracts, not universal exactly-once payment execution.

49. How would you design an order workflow across inventory, payment, and shipping?

A durable order coordinator records each step and participant outcome. Inventory owns reservations, payment owns provider reconciliation, and shipping owns fulfillment; participants alone mutate their databases.

Step

Successful state

Failure response

Reserve stock

Reservation held

Reject unavailable items

Authorize payment

Authorization recorded

Release reservation

Confirm fulfillment

Shipment committed

Cancel before shipment, otherwise handle return

Capture payment before shipment, reconciling uncertain outcomes. Each command carries an operation ID and is safe to repeat. Record local transitions with outgoing commands in an outbox so a coordinator crash does not lose the next action. Inventory expiry races with confirmation; resolve them atomically in the inventory owner before promising the order.

Compensation is a new business action, not a distributed rollback. A payment void or refund can fail and needs retry/reconciliation. Physical shipment cannot be undone by deleting a database row. Expose pending or manual-review states instead of reporting success while required steps remain uncertain.

50. How would you design a notification service for email, SMS, and push?

Persist a notification intent with recipient, template version, channel policy, and business event ID. Generate channel-specific tasks through an outbox, then route them to separate email, SMS, and push worker pools. This isolates a slow provider from unrelated channels.

Apply preferences, legal suppression requirements, and recipient authorization before scheduling and again before sending when policy can change. Define deduplication at (business_event, recipient, channel) rather than by message text. Render templates with escaped untrusted fields and avoid secrets in logs.

Workers obey provider quotas, cap concurrency, and retry transient failures with bounded exponential backoff and jitter. If a send times out after the provider may have accepted it, reconcile by provider reference or reuse a supported idempotency key; without that support, duplicate delivery remains possible. Record accepted, delivered, bounced, and permanently failed separately. Queue age and delivery latency matter more than raw queue depth, and urgent alerts need capacity isolated from bulk campaigns.

51. How would you design a distributed token-bucket rate limiter?

Assume a tenant may burst to 200 requests and sustain 50 requests/s. Keep token balance and last-refill time at a single logical bucket owner; partition different tenants across owners. Refill and admission must be one atomic operation, not separate reads and writes.

Pseudocode example

atomic at bucket owner:
    now = monotonic_seconds()
    elapsed_seconds = now - last_refill
    tokens = min(200, tokens + 50 * elapsed_seconds)
    last_refill = now
    if tokens < 1: reject
    else: tokens = tokens - 1; admit
atomic at bucket owner:
    now = monotonic_seconds()
    elapsed_seconds = now - last_refill
    tokens = min(200, tokens + 50 * elapsed_seconds)
    last_refill = now
    if tokens < 1: reject
    else: tokens = tokens - 1; admit
atomic at bucket owner:
    now = monotonic_seconds()
    elapsed_seconds = now - last_refill
    tokens = min(200, tokens + 50 * elapsed_seconds)
    last_refill = now
    if tokens < 1: reject
    else: tokens = tokens - 1; admit

Use one sampled time per operation. Monotonic timestamps are meaningful only within their clock domain: failover must not subtract timestamps from different hosts. Restore conservatively or use a defined server-clock strategy with skew handling. Redis scripts can serialize bucket operations, but atomic execution alone does not establish clock or failover correctness.

For security-sensitive endpoints, reject or apply a strict local fallback when the store is unavailable. Independent full-size local buckets would multiply the global allowance. A globally hot tenant may require deliberately approximate regional budgets instead.

52. How would you design a background job queue with retries and dead-letter handling?

Assume jobs take up to 30 seconds normally and may perform retried database writes. A durable lease-based queue gives a worker a visibility deadline; the worker renews it while healthy and acknowledges only after the intended effect is committed. A crash or expired lease permits another attempt, so duplicate execution must be safe even when attempts overlap.

Use a stable job ID and an atomic deduplication/business-update transaction for database effects. External calls require their own idempotency or reconciliation contract. Apply bounded backoff with jitter for transient failures, and move repeatedly failing jobs to a dead-letter queue with error classification and attempt history.

Do not endlessly retry malformed input. Alert on oldest-job age and dead-letter growth, and retain failed payloads under an explicit privacy policy. Replay only after fixing the cause, at a controlled rate, while preserving job IDs. Visibility timeouts are a lease-queue mechanism; Kafka consumers instead manage offsets and partition ownership, so their recovery design differs.

53. How would you design a delayed-task scheduler without scanning every future task?

Store tasks durably with task_id, due_at, state, and a partition key. An index on due time supports bounded range reads; alternatively, hierarchical time buckets keep distant tasks off the active heap. Only tasks in the near-term horizon enter each scheduler's in-memory priority queue.

Each partition has one fenced owner. At the due time, a transaction changes a task from scheduled to dispatched and records an outbox item; a relay publishes it to workers. Recovery reconstructs the heap from durable state. Duplicate dispatch remains possible, so workers use the stable task ID for idempotency.

Interpret due times against a documented UTC clock-skew bound; monotonic timers handle waiting, but cannot alone interpret calendar timestamps. Dispatch may be early by the clock-skew bound; a strict no-earlier-than contract waits through that uncertainty. Exact millisecond starts are not promised under overload. Monitor lateness and reserve dispatch capacity for bursts. Cancellation conditionally changes scheduled state; once dispatched, cancellation becomes a worker-level business decision.

54. How would you design cache invalidation when data can be updated concurrently?

The dangerous race is a reader fetching database version 7, a writer committing version 8 and invalidating the cache, then the reader filling the empty cache with version 7. Deleting after the database write does not by itself prevent this stale fill.

For an assumed five-second freshness target, publish versioned invalidations through an outbox and retain a per-key version watermark. A fill is accepted only if its version is at least the watermark and not older than the cached value; the check and write are atomic. A version stored only in a deleted cache value is insufficient because deletion loses the comparison point.

Watermark retention must exceed the maximum allowed fill/replay delay; bound those delays or fetch authoritative state when the watermark is absent. TTL limits exposure but does not prove a strict freshness bound during outages or replica lag. If zero stale reads are required, read the authoritative store or use a stronger coordinated read/write design instead of promising ordinary cache-aside consistency.

55. How would you prevent a cache stampede after a popular key expires?

Assume 5,000 requests/s target one popular key and rebuilding it takes 200 ms. An uncontrolled expiry can generate roughly 1,000 concurrent rebuild attempts. Coalesce misses so one fill runs per key while other callers wait briefly or receive an acceptable stale value.

Use soft expiry to trigger refresh and hard expiry to stop serving data that is too old. Jitter expiration times across keys, cap total concurrent fills, and set a rebuild deadline. A process-local single-flight mechanism only coalesces within that process; 100 application replicas could still initiate 100 fills.

Cross-process coordination can use a short lease, but lease expiry may overlap an old fill with a new one. Publish results conditionally by version so a late worker cannot replace newer data. If the origin is failing, back off rather than refreshing on every request. Stale serving is appropriate for a product description, not automatically for revoked access or a financial authorization decision.

56. How would you estimate the cache capacity and origin load for a read-heavy service?

Assume a hot set of 10 million entries, each containing a 900-byte value and 100 bytes of key/metadata, plus 200 bytes of allocator and cache-engine overhead. That is 12 GB decimal per copy, not 10 GB. Two complete copies require 24 GB; operating at 75% memory occupancy implies about 32 GB provisioned across both copies, before additional engine-specific overhead.

Capacity calculation

origin misses = 20,000 requests/s * (1 - 0.95)
              = 1,000 requests/s
at 80% hits:     4,000 requests/s
with cold cache: 20,000 requests/s
origin misses = 20,000 requests/s * (1 - 0.95)
              = 1,000 requests/s
at 80% hits:     4,000 requests/s
with cold cache: 20,000 requests/s
origin misses = 20,000 requests/s * (1 - 0.95)
              = 1,000 requests/s
at 80% hits:     4,000 requests/s
with cold cache: 20,000 requests/s

The 95% hit ratio is an assumption to validate against the real access distribution, TTLs, evictions, and invalidations. It does not follow merely from fitting an average object count in memory. Origin load also includes writes and refresh work. Plan admission control and staged warming for a cold cache; otherwise the cache can hide an origin that fails during recovery. Measure bytes and latency separately from request hit rate.

57. How would you choose a composite database index for a filtered and ordered query?

Assume a PostgreSQL orders screen lists one tenant's pending orders, newest first, in pages of 50. Equality filters belong before the ordered columns in this candidate B-tree index.

SQL example

CREATE INDEX orders_pending_feed
ON orders (tenant_id, status, created_at DESC, id DESC);

SELECT id, created_at, total
FROM orders
WHERE tenant_id = $1 AND status = 'pending'
ORDER BY created_at DESC, id DESC
LIMIT 50

CREATE INDEX orders_pending_feed
ON orders (tenant_id, status, created_at DESC, id DESC);

SELECT id, created_at, total
FROM orders
WHERE tenant_id = $1 AND status = 'pending'
ORDER BY created_at DESC, id DESC
LIMIT 50

CREATE INDEX orders_pending_feed
ON orders (tenant_id, status, created_at DESC, id DESC);

SELECT id, created_at, total
FROM orders
WHERE tenant_id = $1 AND status = 'pending'
ORDER BY created_at DESC, id DESC
LIMIT 50

The tenant and status prefix narrows the search, while the remaining keys match the requested ordering; the unique ID breaks timestamp ties. This is a proposed index, not a measured execution plan. Use representative data and EXPLAIN to check whether the optimizer chooses it and how many rows it visits.

If nearly all queries select only pending orders, a partial index may save space, but its predicate must match applicable queries. Every extra index increases write and storage cost. Queries across all statuses may need a different index because the status key interrupts the desired global time order.

58. How would you preserve read-your-writes when reads normally use replicas?

After a successful profile update, return a commit/version token and route that session's dependent reads to a source that has applied at least that version. A simple policy temporarily pins the session to the leader; a more scalable policy checks replica progress and waits briefly or falls back to the leader.

Do not assume a fixed delay, such as two seconds, proves a replica is caught up. Lag can grow during load or failure. The token must represent an actual committed write, and progress comparison must use the database's valid replication timeline or equivalent ordering; a plain wall-clock timestamp is insufficient.

Read-your-writes applies only where the token or session context is propagated, including cache lookups. Failover adds a durability question: asynchronous replication may lose an acknowledged write, which routing cannot repair. If that loss is unacceptable, choose an acknowledgment/replication policy that preserves committed writes under the stated failure model. This stronger contract may cost latency or availability.

59. How would you handle a very large tenant without making it a permanent hot shard?

A shard key containing only tenant ID keeps small tenants convenient but traps a large tenant on one shard. Suppose one tenant produces 40% of writes: adding shards does not help if all of its records still route to one owner.

Split that tenant into stable virtual buckets using (tenant_id, hash(entity_id) mod bucket_count). A routing directory maps buckets to physical shards, allowing selected buckets to move without changing every entity's identity. Keep an entity's transaction boundary within one bucket where possible.

Tenant-wide queries now scatter across buckets, so bound fan-out and use derived aggregates for expensive dashboards. A single hot entity remains hot even after tenant splitting; it may need separate serialization or a redesigned operation. Enforce per-tenant queue, connection, and concurrency budgets to stop noisy-neighbor effects. Dedicated capacity is another option for a few exceptional tenants, but it should not replace a migration and growth plan.

60. How would you add shards and move data while requests continue?

Use a versioned routing directory and one authoritative writer for each moving range. Keep the source authoritative while copying a consistent snapshot to the destination, then replay changes from the snapshot's log position. Apply inserts, updates, and deletes in a way that cannot let the snapshot overwrite newer replayed state.

Sequence diagram

source owns writes -> snapshot copy -> change-log catch-up
fence source writes -> drain final changes -> advance routing epoch
destination owns writes -> verify -> retire source copy
source owns writes -> snapshot copy -> change-log catch-up
fence source writes -> drain final changes -> advance routing epoch
destination owns writes -> verify -> retire source copy
source owns writes -> snapshot copy -> change-log catch-up
fence source writes -> drain final changes -> advance routing epoch
destination owns writes -> verify -> retire source copy

The cutover may include a short write pause. Stale routers must receive a redirect or epoch error; they cannot be allowed to keep writing the old copy. Compare counts, checksums, and selected application invariants before cleanup.

Before cutover, rollback means abandoning the destination. After the destination accepts new writes, rollback needs reverse replication or another controlled migration, not merely flipping a pointer. Throttle copying so migration does not exhaust production I/O, and retain the old copy for an explicit recovery window.

61. How would you make a database update and event publication reliable without a dual-write gap?

Write the business update and an outbox row in the same database transaction. For an order confirmation, the row contains event ID, order ID, aggregate version, type, and payload. Either both records commit or neither does.

Data-flow diagram

transaction: update order + insert outbox row
relay: read committed outbox -> publish -> mark dispatched
consumer: deduplicate event -> apply local update
transaction: update order + insert outbox row
relay: read committed outbox -> publish -> mark dispatched
consumer: deduplicate event -> apply local update
transaction: update order + insert outbox row
relay: read committed outbox -> publish -> mark dispatched
consumer: deduplicate event -> apply local update

A relay can crash after the broker accepts an event but before the dispatched marker commits. On restart it publishes again, so consumers still need duplicate handling. Marking dispatched before publishing would instead risk permanent loss.

Preserve required per-order sequencing with aggregate versions and appropriate relay/partition handling; parallel relays must not silently reverse dependent events. Monitor the oldest unpublished row, retry transient failures, and retain rows long enough to diagnose delivery problems. This closes the local database-to-publication gap, not an atomic transaction across every consumer. Sensitive fields should be minimized because event retention can outlive the originating record.

62. How would you build an idempotent consumer when messages can be delivered more than once?

For a consumer updating its database, put the deduplication marker and business change in one transaction. The inbox has a unique key on (consumer_name, event_id), so independently deployed consumers do not suppress one another's work.

Pseudocode example

begin transaction
insert inbox marker if absent
if marker was inserted:
    validate event and apply business update
commit transaction
acknowledge message
begin transaction
insert inbox marker if absent
if marker was inserted:
    validate event and apply business update
commit transaction
acknowledge message
begin transaction
insert inbox marker if absent
if marker was inserted:
    validate event and apply business update
commit transaction
acknowledge message

If validation or the update fails, roll back the marker too. If the commit succeeds but acknowledgment is lost, redelivery finds the marker and safely acknowledges without repeating the update. Concurrent attempts rely on the database uniqueness constraint, not an in-memory set.

Retain markers for at least the permitted redelivery and replay horizon; deleting them too early allows old events to apply again. This boundary covers database effects only. An email or payment call cannot be made atomic by adding an inbox row; record a separate outgoing intent and use downstream idempotency or reconciliation. New event IDs for the same business action may require an additional business uniqueness rule.

63. How would you preserve ordering where it matters without imposing a global order?

Choose the smallest key whose operations must be ordered—for example, account ID for account state or conversation ID for chat history. Route that key to one ordered log partition and process its events serially. Different keys can execute concurrently.

Broker order is the order records enter a partition, not necessarily business causality across concurrent producers. Assign an aggregate sequence in the authoritative transaction and detect duplicates or gaps at the consumer. A gap may require waiting, replay, or fetching current authoritative state rather than guessing.

If version 12 fails, applying dependent version 13 first can violate the business invariant. Block that key, or explicitly redesign the operation to tolerate reordering; do not silently skip the failed event into a dead-letter queue. Adding partitions can change key routing, so migration needs an ordering boundary or stable virtual partitions. The tradeoff is deliberate: a hot account remains serial even when unrelated accounts scale horizontally.

64. How would you choose and implement optimistic concurrency control?

Use optimistic concurrency when conflicting edits are uncommon and users can resolve conflicts. Return a record version with each read, then update only if that version still matches. Authorization and validation still apply to every attempted change.

SQL example

UPDATE documents
SET title = $1, version = version + 1
WHERE tenant_id = $2 AND id = $3 AND version = $4

UPDATE documents
SET title = $1, version = version + 1
WHERE tenant_id = $2 AND id = $3 AND version = $4

UPDATE documents
SET title = $1, version = version + 1
WHERE tenant_id = $2 AND id = $3 AND version = $4

Zero returned rows mean the expected version was not updated; the service distinguishes a conflict from a missing or unauthorized record without leaking another tenant's data. For an HTTP resource, a strong ETag with If-Match can express the precondition, with a failed precondition reported as 412.

A retry must reread and reevaluate the intended change, not blindly replace the new version with old content. This prevents lost updates on the checked row but does not protect arbitrary multi-row invariants. Those may require conditional updates on additional rows, locks, or serializable transactions with whole-transaction retry.

65. How would you prevent overselling the final item in inventory?

Assume one database owns a SKU's available quantity. Reserve stock with an atomic conditional decrement, not a read followed by an unconditional write.

SQL example

UPDATE inventory
SET available = available - 1
WHERE sku = $1 AND available >= 1

UPDATE inventory
SET available = available - 1
WHERE sku = $1 AND available >= 1

UPDATE inventory
SET available = available - 1
WHERE sku = $1 AND available >= 1

Exactly one concurrent request can consume the last unit when this update executes against the same authoritative row under PostgreSQL's row-update concurrency behavior. Zero returned rows means no unit was reserved, assuming the SKU exists. Add a nonnegative constraint as a backstop.

The decrement and reservation record share a transaction. First claim a unique request ID in that transaction; duplicate requests return the original reservation rather than decrementing again. Roll back if stock is unavailable. Expiry and checkout confirmation compete through conditional reservation-state transitions so only one can release or consume the reservation.

Read replicas and caches can display approximate availability but cannot authorize the sale. Multiple independent regional writers require inventory allocation or coordinated transactions; this single-row solution does not magically extend across them.

66. How would you keep a search index synchronized and recover from indexing failures?

Capture catalog changes through an outbox or database change stream, including stable document ID, source version, and deletion records. Checkpoint a batch only after each event is indexed successfully or durably quarantined under an explicit repair policy. A successful request can still contain individual failed items, so inspect per-item outcomes.

Use conditional version handling at the index boundary so an old retry cannot overwrite a newer document. Deletions need tombstone/version protection for the allowed replay horizon; otherwise a delayed update can resurrect a deleted product. Measure freshness through query visibility, since index refresh adds delay after acceptance.

For a rebuild, create a new index from a consistent snapshot and replay changes from its corresponding log position. Validate counts, representative queries, and lag before switching the read alias. Keep the old index during the rollback window. If source-log retention expires before recovery catches up, restart from a new baseline rather than claiming the index is complete.

67. How would you choose between CQRS and a single read-write model?

Start with one read-write model when the domain is straightforward and indexed queries meet the workload. CQRS separates command and query models; it does not inherently require separate databases, an event broker, or event sourcing.

Situation

Likely direction

Simple CRUD, modest traffic

Shared model and indexes

Complex command invariants, many read shapes

Separate command/query models

Expensive dashboards with independent scale

Derived read projections

For an assumed order system, commands enforce stock and payment rules in the authoritative database, while a projection combines order and shipment details for a support dashboard. That projection can lag, fail, or require rebuilding. Show its freshness and route immediate post-update confirmation to authoritative state when needed.

The cost is additional schemas, synchronization, monitoring, and failure recovery. A projection outage must not be misreported as a lost order. Adopt CQRS for a concrete mismatch between read and write needs, not because separating every service sounds more scalable.

68. How would you design an append-only audit trail without leaking sensitive data?

Record who performed which action, on which resource, when, with what outcome and correlation ID. For an assumed administrative permission change, store the actor identity, tenant, target permission, old/new nonsecret values, and authorization decision—not the access token or full request body.

Append audit intent in the same transaction as the sensitive database change when atomic coverage is required. A separate collector writes to restricted retention-controlled storage. Grant application writers append capability without routine update/delete permission, and separately audit privileged access to the audit store.

Append-only application APIs are not inherently tamper-proof: administrators, storage compromise, or retention jobs can alter evidence. Integrity checks, immutable storage policies, and independently held signed checkpoints strengthen detection under stated threats. Hash chaining alone cannot detect truncation without a trusted external anchor.

Define retention, access, redaction, and deletion procedures before collecting personal data. Database write-ahead logs aid recovery but are not a substitute for business-level audit context or a privacy-conscious audit schema.

69. How would you design a multi-tenant service with strong data isolation?

Derive tenant context from the authenticated principal and validated membership, not an unchecked request header. Every resource operation enforces both tenant scope and object-level permissions. Include tenant ID in database keys, cache keys, search filters, queue payloads, and storage paths, but remember that namespacing alone is not authorization.

For shared tables, enforce tenant predicates centrally and consider database row-level security as defense in depth. Connection pools must reset session context; background workers must reestablish the correct tenant before processing a task. Cross-tenant administration uses a separate, narrowly authorized path with audit records.

Assume one tenant can become abusive or accidentally expensive. Per-tenant quotas, bounded query costs, and isolated worker/concurrency pools reduce noisy-neighbor effects. Separate databases or accounts provide stronger isolation boundaries at higher operating cost; choose according to the threat model and contractual requirements. Test adversarial cross-tenant identifiers, export endpoints, cache hits, and asynchronous jobs, not just the main API's happy path.

70. How would you size workers to meet a queue-processing deadline?

Assume 120,000 queued jobs must drain within 10 minutes while 200 new jobs/s continue arriving. Each worker processes 50 jobs/s at sustainable service rate under the assumed workload. The backlog needs another 200 jobs/s beyond incoming traffic.

Capacity calculation

required throughput = 200 + 120,000 / 600 = 400 jobs/s
minimum workers     = ceil(400 / 50) = 8
at 70% utilization  = ceil(400 / (50 * 0.70)) = 12
required throughput = 200 + 120,000 / 600 = 400 jobs/s
minimum workers     = ceil(400 / 50) = 8
at 70% utilization  = ceil(400 / (50 * 0.70)) = 12
required throughput = 200 + 120,000 / 600 = 400 jobs/s
minimum workers     = ceil(400 / 50) = 8
at 70% utilization  = ceil(400 / (50 * 0.70)) = 12

Eight workers meet the arithmetic only if service rate stays steady and there are no failures or retries. Twelve provide the stated utilization headroom; they are not automatically a measured production recommendation. If one worker must fail without breaking the 70% target, provision thirteen.

Verify downstream quotas, database capacity, job-size variation, and partition parallelism. Adding workers cannot accelerate a single serialized hot key. For a changing backlog, drain time is backlog / (processing rate − arrival rate) only when the denominator stays positive. Track oldest-job age and completion deadlines as well as queue length.

71. How would you distribute a request deadline across dependent service calls?

Assume an API has a 500 ms server-side deadline. Reserve 50 ms for local work and response serialization, budget 100 ms for a required authorization call, and allow up to 250 ms for two independent downstream reads running in parallel. The remaining 100 ms covers queueing and uncertainty, not an automatic extra retry.

Propagate the remaining deadline at each hop, subtracting elapsed time and a safety margin. Bound connection setup, pool waits, request execution, and retry delays. Retry only an eligible operation when enough budget remains, and avoid multiplying retries at every service layer. Cancel outstanding work when the caller's deadline expires where supported.

Serial durations add; parallel completion waits for the slowest required branch. Their percentile latencies cannot generally be added to derive an end-to-end percentile because distributions and correlations matter. Measure the whole request. Optional recommendations may return a fallback when their budget expires, while an uncertain payment outcome must remain pending rather than be treated as failure.

72. How would you handle slow consumers in a real-time streaming service?

Give every connection a bounded outbound queue and a maximum lag budget. If an assumed 100,000 clients may each buffer 64 KiB, buffers alone can consume about 6.55 GB decimal; allowing unbounded queues is not a viable backpressure strategy.

For replaceable state such as cursor positions or stock-price snapshots, coalesce updates by key and send the latest value. For durable business events, never silently discard required history: disconnect a lagging client with a resume cursor and let it catch up from retained storage. If its cursor is older than retention, require a fresh snapshot.

Limit per-client send work so one slow connection cannot stall a gateway loop, and apply per-tenant budgets against reconnect storms. WebSocket provides transport framing, not a complete application flow-control policy. Advertise heartbeat and idle behavior, authenticate reconnects, and recheck authorization during catch-up. Track queue bytes, oldest buffered event, disconnect reasons, and recovery lag instead of reporting only the number of open sockets.

73. How would you keep API pagination stable while new records arrive?

Use an immutable, unique ordering tuple such as (created_at, id) rather than an offset that shifts when new records arrive. For descending PostgreSQL pagination, the next page starts strictly after the last tuple already returned.

SQL example

SELECT id, created_at, title
FROM posts
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)
ORDER BY created_at DESC, id DESC
LIMIT 50

SELECT id, created_at, title
FROM posts
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)
ORDER BY created_at DESC, id DESC
LIMIT 50

SELECT id, created_at, title
FROM posts
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)
ORDER BY created_at DESC, id DESC
LIMIT 50

Support this with an index beginning with tenant ID followed by the ordering columns. Encode the last tuple, filters, and cursor version in an opaque integrity-protected cursor. Authorization must be checked again on every page; a cursor is not an access grant.

This live-feed contract avoids duplicates caused by new rows ahead of the cursor, but does not freeze deletions, edits, or backdated inserts. A repeatable export needs a database snapshot or search point-in-time mechanism with bounded lifetime. Null or mutable sort fields complicate the contract, so define their ordering and update semantics explicitly.

74. How would you upload metadata and an object without pretending they share one transaction?

Represent the upload as a state machine: PENDING → VERIFIED → READY, with failure and expiration states. The metadata database owns the state; object storage owns bytes. Do not expose an object merely because its key exists.

Create pending metadata with an upload ID and private immutable object key. The client uploads parts, then a durable finalization task verifies ownership, size, checksum, and required scanning results. A conditional database transaction marks the metadata ready and records any publication event in an outbox. Repeated finalization returns the existing result.

An object may exist after the database commit fails, or metadata may remain pending after a client disappears. A repair worker reconciles both cases by upload ID and object version. Garbage collection deletes only expired, unreferenced candidates after rechecking state and waiting a grace period; it must not race an active finalizer. Once ready, prevent replacement of the verified bytes or bind reads to the verified object version. This provides recoverable coordination, not a shared cross-system transaction.

75. How would you verify and process third-party webhooks reliably?

Validate the provider signature against the exact raw request bytes, using the correct endpoint secret and the provider's recommended timestamp tolerance. Parsing and reserializing JSON before verification can change the signed representation. Rotate secrets safely, bound body size, and avoid logging the secret or full sensitive payload.

After verification, atomically store the event ID and durable work item, then return success promptly. If persistence fails, return a retriable failure rather than acknowledging work that can disappear. Duplicate deliveries return success once durable acceptance is established.

Process asynchronously with per-business-object state checks. Delivery order is not guaranteed: an older payment event must not overwrite a newer authoritative state. Fetch current provider state when the event is insufficient or contradictory, and record reconciliation outcomes. Signature timestamp checks reduce captured-request replay but do not replace event deduplication; providers can legitimately sign later delivery attempts again. Retain deduplication records for the supported replay horizon, and keep manual redelivery subject to the same validation and authorization boundaries.

76. How would you design per-user unread counts or view counters under high write volume?

Exact unread counts and approximate view counts have different semantics. For a conversation whose messages have contiguous visible sequence numbers and no visibility gaps, unread count can be latest_sequence − last_read_sequence. Advance the read marker monotonically; multi-device retries must not move it backward. Deleted, hidden, or selectively visible messages require a visibility-aware query or maintained projection instead of that subtraction.

For an assumed popular post receiving 100,000 views/s, a single counter row can become a write hotspot. Hash stable view-event IDs across counter shards, apply deduplication within the counting contract, and asynchronously sum shards for display. This trades immediate exactness for throughput.

Define what constitutes a view and whether repeated views count before choosing deduplication. Persist source events only if privacy and retention policy permit; otherwise recovery may be limited to checkpoints. Derived counters should expose freshness and support repair against retained authoritative data. Atomic increments prevent lost updates on one counter, but do not by themselves prevent duplicate increments after retries.

77. How would you evolve an event or API schema without breaking older clients?

Treat compatibility as both syntax and meaning. Adding an optional field is safe only when old readers ignore unknown fields and new readers tolerate its absence. Renaming a field, changing units, or reinterpreting an enum value can break clients even when JSON still parses.

Use an expand–migrate–contract sequence: first deploy readers that accept old and new forms, then deploy writers of the new form, then remove the old form only after the supported client and event-retention windows have passed. Store an explicit event schema version and retain historical decoders or upcasters for replay.

Test producers against old consumers and new consumers against retained events. Unknown enum values need a defined fallback rather than a crash. Canary the change with error and business-outcome monitoring; a rollback must still read data already written by the new version. HTTP API versioning does not fix incompatible business semantics automatically. For a currency field changing from dollars to cents, introduce a new explicit field instead of silently changing the existing value's unit.

78. How would you choose health checks that avoid removing every server during an overload?

Separate liveness from readiness. Liveness asks whether the process is irrecoverably stuck; readiness asks whether it can accept useful new work. A slow shared database is not proof that every application process should restart.

Keep checks cheap and bounded. Avoid executing an expensive downstream query on every probe, since probe traffic can worsen the incident. Use consecutive-failure thresholds and recovery hysteresis to avoid flapping, and distinguish local failure from a shared dependency outage. Drain connections before deliberate removal or shutdown.

Under overload, limit admission and return controlled failures rather than immediately marking all capacity unready. Removing too many servers can push remaining servers past their limits and create a cascade. However, a truly broken instance should stop receiving traffic; do not keep it ready solely to preserve a healthy count.

Monitor healthy-capacity fraction, rejected requests, probe failures, and restart rate together. Test partial dependency failures and cold-start bursts, because a health policy that works on one idle server may destabilize the whole fleet.

79. How would you design an observability dashboard for a newly proposed service?

For an assumed checkout API, start with user-visible success rate and latency, excluding neither timeouts nor failed dependency calls merely because the application lacks a response. Define the population and measurement point before choosing targets. Separate payment-pending outcomes from completed purchases so a fast acknowledgment is not mistaken for successful checkout.

View

Signals

User outcome

Availability, latency distribution, completion rate

Capacity

Request rate, queue age, CPU, connection saturation

Dependencies

Error/timeout rate, latency, circuit state

Asynchronous work

Outbox age, consumer lag, repair backlog

Use traces to connect slow requests to dependencies and structured logs for individual failures. Keep metrics labels bounded; user IDs, order IDs, and raw URLs belong in access-controlled diagnostic data, not unbounded label sets. Redact payment and authentication secrets.

Alert on sustained user-impacting symptoms or error-budget burn with actionable context. Dashboard resource metrics support diagnosis; they do not substitute for end-to-end synthetic and real-user evidence that the service fulfills its contract.

80. How would you reduce a design's cost without silently weakening its guarantees?

Measure cost per useful outcome, such as dollars per million successful requests, alongside latency, durability, freshness, and recovery objectives. Assume storage dominates a document service: examine duplicate bytes, compression, and access patterns before removing replicas.

Change

Potential saving

Guarantee to verify

Tier cold objects

Storage cost

Retrieval latency and restore cost

Shorten retention

Stored bytes

Replay, audit, and recovery windows

Batch background work

Request overhead

Processing deadline and burst capacity

Reduce replicas

Storage/compute

Surviving specified failure domains

For example, moving 80% of bytes to a tier costing one-quarter as much yields 0.20 + 0.80 × 0.25 = 0.40 of the original storage charge, before retrieval, request, and transfer costs. This comparison is assumed, not a vendor quote.

Validate changes with representative workloads and recovery exercises. Preserve headroom for peak traffic and failures; average utilization alone cannot justify downsizing. Weaker guarantees require explicit product decisions, not invisible optimizations.

System Design Interview Questions and Answers for Experienced

The experienced answers connect architectural decisions to invariants, failure boundaries, resource limits, and evidence. They explain what must be measured or tested before accepting a proposal and where the stated guarantees stop.

81. How would you design active-passive regional failover with explicit RPO and RTO?

Assume an order service targets a recovery point objective (RPO) of 30 seconds and recovery time objective (RTO) of 10 minutes for loss of its primary region. Keep an asynchronously replicated database and deployable application capacity in a second region. Replication lag is evidence about exposure, not a promise that the RPO will always hold.

Failover requires proving or enforcing that the old primary cannot accept writes, selecting a recoverable replica, promoting it, checking dependencies, and moving traffic. A health-check failure alone does not justify two writable primaries. Measure recovery from the agreed incident start through successful user operations, including routing caches and cold caches.

If observed replication exposure exceeds 30 seconds, explicitly choose between continued unavailability and authorized data loss. Reconcile uncertain operations afterward. Failback is a controlled ownership transfer after resynchronization, not merely switching DNS back.

82. How would you design active-active writes without ignoring conflicting updates?

Active-active traffic does not require unrestricted concurrent writes to every record. For an assumed global account service, assign each account a home region and route balance-changing commands there. A consensus-backed ownership change can move an account after failure; an isolated old owner must be fenced. This sacrifices local write latency and partition-time write availability for that account.

For mergeable preferences, regions may accept concurrent updates, preserve versions, and apply a documented merge rule. A last-writer-wins timestamp rule can discard a valid update and is unsafe for inventory or balances. Independent regional debits cannot preserve a global nonnegative balance merely by merging afterward.

Choose conflict handling per invariant: ownership or coordinated transactions for money, domain-specific merging for suitable preferences. Monitor unresolved conflicts, replication lag, and the time required to transfer ownership safely.

83. How would you place a consensus quorum across three regions?

With three voting replicas, place one in each of three regions; a majority is two. Losing any one region leaves two voters, provided they can communicate and clients can reach them. A leader must obtain a remote acknowledgment, so write latency includes an interregional communication path plus storage and processing time.

For five voters, a 2–2–1 placement also survives any single-region loss: at least three voters remain. A 3–1–1 placement does not survive loss of the three-voter region. Five voters are not automatically faster or safer against correlated failures.

Spread voters across independent zones within regions where possible. Evaluate leader location, slow links, elections, and repair bandwidth. These counts establish quorum feasibility, not whole-service availability: authentication, routing, storage dependencies, and sufficient serving capacity must also survive.

84. What does Raft guarantee, and under which failure assumptions can it make progress?

Raft maintains a replicated log so committed commands are applied in a consistent order. Its safety argument assumes correct non-Byzantine participants and persistence of required state across crashes; it does not tolerate arbitrary malicious behavior or lost stable storage as though those were ordinary network delays.

Progress requires a reachable majority and sufficiently stable communication and scheduling for a leader to remain elected. A five-voter cluster can lose two voters, but three mutually reachable survivors are still necessary. Arbitrarily delayed messages can prevent progress without permitting conflicting committed histories.

A local read from a node that believes it is leader is not automatically linearizable. Use an appropriate quorum-confirmed read protocol or a lease implementation with its required timing assumptions. Client request deduplication is also necessary when retried commands must not execute twice.

85. Why is quorum intersection not by itself a proof of linearizability?

For a fixed replica set of size N, R + W > N means every read set of size R intersects every acknowledged write set of size W. It does not specify which version the reader returns or how concurrent writes are ordered.

For example, with N = 3 and R = W = 2, two writers can overlap at one replica while arriving in different orders elsewhere. A complete protocol must define version ordering, handling of incomplete writes, and any read repair or write-back needed for its guarantee. Clock timestamps alone do not establish real-time ordering.

Sloppy quorums can use substitute nodes outside the intended replica set, invalidating the simple intersection argument. Leader-based consensus addresses ordering through a protocol, not merely through the arithmetic of replica counts.

86. How would you prevent an expired distributed-lock holder from corrupting shared state?

A lease can expire while its holder is paused; the old process may later resume and write after a new owner has acquired the lock. Prevent this at the resource, not just inside the lock client.

Issue a monotonically increasing fencing token through a linearizable ownership service. Each mutation carries that token, and the destination atomically rejects tokens below its stored ownership epoch. Establish the new epoch at the destination before the new owner begins dependent work. Same-epoch retries still need operation identifiers or version checks.

Pseudocode example

atomically at destination:
    require request.epoch == current_owner_epoch
    require request.operation_id not already_applied
    apply mutation; record operation_id
atomically at destination:
    require request.epoch == current_owner_epoch
    require request.operation_id not already_applied
    apply mutation; record operation_id
atomically at destination:
    require request.epoch == current_owner_epoch
    require request.operation_id not already_applied
    apply mutation; record operation_id

The epoch update and mutations must share an enforceable ordering boundary. If an external destination cannot validate ownership epochs, possessing a distributed lock does not by itself make stale effects safe.

87. How do clock skew and clock uncertainty affect IDs, leases, and transaction ordering?

Use wall-clock time for human timestamps, a local monotonic clock for elapsed durations, and logical versions or ordered logs for causality-sensitive decisions. A monotonic clock is not a globally comparable timestamp and does not automatically solve lease safety across machines.

UUIDv7 provides time-ordered structure, not a global transaction order or proof that one event caused another. Clock rollback, concurrent generators, and identifier allocation policy must be considered. Require uniqueness independently of any desired sort order.

For leases, document clock-drift and pause assumptions, renew early, and fence stale owners at the protected resource. For transactions, bounded clock uncertainty can support protocols such as Spanner's commit wait; ordinary timestamp comparison lacks that guarantee. When uncertainty or synchronization health violates the protocol's assumptions, degrade availability rather than inventing precise ordering.

88. What does exactly-once processing mean at a specified system boundary?

Exactly-once must name an observable effect and its atomic boundary. In a Kafka-to-Kafka workflow, transactions can atomically commit output records and consumed offsets; downstream consumers must use the appropriate committed-record isolation. Execution may repeat, while committed results have the intended transactional semantics.

That transaction does not atomically include an unrelated payment API or database. For a database consumer, commit a unique event identifier and the business mutation in the same database transaction. Acknowledge consumption afterward; a crash causes replay, which the identifier suppresses.

Retain deduplication state for the supported replay horizon, including recovery and manual reprocessing. An outbox closes a database-update/publication gap but can publish duplicates. External effects require destination-supported idempotency or reconciliation, with that provider's key scope and retention limits explicitly considered.

89. How would you recover a payment operation whose outcome is unknown after a timeout?

A timeout means the caller lacks an outcome; it does not prove that the provider declined or ignored the operation. Persist a payment attempt before sending it, including a stable business operation ID, provider idempotency key, amount, currency, and state such as pending or unknown.

Retry only the same intended operation with the same key and compatible parameters while the provider's idempotency contract remains applicable. Query the provider using its operation identifier when available. Process verified webhooks idempotently, expecting duplicates and out-of-order delivery, and run reconciliation when callbacks are missing.

Do not create a fresh charge merely because the original call timed out. If the key-retention horizon has passed and the result remains uncertain, hold the operation for investigation rather than risking a duplicate. Record refunds as separate authorized actions, not silent erasures of the original charge.

90. How would you design reconciliation to detect and repair missing or duplicated financial events?

Use reconciliation as an independent comparison of intended operations, internal ledger postings, and external settlement evidence. Join by stable operation and provider identifiers; compare currency and exact amounts, not totals alone. Equal totals can conceal one missing and one duplicated event.

For an assumed daily batch, track both the business date and ingestion checkpoint. Re-scan an overlapping window for late arrivals and classify mismatches as missing, duplicate, amount mismatch, or timing difference. A checkpoint marks processed evidence, not proof that every upstream event has arrived.

Repair through an idempotent workflow: recover missing evidence, publish a missing derived event, or append an approved corrective ledger entry. Never rewrite historical postings merely to make reports agree. Keep discrepancy age, ownership, evidence, and resolution auditable. An outbox reduces one failure gap; it does not replace independent reconciliation.

91. How would you handle a saga whose compensation also fails?

Persist compensation as a durable workflow state, not a best-effort callback. For an order with a reserved item and captured payment, a failed refund should leave an explicit refund-pending state with the original operation identifier, attempt history, and next eligible retry time.

Retry transient failures using bounded backoff and idempotent compensation commands. Permanent rejection, exhausted retry policy, or an unresolved financial outcome moves the case to a repair queue with an owner and alert. Recovery resumes from recorded progress rather than restarting every completed step.

Compensation is a new business action: a refund may incur fees, and a delivered email cannot be unsent. Define irreversible steps and acceptable intermediate visibility before implementation. Report the incomplete state accurately to users, and prevent later steps from treating the saga as successfully reversed.

92. How would you choose between two-phase commit and a saga?

Choose two-phase commit when participating resource managers support it and the business requires one atomic commit decision across their updates. Prepared participants retain durable transaction state and may hold locks while waiting for the decision; coordinator failure can block resolution. Isolation still depends on the participating transaction mechanisms, not on atomic commit alone.

Choose a saga when steps span independently managed services or external actions and the business accepts intermediate states with compensating actions. A saga cannot promise that observers never see partial progress.

Requirement

Better starting point

Atomic transfer within one database

One local transaction

Atomic decision across supported resource managers

Evaluate two-phase commit

Order, shipment, and payment workflow

Durable saga with explicit compensation

For either distributed option, define timeout recovery and operational ownership. A saga trades blocking coordination for application-level recovery complexity; it is not simply a faster transaction.

93. How would you analyze tail latency in a service that fans out to many dependencies?

For an assumed request that waits for all 100 independent, identically distributed dependencies, let each dependency exceed a latency threshold with probability 1%. The probability that at least one exceeds it is:

Capacity calculation

P(at least one slow) = 1 - 0.99^100
                    ≈ 0.634 = 63.4

P(at least one slow) = 1 - 0.99^100
                    ≈ 0.634 = 63.4

P(at least one slow) = 1 - 0.99^100
                    ≈ 0.634 = 63.4

This is not a formula for adding p99 latencies. Parallel completion follows the slowest required branch; sequential stages add durations, while their percentile distributions require joint analysis. Shared hosts, queues, and regions also make independence an imperfect assumption.

Trace the critical path and distinguish queueing, network, and execution time. Reduce mandatory fan-out, cache appropriate results, cap per-dependency deadlines, and allow explicitly partial responses where semantics permit. Measure end-to-end latency under realistic correlated load rather than inferring it solely from individual service dashboards.

94. When are hedged requests helpful, and when can they worsen an outage?

A hedged request sends a second attempt after an initial attempt becomes unusually slow and uses the first acceptable response. It is most suitable for side-effect-free reads to replicas with compatible freshness guarantees and genuinely independent bottlenecks.

Assume 10,000 reads/second and a hedge threshold exceeded by 2% of first attempts. If each qualifying request sends one hedge, the added offered load is approximately 200 requests/second before accounting for races or failed cancellation. This is an estimate, not free capacity.

Use a shared deadline, a small hedge budget, and cancellation where supported. Do not hedge when saturation or a shared database explains the slowness; extra work can deepen the outage. Disable hedging automatically under overload. For mutations, equivalent idempotency and destination coordination are prerequisites, and avoiding hedges is often simpler.

95. How would you keep retry storms from causing cascading failure across services?

Put retries at one deliberate layer of a request path and propagate a deadline. If three nested layers each make up to three attempts, one original request can cause 3 × 3 × 3 = 27 downstream attempts. Jitter spreads attempts in time but does not remove that amplification.

Retry only eligible transient failures and safe operations. Enforce an attempt limit and retry budget, honor overload signals, and stop when the remaining deadline cannot support useful work. Bound concurrency and queue length before exhausted workers accumulate.

Use circuit breakers to avoid repeatedly calling a persistently unhealthy dependency, with limited probes for recovery. Shed low-priority work and return an explicit degraded or unavailable response instead of a misleading success. Monitor original demand separately from attempted calls, plus queue age, rejected work, and recovery probes. Recovery must ramp traffic gradually rather than release every waiting request simultaneously.

96. How would you test whether a capacity plan survives losing an availability zone?

Assume three zones, 12 application instances per zone, and a validated safe serving limit of 800 requests/second per instance at the required latency. At an assumed peak of 18,000 requests/second:

Capacity calculation

Normal capacity: 3 × 12 × 800 = 28,800 requests/second
After one-zone loss: 2 × 12 × 800 = 19,200 requests/second
Survivor utilization: 18,000 / 19,200 = 93.75

Normal capacity: 3 × 12 × 800 = 28,800 requests/second
After one-zone loss: 2 × 12 × 800 = 19,200 requests/second
Survivor utilization: 18,000 / 19,200 = 93.75

Normal capacity: 3 × 12 × 800 = 28,800 requests/second
After one-zone loss: 2 × 12 × 800 = 19,200 requests/second
Survivor utilization: 18,000 / 19,200 = 93.75

The arithmetic shows little remaining headroom; it does not demonstrate a successful failover. Reconnection bursts, cold caches, uneven routing, and replication work may exceed that margin. To keep survivors at or below an assumed 70% utilization, provision at least 17 instances per zone: 18,000 / (2 × 800 × 0.70), rounded up.

Validate database capacity, quotas, network paths, and traffic redistribution in an isolated failure exercise. Count only capacity that is ready before the recovery deadline.

97. How would you design graceful degradation while preserving core correctness?

Separate mandatory correctness work from optional enhancements. For an assumed commerce service, payment authorization, inventory reservation, and ownership checks remain mandatory; personalized recommendations and high-resolution images can be disabled. A fallback must not turn an unavailable authorization decision into approval.

Give optional work separate concurrency pools and bounded queues so it cannot consume the resources needed for checkout. Serve cached catalog descriptions only within a documented staleness contract; validate current price and availability on the authoritative write path.

Trigger degradation from sustained resource pressure or user-facing SLO signals, with hysteresis to avoid rapid switching. Responses should identify partial or stale results where relevant. During recovery, reintroduce features gradually and drain deferred work with limits. Dropping an optional notification is different from silently dropping a committed payment event, which needs durable recovery.

98. How would you design a hot-key strategy when hashing cannot spread one key's traffic?

Hashing spreads distinct keys; it cannot distribute all operations on one indivisible key. For a popular product page, replicate reads through a CDN or local caches, coalesce concurrent cache fills, and bound refresh work. More read replicas help only if freshness and invalidation requirements permit their use.

For a high-volume approximate view counter, stripe increments across subkeys and aggregate them. With 20 balanced stripes, an assumed 100,000 increments/second becomes approximately 5,000 per stripe, but exact synchronous reads now need coordination across stripes. Real skew can defeat that estimate.

Do not stripe a scarce-inventory decrement into independent unconstrained counters. Preserve the invariant through one owner, coordinated transactions, or explicitly allocated inventory rights. Apply per-key admission limits when serialization is unavoidable. Track key-level queueing rather than relying on an average shard utilization metric.

99. How would you manage tombstones, deletions, and stale replicas so deleted data does not return?

Represent deletion as a versioned operation that participates in replication and conflict resolution. Physically removing a row immediately can let an offline replica later reintroduce its older value if the system no longer remembers the deletion.

Retain deletion evidence until every replica allowed to rejoin is known to have incorporated it, or require stale replicas to be rebuilt from a current snapshot. A fixed tombstone lifetime is safe only under explicit assumptions about maximum offline time, repair completion, and retained history. Long partitions invalidate casual time-based assumptions.

Propagate deletion to indexes, caches, and derived stores using idempotent version-aware processing. A restore must replay deletion records before becoming available; otherwise an old backup can resurrect data. Distinguish immediate logical inaccessibility from eventual physical removal, and verify each storage layer rather than claiming one database delete removes every copy.

100. How would you rebuild a large derived read model without stopping writes?

Build a new read-model generation while the existing generation serves traffic. Capture a consistent source snapshot tied to an exact change-log position, then load that snapshot and apply subsequent changes in their required per-entity order. The snapshot/log boundary must prevent both gaps and lost updates.

Data-flow diagram

Source snapshot at position P -> new model
Source changes after P ------> ordered, idempotent replay
Old model -------------------> current user traffic
Source snapshot at position P -> new model
Source changes after P ------> ordered, idempotent replay
Old model -------------------> current user traffic
Source snapshot at position P -> new model
Source changes after P ------> ordered, idempotent replay
Old model -------------------> current user traffic

Track replay lag, deleted entities, schema compatibility, and per-entity versions. Validate counts, representative queries, and stronger checksums where meaningful; equal row counts alone are insufficient. When the new generation reaches the required freshness and passes validation, atomically switch the routing alias.

Keep the old generation temporarily for rollback, but ensure it continues receiving changes if rollback requires current data. If source log retention expires before catch-up, restart from a new consistent snapshot.

101. How would you make a large backfill safe for live serving traffic?

Run the backfill as low-priority, resumable work with bounded batches, concurrency, and transaction duration. Use stable key ranges and durable checkpoints, not a large offset scan whose meaning changes during concurrent writes. Make repeated batches harmless through conditional updates or explicit operation versions.

Assume a tested storage budget of 50,000 writes/second and a serving reserve of 40,000; 10,000 writes/second is a ceiling for the backfill, not an entitlement. Reduce it when p99 latency, replica lag, or storage queue depth crosses a defined threshold. Include index maintenance and downstream event volume in the budget.

Start on a small range, validate results, and increase gradually. Pause without losing progress, then resume from the last completed checkpoint. Avoid replacing newer live values with older snapshot-derived values. Completion requires reconciliation of missed or concurrently changed records, not just exhausting the original scan.

102. How would you perform a zero-downtime schema migration across independently deployed services?

Use expand, migrate, and contract across separately deployed producers and consumers. First add a compatible representation, such as a nullable field or new table. Deploy readers that tolerate both formats, then deploy writers that populate the new representation without breaking old readers.

Backfill in bounded, restartable batches, protecting newer writes with versions or conditional updates. If two stores must be updated, a naive dual write creates a failure gap; prefer one transactional source plus reliable propagation. Validate coverage and compare old and new reads before switching authoritative reads.

Only remove the old field after every deployed consumer, rollback version, and retained event format that needs it is retired or adapted. Schema changes may still require locks or costly rewrites, so inspect database-specific behavior and rehearse. Zero downtime is a measured rollout objective, not a property automatically conferred by adding a nullable column.

103. How would you design disaster-recovery tests that establish recovery rather than merely checking backups?

A recovery test must restore usable service, not merely list backup objects. In an isolated environment, restore a selected backup and the required transaction logs, recover to the intended point, and run business-level integrity and representative application checks.

Measure RPO using the latest recoverable acknowledged operations, and measure RTO from the agreed incident start to verified service readiness. Include key access, credentials, schemas, routing, dependencies, capacity, and operator decision time. A database that starts but cannot decrypt customer records is not a recovered service.

Exercise distinct scenarios: lost region, accidental deletion, and corruption discovered after several backups. Validate that deletion policy survives restores and that external side effects cannot accidentally be repeated by test workers. Record gaps, repair the runbook or design, and repeat the failed scenario. Backup existence establishes neither recoverability nor a recovery-time commitment.

104. How would you avoid a shared control plane becoming a data-plane availability dependency?

Keep ordinary request serving independent of a synchronous control-plane call. Distribute versioned, validated configuration to data-plane processes, apply updates atomically, and retain a last-known-good version. Existing requests can then continue during a configuration-service outage within the configuration's allowed lifetime.

Treat configuration by risk. Routing weights may tolerate staleness; revoked credentials or tenant-disable decisions may require short validity and a fail-closed path. A universal indefinite cache is not safe. Reject malformed updates and preserve the previous valid version rather than pushing one bad configuration everywhere.

Plan bootstrap separately: new instances need a trusted initial configuration without making emergency scaling impossible. Stage updates by cell, retain rollback versions, and expose configuration age and version in telemetry. Also remove hidden dependencies such as per-request quota allocation or key lookup that would reconnect the data plane to the failed control plane.

105. How would you make authentication and authorization resilient during identity-service outages?

For APIs using signed access tokens, cache trusted verification keys and validate signature, issuer, audience, expiry, and relevant claims locally. This lets existing eligible requests avoid a synchronous identity-provider dependency. Unknown key identifiers require controlled refresh; they are not a reason to skip verification.

Authentication does not replace current authorization. A valid token can outlive a revoked permission. Choose token lifetime, authorization-cache lifetime, and revocation propagation according to the operation's risk. For an assumed payment-management API, deny sensitive actions when required authorization freshness cannot be established; a public catalog can remain available independently.

Keep login and token refresh behavior separate from existing-token validation. Bound refresh concurrency to avoid a recovery stampede, protect cached keys from tampering, and monitor their age. Explicitly document which actions stop during the outage rather than describing an unsafe fail-open policy as resilience.

106. How would you design data residency and deletion across replicas, caches, logs, and backups?

Attach residency policy to the tenant and enforce it at routing, storage placement, replication, processing, and export boundaries. A regional primary is insufficient if logs, analytics, support exports, or disaster-recovery copies move the same data elsewhere. Maintain a data inventory and lineage for those derived copies.

Deletion should create a durable, idempotent workflow covering the primary store, replicas, indexes, caches, objects, and downstream processors. Minimize personal data in logs so deletion is not forced to search uncontrolled text archives. Track each target's result and retries.

Backups may require retention-based expiry rather than immediate selective deletion. Restrict access and ensure deletion records are reapplied before any restored data serves users. Define policy-specific deadlines and exceptions with the responsible compliance owners; this architecture alone does not establish legal compliance. Verify completion with evidence from each system, not just acceptance of the initial deletion request.

107. How would you design end-to-end message ordering across regions?

Define the ordering scope first. Chat messages typically need one order per conversation, not a total order across every conversation worldwide. Route each conversation to an ordering owner that assigns a durable sequence, or use a consensus-backed log for that partition.

Acknowledged ordering across region failures requires an explicit replication and commit policy. An asynchronous remote copy can lag and cannot automatically preserve every acknowledged message after primary-region loss. A quorum spanning regions adds latency; a partitioned minority cannot continue committing the same linearizable history.

Consumers must preserve the required per-key order, detect gaps, and deduplicate retries. A broker orders appended records within each partition, but this does not establish order across partitions, producers' real-time or causal order, or the order of external effects. If both isolated regions must accept local writes, use a weaker contract with causal metadata and conflict resolution rather than falsely promising one immediately agreed global order.

108. How would you use CRDTs without assuming every business invariant is mergeable?

CRDTs make suitable replicated state converge when updates eventually reach all replicas under the required merge or delivery conditions; arbitrary business rules are not automatically coordination-free. A state-based grow-only counter can keep one component per replica, merge components by maximum, and sum them. Its merge is associative, commutative, and idempotent, so repeated and reordered state merges are harmless.

An observed-remove set can preserve concurrent additions according to its chosen semantics. That behavior must match the product requirement; “delete wins” is not implied by the word CRDT. Operation-based designs have their own delivery and causal-order requirements.

A convergent counter does not by itself prevent two regions from overselling the last item. Use coordination or carefully allocated rights for that invariant. Plan replica identity changes, metadata growth, and safe removal of causal history. Garbage-collecting tombstones without establishing that old operations cannot return can break the intended behavior.

109. How would you design an event-time analytics pipeline for late and out-of-order events?

Keep event time separate from processing time. For an assumed five-minute purchase-count window, use event timestamps for assignment and a watermark to estimate progress. Configure, for example, ten minutes of allowed lateness as a product choice; it does not prove that every event has arrived.

Emit provisional results, then update or retract them when late data changes the aggregate. Use a deterministic window key and result revision at the sink so replayed outputs do not double-count. Preserve event identifiers when duplicate source events are possible.

Handle idle partitions so one inactive source does not indefinitely hold back progress, while documenting the consequences when it resumes. Route arrivals beyond the supported lateness horizon to a correction or audit process. Separate ingestion durability, processing checkpoints, and business completeness: a committed offset proves progress through received records, not that upstream systems emitted all expected purchases.

110. How would you design an event-sourced service and evolve its historical events?

Treat the event log as the authoritative history of accepted domain changes, not as a dump of arbitrary database updates. For an assumed order aggregate, append events with an expected aggregate version so concurrent commands cannot silently overwrite each other. Persist command deduplication with the append when retries must represent one business action.

Build query projections asynchronously and expose their lag. Snapshots accelerate recovery but need an event position and schema version; they are disposable only if the retained history is sufficient to rebuild them.

Version event schemas and use tested upcasters or version-aware handlers to interpret old records. Avoid rewriting history merely to simplify current code. Replay must not resend payments or emails: separate projection rebuilding from external-effect execution. Plan retention, privacy, and deletion requirements before choosing an immutable history model, because permanently storing every personal detail creates avoidable operational difficulty.

111. How would you scale a chat system while keeping history, reconnect, and delivery semantics coherent?

Separate connection gateways from durable conversation storage. Gateways maintain WebSocket sessions; a partition owner assigns conversation sequence numbers and persists accepted messages before issuing the acknowledgment promised by the API. The acknowledgment's durability depends on the configured replication policy.

A sender supplies a stable message ID so reconnect retries return the original acceptance instead of creating another message. Each device records its highest contiguous delivered sequence, detects gaps before advancing that cursor, and requests subsequent history on reconnect. Push delivery is an optimization; the durable history repairs missed pushes.

Architecture diagram

Device -> Gateway -> Conversation owner -> Durable log
Device <- Gateway <- Delivery workers   <- Durable log
Reconnect -> history after device cursor
Device -> Gateway -> Conversation owner -> Durable log
Device <- Gateway <- Delivery workers   <- Durable log
Reconnect -> history after device cursor
Device -> Gateway -> Conversation owner -> Durable log
Device <- Gateway <- Delivery workers   <- Durable log
Reconnect -> history after device cursor

Partition by conversation to preserve local order, and split very large rooms' delivery work from their ordered append path. Distinguish accepted, delivered-to-device, and read receipts. A live socket or successful broker publish alone does not establish all three.

112. How would you evolve a feed design when celebrity fan-out and privacy changes dominate cost?

Use fan-out-on-write for ordinary authors and merge celebrity posts at read time, with the threshold driven by follower activity and write amplification. Under an assumed 10 million followers, one celebrity post could otherwise require 10 million timeline-reference writes, including many for inactive users.

Materialize only a bounded recent window and paginate with a stable cursor contract. Store post identifiers rather than duplicating all content so edits and deletions have fewer copies to reconcile. Consumers must tolerate duplicate fan-out events and resume from durable checkpoints.

Privacy changes are correctness events, not merely cache refreshes. Recheck current viewer authorization before returning protected posts; remove stale timeline references asynchronously as cleanup. A stale feed index must not grant access to revoked content. Provide fallback ranking or a chronological subset when enrichment fails, while preserving authorization and explicitly defining freshness expectations.

113. How would you design a search service that tolerates partial shard failures?

Give each search request a deadline and dispatch bounded parallel shard queries. If an optional shard misses the deadline, return results only when the API explicitly permits partial coverage, including a partial-results indicator and failed-shard information. Never present a partial count as an exact global total.

Merging top candidates from available shards cannot recover documents from a missing shard. Ranking may also depend on distributed statistics, so degraded coverage can change ordering. For compliance-sensitive or exhaustive queries, fail rather than silently omit results.

Use replica selection and a tightly budgeted retry or hedge where independent capacity exists. For pagination, a point-in-time view plus a stable tie-breaker can reduce inconsistency, but it does not guarantee that an unavailable shard suddenly contributes data. Bound snapshot retention and request users to restart when the pagination context expires. Monitor coverage alongside latency and error rate.

114. How would you design a global file service with replication, deduplication, and resumable uploads?

Separate immutable file chunks from authoritative metadata. Create an upload session with owner, permitted size, expected checksums, expiry, and residency policy. Upload chunks directly using narrowly scoped temporary credentials or signed URLs, then finalize only after verifying the required parts and integrity.

A multipart object-store API can support resumable transfer, but application metadata still needs an atomic transition from uploading to available. Record the exact object version or immutable content reference. Garbage-collect abandoned sessions and unreferenced chunks after a safe grace period.

Replicate bytes according to the durability and residency policy; distinguish local availability from verified remote replication. Tenant-scoped deduplication avoids exposing whether another tenant possesses particular content. Do not authorize downloads solely because a caller knows a content hash. For concurrent edits, create versions with a conditional metadata update rather than overwriting an ambiguous global filename.

115. How would you separate a payment ledger's correctness path from asynchronous analytics?

Keep posting validation and durable ledger writes inside one transactional correctness boundary. For an assumed same-currency transfer, record equal-and-opposite postings in integer minor units or a suitably constrained exact decimal type. Enforce the balance invariant per transaction and currency, plus uniqueness of the business operation identifier. Cross-currency activity needs explicit conversion and balancing accounts rather than adding unlike units.

Protect overdraft or spending-limit checks with appropriate locking, conditional updates, or serializable transactions with retries. Double-entry balance alone does not prevent unauthorized spending.

Commit an outbox record with the ledger transaction, then publish asynchronously to analytics and reporting. Derived consumers deduplicate and expose their lag; their dashboards never authorize new spending from stale aggregates. Correct mistakes through new auditable postings. Durable storage protects committed history under its configured failure model, while reconciliation detects missing or inconsistent business evidence outside that transaction boundary.

116. How would you choose a failure domain and blast-radius strategy for a large service?

Partition the service into cells with bounded tenant populations, dedicated serving capacity, and separate critical data paths. Route each tenant to a cell using a controlled mapping, and stage software and configuration changes across cells rather than updating all of them together.

Assume ten equally loaded cells. Losing one initially affects roughly 10% of traffic, but only if shared identity, routing, storage, and deployment dependencies do not fail with it. Uneven tenants or cross-cell calls can make the real blast radius larger.

Choose between failing the affected cell closed and moving traffic to prepared spare capacity. Automatic evacuation without headroom can convert a local failure into a global overload. Preserve tenant ownership and write fencing during movement. Test correlated failures, including a bad shared configuration, and measure both affected users and recovery time rather than counting boxes labeled as independent.

117. How would you evaluate a new storage engine without relying on a vendor benchmark?

Start with required semantics and representative access patterns, not a vendor throughput number. Specify record sizes, key skew, read/write mix, transactions, indexes, retention, consistency, and durability settings. Compare systems under equivalent guarantees and a dataset larger than their caches where relevant.

Measure sustained throughput at the required p99 latency, resource cost, compaction behavior, and performance during backup, repair, and resharding. Include hot keys and bursts rather than a uniform-key average alone.

In an isolated test environment, crash nodes, delay replication, interrupt upgrades, and restore backups. Verify acknowledged writes, read behavior, and measured recovery objectives against the stated failure model. Inspect operational tooling, schema evolution, exportability, and staffing burden. A fast steady-state result does not establish recoverability. Select the simplest engine that meets the actual constraints, and retain the workload definition so later versions can be evaluated consistently.

118. How would you test a distributed design's invariants under duplicated, delayed, and reordered messages?

Write safety properties before scenarios: a payment operation posts at most once, a committed conversation sequence never changes, and acknowledged inventory reservations never exceed allocated stock. State liveness separately, such as eligible pending work eventually completing after dependencies recover and communication stabilizes.

Use a deterministic simulator or isolated test cluster to duplicate, drop, delay, and reorder messages; crash processes before and after durable writes; and expire deduplication state at its boundary. Record a reproducible seed and complete operation history. Model uncertain responses rather than treating every timeout as failure.

Check observed histories against the claimed consistency model and inspect final business invariants. Exercise restart, replay, and ownership transfer, not only steady-state delivery. Passing these tests is evidence, not proof of all possible schedules. Combine testing with protocol reasoning and explicitly documented assumptions about storage durability, clocks, and failure recovery.

119. How would you recognize when a distributed architecture is more complex than the workload requires?

Begin with a modular application and a transactional database unless measured requirements justify more distribution. An assumed service handling 200 peak requests/second with one engineering team does not automatically need sharding, independent deployment of every module, or globally writable replicas. The relevant question is whether its workload and reliability objectives fit the simpler design.

List the concrete reason for each boundary: independent scaling, fault isolation, data ownership, deployment cadence, or residency. Remove components whose only justification is anticipated popularity. Distribution adds network failure modes, replay, consistency decisions, observability, and operational ownership.

Define evolution triggers such as a sustained database bottleneck after query optimization or a clearly separate team's release needs. Preserve module interfaces and data ownership now so extraction remains possible later. Simplicity is not refusal to scale; it is delaying expensive coordination mechanisms until their benefits exceed their reliability and maintenance costs.

120. How would you defend a final architecture when latency, consistency, availability, and cost conflict?

Defend decisions against explicit requirements and failure behavior, not product names. For an assumed global order service, distinguish correctness-sensitive writes from latency-sensitive browsing:

Requirement

Decision

Cost or risk

Prevent duplicate orders

Transactional operation-ID uniqueness

Shared correctness boundary

Fast global browsing

Regional caches

Bounded or disclosed staleness

Preserve one write history

Quorum-backed ownership

Remote latency; minority stops writing

Recover regional loss

Tested replication and restore plan

Capacity and operational cost

During a network partition, linearizable operations cannot also guarantee successful service to every nonfailed partition. State which requests wait or fail and which weaker reads remain available. Outside partitions, latency and cost still require measurement.

Compare at least one rejected alternative, identify the riskiest assumption, and specify a validating load or recovery test. A defensible design makes its compromises observable and reversible where practical; it does not promise every desirable property simultaneously.

System Design Documentation and Further Reading

This guide covers software and distributed-system design, not an IT systems-administration syllabus. Designs are vendor-neutral unless a product-specific example is explicitly identified. State the workload, units, latency targets, consistency requirements, failure model, and operating constraints before applying a pattern. Capacity figures are illustrative estimates; replication, indexing, protocol overhead, headroom, and skew must be included when relevant. No database, queue, cache, or deployment topology has identical guarantees across all implementations. Verify the selected product and version before relying on a specific feature. Diagrams and pseudocode are explanatory artifacts, not deployed or formally verified systems. Arithmetic or static checks do not establish throughput, durability, fault tolerance, security, or production cost. Validate those properties with representative workloads and failure tests.

Use these curated primary references to examine distributed-system guarantees, protocols, capacity reasoning, and reliability trade-offs. Product and research examples have their own assumptions; apply the underlying reasoning rather than copying an architecture without checking its constraints.

Team Sherlock.sh

Building fraud intelligence for live high-stake video interactions (financial discussions, hiring interviews, board meeting etc).

Catch candidate fraud before it costs you.

Sherlock AI helps you detect AI-assisted, proxy, and deepfake candidates, so every hiring decision is backed by evidence.

Ask AI about Sherlock.sh

Similar Posts