System Design Arena
System Design: Mobile OS Update Rollout
Design the services that can push an urgent OS patch to 500 million devices worldwide under tight deadlines.
Case Study
Mobile OS Update
Same conversation, diagrams, and wrap-up you expect—now framed with clearer scaffolding and iconography.
Prompt
You must deliver a zero-day security patch to 500M mobile devices. 80% must update within 3 days, 100% within 5 days. Coordinator service must respect carrier/CDN constraints, handle phased rollouts, and provide real-time telemetry.
Interview snapshot
- • Topic: Mobile OS Update
- • Expected depth: 45 - 60 minutes
- • Focus areas: APIs, scale estimation, resilient architecture
- • Wrap-up: risk, monitoring, disaster recovery
Key takeaways
- • Package OS delta (full image + differential) and publish manifests per device model.
- • Deadline: 80% adoption by day 3 -> priority on high-density markets (NA/EU) first.
- • POST /api/devices/register { device_id, model, carrier, region, build } -> issue signed token for update polling.
🎙️ Interview mode
Practice this like a real interview
Don't read the answer first. Work the framework against the bare prompt, then compare against what the interviewer expected at each step.
Scorecard
What the interviewer is evaluating
Staff-level signal
Does the candidate treat this as a risk-management problem — staged rollout, health signals, halt criteria — rather than a file-distribution problem?
🧠 Staff engineer judgment
You cannot roll back a bricked phone remotely. Every design decision here should be justified by "how do we make failure recoverable" — not by throughput.
Calibration
The common (bad) answer
Build server → CDN → 1B devices download
❌ Why this scores poorly: Distribution is the easy 20%. The hard 80% is not bricking a billion phones: cohorts, canaries, rollback, and delta patches never appear in this answer.
✓ What a strong answer adds
- Staged rollout: 0.1% canary → 1% → 10% → 100%, each gated on health metrics.
- Delta updates: ship the diff, not the OS — size math changes the whole CDN story.
- A/B partitions on device so a failed update falls back to the working slot.
- Define halt criteria numerically (crash rate, boot loops) and automate the stop.
- Jittered check-in schedule so a billion devices do not poll at midnight.
Build it up
Step-by-Step Walkthrough
Pushing one build to ~500M phones is really two systems in a trench coat: a bytes problem (petabytes through CDNs) and a control problem (who updates, when, and how fast you can stop it). Keep them separate and the design falls out.
Naive: devices poll your server, download from your server
Every device polls GET /api/updates/check; if a new build exists, it downloads the ~200 MB image from the same API fleet and installs it. For an internal beta of a thousand devices this is completely fine.
Now the fleet math: ~500M devices times ~200 MB is on the order of ~100 PB. Hitting 80% adoption in three days means moving ~27 PB/day — roughly ~2.5 Tbps sustained. No origin fleet serves that, and worse: if the build has a bricking bug, every device that polls gets it before you even know.
?Split control plane from data plane immediately
The manifest (a few KB: version, URL, start time) and the image (~200 MB) have wildly different scale profiles. Saying "the API serves manifests, CDNs serve bytes" in the first five minutes structures the whole interview.
Data plane: signed packages, deltas, multi-CDN
A build pipeline signs each release, and a manifest service publishes per-model manifests: full image plus differential patches, because a delta from the previous build is often a fraction of the full ~200 MB and can cut that ~100 PB dramatically. Packages land in object storage and replicate to multiple CDNs (Akamai, CloudFront, Cloudflare) — at ~2.5 Tbps you are past any single provider's comfortable contract in some regions.
Clients verify the signature before installing — the CDN is untrusted transport — resume interrupted downloads, and fall back to a backup CDN on checksum mismatch. Polling stays cheap: ~500M devices on a ~6-hour poll interval is only ~23K QPS of cacheable manifest reads.
!The security question is not optional
An OS update system is the highest-value supply-chain target imaginable. Interviewers expect: packages signed at build time, devices verify before install, mutual TLS on the update API, and no code path where an unsigned image can ever be applied.
Control plane: cohorts, staged rollout, instant pause
Never ship to everyone at once. A rollout controller segments the fleet into cohorts by region, carrier, and device model, and advances on a schedule — ~1% canary, then ~10%, then broad — with high-density markets sequenced to hit the 80%-by-day-3 deadline. The manifest a device receives is a function of its cohort and the rollout's current stage.
Cohort membership and rollout state live in a metadata store built for the fleet size (Cassandra or Spanner, sharded by region/model — ~1 KB per device is ~500 GB). The one non-negotiable property: pause is instant. Pausing flips rollout state, and every subsequent manifest check returns nothing new — no CDN purge, no client push needed.
Telemetry: the rollout watches itself
Every device reports install progress, failure codes, even battery and temperature. At peak that is on the order of ~500K events/sec — an ingestion API feeds Kafka, stream processing aggregates per-cohort, and a time-series store (ClickHouse-style) drives live adoption dashboards and the compliance reports proving X% of devices patched per jurisdiction.
The payoff is closing the loop: anomaly detection on per-cohort failure rates can auto-pause the rollout before a human ever sees the graph. A bad build caught at the 1% canary stage is an incident; caught at 100% it is a headline. This feedback loop is the difference between a download service and a rollout system.
Ops console and surviving your own outage
Operators need levers, not dashboards alone: per-cohort throttling, pause/resume, manual overrides — all backed by feature flags in a config service (Consul/etcd) so changes propagate in seconds. The ops console is just a client of the same rollout controller API the automation uses.
Finally, the control plane itself must not be a single point of failure during the world's most visible three days: active-active across two regions (or two clouds), with CDN contracts spanning multiple providers so a provider incident becomes a traffic shift, not a stalled rollout. Closing on DR and chaos-testing the processing fleet is the strongest possible last note.
Blueprint
Architecture Diagram
The narrated components above, laid out as an actual flow — so you can see how a request moves through the system, not just read a list of pieces.
Device
Device
Update Gateway (global
Update Gateway (global LB)
API service
rollout controller
metadata store (Cassandra
metadata store (Cassandra / Spanner)
Manifest service signs
Manifest service signs updates
Manifest service signs updates, stores in object storage, replicates to multiple CDNs (Akamai, CloudFront, Cloudflare)
Clients download from
Clients download from CDN
if checksum mismatch
if checksum mismatch, fallback to backup CDN or delta rollback
Telemetry path
device
ingestion API
Kafka
stream processing
adoption dashboards +
adoption dashboards + alerting
Ops console allows
Ops console allows per-cohort throttling
Ops console allows per-cohort throttling, pause/resume, manual overrides
Feature flags stored in config service (Consul/Etcd).
DR: active-active control
DR: active-active control plane across 2
DR: active-active control plane across 2 clouds
CDN contracts span multiple providers.
Flow Diagram
How the API maps to this flow
- 1
POST /api/devices/register { device_id, model, carrier, region, build }— > issue signed token for update polling. - 2
GET /api/updates/check?device_id=...— > returns manifest (version, download_url, start_time, rollout_id). - 3
POST /api/updates/report { device_id, rollout_id, status, metrics }— > telemetry ingestion. - 4
POST /api/rollouts/{id}/pause or /resume— > operations control. - 5
GET /api/rollouts/{id}/cohort— stats -> summarized adoption, failure rate, bandwidth consumption.
?But why does this actually hold up at scale?
Traffic: 500M devices * 200 MB -> 100 PB. 80% in 3 days -> 80 PB / 3 ≈ 26.7 PB/day ≈ 309 GBytes/s ≈ 2.5 Tbps. Need multi-CDN contract with regional POPs.
Interview flow
Dialogue timeline
Interviewer
Summarize the challenge.
Candidate
Distribute an OS delta to 500M devices, ensuring 80% install by day 3 and the remainder by day 5. Need patch packaging, staged rollout control, CDN delivery, and compliance reporting.
Interviewer
Where do you start sizing?
Candidate
Assume 2 GB delta for worst case, but we can ship differential ~200 MB average. 500M devices * 200 MB = 100 PB of traffic. With 80% in 3 days -> ~133 PB/day -> 1.54 Tbps sustained. Need multi-CDN + regional staging.
Interviewer
How do you orchestrate phased rollouts?
Candidate
Control plane maintains cohorts (region, device model, carrier). Clients poll update API, receive signed manifest with start time + CDN URL. Feature flags allow halting per cohort if telemetry spikes.
Interviewer
Telemetry coverage?
Candidate
Clients post installation status + health metrics. Stream into Kafka, aggregate per cohort to verify 80% target by day 3. Alert if adoption lagging or failure rate >1%.
Scoping
Requirements & trade-offs
Functional Requirements
- —Package OS delta (full image + differential) and publish manifests per device model.
- —Clients securely poll update availability, download via nearest CDN edge, verify signature, install, and reboot.
- —Rollout controller enforces cohort schedule (region/carrier/device) and can pause/resume instantly.
- —Telemetry pipeline ingests install progress, failure codes, battery/temperature metrics, and surfaces dashboards for ops.
- —Compliance: ability to prove X% of devices patched per jurisdiction.
Non-Functional Requirements
- —Deadline: 80% adoption by day 3 -> priority on high-density markets (NA/EU) first.
- —Security: updates signed, clients verify before installing. Mutual TLS between device and update API.
- —Scalability: handle 10M concurrent download sessions per CDN, 500K telemetry events/sec.
- —Resilience: ability to throttle or switch CDN per region; failover control plane cross-region active-active.
- —Observability: per-cohort dashboards, anomaly detection on failure rates, live map of adoption.
Blueprint
Architecture modules
Module 1
API Endpoints
- •POST /api/devices/register { device_id, model, carrier, region, build } -> issue signed token for update polling.
- •GET /api/updates/check?device_id=... -> returns manifest (version, download_url, start_time, rollout_id).
- •POST /api/updates/report { device_id, rollout_id, status, metrics } -> telemetry ingestion.
- •POST /api/rollouts/{id}/pause or /resume -> operations control.
- •GET /api/rollouts/{id}/cohort-stats -> summarized adoption, failure rate, bandwidth consumption.
Module 2
Back-of-the-Envelope
- •Traffic: 500M devices * 200 MB -> 100 PB. 80% in 3 days -> 80 PB / 3 ≈ 26.7 PB/day ≈ 309 GBytes/s ≈ 2.5 Tbps. Need multi-CDN contract with regional POPs.
- •Clients poll: assume 500M devices poll every 6 hours -> ~23K QPS to update API (per region). Cache manifest responses.
- •Telemetry: each install sends 2 KB payload. 500M installs -> 1 TB over rollout; peak 500K events/sec. Kafka + time-series DB (ClickHouse) for real-time dashboards.
- •Control plane storage: metadata per device (~1 KB) -> 500 GB; shard by region/device model.
Module 3
System Diagram Notes
- •Device -> Update Gateway (global LB) -> API service -> rollout controller -> metadata store (Cassandra / Spanner).
- •Manifest service signs updates, stores in object storage, replicates to multiple CDNs (Akamai, CloudFront, Cloudflare).
- •Clients download from CDN; if checksum mismatch, fallback to backup CDN or delta rollback.
- •Telemetry path: device -> ingestion API -> Kafka -> stream processing -> adoption dashboards + alerting.
- •Ops console allows per-cohort throttling, pause/resume, manual overrides. Feature flags stored in config service (Consul/Etcd).
- •DR: active-active control plane across 2 clouds; CDN contracts span multiple providers.
Module 4
Design Playbook
- •Clarify device segmentation (region, carrier, hardware) and how cohorts are scheduled.
- •Explain packaging: full image vs delta, signing pipeline, distribution to object storage/CDNs.
- •Discuss client behavior: exponential backoff, resume downloads, verify signatures, rollback on failure.
- •Highlight telemetry/monitoring: adoption goal tracking, failure alerts, auto-pause logic.
- •Cover compliance: proof-of-installation logs, ability to export per-country adoption report.
- •Wrap with risk mitigation: staged rollout (10% -> 30% -> 100%), rollback, chaos testing, bandwidth throttling agreements.