Latency and throughput are the two numbers you use to describe how a system performs. Latency is how long one request takes. Throughput is how many requests the system handles per second. They are related but not the same, and improving one can hurt the other.
Every design discussion about performance comes back to these two, plus the targets you promise around them.
Definitions#
The two terms answer different questions.
| Latency | Throughput | |
|---|---|---|
| Question | How fast does one request finish? | How much work gets done overall? |
| Measured as | Time: ms, µs, s | Rate: requests/s (RPS), queries/s (QPS), transactions/s (TPS) |
| Fast-food analogy | Time from ordering to getting your burger | Burgers the restaurant serves per minute |
Latency is usually reported as percentiles, not averages. P50 is the median. P95 means 95% of requests are faster than this value. P99 catches the slow tail that averages hide.
Why the distinction matters#
You can't optimize one number and assume the other follows.
- They trade off. Giving each request more dedicated resources can lower its latency but reduce total throughput. Batching and queueing raise throughput but make individual requests wait longer.
- Every design needs targets for both, for example "P95 under 200 ms at 10,000 RPS".
Where they show up#
Every component in a design has both a latency and a throughput.
| System | Latency example | Throughput example |
|---|---|---|
| Web API | Time to return JSON for a GET |
Requests handled per second |
| Database | Time to run one query | Queries per second |
| Message queue | Time to enqueue or dequeue a message | Messages processed per second |
| Video streaming | Delay before playback starts | Concurrent streams served |
How to improve each#
Different techniques move each number.
Lower latency (faster per request)#
- Cache frequently read data (Redis, Memcached).
- Shorten call chains: fewer hops between microservices.
- Load balance so no node is overloaded.
- Use better algorithms and indexes.
- Return partial results early (progressive rendering).
- Move servers and data closer to users (Regionalization and Latency).
Higher throughput (more requests overall)#
- Scale horizontally: add servers.
- Batch work into chunks.
- Process asynchronously with queues and workers.
- Partition or shard data so work spreads across machines.
Ballpark latency targets#
These are rough, commonly used expectations for user-facing systems, usually stated at P95 or P99. Treat them as a starting point for discussion, not as standards. Real targets come from your product's requirements.
| System | Operation | Rough target |
|---|---|---|
| Web APIs | Standard CRUD (profile lookup, settings update) | 100–300 ms |
| Read-heavy, served from cache | < 50–150 ms | |
| Write-heavy (posting a comment) | 150–500 ms | |
| Search | Small dataset | 100–300 ms |
| Large, many shards | 300–800 ms | |
| Autocomplete / typeahead | < 100 ms (must feel instant) | |
| Messaging | 1:1 chat message | < 100 ms |
| Typing indicator, presence | < 50 ms | |
| Push notification | < 500 ms | |
| In-app notification | 200–500 ms | |
| Files and media | Acknowledge a small upload | < 500 ms |
| Large upload | Chunked and progressive | |
| Thumbnail fetch | < 100 ms | |
| Video playback start | 1–2 s | |
| Payments | Authorization, confirmation | < 2 s |
| Balance inquiry | < 500 ms | |
| Fraud check | < 300 ms (sometimes async) | |
| URL shortener | Redirect | < 50 ms |
| CDN | Edge cache hit | < 50 ms |
| Edge miss (origin fetch) | 100–500 ms | |
| Auth | Token generation | < 200 ms |
| Session validation | < 100 ms | |
| 2FA code check | < 500 ms |
A handy way to state a target: "This should respond in about X ms at P95."
SLA, SLO and SLI#
Three related terms describe the targets you set and how you track them.
| Term | What it is | Who it's for | Example |
|---|---|---|---|
| SLA (service level agreement) | A formal promise to customers, often with penalties (credits, refunds) if broken | External, contractual | "99.9% of requests complete in < 500 ms each month." |
| SLO (service level objective) | The internal target your team aims for, stricter than the SLA | Internal, measurable | "99.95% of requests complete in < 500 ms." |
| SLI (service level indicator) | The measured value | Dashboards and alerts | "Last month, 99.97% completed in < 500 ms." |
Why the SLO is stricter than the SLA#
You never want to run exactly at the SLA. If you promise 99.9% and engineer for exactly 99.9%, any bad deploy or cloud hiccup immediately breaks the contract. Setting a tighter SLO gives you a safety margin.
| Level | Value | Role |
|---|---|---|
| SLA (promise to customers) | 99.9% | The contract |
| SLO (internal target) | 99.95% | Stricter, so you get early warning |
| SLI (what you measured) | 99.98% | Compare against the SLO and alert before the SLA is at risk |
With this margin you have room for incidents, you spot problems before customers do, and you avoid penalties.
Key takeaways#
- Latency is time per request. Throughput is requests per unit of time.
- Measure latency with percentiles (P95, P99), not averages.
- Caching, fewer hops and proximity cut latency. Horizontal scaling, batching, async work and sharding raise throughput.
- Batching and queueing usually trade higher latency for higher throughput.
- SLA is the external promise, SLO the stricter internal target, SLI the measurement.