Sharding

When your data or write volume gets too large for one database, you shard it: split the rows across several machines, each owning a slice. Data modeling matters here because the shard key decides which queries stay cheap. A good key keeps related data together; a bad one turns everyday queries into fan-outs across every shard.

Shard by your primary access pattern#

Pick the shard key from the query you run most often.

• If you mostly query "posts by user", shard by user_id. Each user's posts live on one shard, so the query hits one database.
• If you mostly query recent posts across all users, consider partitioning by time ranges instead. Be aware that the newest range gets all the writes.

Avoid cross-shard queries#

Say you've sharded by user_id and the home timeline needs posts from everyone a user follows. Those posts live on many shards, so you query each and merge the results (scatter-gather). That's slower and more complex. Designs usually work around it, for example by precomputing timelines or caching them, rather than fanning out on every read.

Choose carefully#

The shard key is often effectively permanent: it shapes every query, and changing it means migrating live data. A good key:

• has high cardinality (many distinct values),
• spreads load evenly (no single hot value),
• matches the WHERE clause of most queries, and
• co-locates data that's read together.

For hash vs range partitioning, hotspots and consistent hashing, see Scaling Writes.

Key takeaways#

• Shard by the access pattern you run most often.
• Keep related data on the same shard so common queries touch one database.
• Cross-shard queries mean scatter-gather. Design to avoid them on hot paths.
• Treat the shard key as a long-term decision.