Thread Pools

A thread pool creates a fixed set of worker threads once and feeds them tasks through a queue. Instead of paying to create and destroy a thread per request, you reuse a handful of threads and let extra work wait cheaply in line. Almost every server framework, database driver and job runner uses one, and sizing it well is a common design question.

The problem: one thread per task#

The naive approach gives every request its own thread. Each request goes through the same four steps:

1. The request arrives.
2. A thread is created: allocate a stack (often ~1 MB reserved) and register with the OS (~1–10 µs).
3. The thread does the work.
4. The thread dies: free the stack, remove it from the scheduler, clean up.

So 1,000 requests means 1,000 creates and 1,000 destroys.

Three costs add up:

1. Memory: 1,000 threads × ~1 MB ≈ 1 GB just in stacks.
2. Overhead: creating and destroying threads thousands of times.
3. Scheduling: the OS juggling 1,000 threads, with context switches between them.

Nothing stops you from creating 1,000 raw threads; it's just wasteful. A thread pool is a design pattern, not a language requirement.

How a pool works#

Create N threads at startup. Each one loops forever: take a task from the queue, run it, take the next.

When the queue is empty, queue.take() puts the worker in the Waiting state, using no CPU. When a task arrives, one waiting worker wakes and takes it. The queue itself is a producer–consumer buffer.

Queue entries are far cheaper than threads#

Each queued task is a small object (a reference to the work), while each thread reserves a whole stack.

In Python#

Here is the same loop written both ways.

import threading

# Without a pool: you create and lose track of every thread
for req in requests:
    t = threading.Thread(target=handle_request, args=(req,))
    t.start()  # 1,000 requests -> 1,000 threads created and destroyed

from concurrent.futures import ThreadPoolExecutor

# With a pool: 4 threads handle every task
with ThreadPoolExecutor(max_workers=4) as pool:
    for req in requests:
        # Goes into the pool's queue; a free worker picks it up
        pool.submit(handle_request, req)
# Leaving the block calls shutdown(wait=True): queued tasks finish first

An analogy: without a pool, you hire 1,000 people for one task each, and they all quit after. With a pool, you hire 4 people who work through all 1,000 tasks.

Sizing the pool#

The right size depends on whether your tasks mostly compute or mostly wait.

CPU-bound work (pure computation)#

Use threads ≈ number of cores:

• 4 cores, 4 threads: every core is 100% busy.
• 4 cores, 8 threads: the same work gets done, plus switching overhead.

More threads than cores can't add parallelism; they just take turns on the same cores.

I/O-bound work (DB calls, HTTP calls)#

Use this formula as a starting point:

For example, with 4 cores and requests that take 5 ms of compute plus 50 ms waiting on the DB: threads ≈ 4 × (1 + 50/5) = 4 × 11 = 44.

Workers spend most of their time asleep waiting for I/O, so you need enough of them to keep every core busy:

Treat the formula as a starting point and tune with load tests; downstream limits (like the size of your database connection pool) often cap you first.

When the queue is full#

Pools usually use a bounded queue so memory can't grow without limit. When every worker is busy and the queue is full, a rejection policy decides what happens:

One way to bound it is a semaphore around submit() that implements the abort policy:

import threading
from concurrent.futures import ThreadPoolExecutor

MAX_WORKERS, MAX_QUEUED = 4, 100
pool = ThreadPoolExecutor(max_workers=MAX_WORKERS)
slots = threading.BoundedSemaphore(MAX_WORKERS + MAX_QUEUED)

def submit_or_reject(fn, *args):
    # Abort policy: fail fast when every worker and queue slot is taken
    if not slots.acquire(blocking=False):
        raise RuntimeError("pool is full, task rejected")
    future = pool.submit(fn, *args)
    future.add_done_callback(lambda _: slots.release())
    return future

Calling slots.acquire() with blocking left on instead makes producers wait, which gives the same back-pressure as caller runs.

In practice#

Serverless platforms apply the same idea at a larger scale. AWS Lambda, for example, keeps warm execution environments and reuses them for new requests. If none is free it starts a new one (a cold start), up to your concurrency limit, and throttles beyond that. The concurrency limit plays the role of pool size, and throttling is the rejection policy.

Key takeaways#

• A pool pre-creates N workers that take tasks from a shared queue, avoiding per-task thread creation and bounding memory.
• Queued tasks cost bytes; threads cost a stack each.
• CPU-bound: about one thread per core. I/O-bound: about cores × (1 + wait/compute), then tune.
• Use a bounded queue plus a rejection policy so overload fails predictably.
• Pools are a pattern you could build yourself from threads, a blocking queue and a loop.