Autoscaling: Policies, Cooldowns and Health Checks

Autoscaling adds and removes servers (or containers) automatically as demand changes. You get enough capacity for peaks without paying for idle machines at night. Autoscaling also replaces broken servers, so it doubles as a self-healing tool. This note covers the policies that decide capacity, the metrics that drive them, and the settings that keep scaling stable and safe.

Prerequisites#

Before any of that works, the tier you scale has to be ready for it. Autoscaling works only on a horizontally scaled, stateless tier:

• Interchangeable servers. Any instance can serve any request. Keep sessions and other state in a shared store, not in server memory.
• A load balancer in front. New instances register with it and start getting traffic. Removed instances deregister. See load balancing.
• A machine image or container image. Every new instance starts from the same template with the same software and config.
• Fast startup. Capacity that takes ten minutes to boot cannot absorb a spike that lasts five.

Min, max and desired capacity#

Once the tier is ready, you put its instances in a scaling group. Every scaling group has three numbers:

The desired count always stays between minimum and maximum. If a policy asks for more than the maximum, the group stops at the maximum.

Choose the minimum for availability, not only for load. Spread at least two instances across separate zones so one zone failure does not take you offline. Choose the maximum with your database and other dependencies in mind. Doubling the app tier also doubles the connections it opens to the database.

Three ways to decide capacity#

A policy is the rule that changes the desired count. Policies fall into three families, and most production systems combine them.

• Scheduled scaling sets capacity at fixed times. Example: raise the minimum to 10 at 08:30 on weekdays and drop it to 2 at 19:00.
• Reactive scaling watches metrics and responds after load changes. It handles surprises, but it always lags a little behind demand.
• Predictive scaling learns patterns from past metrics, often with machine learning. It adds capacity before the forecast rise, which helps when instances boot slowly.

Manual scaling (an operator changes the desired count) still has a place for one-off events and emergencies.

A common mix: predictive or scheduled scaling sets the baseline, and a reactive policy handles anything the forecast missed.

Reactive policies: target tracking vs step#

Reactive policies are the ones you tune most, since they handle the surprises. Two types cover most needs.

Target tracking#

You pick a metric and a target value, such as "average CPU at 50%". The autoscaler works like a thermostat. It adds capacity when the metric is above target and removes it when the metric is below.

For a metric that falls as you add instances, the math is a proportion. Kubernetes' Horizontal Pod Autoscaler uses this form, and the function below shows it with the min and max clamp:

import math


def desired_replicas(current: int, metric: float, target: float,
                     min_n: int, max_n: int) -> int:
    """Proportional target tracking, clamped to [min_n, max_n]."""
    wanted = math.ceil(current * metric / target)
    return max(min_n, min(max_n, wanted))


# 4 instances at 80% CPU, target 50% -> 7 instances
print(desired_replicas(4, 80, 50, min_n=2, max_n=10))

Target tracking is the usual default. You state the goal, and the autoscaler works out the step size.

Step scaling#

Step scaling hands you the step size instead. You define thresholds and an action for each band, and bigger breaches get bigger steps:

Step scaling gives you direct control. It also means more tuning, and badly chosen bands can over-scale or under-scale. An older variant, simple scaling, takes one fixed action per alarm and then waits for a cooldown before it reacts again.

Choosing a metric#

Both policy types are only as good as the metric they watch. A good scaling metric rises with load and falls when you add capacity:

Scaling workers on queue depth#

The queue row in that table needs more explanation. CPU is a poor signal for queue workers, because a worker waiting on a slow API uses little CPU while the backlog grows. Scale on backlog per worker instead. Here's how to set the target:

1. Measure how long one worker takes per message, for example 0.5 s.
2. Decide the longest acceptable wait, for example 60 s.
3. Each worker can then hold 60 / 0.5 = 120 messages of backlog.
4. Set the target: backlog per worker = 120.
5. With 3,600 messages waiting, the autoscaler aims for 3,600 / 120 = 30 workers.

Raw queue depth alone is not enough. The same depth means a lot with 2 workers and very little with 50.

Cooldowns, warm-up and flapping#

Scaling actions take time to show up in metrics. A new instance must boot, start the app and warm its caches. If the autoscaler reacts again before that, it over-corrects.

• Cooldown. After a scaling action, the autoscaler waits a set time before the next one. Make it at least as long as your startup time.
• Warm-up. Some autoscalers ignore a new instance's metrics until it has warmed up, so its cold-start CPU does not trigger more scaling.
• Scheduled actions run at their set time and do not wait for a cooldown.

Get these wrong and you get flapping: the group keeps adding and removing instances in a loop. Typical causes and fixes:

Scaling out too late hurts users. Scaling in too late only costs money. So bias the policy toward fast scale-out and cautious scale-in.

Health checks and replacement#

Autoscaling doesn't only react to load. It keeps the group at its desired count of healthy instances. When an instance fails its health check, the group terminates it and launches a replacement. With desired capacity 4, one failed instance leads to one new instance, back to 4.

How the group decides an instance is unhealthy depends on which of two checks you use:

• Instance checks ask the platform whether the VM or container is running. They miss an app that is up but broken.
• Application checks come from the load balancer calling an endpoint such as /health. They catch a hung or failing app. Use them for any service that takes traffic.

A grace period pauses health checks right after launch. Without it, a slow-booting instance fails its first checks and gets killed before it ever starts, in an endless loop.

Some platforms let you move an instance to standby instead of terminating it. It stops taking traffic, and you can debug it and return it later.

Graceful scale-in#

Adding instances is the easy direction. Removing one mid-request drops that request, so scale-in needs a clean exit:

1. The autoscaler picks an instance to remove. A termination policy decides which one, for example to keep zones balanced or to remove the oldest configuration first.
2. The load balancer stops sending new requests to it.
3. Connection draining (deregistration delay) lets in-flight requests finish.
4. A termination hook can pause the shutdown so the instance can flush logs, finish a job or deregister from other systems.
5. The process receives a stop signal (SIGTERM), finishes its work and exits.
6. The instance is terminated.

Queue workers need the same care. On SIGTERM, stop pulling new messages, finish or release the current one, then exit. An unfinished message that was never acked is redelivered to another worker. For long jobs, some platforms offer scale-in protection to keep a busy instance from being chosen.

Trade-offs and pitfalls#

Autoscaling solves capacity, but it can create new problems. Most of them come from the settings covered above.

• Downstream overload. The app tier scales, the database does not. Set the maximum with your shared dependencies in mind, and use connection pools.
• Cold starts. Reactive scaling lags. Keep headroom, bake dependencies into the image, or add predictive or scheduled scaling.
• Bad health checks. A check that depends on a shared database can mark every instance unhealthy at once and trigger mass replacement.
• Runaway cost. A bug or traffic flood can scale you to the maximum. Alert on group size and set a sane maximum.
• State on instances. Scale-in deletes local disks and memory. Anything stored there is lost.

Key takeaways#

• Autoscaling keeps a stateless tier between a minimum (availability) and a maximum (cost and downstream safety).
• Combine scheduled or predictive scaling for known patterns with a reactive policy for surprises.
• Target tracking is the usual default. Pick a metric that falls as capacity rises; for workers, use backlog per worker.
• Cooldowns, warm-up, a wide threshold gap and slow scale-in prevent flapping.
• Use application health checks with a grace period, and drain connections before you remove an instance.