bigonotes
Engineering PracticesCI/CD and Releases

Feature Flags

How feature flags let you turn code on and off without a deploy, roll out to a percentage of users, and avoid flag debt.

5 min readLast updated

A feature flag (also called a feature toggle) is a condition in your code that turns a feature on or off at runtime. The flag value comes from config or a flag service, not from the code itself, so you can change behavior without a new deploy. Flags let you merge unfinished work, release to a few users first, and switch off a broken feature in seconds.

How a flag works#

At its simplest, a flag is an if statement around new code.

if flags.is_enabled("new_checkout", user_id=user.id):
    return new_checkout(cart)
return old_checkout(cart)

The code only asks the question. The flag service decides the answer for each call. It can return the same value for everyone, or a different value per user, region or account. Teams often use a hosted service such as LaunchDarkly or Unleash, or a simple table in their own config system.

Flag types#

Not every flag does the same job. Flags differ in how long they live and who changes them, so treat each type differently.

Type Purpose Lifetime Changed by
Release Hide unfinished or new code Days to weeks Developers
Experiment A/B test two versions Length of the test Product, data
Ops (kill switch) Turn off a costly or risky path Long On-call
Permission Give features to some users Long or permanent Product, sales

An A/B test shows version A to one group and version B to another, then compares a metric such as conversion rate. A kill switch lets on-call disable a feature during an incident. It's often the fastest mitigation after a rollback.

Percentage rollouts and targeting#

Release flags rarely go from off to on for everyone at once. A percentage rollout turns a flag on for a share of users, such as 5%. You then increase the share in steps, like a canary deploy but at the feature level. Targeting rules turn the flag on for a named group first: employees, beta users, or one region.

The rollout must be sticky. A user who sees the new feature must keep seeing it on every request. Random choice per request breaks this, so flag systems hash a stable key such as the user ID.

Example: a flag check with stable hashing#

Here's how a sticky percentage rollout can work. This example hashes the flag name and user ID into a bucket from 0 to 9999. A user is in the rollout when their bucket is below the percentage limit. The same user always gets the same bucket, so the result is stable.

import hashlib
from dataclasses import dataclass, field


@dataclass
class Flag:
    name: str
    enabled: bool = False
    percent: float = 0.0  # 0 to 100
    allow_users: set[str] = field(default_factory=set)


def bucket(flag_name: str, user_id: str) -> int:
    # Include the flag name so each flag gets a different 5% of users.
    key = f"{flag_name}:{user_id}".encode()
    digest = hashlib.sha256(key).digest()
    return int.from_bytes(digest[:8], "big") % 10_000


def is_enabled(flag: Flag, user_id: str) -> bool:
    if not flag.enabled:
        return False  # Global off acts as the kill switch.
    if user_id in flag.allow_users:
        return True
    return bucket(flag.name, user_id) < flag.percent * 100


checkout = Flag("new_checkout", enabled=True, percent=5.0,
                allow_users={"qa-tester-1"})
users = [f"user-{i}" for i in range(10_000)]
share = sum(is_enabled(checkout, u) for u in users) / len(users)
print(f"{share:.1%} of users see the new checkout")

You might wonder why the code doesn't use Python's built-in hash(). It's randomized per process for strings, so two servers would put the same user in different buckets. Use a fixed hash such as SHA-256. When you raise the percentage from 5 to 20, the first 5% stay in, because their buckets don't change.

Flag debt and cleanup#

Once a rollout reaches 100%, the flag has done its job, but it's still in your code. Every flag adds a branch. Old release flags that nobody removes are flag debt: dead code paths, confusing logic, and more combinations to test.

  • Give each release flag an owner and an expiry date when you create it.
  • Remove the flag and the old code path soon after the rollout reaches 100%.
  • Add a cleanup ticket at the same time as the flag.
  • List stale flags regularly, for example flags with no change in 30 days.

Old flags aren't only messy; they can be dangerous. In a famous example from 2012, Knight Capital reused an old flag name. One server still had old code behind that flag, and the firm lost about $440 million in under an hour.

Testing both paths#

Until you remove it, a flag means two versions of your code, and both can run in production. Test both of them.

  • Write unit tests with the flag on and with the flag off.
  • In CI, run key integration tests with the flag set to its production value and to its next value.
  • Don't try to test every combination of every flag. Test the combinations you plan to ship.

Risks#

Flags make releases safer, but they add their own failure modes.

  • Config is code. A flag change can break production like a deploy can. Log every change, with who made it and when.
  • Flag service outage. Decide on a safe default value for when the service can't answer. Cache the last known values locally.
  • Nested flags. A flag inside another flag is hard to reason about. Avoid it.
  • Leaking hidden features. If you send flag rules to the browser, users can read them. Evaluate sensitive flags on the server.

Key takeaways#

  • A feature flag changes behavior at runtime, so you can release without a deploy.
  • Release, experiment, ops and permission flags have different lifetimes and owners.
  • Percentage rollouts must be sticky: hash a stable key, not a random number.
  • Remove release flags soon after full rollout, or they become flag debt.
  • Test both paths, log every flag change, and choose safe defaults.
Revision notes · 2 min

Core idea#

A feature flag is a condition in code that turns a feature on or off at runtime. Its value comes from config or a flag service, so you change behavior without a deploy.

Key points#

  • Flags let you merge unfinished work, release to a few users first, and switch off a broken feature in seconds.
  • A percentage rollout turns a flag on for a share of users, then raises it in steps. It is a canary at the feature level.
  • Targeting rules turn a flag on for a named group first: employees, beta users or one region.
  • Rollouts must be sticky. Hash a stable key such as flag_name:user_id into a bucket.
  • Include the flag name in the key, so each flag gets a different set of users.
  • When you raise the percentage, users already in stay in, because their buckets do not change.

Flag types#

Type Purpose Lifetime Changed by
Release Hide unfinished or new code Days to weeks Developers
Experiment A/B test two versions Length of the test Product, data
Ops (kill switch) Turn off a costly or risky path Long On-call
Permission Give features to some users Long or permanent Product, sales

Flag debt and testing#

  • Every flag adds a branch. Old release flags are flag debt: dead paths and more combinations to test.
  • Give each release flag an owner, an expiry date and a cleanup ticket when you create it.
  • Remove the flag and the old path soon after rollout reaches 100%.
  • Test both paths: unit tests with the flag on and off.
  • Test the combinations you plan to ship, not every combination.

Pitfalls#

  • Python's hash() is randomized per process for strings. Two servers bucket the same user differently. Use SHA-256.
  • Config is code. Log every flag change with who and when.
  • Pick a safe default for a flag service outage, and cache the last known values.
  • Flag rules sent to the browser leak hidden features. Evaluate sensitive flags on the server. Avoid nested flags.

Say it in an interview#

  • "A flag changes behavior at runtime, so we release without a deploy."
  • "Rollouts hash the user ID, so each user gets the same answer every time."
  • "A kill switch is often the fastest mitigation after a rollback."

Found a mistake? Email ask@bigonotes.com

Further reading#