Logs, Metrics and Traces

Observability is how well you can understand what a running system is doing from the data it sends out. That data is called telemetry, and it comes in three main kinds: logs, metrics and traces. When a page fires at 2 a.m., these three signals are all you have. Good telemetry turns "something is broken" into "this endpoint fails for these users because of this dependency".

The three pillars#

Each signal answers a different question, so you need all three.

The three work best together. A typical investigation uses them in order:

1. A metric shows a problem, e.g. the error rate rose to 5%.
2. A trace shows where, e.g. the payment service is slow.
3. Logs show why, e.g. the payment client hit a connection timeout.

Structured logging#

Logs come first because they're the signal you'll write most often. A structured log is a log line written as key-value data, usually JSON, instead of free text. Log tools can then filter and group by any field. A free-text line like user 42 failed checkout is easy to read but hard to search across millions of lines.

Request IDs#

The most useful field in a structured log is one that ties lines together. A request ID (also called a correlation ID) is a unique value attached to every log line for one request. Generate it at the edge of your system, or read it from an incoming header. Pass it to every downstream call. You can then search for one ID and see the full story of one request across all services.

A JSON formatter in Python#

Here's how both ideas look in code. This example uses only the standard library. A ContextVar holds the request ID, so each request sees its own value, even with threads or asyncio.

import json
import logging
import uuid
from contextvars import ContextVar

request_id: ContextVar[str] = ContextVar("request_id", default="-")


class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        entry = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "msg": record.getMessage(),
            "request_id": request_id.get(),
        }
        # Extra fields passed with logger.info(..., extra={...})
        entry.update(getattr(record, "fields", {}))
        if record.exc_info:
            entry["exc"] = self.formatException(record.exc_info)
        return json.dumps(entry)


handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("checkout")
log.addHandler(handler)
log.setLevel(logging.INFO)


def handle_request(user_id: int) -> None:
    # Set once per request; every log call below picks it up.
    request_id.set(uuid.uuid4().hex)
    log.info("checkout started", extra={"fields": {"user_id": user_id}})


handle_request(42)

The output is one JSON object per line (wrapped here to fit):

{"ts": "2026-10-17T09:15:02+0000", "level": "INFO", "logger": "checkout",
 "msg": "checkout started", "request_id": "3f9c...", "user_id": 42}

Log levels#

The formatter above writes a level field on every line. Use levels in the same way across your team, so filters mean the same thing everywhere.

What not to log#

Structured logs make it easy to add fields, which also makes it easy to log too much. Logs are copied to many systems, kept for a long time, and readable by many people. Never log:

• passwords, API keys, tokens, session cookies or private keys
• full card numbers or bank details
• PII (personally identifiable information) such as emails, addresses and government IDs, unless policy allows it and the data is masked
• full request or response bodies, which often hold all of the above

Logging habits that pay off#

Beyond what goes in a log, a few habits keep logs cheap to write and easy to read later.

Lazy and guarded logging. A disabled log call should cost almost nothing. If you build the message with an f-string, Python formats it before logging checks the level, so the work is wasted when DEBUG is off. Pass arguments instead, and logging formats them only if the record is emitted. If computing a value is itself expensive, guard the call with isEnabledFor.

import logging

log = logging.getLogger("orders")
logging.basicConfig(level=logging.INFO)

order = {"id": 7, "items": ["book", "pen"]}

# Wasteful: the f-string is built even though DEBUG is off.
log.debug(f"order details: {order}")

# Lazy: %s is filled in only if the record is emitted.
log.debug("order details: %s", order)


def expensive_summary(data: dict) -> str:
    return ", ".join(sorted(map(str, data.values())))


# Guarded: skip the expensive call when DEBUG is off.
if log.isEnabledFor(logging.DEBUG):
    log.debug("summary: %s", expensive_summary(order))

Timestamps in UTC. Servers run in different time zones, and local time jumps at daylight saving changes. Write every timestamp in UTC, in ISO 8601 form, so events from all servers sort into the right order. Python's logging uses local time by default. Set formatter.converter = time.gmtime on your formatter to switch to UTC.

One event, one line. Log pipelines often split input on newlines. A plain-text stack trace then becomes dozens of separate events that are hard to search and index. JSON logging avoids this, because newlines inside a field are escaped. If you must use plain text, keep each event on one line.

Write to a local file, then ship. If your app sends logs straight over the network, a network outage can block it or lose the logs. Writing to a local file, or to standard output that the platform collects, gives you a buffer. A separate agent then ships the file to your central log store. Rotate local files so they can't fill the disk.

Metrics#

Logs record each event, which gets expensive at scale. Metrics take the opposite approach. A metric is a number that you record over time, such as requests per second or memory used. Metrics are cheap because the system stores totals and counts, not each event. That makes them the right source for dashboards and alerts.

Metric types#

Prometheus, a widely used open source metrics system, has four metric types. These three cover most needs; the fourth, the summary, is a less flexible cousin of the histogram.

Each type is read differently. You rarely read a counter's raw value. You read its rate, e.g. errors per second over the last 5 minutes. A histogram lets you estimate percentiles such as p99 latency: the value that 99% of requests are faster than. Averages hide slow outliers, so prefer percentiles for latency.

A Prometheus example in Python#

Here's a counter and a histogram in a small service. This example uses the prometheus_client library. It exposes metrics on an HTTP endpoint that a Prometheus server scrapes (pulls from on a schedule).

import random
import time

from prometheus_client import Counter, Histogram, start_http_server

REQUESTS = Counter(
    "http_requests_total",
    "HTTP requests handled",
    ["route", "status"],
)
LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["route"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)


def handle_checkout() -> None:
    # time() records the block's duration into the histogram.
    with LATENCY.labels(route="/checkout").time():
        time.sleep(random.uniform(0.01, 0.3))
        status = "500" if random.random() < 0.02 else "200"
    REQUESTS.labels(route="/checkout", status=status).inc()


if __name__ == "__main__":
    start_http_server(8000)  # serves /metrics on port 8000
    while True:
        handle_checkout()

RED and USE#

Knowing how to record metrics leaves the question of which ones to record. Two short checklists tell you which metrics to collect first.

• RED for request-driven services: Rate (requests per second), Errors (failed requests per second), Duration (latency distribution).
• USE for resources such as CPU, disk and queues: Utilization (percent busy), Saturation (work waiting), Errors (error events).

Cardinality pitfalls#

The labels in the example (route, status) are where metrics get expensive. Cardinality is the number of unique label combinations for a metric. Each combination is a separate time series that the system must store. A route label with 20 values is fine. A user_id label with a million values creates a million series and can overload your metrics backend.

• Keep label values to a small, fixed set: route, status code, region.
• Never use user IDs, request IDs, emails or raw URLs as labels.
• Put high-cardinality detail in logs and traces instead.

Distributed tracing#

Metrics tell you something is slow, but not which service is slow. That's the job of tracing. A trace records the full path of one request as it moves through your services. It's made of spans. A span is one unit of work, such as an HTTP call or a database query, with a start time, a duration and attributes. Spans have parent-child links, so the trace forms a tree.

In the trace above, the slow part is clear at once: payment.charge takes most of the time. Metrics alone would only tell you that checkout is slow.

Context propagation#

The waterfall only works if every service knows which trace it belongs to. For a trace to cross services, each service must pass the trace context to the next one. The W3C Trace Context standard defines a traceparent HTTP header for this. It holds a version, the trace ID, the parent span ID and flags:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

If one service in the chain drops the header, the trace breaks into two unrelated pieces. Tracing libraries add and read the header for you, as long as you instrument every hop.

Sampling#

Traces have the same cost problem as logs: recording every span of every request adds up at high traffic. Sampling keeps only some traces.

• Head sampling decides at the start of the request, e.g. keep 10% at random. It's simple and cheap, but it can miss rare errors.
• Tail sampling decides after the request ends, so it can keep all errors and slow requests. It needs a collector that buffers whole traces, so it costs more to run.

OpenTelemetry#

So far each signal has used its own library. OpenTelemetry (OTel) is an open standard and set of libraries for producing all three: logs, metrics and traces. You instrument your code once with the OTel API. Then you send the data to any compatible backend, such as Jaeger, Prometheus, Grafana or a commercial vendor. This avoids lock-in to one vendor's agent.

The example below creates a span and prints it to the console. It needs the opentelemetry-sdk package.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("checkout")


def charge_card(amount_cents: int) -> None:
    # Child span: nests under whatever span is current.
    with tracer.start_as_current_span("payment.charge") as span:
        span.set_attribute("payment.amount_cents", amount_cents)


with tracer.start_as_current_span("GET /checkout"):
    charge_card(1999)

In a real service you'd export to an OTel Collector (a separate process that receives telemetry and forwards it to your backends) instead of the console. Auto-instrumentation packages exist for common frameworks such as Flask, Django, FastAPI and requests, so most HTTP and database spans need no manual code.

Key takeaways#

• Metrics tell you that something is wrong, traces tell you where, and logs tell you why.
• Write logs as JSON with a request ID on every line, and never log secrets or PII.
• Pass log arguments lazily, stamp events in UTC, keep one event per line, and buffer logs in a local file.
• Use counters, gauges and histograms; read latency as percentiles, not averages.
• Start with RED for services and USE for resources.
• Keep metric labels low-cardinality; put per-user detail in logs and traces.
• Propagate trace context on every hop, and instrument with OpenTelemetry to stay vendor-neutral.