Python Correlation IDs — Core Concepts

When a user clicks a button, that single action can trigger an HTTP request, a database query, a message to a task queue, and calls to three other services. Each component writes its own logs. A correlation ID is the thread that ties all those log lines together.

How correlation IDs work

1. The first service to receive a request generates a unique ID (typically a UUID4).
2. The ID travels in HTTP headers (X-Request-ID or X-Correlation-ID) to downstream services.
3. Every service includes the ID in its log entries.
4. When something breaks, you search your log aggregator for that one ID and see the full trace.

Generating and propagating the ID in Python

FastAPI example

import uuid
from starlette.middleware.base import BaseHTTPMiddleware
from contextvars import ContextVar

correlation_id: ContextVar[str] = ContextVar("correlation_id", default="")

class CorrelationMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        cid = request.headers.get("X-Correlation-ID", str(uuid.uuid4()))
        correlation_id.set(cid)
        response = await call_next(request)
        response.headers["X-Correlation-ID"] = cid
        return response

Using ContextVar ensures the ID is available throughout the request lifecycle without passing it as a function argument everywhere.

Injecting into logs

Add the correlation ID to every log line using a filter or formatter:

import logging

class CorrelationFilter(logging.Filter):
    def filter(self, record):
        record.correlation_id = correlation_id.get("")
        return True

handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(
    "%(asctime)s [%(correlation_id)s] %(levelname)s %(message)s"
))
handler.addFilter(CorrelationFilter())

Now every log line includes the correlation ID automatically — no manual passing needed.

Propagating to downstream services

When your service calls another service, include the ID in the outgoing request:

import httpx

async def call_payment_service(order_id: str, amount: float):
    cid = correlation_id.get()
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://payments.internal/charge",
            json={"order_id": order_id, "amount": amount},
            headers={"X-Correlation-ID": cid}
        )
    return response.json()

The payment service’s middleware picks up the same ID, logs it, and passes it further downstream.

Propagating to task queues

Background jobs (Celery, RQ) need the ID too:

from celery import Celery, signals

app = Celery("tasks")

@signals.before_task_publish.connect
def add_correlation_id(headers, **kwargs):
    headers["correlation_id"] = correlation_id.get("")

@signals.task_prerun.connect
def set_correlation_id(task, **kwargs):
    cid = task.request.get("correlation_id", str(uuid.uuid4()))
    correlation_id.set(cid)

This ensures async work triggered by a request carries the same correlation ID through the task queue.

Header naming conventions

There is no universal standard, but common choices include:

Header	Used by
X-Request-ID	Heroku, Nginx, many APIs
X-Correlation-ID	Azure, enterprise systems
traceparent	W3C Trace Context (OpenTelemetry)

If you’re adopting OpenTelemetry, use traceparent and let the OTel SDK handle generation. For simpler setups, X-Correlation-ID is clear and widely recognized.

Common misconception

“Correlation IDs and trace IDs are the same thing.” They overlap but differ in scope. A correlation ID is a business-level concept — one per user request. A trace ID (from OpenTelemetry or Jaeger) is an observability-level concept that may include parent/child span relationships. You can use a trace ID as your correlation ID, but not every correlation ID system provides distributed tracing spans.

One thing to remember: A correlation ID costs almost nothing to implement — one middleware, one context variable, one log filter — but it transforms debugging from guesswork into grep.