Python API Error Handling Standards — Deep Dive

Implement production-grade error handling in Python APIs with RFC 9457, custom exception hierarchies, and observability integration.

Technical foundation

Error handling in production APIs is not about catching exceptions — it is about building a systematic contract between your service and its consumers that covers every failure mode, provides actionable information, and integrates with observability infrastructure.

Building an exception hierarchy

A well-designed Python API defines a custom exception tree that maps to HTTP semantics:

class APIError(Exception):
    """Base for all API errors."""
    status_code: int = 500
    error_type: str = "INTERNAL_ERROR"
    
    def __init__(self, detail: str, errors: list[dict] | None = None, headers: dict | None = None):
        self.detail = detail
        self.errors = errors or []
        self.headers = headers or {}

class ValidationError(APIError):
    status_code = 422
    error_type = "VALIDATION_ERROR"

class NotFoundError(APIError):
    status_code = 404
    error_type = "NOT_FOUND"

class ConflictError(APIError):
    status_code = 409
    error_type = "CONFLICT"

class AuthenticationError(APIError):
    status_code = 401
    error_type = "AUTHENTICATION_REQUIRED"

class AuthorizationError(APIError):
    status_code = 403
    error_type = "FORBIDDEN"

class RateLimitError(APIError):
    status_code = 429
    error_type = "RATE_LIMIT_EXCEEDED"
    
    def __init__(self, retry_after: int = 60):
        super().__init__(detail=f"Rate limit exceeded. Retry after {retry_after} seconds.")
        self.headers = {"Retry-After": str(retry_after)}

This hierarchy lets you raise domain-specific errors in business logic without importing HTTP concepts:

async def transfer_funds(from_id: int, to_id: int, amount: int):
    sender = await get_account(from_id)
    if not sender:
        raise NotFoundError(f"Account {from_id} not found")
    if sender.balance < amount:
        raise ConflictError(
            "Insufficient funds",
            errors=[{"field": "amount", "message": f"Available: {sender.balance}, requested: {amount}"}]
        )

RFC 9457 implementation in FastAPI

RFC 9457 (the successor to RFC 7807) defines the application/problem+json media type. Here is a complete implementation:

import uuid
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from fastapi.exceptions import RequestValidationError

app = FastAPI()

def problem_response(
    request: Request,
    status: int,
    error_type: str,
    title: str,
    detail: str,
    errors: list[dict] | None = None,
    headers: dict | None = None,
) -> JSONResponse:
    body = {
        "type": f"https://api.example.com/errors/{error_type.lower().replace('_', '-')}",
        "title": title,
        "status": status,
        "detail": detail,
        "instance": str(request.url.path),
        "request_id": request.state.request_id if hasattr(request.state, "request_id") else str(uuid.uuid4()),
    }
    if errors:
        body["errors"] = errors
    return JSONResponse(
        status_code=status,
        content=body,
        headers={"Content-Type": "application/problem+json", **(headers or {})},
    )

@app.exception_handler(APIError)
async def handle_api_error(request: Request, exc: APIError):
    return problem_response(
        request=request,
        status=exc.status_code,
        error_type=exc.error_type,
        title=exc.error_type.replace("_", " ").title(),
        detail=exc.detail,
        errors=exc.errors,
        headers=exc.headers,
    )

@app.exception_handler(RequestValidationError)
async def handle_validation_error(request: Request, exc: RequestValidationError):
    errors = []
    for err in exc.errors():
        field = " -> ".join(str(loc) for loc in err["loc"])
        errors.append({"field": field, "message": err["msg"]})
    return problem_response(
        request=request,
        status=422,
        error_type="VALIDATION_ERROR",
        title="Validation Error",
        detail=f"{len(errors)} validation error(s) in request",
        errors=errors,
    )

@app.exception_handler(Exception)
async def handle_unhandled(request: Request, exc: Exception):
    # Log full traceback but return sanitized response
    import traceback
    traceback.print_exc()
    return problem_response(
        request=request,
        status=500,
        error_type="INTERNAL_ERROR",
        title="Internal Server Error",
        detail="An unexpected error occurred. Contact support with the request_id.",
    )

Request ID middleware

Every error response should include a request ID for traceability:

from starlette.middleware.base import BaseHTTPMiddleware

class RequestIDMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
        request.state.request_id = request_id
        response = await call_next(request)
        response.headers["X-Request-ID"] = request_id
        return response

app.add_middleware(RequestIDMiddleware)

Clients include the request ID in support tickets. Your logging infrastructure indexes it, so finding the exact request across distributed services takes seconds.

Error handling in Django REST Framework

DRF uses a custom exception handler function:

from rest_framework.views import exception_handler
from rest_framework.response import Response

def custom_exception_handler(exc, context):
    response = exception_handler(exc, context)
    
    if response is None:
        # Unhandled exception
        return Response(
            {
                "type": "INTERNAL_ERROR",
                "title": "Internal Server Error",
                "status": 500,
                "detail": "An unexpected error occurred.",
            },
            status=500,
            content_type="application/problem+json",
        )
    
    request = context.get("request")
    body = {
        "type": _error_type_from_status(response.status_code),
        "title": _title_from_status(response.status_code),
        "status": response.status_code,
        "detail": _extract_detail(response.data),
        "instance": request.path if request else None,
    }
    
    if isinstance(response.data, dict) and "errors" in response.data:
        body["errors"] = response.data["errors"]
    
    response.data = body
    response["Content-Type"] = "application/problem+json"
    return response

Configure it in settings: REST_FRAMEWORK = {"EXCEPTION_HANDLER": "myapp.errors.custom_exception_handler"}

Security considerations in error responses

Production error responses must balance helpfulness with security:

  • Never expose stack traces in non-debug environments. Log them server-side, return a sanitized message.
  • Never reveal infrastructure details like database names, internal IPs, or library versions.
  • Rate limit error detail for authentication endpoints. Returning “user not found” vs “wrong password” tells attackers whether an email exists.
  • Sanitize SQL and validation errors from ORMs. A raw SQLAlchemy IntegrityError can reveal table and column names.
@app.exception_handler(IntegrityError)
async def handle_integrity(request: Request, exc: IntegrityError):
    # Don't expose: "duplicate key value violates unique constraint users_email_key"
    return problem_response(
        request=request,
        status=409,
        error_type="CONFLICT",
        title="Conflict",
        detail="A resource with the provided identifier already exists.",
    )

Observability integration

Error handling should feed directly into monitoring:

from opentelemetry import trace
from prometheus_client import Counter

error_counter = Counter(
    "api_errors_total",
    "Total API errors",
    ["status_code", "error_type", "endpoint"],
)

@app.exception_handler(APIError)
async def handle_api_error_with_metrics(request: Request, exc: APIError):
    # Increment Prometheus counter
    error_counter.labels(
        status_code=str(exc.status_code),
        error_type=exc.error_type,
        endpoint=request.url.path,
    ).inc()
    
    # Add error info to current OpenTelemetry span
    span = trace.get_current_span()
    span.set_attribute("error.type", exc.error_type)
    span.set_attribute("error.detail", exc.detail)
    if exc.status_code >= 500:
        span.set_status(trace.StatusCode.ERROR, exc.detail)
    
    return problem_response(...)

This gives you dashboards showing error rates by type, endpoint, and status code — and distributed traces that include error context.

Retry semantics

Your error responses should guide client retry behavior:

  • 4xx errors (except 429): Do not retry. The request is wrong.
  • 429 Too Many Requests: Retry after the Retry-After header value.
  • 500 Internal Server Error: Retry with exponential backoff (the server might recover).
  • 503 Service Unavailable: Retry after Retry-After if present, otherwise backoff.
  • 504 Gateway Timeout: Retry with backoff, but check idempotency first.

Include retry guidance in your API documentation. Better yet, include it in the error response itself:

{
  "type": "https://api.example.com/errors/rate-limit",
  "title": "Rate Limit Exceeded",
  "status": 429,
  "detail": "You have exceeded 100 requests per minute.",
  "retry_after": 30,
  "retry_strategy": "Wait at least 30 seconds before retrying."
}

Testing error paths

Error paths deserve the same test coverage as happy paths:

import pytest
from httpx import AsyncClient

@pytest.mark.anyio
async def test_not_found_returns_problem_json(client: AsyncClient):
    response = await client.get("/users/999999")
    assert response.status_code == 404
    body = response.json()
    assert body["type"] == "https://api.example.com/errors/not-found"
    assert body["status"] == 404
    assert "request_id" in body
    assert response.headers["content-type"] == "application/problem+json"

@pytest.mark.anyio
async def test_validation_error_includes_field_details(client: AsyncClient):
    response = await client.post("/users", json={"email": "not-an-email"})
    assert response.status_code == 422
    body = response.json()
    assert len(body["errors"]) > 0
    assert any(e["field"] == "body -> email" for e in body["errors"])

Tradeoffs

Standardized error handling adds upfront work: defining exception hierarchies, writing middleware, configuring formatters. The payoff is dramatically lower integration friction, faster debugging, and error metrics that actually mean something. Teams that skip this step pay the cost in support tickets and client developer frustration.

The one thing to remember: Treat error responses as a first-class API contract — define a standard format (RFC 9457), build a custom exception hierarchy, wire errors into observability, and test error paths as rigorously as success paths.

pythonapierror-handlingbest-practices

See Also

Related Topics

ab-testing How tech companies run thousands of experiments at once to improve their products — the scientific method applied to everything from button colors to recommendation algorithms. 3 levels → activation-functions Why neural networks need these tiny mathematical functions — and how ReLU's simplicity accidentally made deep learning possible. 3 levels → ai-agents-architecture How AI systems go from answering questions to actually doing things — the design patterns that turn language models into autonomous agents that browse, code, and plan. 3 levels →