Python Capsule API — Core Concepts

The PyCapsule API provides a way for C extension modules to share C-level data — pointers, function tables, and opaque handles — through Python’s import system. It is the standard mechanism for inter-extension communication without going through Python’s type system.

The Problem Capsules Solve

Consider two C extensions: fast_math provides optimized math functions, and renderer needs to call them. Without Capsules, the options are:

1. Call through Python: renderer imports fast_math and calls functions via Python’s call protocol. This works but adds overhead for each call (argument parsing, reference counting, type checking).

2. Link at compile time: renderer links against fast_math’s shared library directly. This creates a build dependency and doesn’t work if both are distributed as separate pip packages.

3. Use Capsules: fast_math publishes a C pointer wrapped in a Capsule. renderer retrieves the pointer at import time and calls the C function directly — no Python overhead, no link dependency.

How Capsules Work

Creating a Capsule (Producer)

// fast_math.c
static double fast_sin(double x) {
    // Optimized sin implementation
    return __builtin_sin(x);
}

PyMODINIT_FUNC PyInit_fast_math(void) {
    PyObject *module = PyModule_Create(&fast_math_module);
    if (!module) return NULL;
    
    // Wrap the function pointer in a Capsule
    PyObject *capsule = PyCapsule_New(
        (void*)fast_sin,                    // The pointer to share
        "fast_math.fast_sin",               // Unique name (dotted path)
        NULL                                // Optional destructor
    );
    
    if (!capsule) { Py_DECREF(module); return NULL; }
    
    PyModule_AddObject(module, "fast_sin", capsule);
    return module;
}

Retrieving a Capsule (Consumer)

// renderer.c
typedef double (*sin_func)(double);

static sin_func imported_sin = NULL;

static int import_fast_math(void) {
    PyObject *capsule = PyCapsule_Import("fast_math.fast_sin", 0);
    if (!capsule) return -1;
    
    imported_sin = (sin_func)capsule;
    return 0;
}

// Now renderer can call imported_sin(x) at C speed

PyCapsule_Import handles the entire import chain: it imports the fast_math module, retrieves the fast_sin attribute, validates the name, and returns the raw pointer.

The Name System

Capsule names serve as a type-safety mechanism. They follow the convention "module.attribute":

// Creating with a name
PyCapsule_New(ptr, "mylib.MyAPI", NULL);

// Retrieving checks the name
void *p = PyCapsule_GetPointer(capsule, "mylib.MyAPI");
// Returns NULL and sets TypeError if name doesn't match

If a consumer requests a Capsule with the wrong name, the call fails safely instead of returning a pointer to the wrong thing. This prevents accidentally casting a database handle as a math function, for example.

Common Use Patterns

Function Table Pattern

Instead of one function, share an entire API through a struct pointer:

// Shared header: fast_math_api.h
typedef struct {
    double (*sin)(double);
    double (*cos)(double);
    double (*sqrt)(double);
    int version;
} FastMathAPI;

// Producer: packages the struct as a Capsule
static FastMathAPI api = {fast_sin, fast_cos, fast_sqrt, 2};
PyCapsule_New(&api, "fast_math._C_API", NULL);

// Consumer: gets the whole API in one import
FastMathAPI *math = (FastMathAPI*)PyCapsule_Import("fast_math._C_API", 0);
double result = math->sin(3.14);

This is exactly how NumPy exposes its C API. The numpy/_core/include/numpy/ndarrayobject.h header defines a function table that other extensions import via Capsule.

Opaque Handle Pattern

Share a handle to an internal resource:

// Database extension creates a handle
DBHandle *handle = db_open("mydata.db");
PyObject *capsule = PyCapsule_New(handle, "mydb.handle", capsule_destructor);

The destructor is called when the Capsule is garbage collected:

static void capsule_destructor(PyObject *capsule) {
    DBHandle *handle = PyCapsule_GetPointer(capsule, "mydb.handle");
    if (handle) db_close(handle);
}

Real-World Examples

Library	What It Shares via Capsule
NumPy	C API function table (PyArray_API) with ~300+ functions
datetime	C API for creating date/time objects efficiently
Pillow	Imaging C API for pixel manipulation
lxml	libxml2 tree pointers for extensions that need direct XML tree access

Common Misconception

“Capsules are just for function pointers.” Capsules can hold any void* — function pointers, data structures, opaque handles, or even pointers to allocated memory buffers. The pointer type is entirely up to the producer and consumer.

One Thing to Remember

PyCapsule is the standard way for C extensions to share raw pointers through Python’s module system. It provides name-based type safety, optional destructors for cleanup, and enables extensions to call each other’s C code directly without Python overhead.