Python Capsule API — Deep Dive

PyCapsule is deceptively simple — it wraps a void* in a Python object. But building a reliable, versioned C API on top of Capsules requires careful design around compatibility, lifecycle management, and error handling. This guide examines production patterns, starting with the architecture NumPy uses to serve hundreds of C extensions.

The NumPy Pattern: Function Pointer Tables

NumPy is the canonical example of Capsule-based API design. Its approach has been stable for over 15 years and serves as the template for other projects.

How NumPy Exports Its C API

NumPy defines a static array of void* pointers, each slot holding a function pointer:

// Simplified from numpy/core/src/multiarray/multiarraymodule.c
static void *PyArray_API[NPY_API_SIZE];

// During module initialization:
PyArray_API[0] = (void*)&PyArray_Type;
PyArray_API[1] = (void*)PyArray_NewFromDescr;
PyArray_API[2] = (void*)PyArray_SetBaseObject;
// ... 300+ entries ...

PyObject *c_api = PyCapsule_New(PyArray_API, "numpy._core._multiarray_umath._ARRAY_API", NULL);
PyModule_AddObject(module, "_ARRAY_API", c_api);

How Consumers Import It

NumPy provides a header (ndarrayobject.h) that defines macros:

// Generated by NumPy's header system
static void **PyArray_API;

#define PyArray_Type   (*(PyTypeObject *)PyArray_API[0])
#define PyArray_NewFromDescr \
    (*(PyObject* (*)(PyTypeObject*, PyArray_Descr*, int, npy_intp*, npy_intp*, void*, int, PyObject*)) \
     PyArray_API[1])

static int _import_array(void) {
    PyArray_API = (void**)PyCapsule_Import(
        "numpy._core._multiarray_umath._ARRAY_API", 0);
    return (PyArray_API == NULL) ? -1 : 0;
}

#define import_array() { if (_import_array() < 0) return NULL; }

Consumer code simply calls import_array() in its PyInit_* function, then uses PyArray_NewFromDescr and other macros as if they were regular function calls.

Why This Design Works

1. Binary compatibility: The table is an array of pointers. New functions are appended at the end, never inserted in the middle. Old consumers see the same offsets.
2. No link dependency: Consumers don’t link against NumPy’s .so. The pointers are resolved at runtime through Python’s import system.
3. Version checking: The table includes a version number at a known slot. Consumers check compatibility at import time.

Versioned API Design

Version Field Pattern

// api.h (shared header)
#define MY_API_VERSION 3

typedef struct {
    int version;
    // v1 functions
    double (*compute)(const double*, size_t);
    void   (*reset)(void);
    // v2 additions
    int    (*configure)(const char* key, const char* value);
    // v3 additions
    void   (*set_callback)(void (*cb)(int));
} MyAPI;

Producer (Version 3)

static MyAPI api = {
    .version = MY_API_VERSION,
    .compute = my_compute,
    .reset = my_reset,
    .configure = my_configure,
    .set_callback = my_set_callback,
};

PyCapsule_New(&api, "mylib._C_API", NULL);

Consumer (Handles Multiple Versions)

static MyAPI *api = NULL;

static int import_mylib(void) {
    api = (MyAPI*)PyCapsule_Import("mylib._C_API", 0);
    if (!api) return -1;
    
    if (api->version < 2) {
        PyErr_SetString(PyExc_ImportError,
            "mylib >= 2.0 required (found version 1)");
        return -1;
    }
    return 0;
}

// Safe to call api->configure (v2+)
// Must check version before calling api->set_callback (v3+)
if (api->version >= 3) {
    api->set_callback(my_handler);
}

Destructor Lifecycle

Basic Destructor

static void capsule_free_buffer(PyObject *capsule) {
    void *ptr = PyCapsule_GetPointer(capsule, "mylib.buffer");
    if (ptr) {
        free(ptr);
    }
}

PyObject *cap = PyCapsule_New(
    malloc(4096),
    "mylib.buffer",
    capsule_free_buffer  // Called when Capsule is GC'd
);

Destructor Safety Rules

1. Do not call Python API in destructors during interpreter shutdown. The interpreter state may be partially torn down. Check with Py_IsInitialized() if necessary.

2. Handle NULL pointers: PyCapsule_GetPointer can return NULL if the name doesn’t match or if PyCapsule_SetPointer was called with NULL (marking the capsule as consumed).

3. Idempotent cleanup: The destructor should be safe to call even if the resource was already cleaned up:

static void safe_destructor(PyObject *capsule) {
    DBHandle *h = PyCapsule_GetPointer(capsule, "mydb.handle");
    if (h && h->is_open) {
        db_close(h);
        h->is_open = 0;
    }
}

Context Pointer

Capsules support an additional context pointer for destructor metadata:

// Store cleanup metadata
typedef struct { int pool_id; size_t alloc_size; } AllocInfo;
AllocInfo *info = malloc(sizeof(AllocInfo));
info->pool_id = 7;
info->alloc_size = 4096;

PyObject *cap = PyCapsule_New(buffer, "mylib.pooled_buffer", pooled_destructor);
PyCapsule_SetContext(cap, info);

static void pooled_destructor(PyObject *capsule) {
    void *buf = PyCapsule_GetPointer(capsule, "mylib.pooled_buffer");
    AllocInfo *info = PyCapsule_GetContext(capsule);
    pool_free(info->pool_id, buf, info->alloc_size);
    free(info);
}

Cross-Module Type Systems

When multiple extensions need to recognize each other’s types, Capsules can share PyTypeObject*:

// Module A exports its custom type
PyObject *type_capsule = PyCapsule_New(
    &MyCustomType, "module_a.MyCustomType", NULL
);

// Module B imports and uses it for isinstance checks
PyTypeObject *CustomType = (PyTypeObject*)PyCapsule_Import(
    "module_a.MyCustomType", 0
);

static PyObject* process(PyObject *self, PyObject *arg) {
    if (!PyObject_IsInstance(arg, (PyObject*)CustomType)) {
        PyErr_SetString(PyExc_TypeError, "Expected MyCustomType");
        return NULL;
    }
    // Safe to cast
    MyCustomObject *obj = (MyCustomObject*)arg;
    // ...
}

This pattern is used by datetime (datetime.datetime_CAPI), which exports a Capsule so other extensions can create datetime objects without going through the Python constructor.

The datetime C API Pattern

// datetime exports a struct of factory functions
typedef struct {
    PyTypeObject *DateType;
    PyTypeObject *DateTimeType;
    PyObject *(*Date_FromDate)(int, int, int, PyTypeObject*);
    PyObject *(*DateTime_FromDateAndTime)(int, int, int, int, int, int, int,
                                          PyObject*, PyTypeObject*);
    // ... more functions
} PyDateTime_CAPI;

// Consumer:
static PyDateTime_CAPI *datetime_api = NULL;

static int init_datetime(void) {
    datetime_api = (PyDateTime_CAPI*)PyCapsule_Import("datetime.datetime_CAPI", 0);
    return datetime_api ? 0 : -1;
}

// Create a datetime at C speed:
PyObject *dt = datetime_api->DateTime_FromDateAndTime(
    2026, 3, 28, 12, 0, 0, 0, Py_None, datetime_api->DateTimeType
);

Error Handling

Capsule Validation

// Check if an object is a Capsule
if (!PyCapsule_IsValid(obj, "expected.name")) {
    PyErr_SetString(PyExc_TypeError, "Invalid capsule");
    return NULL;
}

// PyCapsule_GetPointer returns NULL on name mismatch
void *ptr = PyCapsule_GetPointer(capsule, "expected.name");
if (!ptr) return NULL;  // TypeError already set

Common Failure Modes

Problem	Symptom	Fix
Wrong capsule name	TypeError: PyCapsule_GetPointer called with incorrect name	Verify name string matches exactly
Capsule from wrong module version	Segfault or wrong behavior	Add version field to API struct
Destructor called during shutdown	Segfault in cleanup code	Guard with Py_IsInitialized()
Consumer loaded before producer	ImportError	Ensure import order in __init__.py
Capsule pointer set to NULL	NULL dereference	Check return value of every GetPointer call

Performance Characteristics

Capsule-based function calls have essentially zero overhead compared to direct C function calls:

Direct C call:          ~1 ns
Via Capsule pointer:    ~1 ns (one pointer dereference)
Via Python call:        ~100-300 ns (argument parsing, ref counting)

The pointer dereference is negligible because it’s a single memory load, typically from L1 cache after the first call.

Modern Alternatives

HPy (Handle-based API)

HPy is a new C API designed for multiple Python implementations (CPython, PyPy, GraalPy). It provides its own Capsule-like mechanism but with stronger guarantees about handle validity.

Cython’s cdef exports

Cython modules can export C-level functions to other Cython modules via .pxd declaration files:

# fast_math.pxd
cdef double fast_sin(double x)

Under the hood, Cython uses Capsules for this, but the user never sees them.

One Thing to Remember

PyCapsule is the glue that lets C extensions build on each other’s native code without Python overhead. The key to a robust Capsule-based API is versioned function tables, careful name conventions, and destructor safety — patterns proven at scale by NumPy, datetime, and dozens of other foundational libraries.