Python Extension Modules API — Core Concepts

The Python C Extension API is the official interface for writing Python modules in C. Every high-level binding tool (pybind11, PyO3, Cython, SWIG) ultimately generates code that targets this API. Understanding it directly gives you insight into how Python works at its lowest level.

What Is an Extension Module?

An extension module is a shared library (.so on Linux, .pyd on Windows) that Python can import like a regular .py file. The difference: its code is compiled C that runs at native speed.

When you type import mymodule, Python:

1. Searches for mymodule.cpython-312-x86_64-linux-gnu.so (or similar platform-specific name)
2. Loads the shared library via dlopen
3. Calls the PyInit_mymodule function exported by the library
4. Receives a PyObject* representing the module

Anatomy of an Extension Module

#define PY_SSIZE_T_CLEAN
#include <Python.h>

// The actual function implementation
static PyObject* mymod_add(PyObject* self, PyObject* args) {
    long a, b;
    if (!PyArg_ParseTuple(args, "ll", &a, &b))
        return NULL;  // Exception already set
    return PyLong_FromLong(a + b);
}

// Method table
static PyMethodDef MyMethods[] = {
    {"add", mymod_add, METH_VARARGS, "Add two integers"},
    {NULL, NULL, 0, NULL}  // Sentinel
};

// Module definition
static struct PyModuleDef mymodule = {
    PyModuleDef_HEAD_INIT,
    "mymodule",      // module name
    "Example module", // docstring
    -1,              // per-interpreter state size (-1 = global)
    MyMethods
};

// Module initialization function
PyMODINIT_FUNC PyInit_mymodule(void) {
    return PyModule_Create(&mymodule);
}

Key Mechanisms

Argument Parsing

PyArg_ParseTuple converts Python objects to C types using format codes:

Code	Python Type	C Type
i	int	int
l	int	long
d	float	double
s	str	const char*
O	any object	PyObject*
O!	specific type	PyObject* (type-checked)

For keyword arguments, use PyArg_ParseTupleAndKeywords:

static PyObject* mymod_greet(PyObject* self, PyObject* args, PyObject* kwargs) {
    const char* name;
    int excited = 0;
    static char* kwlist[] = {"name", "excited", NULL};
    
    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s|p", kwlist, &name, &excited))
        return NULL;
    
    if (excited)
        return PyUnicode_FromFormat("Hello, %s!!!", name);
    return PyUnicode_FromFormat("Hello, %s.", name);
}

Reference Counting

Python manages memory through reference counting. Every PyObject* has a reference count:

• Py_INCREF(obj): Increment — “I’m using this object.”
• Py_DECREF(obj): Decrement — “I’m done with this object.”
• When the count reaches zero, the object is deallocated.

The critical rules:

• Functions that create new objects return a new reference (caller must eventually DECREF).
• Functions like PyList_GetItem return a borrowed reference (caller must NOT DECREF unless they INCREF first).

Getting this wrong causes either memory leaks (missing DECREF) or crashes (premature DECREF).

Error Handling

C extension functions signal errors by returning NULL after setting an exception:

if (denominator == 0) {
    PyErr_SetString(PyExc_ZeroDivisionError, "division by zero");
    return NULL;
}

Every C API function that can fail returns NULL (for object returns) or -1 (for integer returns). You must check return values and propagate errors.

Building an Extension

The simplest approach uses setuptools:

from setuptools import setup, Extension

setup(
    name="mymodule",
    ext_modules=[Extension("mymodule", sources=["mymodule.c"])],
)

Build with: pip install -e . or python setup.py build_ext --inplace.

When to Use the C API Directly

Scenario	Use C API?
Maximum performance, zero overhead	Yes
Building a binding tool (like pybind11)	Yes
One-off extension for a project	Probably not — use pybind11 or Cython
Wrapping an existing C library	Maybe — ctypes or cffi might be simpler

Common Misconception

“The C API is unstable and changes every release.” CPython distinguishes between the stable ABI (functions guaranteed across versions) and the full API (may change). Extensions targeting the stable ABI (using Py_LIMITED_API) can work across multiple Python versions without recompilation.

One Thing to Remember

The C Extension API is the foundation that all Python-C interop builds on. It’s verbose and requires careful reference counting, but it provides complete control over how C code integrates with CPython’s runtime — and understanding it makes every higher-level tool more transparent.