Python Boost.Python Bindings — Deep Dive

Boost.Python’s template-driven design gives fine-grained control over how C++ objects, memory, and exceptions cross into Python. This depth is both its strength and its complexity. Understanding call policies, custom converters, and the object lifecycle model separates toy wrappers from production bindings.

Call Policies: Controlling Lifetime and Ownership

When a C++ function returns a pointer or reference, Python needs to know: who owns this memory? Call policies answer that question.

Built-in Policies

#include <boost/python.hpp>
using namespace boost::python;

class Engine {
public:
    Config& get_config() { return config_; }
    Wheel* create_wheel() { return new Wheel(); }
    const std::string& name() const { return name_; }
private:
    Config config_;
    std::string name_;
};

BOOST_PYTHON_MODULE(engine) {
    class_<Engine>("Engine")
        // Reference to internal data — Python must not outlive Engine
        .def("get_config", &Engine::get_config,
             return_internal_reference<>())
        
        // Caller owns the new object
        .def("create_wheel", &Engine::create_wheel,
             return_value_policy<manage_new_object>())
        
        // Copy the value (safe but may be expensive)
        .def("name", &Engine::name,
             return_value_policy<copy_const_reference>());
}

Policy	When to Use	Risk if Wrong
return_internal_reference<>()	Returned ref/ptr is owned by this	Dangling pointer if parent is GC’d
manage_new_object	Function returns new-allocated object	Memory leak if not set
copy_const_reference	Return a copy of the referenced value	Unnecessary copy overhead
reference_existing_object	Pointer to long-lived external object	Dangling if object is destroyed
return_by_value	Return by value (default for non-pointer)	Usually safe

Custom Call Policies

For complex ownership patterns (e.g., shared ownership via shared_ptr):

// Register shared_ptr<Widget> as a holder type
class_<Widget, boost::shared_ptr<Widget>>("Widget")
    .def("process", &Widget::process);

// Factory function returning shared_ptr
def("create_widget", &create_widget);
// Boost.Python detects shared_ptr return and manages reference counting

Custom Type Converters

For types not covered by Boost.Python’s defaults, register converters:

Python → C++ (from-python converter)

struct NumpyArrayToVector {
    static void* convertible(PyObject* obj) {
        // Check if obj is a numpy array of float64
        if (!PyArray_Check(obj)) return nullptr;
        if (PyArray_TYPE((PyArrayObject*)obj) != NPY_FLOAT64) return nullptr;
        return obj;
    }
    
    static void construct(PyObject* obj,
                          converter::rvalue_from_python_stage1_data* data) {
        PyArrayObject* arr = (PyArrayObject*)obj;
        double* raw = (double*)PyArray_DATA(arr);
        npy_intp size = PyArray_SIZE(arr);
        
        void* storage = ((converter::rvalue_from_python_storage<std::vector<double>>*)data)
            ->storage.bytes;
        new (storage) std::vector<double>(raw, raw + size);
        data->convertible = storage;
    }
};

// Register in module init
converter::registry::push_back(
    &NumpyArrayToVector::convertible,
    &NumpyArrayToVector::construct,
    type_id<std::vector<double>>()
);

C++ → Python (to-python converter)

struct VectorToList {
    static PyObject* convert(const std::vector<double>& vec) {
        list result;
        for (double v : vec) result.append(v);
        return incref(result.ptr());
    }
};

to_python_converter<std::vector<double>, VectorToList>();

Cross-Language Exception Handling

C++ Exceptions → Python

Register exception translators:

void translate_db_error(const DatabaseError& e) {
    PyErr_SetString(PyExc_RuntimeError, e.what());
}

void translate_not_found(const NotFoundError& e) {
    PyErr_SetString(PyExc_KeyError, e.what());
}

BOOST_PYTHON_MODULE(mydb) {
    register_exception_translator<DatabaseError>(&translate_db_error);
    register_exception_translator<NotFoundError>(&translate_not_found);
    // Translators are tried in reverse registration order (most specific last)
}

Python Exceptions → C++

When calling Python from C++ (e.g., in director-like patterns):

try {
    object result = callback(arg1, arg2);
} catch (error_already_set&) {
    // A Python exception was raised
    PyErr_Print();
    // Or inspect: PyObject *type, *value, *tb; PyErr_Fetch(&type, &value, &tb);
}

Iterator and Container Support

Exposing C++ Iterators

class Collection {
public:
    typedef std::vector<Item>::iterator iterator;
    iterator begin() { return items_.begin(); }
    iterator end() { return items_.end(); }
    size_t size() const { return items_.size(); }
private:
    std::vector<Item> items_;
};

BOOST_PYTHON_MODULE(mylib) {
    class_<Collection>("Collection")
        .def("__iter__", range(&Collection::begin, &Collection::end))
        .def("__len__", &Collection::size);
}

STL Container Wrappers

#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
#include <boost/python/suite/indexing/map_indexing_suite.hpp>

class_<std::vector<int>>("IntVector")
    .def(vector_indexing_suite<std::vector<int>>());

class_<std::map<std::string, double>>("StringDoubleMap")
    .def(map_indexing_suite<std::map<std::string, double>>());

These provide __getitem__, __setitem__, __len__, __contains__, and iteration — making C++ containers behave like Python sequences and mappings.

Wrapping Callback-Heavy APIs

For C++ APIs that accept function pointers or std::function:

typedef std::function<double(double)> TransformFn;

class Processor {
public:
    void set_transform(TransformFn fn) { transform_ = fn; }
    double apply(double x) { return transform_(x); }
private:
    TransformFn transform_;
};

BOOST_PYTHON_MODULE(proc) {
    class_<Processor>("Processor")
        .def("set_transform", &Processor::set_transform)
        .def("apply", &Processor::apply);
}

Python can pass a lambda directly:

p = Processor()
p.set_transform(lambda x: x ** 2 + 1)
print(p.apply(3.0))  # 10.0

Boost.Python wraps the Python callable in a boost::python::object stored inside the std::function.

Build System Integration

CMake (Modern)

find_package(Python3 COMPONENTS Development)
find_package(Boost REQUIRED COMPONENTS python${Python3_VERSION_MAJOR}${Python3_VERSION_MINOR})

add_library(mymodule SHARED bindings.cpp mylib.cpp)
target_link_libraries(mymodule
    Boost::python${Python3_VERSION_MAJOR}${Python3_VERSION_MINOR}
    Python3::Module
)
set_target_properties(mymodule PROPERTIES
    PREFIX ""  # Python modules don't use "lib" prefix
    SUFFIX ".so"
)

Linking Pitfalls

• Boost.Python must be compiled against the same Python version you’re targeting. Mismatched versions cause cryptic segfaults.
• On macOS, use -undefined dynamic_lookup to avoid linking against libpython directly.
• Static vs. dynamic Boost: if Boost.Python is statically linked into multiple extension modules loaded by the same Python process, you get duplicate type registries and converter conflicts.

Migration Path to pybind11

Many teams maintain Boost.Python bindings but want to migrate to pybind11 for faster compilation and simpler dependencies. The APIs are deliberately similar:

Boost.Python	pybind11 Equivalent
class_<T>("Name", init<Args...>())	py::class_<T>(m, "Name").def(py::init<Args...>())
def("name", &func)	m.def("name", &func)
return_internal_reference<>()	py::return_value_policy::reference_internal
manage_new_object	py::return_value_policy::take_ownership
register_exception_translator	py::register_exception<T>(m, "Name")

Incremental Migration Strategy

1. Start with leaf modules (no cross-module type sharing).
2. Port one module at a time; both Boost.Python and pybind11 modules can coexist in the same Python process.
3. For shared types, use opaque pointer passing until both sides are migrated.
4. Automate with sed/regex for mechanical transformations, then fix call policies manually.

One Thing to Remember

Boost.Python gives you complete control over the C++/Python interface through call policies, custom converters, and template-driven declarations. Its patterns directly influenced pybind11, making migration feasible — but for existing codebases, Boost.Python remains a fully capable production solution that handles the hardest interop scenarios.