Pybinding isn't a single, monolithic tool, but rather a concept that has been realized by several different libraries. The most prominent and widely used among them is pybind11. The story of pybind11 is a testament to the enduring need for high-performance computing in Python and the elegant solutions that can arise from a desire for simplicity and efficiency.
The story starts with a common problem: Python is a fantastic language for rapid prototyping, data analysis, and orchestrating complex tasks. However, when it comes to raw computational speed, especially for number-crunching or highly parallelized operations, it can fall short. C++ and other compiled languages, on the other hand, excel in these areas. The question was, how do you get the best of both worlds? How do you write the performance-critical parts of your application in C++ while still enjoying the development speed and ecosystem of Python?
The answer was to create a "binding" – a bridge that allows Python to call C++ code as if it were native Python. Early efforts in this space, such as Boost.Python, were powerful but often came with a steep learning curve and significant compilation overhead. They were a bit like using a sledgehammer to crack a nut – effective, but perhaps a bit unwieldy for many use cases.
This is where the next chapter of the story begins, with the emergence of pybind11. The creator of pybind11, Wenzel Jakob, had a vision for a library that would be:
Lightweight and Header-Only: No need to link against a large library. You just include the headers, and the magic happens at compile time.
Easy to Use: The syntax should feel "Pythonic" and intuitive, allowing developers to create bindings with minimal boilerplate code.
Efficient: The generated code should be fast and have a small binary footprint.
Modern: It should fully embrace modern C++ standards, like C++11 and beyond, to take advantage of features like move semantics and templates.
Pybind11 was born from this vision, and it quickly gained traction. It told a new story about how C++ and Python could work together. Instead of a cumbersome, complex process, creating a Python wrapper became a straightforward task.
The story of using pybind11 often goes something like this:
The Setup: A scientist or developer has a C++ library with a highly optimized algorithm. Let's say it's a function that performs a complex matrix multiplication. They want to use this function from their Python scripts, where they are handling data loading, visualization, and other tasks.
The Binding: They write a small C++ file that acts as the "wrapper." This file includes the pybind11 headers and uses its simple syntax to expose the C++ function to Python. They define a Python module, give their C++ function a Python-friendly name, and specify the types of arguments it takes. Pybind11 automatically handles the conversion between Python's data types (like NumPy arrays) and C++'s native types (like std::vector or Eigen matrices).
The Compilation: Using a build system like CMake, they compile this wrapper file. The result is a shared library (.so or .dll) that Python can import.
The Payoff: Back in their Python script, they can now simply import the newly created module and call the C++ function directly. The performance-critical work is handled by the blazing-fast C++ code, while the rest of the application remains in the flexible and easy-to-manage world of Python.
So... here we go...
My exploration of Pybinding11 to refactor one of my projects called FactoryDesignPattern.
Here's the source code...
/*
* Food.h
*
* Created on: Mar 10, 2021
* Author: som
*/
#ifndef FOOD_H_
#define FOOD_H_
#include <string>
using namespace std;
class Food {
public:
virtual string getName() = 0;
virtual ~Food(){
}
};
#endif /* FOOD_H_ */
/*
* Biscuit.h
*
* Created on: Mar 10, 2021
* Author: som
*/
#ifndef BISCUIT_H_
#define BISCUIT_H_
#include "Food.h"
class Biscuit: public Food {
public:
Biscuit();
string getName();
~Biscuit();
};
#endif /* BISCUIT_H_ */
/*
* Biscuit.cpp
*
* Created on: Mar 10, 2021
* Author: som
*/
#include <iostream>
#include "Biscuit.h"
using namespace std;
Biscuit::Biscuit() {
// TODO Auto-generated constructor stub
cout<<"Biscuit is made..."<<endl;
}
Biscuit::~Biscuit(){}
string Biscuit::getName(){
return "It's a Biscuit";
}
/*
* Chocolate.h
*
* Created on: Mar 10, 2021
* Author: som
*/
#ifndef CHOCOLATE_H_
#define CHOCOLATE_H_
#include <iostream>
#include "Food.h"
class Chocolate: public Food {
public:
Chocolate();
virtual ~Chocolate();
string getName();
};
#endif /* CHOCOLATE_H_ */
/*
* Chocolate.cpp
*
* Created on: Mar 10, 2021
* Author: som
*/
#include "Chocolate.h"
Chocolate::Chocolate() {
// TODO Auto-generated constructor stub
cout<<"Chocolate is made..."<<endl;
}
Chocolate::~Chocolate() {
// TODO Auto-generated destructor stub
}
string Chocolate::getName(){
return "It's a Chocolate";
}
/*
* Factory.h
*
* Created on: Mar 10, 2021
* Author: som
*/
#ifndef FACTORY_H_
#define FACTORY_H_
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
#include "Biscuit.h"
#include "Chocolate.h"
using namespace std;
class Factory{
public:
static Factory* instance;
static Factory* getInstance();
Food* makeFood(const string& type);
private:
Factory(){}
// Delete copy constructor & assignment operator (Singleton pattern)
Factory(const Factory&) = delete;
Factory& operator=(const Factory&) = delete;
};
//Factory* Factory:: instance = NULL;
#endif /* FACTORY_H_ */
/*
* Factory.cpp
*
* Created on: Jan 30, 2025
* Author: som
*/
#include "Factory.h"
Factory* Factory::instance = NULL;
Factory* Factory:: getInstance(){
if(Factory::instance == NULL){
Factory::instance = new Factory();
}
return Factory::instance;
}
Food* Factory::makeFood(const string& type){
if(type.compare("bi") == 0){
return new Biscuit();
}
if(type.compare("ch") == 0){
return new Chocolate();
}
return NULL;
}
//bindings.cpp
#include <pybind11/pybind11.h>
#include <memory>
#include "Food.h"
#include "Factory.h"
#include "Biscuit.h"
#include "Chocolate.h"
namespace py = pybind11;
PYBIND11_MODULE(libfoodfactory, m) {
py::class_<Food, std::shared_ptr<Food>>(m, "Food")
.def("get_name", &Food::getName);
py::class_<Biscuit, Food, std::shared_ptr<Biscuit>>(m, "Biscuit")
.def(py::init<>());
py::class_<Chocolate, Food, std::shared_ptr<Chocolate>>(m, "Chocolate")
.def(py::init<>());
m.def("make_food", [](const std::string& type) -> std::shared_ptr<Food> {
Food* f = Factory::getInstance()->makeFood(type);
if (!f) throw std::runtime_error("Unknown food type");
return std::shared_ptr<Food>(f); // manage memory in Python
});
}
Here's the CMakeLists.txt
cmake_minimum_required(VERSION 3.10)
# Set some basic project attributes
project (FactoryPattern_pybind)
set(CMAKE_CXX_STANDARD 14)
# Set the prefix path to where pybind11 is installed,
# allowing CMake to find its configuration files.
# This assumes pybind11 was installed via pip.
list(APPEND CMAKE_PREFIX_PATH "/home/som/.local/lib/python3.8/site-packages")
# Find the pybind11 package. The REQUIRED keyword ensures
# the build fails if the package isn't found.
find_package(pybind11 REQUIRED)
# Add the shared library (MODULE is for Python extensions)
add_library(foodfactory MODULE
Biscuit.cpp
Chocolate.cpp
Factory.cpp
bindings.cpp
)
# Set include directories for your source files.
# No need to add pybind11's include path here, as it's
# handled by the pybind11::pybind11 target.
target_include_directories(foodfactory PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
# Link the library to the pybind11 provided target.
# This ensures all necessary compiler flags and include paths
# from pybind11 are used.
target_link_libraries(foodfactory PRIVATE pybind11::pybind11)
The story of pybind11 is not just about a technical tool ; it's about
a paradigm shift. It showed that the barrier between C++ and Python
could be lowered significantly, making it easier for developers to
leverage the strengths of both languages. It's a story of how a
well-designed, modern library can solve a long-standing problem
and become the de facto standard in its domain.
Enjoy...
Read my other exploration regarding Python and C++ bindings.
Here we go...
Let's start with SWIG...
And now let's delve into Boost.Python...
