Embedding Python program in a C/C++ code

• Download EmbeddedPython3.zip - 2.7 MB

Introduction

In this Article, we will discuss the following topics:

• Getting the Python C/C++ API for working.
• Initialize and Destroy a Python environment.
• Running a Simple inline Python code from C/C++.
• Running a Simple Python program from file from C/C++ program.
• Call a Python method from C/C++.
• Call a C/C++ function from Python code.
• Why are we doing this???? (Points of Interest)

 

Background

Before we continue, the reader must be well versed in C/C++ programming. And have basics programming knowledge in Python.

Before we begin : What we need???

 

• Create a Console Application.
• Assuming that Python 3 is installed on the system, completely, at a location "D:\Python34".
• Now add the path of "D:\Python34\include" to Include path of the C/C++ project
• Add the path of "D:\Python34\libs" to Lib path of the project
• Try compiling the project. If it builds correctly, we are ready to begin our work!!!

 

Initialize and Clean the Python Environment

The simplest program to embed is:

#include <stdio.h>
#include <conio.h>
#include <Python.h>

int main()
{
	PyObject* pInt;

	Py_Initialize();

	PyRun_SimpleString("print('Hello World from Embedded Python!!!')");
	
	Py_Finalize();

	printf("\nPress any key to exit...\n");
	if(!_getch()) _getch();
	return 0;
}

• The Python header file is "Python.h" which includes whatever code we need. The linker library file is Python34.lib (in general, PythonXX.lib, XX=34 here).
• Initialize Python environment by calling Py_Initialize()
• Destroy and cleanup the environment by calling Py_Finalize()
• That's all, to get Python Interpreter to get up and running in our C Code.

As you can see that, we have an embedded Python code in our program. This code is:

print('Hello World from Embedded Python!!!')

This code in Python will print the line "Hello World from Embedded Python!!!" on the Console screen. This program is executed by our code using PyRun_SimpleString(char* our_python_code_to_run). But the caller must ensure that, this function is called after Python interpreter is initialized and before it is destroyed.
Thus, we now know how to execute a simple Python code from String in C. Next we will examine how to execute a Python code from an external text file.

 

Execute a Python Program program from file

Consider the Simple Python program to execute, which is stored in the file "pyemb7.py":

print('External Python program running...')
print('Hello World from Python program')

We can run the above program from C/C++ code using the following program:

#include <stdio.h>
#include <conio.h>
#include <Python.h>

int main()
{
	char filename[] = "pyemb7.py";
	FILE* fp;

	Py_Initialize();

	fp = _Py_fopen(filename, "r");
	PyRun_SimpleFile(fp, filename);

	Py_Finalize();
	return 0;
}

• Initialize the Python Environment as discussed before.
• Declare a FILE* to store our program file object.
• Now open the Python program file using _Py_fopen(char* program_filename_with_py_extension, char* file_open_mode). This function is similar to the fopen function of standard C/C++.Here we have opened the pyemb7.py in read mode.
• Check the FILE* object returned. If it is NULL, the file cannot be opened, so we cannot proceed further. Report an error and abort.
• Now we have the file opened. We have to execute it using PyRun_SimpleFile(opened_python_program_file_pointer, char* program_filename_which_becomes_argv_0).
• Destroy the Python Environment, as previously shown.

Now we, are done executing a Python code from external file from within the native code. Next we will discuss a code, to call a specific Python function from a Python file.

 

C++ Helper for Python

Content of C++ Header file pyhelper.hpp:

#ifndef PYHELPER_HPP
#define PYHELPER_HPP
#pragma once

#include <Python.h>

class CPyInstance
{
public:
	CPyInstance()
	{
		Py_Initialize();
	}

	~CPyInstance()
	{
		Py_Finalize();
	}
};


class CPyObject
{
private:
	PyObject *p;
public:
	CPyObject() : p(NULL)
	{}

	CPyObject(PyObject* _p) : p(_p)
	{}

	
	~CPyObject()
	{
		Release();
	}

	PyObject* getObject()
	{
		return p;
	}

	PyObject* setObject(PyObject* _p)
	{
		return (p=_p);
	}

	PyObject* AddRef()
	{
		if(p)
		{
			Py_INCREF(p);
		}
		return p;
	}

	void Release()
	{
		if(p)
		{
			Py_DECREF(p);
		}

		p= NULL;
	}

	PyObject* operator ->()
	{
		return p;
	}

	bool is()
	{
		return p ? true : false;
	}

	operator PyObject*()
	{
		return p;
	}

	PyObject* operator = (PyObject* pp)
	{
		p = pp;
		return p;
	}

	operator bool()
	{
		return p ? true : false;
	}
};


#endif

From now on we will use C++ project, and above file instead.
To automatically initialize Python instance we will just declare an object of CPyInstance pyInstance. When this object goes out of scope, the Python environment is destroyed.
Now we can write the first program above as:

#include <stdio.h>
#include <conio.h>
#include <pyhelper.hpp>

int main()
{
    CPyInstance pyInstance;

	PyRun_SimpleString("print('Hello World from Embedded Python!!!')");
	

	printf("\nPress any key to exit...\n");
	if(!_getch()) _getch();
	return 0;
}

See, how we have removed the 2 functions: Py_Initialize() and Py_Finalize(). Advantage of the CPyInstance class is that, the Python is deinitialized automatically and programmer is free to focus on rest of the code.
In the next section we will discuss the use of Python object PyObject and the corresponding helper CPyObject class.

 

PyObject and the CPyObject helper

PyObject is a Python object used to access the Python objects used or returned by Python interpreter. Python uses references to track the objects returned. When an object is no more needed, it must be deferenced, using DECREF and XDECREF. Consider the code sample below:

PyObject* pObj = ...;
.... the pObj is used in this part of code ....

DECREF(pObj);

In this above code, assume that pObj is returned by a Python function, is used in the further code path. Once the object is no more required, we have called DECREF macro and passed the pObj pointer to it, and we are done. After this, the pObj code will become useless, and caller should never use it further. What about XDECREF macro? Calling DECREF requires that the passed PyObject pointer should never be null. But XDECREF is safe. XDECREF checks for NULL pointer and then internally call DECREF if the PyObject pointer passed is NOT NULL.

If we want to re-reference the PyObject further, or pass it to another function, then we must increase its reference. Hence use INCREF or XINCREF, and pass the PyObject pointer to it. XINCREF is safe, and you can pass a NULL pointer too. But INCREF cannot process NULL pointer.

To be safe, we will use the C++ class CPyObject. Consider the code sample:

CPyObject pObj = ...;
... use the pObj in further code ...

... forget about pObj it will be deferenced when pObj goes out of scope ...

Thus, in above code CPyObject is used instead of PyObject*. This helper class encapsulate the PyObject* within CPyObject class. Once the class goes out of scope, the object is automatically dereferenced. Also, the NULL object is automatically taken care of, and can also be automatically type casted to PyObject*. But to increase the reference pointer, call the AddReff(), method of the CPyObject class.

#include <pyhelper.hpp>

int main()
{
	CPyInstance pInstance;

	CPyObject p;
	p = PyLong_FromLong(50);
	printf_s("Value =  %ld\n", PyLong_AsLong(p));

	return 0;
}

• Initialize a Python environment by declaring an object of CPyInstance.
• Now we write an object of CPyObject called p.
• Now we create a Python object from long value using a call to Python_FromLong(...). This function has the signature PyObject* PyLong_FromLong(long v). The function takes in a long value and returns a PyObject.
• Next we want to convert PyObject* to long value. We know that the p object represents a long value. So we call PyLong_AsLong(...), to get the long value. The function signature is long PyLong_AsLong(PyObject *pylong). This function takes in a PyObject* and returns a long value. In the above program, we have printed this Python object's value.
• After we are done, we just forget about the object, it will automatically be dereferenced.

Next, we will call a Python function from C++.

 

Call a Python function (method) from C++

Consider the following Python program, stored in pyemb3.py:

def getInteger():
    print('Python function getInteger() called')
    c = 100*50/30
    return c

Now we want to call the function getInteger() from the following C++ code and print the value returned this function. This is the client C++ code:

#include <stdio.h>
#include <Python.h>
#include <pyhelper.hpp>

int main()
{
	CPyInstance hInstance;

	CPyObject pName = PyUnicode_FromString("pyemb3");
	CPyObject pModule = PyImport_Import(pName);

	if(pModule)
	{
		CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger");
		if(pFunc && PyCallable_Check(pFunc))
		{
			CPyObject pValue = PyObject_CallObject(pFunc, NULL);

			printf_s("C: getInteger() = %ld\n", PyLong_AsLong(pValue));
		}
		else
		{
			printf("ERROR: function getInteger()\n");
		}

	}
	else
	{
		printf_s("ERROR: Module not imported\n");
	}

	return 0;
}

So, we will use the follow the following rule to call a Python function:

• Initialize the Python environment.
• Import the Python module.
• Get the reference to Python function, to call.
• Check if the function can be called, and call it.
• Then object the returned Python object, returned by the function, after execution.
• Convert returned Python object to C++ object, and print it on screen.

Now, we explain the program code:

• First, we initialize the Python environment using CPyInstance hInstance.
• Next, we get a Python object representing the module to call. Our module name is pyemb3.py, so we get Python object using from the function PyObject *PyUnicode_FromString(const char *u). This function returns PyObject* in CPyObject as CPyObject pName = PyUnicode_FromString("pyemb3");.
• Next, using PyObject* PyImport_Import(PyObject *name), we will import the module and return an object handle representing the imported module. We use this function as CPyObject pModule = PyImport_Import(pName); and if the module is loaded then a valid Python object is returned. Here, no null check is done, programmers may check for it, if they want.
• Next, we get the function attribute object using the function PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name). So, we use this as CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger");. The first is the handle to the module being used, then the second argument is the name of the function being called.
• Next, we check if the returned function object isNULL. If it is not NULL, then check if we can call the function. For this we use int PyCallable_Check(PyObject *o). We pass the function object pointer to this function. If we get a 1 in return, then we can call the function. Only if we can call the function, we can proceed further.
• Next, we call the Python function using PyObject* PyObject_CallObject(PyObject *callable_object, PyObject *args). The first argument here is the function object, we obtained previously. The second is the arguments that will be passed to the Python method. But we do not pass any argument, so we pass NULL to this. The function after execution, will return an object that is returned by the Python method. The result of the Python function.
• Since we know, that the function returns a long value, so we print the value using the program line printf_s("C: getInteger() = %ld\n", PyLong_AsLong(pValue));.

Thus we see the output of the program above is:

Python function getInteger() called
C: getInteger() = 166

Next, we will discuss, how we can call a C/C++ function from an embedded Python program.

 

Calling a C/C++ function from Python code

Consider the following Python code pyemb6.py:

import arnav
print('in python: ')

val = arnav.foo()
print('in python:arnav.foo() returned ', val)

arnav.show(val*20+80)

And now consider the C++ program below, that calls the code above:

#include <stdio.h>
#include <Python.h>
#include <pyhelper.hpp>

static PyObject* arnav_foo(PyObject* self, PyObject* args)
{
	printf_s("... in C++...: foo() method\n");
	return PyLong_FromLong(51);
}

static PyObject* arnav_show(PyObject* self, PyObject* args)
{
	PyObject *a;
	if(PyArg_UnpackTuple(args, "", 1, 1, &a))
	{
		printf_s("C++: show(%ld)\n", PyLong_AsLong(a));
	}

	return PyLong_FromLong(0);
}

static struct PyMethodDef methods[] = {
	{ "foo", arnav_foo, METH_VARARGS, "Returns the number"},
	{ "show", arnav_show, METH_VARARGS, "Show a number" },
	{ NULL, NULL, 0, NULL }
};

static struct PyModuleDef modDef = {
	PyModuleDef_HEAD_INIT, "arnav", NULL, -1, methods, 
	NULL, NULL, NULL, NULL
};

static PyObject* PyInit_arnav(void)
{
	return PyModule_Create(&modDef);
}

int main()
{
	PyImport_AppendInittab("arnav", &PyInit_arnav);

	CPyInstance hInstance;

	const char pFile[] = "pyemb6.py";
	FILE* fp = _Py_fopen(pFile, "r");
	PyRun_AnyFile(fp, pFile);

	return 0;
}

First, concentrating on the main(). The familiar part of the code are, initializing Python environment, then load a Python program and run the program. But now we will conventrate on how to add a new module to Python environment.

First of all, the imported module must be added before the Python instance is initialized. Now how we do it:

• The C function must have the format-
  • PyObject* self as first object, representing the function itself.
  • PyObject* args as the second object, representing the arguments that is passed by the Python program.
  • PyObject* returned by the function itself. This represents the data to be returned to the Python program calling this function.
  • The function name following the format moduleNamespace_functionName.
  So we have a function foo in module namespace arnav, which is oc C type long arnav.foo(void), the code is:

  static PyObject* arnav_foo(PyObject* self, PyObject* args)
  {
  	printf_s("... in C++...: foo() method\n");
  	return PyLong_FromLong(51);
  }

  Next, we declare another function show in arnav which is of type long arnav.show(long value):

  static PyObject* arnav_show(PyObject* self, PyObject* args)
  {
  	PyObject *a;
  	if(PyArg_UnpackTuple(args, "", 1, 1, &a))
  	{
  		printf_s("C++: show(%ld)\n", PyLong_AsLong(a));
  	}
  
  	return PyLong_FromLong(0);
  }

  Note the use of the function PyArg_UnpackTuple, which ise used to parse the arguments. Its signature is int PyArg_UnpackTuple(PyObject *args, const char *name, Py_ssize_t min, Py_ssize_t max, ...). First argument is the reference to the args pointer. The parameter name is not used, it refers to access mode possibly. The min represents minimum number of arguments (tuples) to be passed by Python code, and max must be maximum number of arguments that can be passed. We need only 1 argument to our function, so we set min = max =1.
• Next, we declare a method table:

  static struct PyMethodDef methods[] = {
  	{ "foo", arnav_foo, METH_VARARGS, "Returns the number"},
  	{ "show", arnav_show, METH_VARARGS, "Show a number" },
  	{ NULL, NULL, 0, NULL }
  };

  This is done using PyMethodDef structure array. It's fields are:
  • First field represents the name of the method, that the Python code will use to call our native code.
  • The C/C++ function pointer, that will be invoked when Python calls in.
  • Method can pass in variable numbers of arguments. This is must.
  • Last field, represents a help string. Specify what our function will do. Can be thought of as a Help for our function, in Python environment.
• A newly introduced way to develop import module in Python 3, is to declare PyModuleDef:

      static struct PyModuleDef modDef = {
  	PyModuleDef_HEAD_INIT, "arnav", NULL, -1, methods, 
  	NULL, NULL, NULL, NULL
  };

  At the simplest we will use only few fields of the PyModuleDef structure.
  • First argument - Specify PyModuleDef_HEAD_INIT must be specified.
  • Second argument- The name of the module/namespace to be used in Python code. In our case, it is arnav.
  • Third argument - We are not using it. Specify NULL.
  • Fourth argument - Specify -1.
  • Fifth argument - We pass the pointer to our method declaration structure PyMethodDef. In our case we pass our static variable method.
  • Last 4 parameters, we do not use. So we specify NULL for these.
  NOTE: We are using a minimal declaration, which will solve most of our problems. So, in future article, we will use more fields of this structure and describe in details about each field and their effect.
• Next, we declare the following code:

  static PyObject* PyInit_arnav(void)
  {
  	return PyModule_Create(&modDef);
  }

  This function will be called in, whenever we need to load our module. Here we declare a function of format PyInit_moduleName(void). In our case for namespace arnav, we use function name PyInit_arnav(void). This function does not take anything, but returns a PyObject*. Note the use of PyObject* PyModule_Create(PyModuleDef *module). Here, we pass our PyModuleDef structure. This function will create a new module. In our case we pass the address of our modDef variable. The function returns a PyObject* when the module is loaded, the function PyInit_arnav(void) returns this object created.
• Now, within main() function, before the Python environment is initialized. This is done through the use of function PyImport_AppendInittab. This function has the signature int PyImport_AppendInittab(const char *name, PyObject* (*initfunc)(void)). We pass our initialization function PyInit_arnav pointer to this function. And, if successful (function will not return -1), the new module will be added to the Python environment.
• Finally, the module will become callable from the Python module.

Now we will examine how the Python source utilizes this new namespace arnav:

• First, import the new namespace:

  import arnav
  

• Now we can call the methods within arnav module, using format moduleName.functionName(...) -

  val = arnav.foo()
  print('in python:arnav.foo() returned ', val)
  
  arnav.show(val*20+80)

The output of the program is:

in python:
... in C++...: foo() method
in python:arnav.foo() returned  51
C++: show(1100)

 

Why are we doing this? (Points to remember)

Python is a very popular language with an extremely large code base. Also, the Python API is elaborate yet easy to use. Further, embedding an interpreter within a Native language will enable us to change the course of the program after it had been designed, makes dynamic programming approach to the programming.
In future, we would investigate more into the embedded Python and examine the above discussed structures in details.
 

• Download the attached ZIP file. It has the Python3 SDK bundled along with some demo applications, discussed in this article. You can directly use it.
• If you are installing Python3, use the x86 build, for building x86 based applications.
• Boost and other API alternatives are available for the same work, but since I have never used one; I am sticking to the Python official SDK.
• Always distribute Python34.dll with your C/C++ application. Your native application depends on it. Otherwise, the client system must have Python3 installed correctly.