Concurrency with multiprocessing

The Python language provides some quiet easy ways to achieve concurrency in applications. We saw this with the Python threading library and the same is true for the Python multiprocessing capabilities too.

If you want to build concurrency in your program with the help of multiprocessing, it is quite easy to achieve, all thanks to the Python multiprocessing library and the APIs exposed by the library.

So, what do we mean when we say that we will implement concurrency by using multiprocessing. Let's try to answer this. Usually, when we talk about concurrency, there are two methods that can help us achieve it. One of those methods is running a single application instance and allowing it to use multiple threads. The threads provide a lightweight solution to multithreading and usually live inside the memory space of the application instance only. Most of the time, the executing interpreter or the OS is responsible for scheduling and executing the threads. But there is one other way to achieve concurrency. With this method, instead of using lightweight threads that live inside the application instance itself, we go for multiple processes. Whenever we talk about multiple processes in the context of Python, we essentially mean that every single process executes in its own instance of the Python interpreter. All these processes have their own exclusive memory regions and are usually a bit heavier in comparison to threads, since they need to maintain their own data structures.

The use of multiple processes also allows us to sidestep the GIL in Python, since every process executes in its own instance of the Python interpreter, allowing us to fully exploit the power of a multiprocessor system.

Now, let's take a look at how Python's multiprocessing library helps us to implement concurrency in our applications.