Python Multiprocessing

Multiprocessing runs several separate copies of work at once, like hiring more cooks in different kitchens. Each process has its own space and can use a different part of the computer.

In this page:

Process vs. Thread
Creating a Basic Process
Joining Processes
Multiprocessing with Arguments
The main Guard

Syntax

python

from multiprocessing import Process

process = Process(target=function_name, args=(arg,))
process.start()
process.join()

Process vs. Thread

Threads in Python share the same memory space but are limited by the Global Interpreter Lock (GIL), so only one thread executes Python bytecode at a time.

Multiprocessing sidesteps the GIL entirely by running each task in its own process with its own memory space, letting CPU-bound work actually use multiple cores in parallel.

Example: Process vs. Thread

python

import multiprocessing
print(multiprocessing.cpu_count())

Creating a Basic Process

Creating a process starts with instantiating a Process object and passing your target function via the target parameter, optionally with args for that function's arguments.

Calling start() spawns the new OS process and begins running your function immediately, independent of the parent.

Example: Creating a Basic Process

python

import multiprocessing

def task():
    print("Running in a process")

if __name__ == "__main__":
    p = multiprocessing.Process(target=task)
    p.start()  # spawns a new OS process
    p.join()

Joining Processes

By default the main script keeps running immediately after start(), so processes execute in the background while your program moves on.

Calling join() blocks the main script at that point until the named process finishes, which matters whenever later code depends on the process's results being ready.

Example: Joining Processes

python

import multiprocessing

def task():
    print("Process work done")

if __name__ == "__main__":
    p = multiprocessing.Process(target=task)
    p.start()
    p.join()  # blocks until the process finishes
    print("Main continues after process finishes")

Multiprocessing with Arguments

Passing data into a process works the same as calling a function: bundle the values into a tuple and hand them to the args parameter of the Process constructor.

Because each process has separate memory, this is the only reliable way to get input into it -- direct variable sharing across processes doesn't work like it does with threads.

Example: Multiprocessing with Arguments

python

import multiprocessing

def greet(name):
    print("Hello,", name)

if __name__ == "__main__":
    p = multiprocessing.Process(target=greet, args=("Alex",))  # only way to pass data in, since memory isn't shared
    p.start()
    p.join()

The main Guard

Wrapping your multiprocessing entry point in if __name__ == __main__: matters most on Windows and macOS, where new processes re-import your script from scratch.

Without the guard, each child process would re-execute the module-level code that spawns processes, triggering runaway recursive process creation.

Example: The main Guard

python

import multiprocessing

def task():
    print("Safe process spawn")

if __name__ == "__main__":  # prevents runaway recursive process creation on Windows/macOS
    p = multiprocessing.Process(target=task)
    p.start()
    p.join()

Chapter Quiz — Complete all 12 topics to unlock

0/12 topics done

Complete these topics first:

← Previous Lesson Python Threading Next Lesson → Python async & await

Python Multiprocessing

Process vs. Thread

Creating a Basic Process

Joining Processes

Multiprocessing with Arguments

The main Guard

Chapter Quiz — Complete all 12 topics to unlock

Login to run this code