Python Multiprocessing
In this page:
Process vs. Thread
Threads in Python share the same memory space but are limited by the Global Interpreter Lock (GIL), so only one thread executes Python bytecode at a time. Multiprocessing sidesteps the GIL entirely by running each task in its own process with its own memory space, letting CPU-bound work actually use multiple cores in parallel.
Example: Process vs. Thread
import multiprocessing
print(multiprocessing.cpu_count())Creating a Basic Process
Creating a process starts with instantiating a Process object and passing your target function via the target parameter, optionally with args for that function's arguments. Calling start() spawns the new OS process and begins running your function immediately, independent of the parent.
Example: Creating a Basic Process
import multiprocessing
def task():
print("Running in a process")
if __name__ == "__main__":
p = multiprocessing.Process(target=task)
p.start()
p.join()Joining Processes
By default the main script keeps running immediately after start(), so processes execute in the background while your program moves on. Calling join() blocks the main script at that point until the named process finishes, which matters whenever later code depends on the process's results being ready.
Example: Joining Processes
import multiprocessing
def task():
print("Process work done")
if __name__ == "__main__":
p = multiprocessing.Process(target=task)
p.start()
p.join()
print("Main continues after process finishes")Multiprocessing with Arguments
Passing data into a process works the same as calling a function: bundle the values into a tuple and hand them to the args parameter of the Process constructor. Because each process has separate memory, this is the only reliable way to get input into it -- direct variable sharing across processes doesn't work like it does with threads.
Example: Multiprocessing with Arguments
import multiprocessing
def greet(name):
print("Hello,", name)
if __name__ == "__main__":
p = multiprocessing.Process(target=greet, args=("Alex",))
p.start()
p.join()The main Guard
Wrapping your multiprocessing entry point in if __name__ == __main__: matters most on Windows and macOS, where new processes re-import your script from scratch. Without the guard, each child process would re-execute the module-level code that spawns processes, triggering runaway recursive process creation.
Example: The main Guard
import multiprocessing
def task():
print("Safe process spawn")
if __name__ == "__main__":
p = multiprocessing.Process(target=task)
p.start()
p.join()Chapter Quiz — Complete all 12 topics to unlock
0/12 topics done
Complete these topics first: