Multiprocessing: Processes and Pools
25 min · Intermediate
Need help? Ask a tutor on WhatsApp (opens in a new tab)Threads share one interpreter. Processes do not: each has its own memory and runs on its own core, so heavy calculations can finish sooner.
Start a separate process
Process is a running program with its own memory and its own Python interpreter. A parent process can start child processes that run at the same time on other CPU cores.
A process is a whole running copy of Python with its own memory. multiprocessing.Process starts one the way Thread starts a thread.
from multiprocessing import Process, Queue
def square(n, out): out.put(n * n)
if __name__ == "__main__": out = Queue() p = Process(target=square, args=(7, out)) p.start() print(out.get()) p.join()Because memory is separate, a child sends results back through a queue.
Example 1 · Results back through a Queue
from multiprocessing import Process, Queue
def square(n, out): out.put((n, n * n))
if __name__ == "__main__": out = Queue() procs = [Process(target=square, args=(n, out)) for n in (3, 1, 2)] for p in procs: p.start() results = sorted(out.get() for _ in procs) for p in procs: p.join() print(results)[(1, 1), (2, 4), (3, 9)]
What this code does
- Three processes each put a pair on the shared queue.
- The parent reads three results first, then joins.
- Each result carries its input, and the list is sorted, so arrival order does not matter.
Read from the queue before join(). A child waiting to hand over a large result can block forever if you join first.
The main guard
Spawn start method makes a new, empty Python process and imports your module in it. It is the default on Windows and macOS. Linux has used fork, which copies the parent, though newer versions are moving to spawn.
On Windows and macOS, a child starts by importing your file again. Without a guard, the import would start more children.
Put the starting code under if __name__ == "__main__":.
Function definitions stay outside the guard, so the child can find them. Forgetting the guard is the most common multiprocessing error.
Pool: map work over processes
Pool is a fixed group of worker processes that share a list of jobs. You give it a function and data, and it returns the results in order.
A pool starts a few worker processes and hands them jobs. pool.map(func, items) runs func on each item and returns the results in the order of the items.
from multiprocessing import Pool
def cube(n): return n ** 3
if __name__ == "__main__": with Pool(2) as pool: print(pool.map(cube, [1, 2, 3, 4]))A CPU-heavy job finishes sooner.
Example 2 · Pool.map on CPU-heavy work
from multiprocessing import Pool
def count_primes(limit): count = 0 for n in range(2, limit): if all(n % d for d in range(2, int(n ** 0.5) + 1)): count += 1 return count
if __name__ == "__main__": with Pool(2) as pool: print(pool.map(count_primes, [10, 100, 1000]))[4, 25, 168]
What this code does
- Each limit is one job. Two workers share the three jobs.
mapkeeps the input order, so the answers line up with[10, 100, 1000].- Counting primes uses the CPU the whole time, which is the kind of work that benefits from processes.
Example 3 · Several arguments with starmap
from multiprocessing import Pool
def area(width, height): return width * height
if __name__ == "__main__": sizes = [(2, 3), (4, 5), (10, 10)] with Pool(2) as pool: print(pool.starmap(area, sizes))[6, 20, 100]
What this code does
mappasses one argument.starmapunpacks each tuple into several arguments.- The result list is in the same order as
sizes.
The with block shuts the pool down. The function must be defined at the top of the module, so Python can send its name to the workers.
Processes do not share variables
Pickling is turning an object into bytes so it can be sent to another process, and rebuilding it there. Lambdas, nested functions and open files cannot be pickled.
A child gets a copy of the data it needs. Changing a variable in the child never changes the parent's variable.
Data sent between processes is pickled: turned into bytes and rebuilt on the other side.
Example 4 · Memory is not shared
from multiprocessing import Process
counter = 0
def bump(): global counter counter += 1 print("child sees", counter)
if __name__ == "__main__": p = Process(target=bump) p.start() p.join() print("parent sees", counter)child sees 1 parent sees 0
What this code does
- The child adds 1 to its own copy of
counter. - The parent's
counteris still 0, because processes do not share variables. join()comes before the parent's print, so the two lines always appear in this order.
So arguments and results must be picklable: numbers, strings, lists, dicts and most classes defined at module level. Lambdas and open files are not.
Next step
Check your understanding
10 quick questions