Running code in parallel with Python multiprocessing

Using Python's multiprocessing module can improve performance, but attention should be paid to startup methods, Pool usage, process communication and exception handling. 1. Choose the appropriate startup method: fork (Unix fast but unstable), spawn (cross-platform recommendation), forkserver (profitable for frequent creation); 2. Use Pool to manage concurrent tasks, control the number of processes, and reasonably select map or apply_async; 3. Inter-process communication can be used to communicate with Queue, Pipe, Value, Array or Manager, pay attention to performance and security; 4. Strengthen exception handling, use logging to debug, and can be simulated by a single process during development.

Python's multiprocessing module can indeed make your code run faster, provided that the correct method is used. It bypasses GIL (global interpreter lock) by creating multiple processes, truly implementing parallel computing. But many people are prone to traps when using it, such as inter-process communication, resource competition, efficiency loss, etc.

The following key points are the most common and most prone to problems in actual use. Let’s take a look at them one by one.

How to choose the right startup method

There are three ways to start multiprocessing: fork , spawn and forkserver . The default values of different systems are different, and the behaviors are also different.

• fork (Unix only) copies the status of the current process, including all variables and threads. Although it is fast, it is easy to cause instability. For example, fork may cause deadlocks in a multi-threaded environment.
• spawn creates a clean new process that inherits only the necessary resources. Good cross-platform compatibility and is recommended for production environments.
• forkserver is a compromise solution, suitable for scenarios where child processes need to be created frequently.

If you find that the program is not running properly on some platforms, you can try to explicitly specify the startup method:

 import multiprocessing as mp

mp.set_start_method('spawn')

Use Pool to manage concurrent tasks more efficiently

When you have a bunch of independent tasks to process, such as batch download, image processing or data analysis, using Pool is the most worry-free way.

It allows you to set the maximum number of concurrencies and automatically schedule tasks to various processes. Commonly used methods include map , apply_async , etc.

For example, suppose you want to process a set of files in parallel:

 from multiprocessing import Pool

def process_file(filename):
    # Assume that this is the time-consuming processing logic return f"Processed {filename}"

if __name__ == '__main__':
    files = ['file1.txt', 'file2.txt', 'file3.txt']
    with Pool(4) as pool:
        results = pool.map(process_file, files)
    print(results)

A few suggestions:

• Control the size of Pool well and don't open too many processes, which will slow down the system.
• If there is no dependency between tasks, use map or imap first, which is more concise.
• If you need asynchronous callbacks or error handling, you can use apply_async , but remember to add join() and exception capture.

Pay attention to communication and data sharing between processes

Sometimes you need to exchange data between multiple processes, so you should pay attention to how to transmit data to be safe and efficient.

• Queue and Pipe are commonly used communication methods, Queue is more suitable for multiple consumer/producer models.
• If you just read shared data, you can use Value or Array , but the write operation must be locked.
• Another type is to use Manager , which can create shared objects that support multiple types (dictionaries, lists, etc.), but there will be some performance losses.

Let's give a simple queue example:

 from multiprocessing import Process, Queue

def worker(q):
    item = q.get()
    print(f"Processing: {item}")

if __name__ == '__main__':
    q = Queue()
    for i in range(5):
        q.put(i)

    processes = [Process(target=worker, args=(q,)) for _ in range(3)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

Remember, try to avoid frequent replication when passing big data between processes, otherwise it will affect performance. Consider using shared memory if necessary.

Don't ignore exception handling and debugging

There is a problem with a multi-process program, and debugging is much more troublesome than single-threaded ones. The most common ones are "stuck" or "silent failure".

You can do this:

• Add try-except to each child process and print out the exception.
• Use logging instead of print to facilitate viewing of the output of each process.
• If the program is stuck, check whether it is because of deadlock, queue blocking, or join() does not end.

Another trick: first use a single process to simulate the execution process during the development stage, and then switch back to multi-process mode after confirming that there is no problem.

Basically that's it. multiprocessing is powerful, but also has some complexity, especially in cross-platform and resource control. If used well, it can significantly improve program performance; if used poorly, it will increase maintenance costs.

The above is the detailed content of Running code in parallel with Python multiprocessing. For more information, please follow other related articles on the PHP Chinese website!