What is a python distributed process? (Example analysis)

In this article, let’s take a look at what python distributed process is. Learn about python distributed processes and what role distributed processes can play in python programming.

Among Thread and Process, Process should be preferred because Process is more stable and Process can be distributed to multiple machines, while Thread can only be distributed to multiple CPUs on the same machine at most.

Python's multiprocessing module not only supports multiple processes, but the managers submodule also supports distributing multiple processes to multiple machines. A service process can act as a scheduler, distributing tasks to multiple other processes, relying on network communication. Since the managers module is well encapsulated, you can easily write distributed multi-process programs without knowing the details of network communication.

For example: If we already have a multi-process program that communicates through Queue running on the same machine, now, because the process that handles the task has a heavy workload, we want to separate the process that sends the task and the process that handles the task. Distributed to two machines. How to implement it using distributed process?

The original Queue can continue to be used, but by exposing the Queue through the network through the managers module, processes on other machines can access the Queue.

Let’s look at the service process first. The service process is responsible for starting the Queue, registering the Queue on the network, and then writing tasks into the Queue:

# task_master.py

import random, time, queue
from multiprocessing.managers import BaseManager
# 发送任务的队列:
task_queue = queue.Queue()
# 接收结果的队列:
result_queue = queue.Queue()
# 从BaseManager继承的QueueManager:
class QueueManager(BaseManager):
    pass
# 把两个Queue都注册到网络上, callable参数关联了Queue对象:
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000, 设置验证码'abc':
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue:
manager.start()
# 获得通过网络访问的Queue对象:
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去:
for i in range(10):
    n = random.randint(0, 10000)
    print('Put task %d...' % n)
    task.put(n)
# 从result队列读取结果:
print('Try get results...')
for i in range(10):
    r = result.get(timeout=10)
    print('Result: %s' % r)
# 关闭:
manager.shutdown()

Please note that when we are on a machine When writing a multi-process program, the created Queue can be used directly. However, in a distributed multi-process environment, adding tasks to the Queue cannot directly operate on the original task_queue, which bypasses the encapsulation of the QueueManager and must be passed The Queue interface obtained by manager.get_task_queue() is added.

Then, start the task process on another machine (it can also be started on this machine):

# task_master.py
import random, time, queue
from multiprocessing.managers import BaseManager
# 发送任务的队列:
task_queue = queue.Queue()
# 接收结果的队列:
result_queue = queue.Queue()
# 从BaseManager继承的QueueManager:
class QueueManager(BaseManager):
    pass
# 把两个Queue都注册到网络上, callable参数关联了Queue对象:
QueueManager.register('get_task_queue', callable=lambda: task_queue)
QueueManager.register('get_result_queue', callable=lambda: result_queue)
# 绑定端口5000, 设置验证码'abc':
manager = QueueManager(address=('', 5000), authkey=b'abc')
# 启动Queue:
manager.start()
# 获得通过网络访问的Queue对象:
task = manager.get_task_queue()
result = manager.get_result_queue()
# 放几个任务进去:
for i in range(10):
    n = random.randint(0, 10000)
    print('Put task %d...' % n)
    task.put(n)
# 从result队列读取结果:
print('Try get results...')
for i in range(10):
    r = result.get(timeout=10)
    print('Result: %s' % r)
# 关闭:
manager.shutdown()

The task process needs to connect to the service process through the network, so the IP of the service process must be specified. .

Now, you can try the working effect of the distributed process. First start the task_master.py service process:

$ python3 task_master.py 
Put task 3411...
Put task 1605...
Put task 1398...
Put task 4729...
Put task 5300...
Put task 7471...
Put task 68...
Put task 4219...
Put task 339...
Put task 7866...
Try get results...

After the task_master.py process sends the task, it starts waiting for the results of the result queue. Now start the task_worker.py process:

$ python3 task_worker.pyConnect to server 127.0.0.1...
run task 3411 * 3411...
run task 1605 * 1605...
run task 1398 * 1398...
run task 4729 * 4729...
run task 5300 * 5300...
run task 7471 * 7471...
run task 68 * 68...
run task 4219 * 4219...
run task 339 * 339...
run task 7866 * 7866...
worker exit.

task_worker.py process ends, and the results will continue to be printed in the task_master.py process:

Result: 3411 * 3411 = 11634921
Result: 1605 * 1605 = 2576025
Result: 1398 * 1398 = 1954404
Result: 4729 * 4729 = 22363441
Result: 5300 * 5300 = 28090000
Result: 7471 * 7471 = 55815841
Result: 68 * 68 = 4624
Result: 4219 * 4219 = 17799961
Result: 339 * 339 = 114921
Result: 7866 * 7866 = 61873956

What is the use of this simple Master/Worker model? In fact, this is a simple but truly distributed computing. By slightly modifying the code and starting multiple workers, the tasks can be distributed to several or even dozens of machines. For example, the code for calculating n*n can be replaced by sending emails. , realizing asynchronous sending of mail queues.

The reason why Queue can be accessed through the network is through QueueManager. Since QueueManager manages more than one Queue, it is necessary to give the network call interface of each Queue a name, such as get_task_queue.

What is the use of authkey? This is to ensure that the two machines communicate normally and are not maliciously interfered by other machines. If the authkey of task_worker.py is inconsistent with the authkey of task_master.py, the connection will definitely fail.

Python’s distributed process interface is simple and well-encapsulated, making it suitable for environments where heavy tasks need to be distributed to multiple machines.

Note that the role of Queue is to deliver tasks and receive results. The amount of data describing each task should be as small as possible. For example, when sending a task to process a log file, instead of sending the log file itself, which is several hundred megabytes in size, send the complete path where the log file is stored, and the Worker process will then read the file from the shared disk.

The above is all the content described in this article. This article mainly introduces the relevant knowledge of python distributed process. I hope you can use the information to understand the above content. I hope what I have described in this article will be helpful to you and make it easier for you to learn python.

For more related knowledge, please visit the Python tutorial column on the php Chinese website.

The above is the detailed content of What is a python distributed process? (Example analysis). For more information, please follow other related articles on the PHP Chinese website!