Briefly talk about multi-process in python-Python Tutorial-php.cn

Briefly talk about multi-process in python

高洛峰

Release： 2017-02-22 10:43:11

Original

1313 people have browsed it

The multiprocessing module is one of the most advanced and powerful modules in the python library. This article will give you a brief introduction to the general skills of multiprocessing

The process is managed by the system itself.

1: The most basic way of writing

from multiprocessing import Pool

def f(x):
  return x*x

if __name__ == &#39;__main__&#39;:
  p = Pool(5)
  print(p.map(f, [1, 2, 3]))
[1, 4, 9]

Copy after login

2. In fact, the process is generated through the os.fork method

## In #unix, all processes are generated through the fork method.

multiprocessing Process
os

info(title):
  title
  , __name__
  (os, ): , os.getppid()
  , os.getpid()

f(name):
  info()
  , name

__name__ == :
  info()
  p = Process(=f, =(,))
  p.start()
  p.join()

Copy after login

3. Thread shared memory

threading

run(info_list,n):
  info_list.append(n)
  info_list

__name__ == :
  info=[]
  i ():
    p=threading.Thread(=run,=[info,i])
    p.start()
[0]
[0, 1]
[0, 1, 2]
[0, 1, 2, 3]
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4, 5]
[0, 1, 2, 3, 4, 5, 6]
[0, 1, 2, 3, 4, 5, 6, 7]
[0, 1, 2, 3, 4, 5, 6, 7, 8]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Copy after login

The process does not share memory:

multiprocessing Process
run(info_list,n):
  info_list.append(n)
  info_list

__name__ == :
  info=[]
  i ():
    p=Process(=run,=[info,i])
    p.start()
[1]
[2]
[3]
[0]
[4]
[5]
[6]
[7]
[8]
[9]

Copy after login

If you want to share memory, you need to use the Queue in the multiprocessing module

multiprocessing Process, Queue
f(q,n):
  q.put([n,])

__name__ == :
  q=Queue()
  i ():
    p=Process(=f,=(q,i))
    p.start()
  :
    q.get()

Copy after login

4, Lock: only for screen sharing, because the process is independent, it is not useful for multiple processes

multiprocessing Process, Lock
f(l, i):
  l.acquire()
  , i
  l.release()

__name__ == :
  lock = Lock()

  num ():
    Process(=f, =(lock, num)).start()
hello world 0
hello world 1
hello world 2
hello world 3
hello world 4
hello world 5
hello world 6
hello world 7
hello world 8
hello world 9

Copy after login

5. Inter-process memory sharing: Value, Array

multiprocessing Process, Value, Array

f(n, a):
  n.value = i ((a)):
    a[i] = -a[i]

__name__ == :
  num = Value(, )
  arr = Array(, ())

  num.value
  arr[:]

  p = Process(=f, =(num, arr))
  p.start()
  p.join()
0.0
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
3.1415927
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]

Copy after login

#manager shared method, but slow

multiprocessing Process, Manager

f(d, l):
  d[] = d[] = d[] = l.reverse()

__name__ == :
  manager = Manager()

  d = manager.dict()
  l = manager.list(())

  p = Process(=f, =(d, l))
  p.start()
  p.join()

  d
  l
# print &#39;-------------&#39;这里只是另一种写法
# print pool.map(f,range(10))
{0.25: None, 1: &#39;1&#39;, &#39;2&#39;: 2}
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Copy after login

#Async: This This writing method is not used much

multiprocessing Pool
time
f(x):
  x*x
  time.sleep()
  x*x

__name__ == :
  pool=Pool(=)
  res_list=[]
  i ():
    res=pool.apply_async(f,[i])  res_list.append(res)

  r res_list:
    r.get(timeout=10) #超时时间

Copy after login

The synchronization is apply

For more articles related to simply talking about multi-process in python, please pay attention to PHP Chinese website!