這是單一進程順序執行的程式碼:
import requests,time,os,random def img_down(url): with open("{}".format(str(random.random())+os.path.basename(url)),"wb") as fob: fob.write(requests.get(url).content) urllist=[] with open("urllist.txt","r+") as u: for a in u.readlines(): urllist.append(a.strip()) s=time.clock() for i in range(len(urllist)): img_down(urllist[i]) e=time.clock() print ("time: %d" % (e-s))
這是多進程的程式碼:
from multiprocessing import Pool import requests,os,time,random def img_down(url): with open("{}".format(str(random.random())+os.path.basename(url)),"wb") as fob: fob.write(requests.get(url).content) if __name__=="__main__": urllist=[] with open("urllist.txt","r+") as urlfob: for s in urlfob.readlines(): urllist.append(s.strip()) s=time.clock() p=Pool() for i in range(len(urllist)): p.apply_async(img_down,args=(urllist[i],)) p.close() p.join() e=time.clock() print ("time: {}".format(e-s))
但是單一進程和多進程花費的時間幾乎沒差別,問題大概是requests阻塞IO,請問理解的對不對,程式碼該怎麼修改達到多進程的目的?
謝謝!
寫檔案的瓶頸在磁碟IO,並不在CPU,你並行並沒有多大作用,你可以試試不要寫入檔案再對比時間
Pool 不帶參數的話 是採用
os.cpu_count() or 1
如果是單核心CPU,或是採集不到數量 就只有1個進程而已。
應該是這個原因。