这是单进程顺序执行的代码:
import requests,time,os,random def img_down(url): with open("{}".format(str(random.random())+os.path.basename(url)),"wb") as fob: fob.write(requests.get(url).content) urllist=[] with open("urllist.txt","r+") as u: for a in u.readlines(): urllist.append(a.strip()) s=time.clock() for i in range(len(urllist)): img_down(urllist[i]) e=time.clock() print ("time: %d" % (e-s))
这是多进程的代码:
from multiprocessing import Pool import requests,os,time,random def img_down(url): with open("{}".format(str(random.random())+os.path.basename(url)),"wb") as fob: fob.write(requests.get(url).content) if __name__=="__main__": urllist=[] with open("urllist.txt","r+") as urlfob: for s in urlfob.readlines(): urllist.append(s.strip()) s=time.clock() p=Pool() for i in range(len(urllist)): p.apply_async(img_down,args=(urllist[i],)) p.close() p.join() e=time.clock() print ("time: {}".format(e-s))
但是单进程和多进程花费的时间几乎没区别,问题大概是requests阻塞IO,请问理解的对不对,代码该怎么修改达到多进程的目的?
谢谢!
写文件的瓶颈在磁盘IO,并不在CPU,你并行并没有多大作用,你可以试试不要写入文件再对比时间
Pool 不带参数的话 是采用
os.cpu_count() or 1
如果是单核CPU,或者采集不到数量 就只有1个进程而已。
应该是这个原因。