Apabila menulis crawler, saya ingin menyimpan data yang dirangkak ke dalam pangkalan data Terdapat banyak entri dalam setiap halaman Sebagai contoh, seseorang mungkin mempunyai ramai pelawat, jadi saya memasukkannya ke dalam gelung pemunggahan,

                try:
                    sql_visitor='INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ("%s",%d,"%s",%d,"%s")'%(ownername,owneruid,visitorname,visitoruid,visitortime)
                    print sql_visitor
                    self.cursor.execute(sql_visitor)
                    self.connect.commit()
                except Exception as e:
                    print e

Satu halaman dan satu utas saya tidak menyukai kelambatan dan membuka 5 daripadanya

max_threads=5
        while uid < 8000000 or threadlist:
            for thread1 in threadlist:
                if not thread1.is_alive():
                    threadlist.remove(thread1)
            while len(threadlist) < max_threads and uid < 8000000:
                uid+=1
                thread2=threading.Thread(target=run,args=(uid,))
                thread2.setDaemon(True)
                thread2.start()
                threadlist.append(thread2)
            time.sleep(5)

Berjalan dengan lancar:

MASUKKAN KE DALAM pelawat (nama pemilik,pemilik,nama pelawat,pelawat,masa lawatan) NILAI ("huosai7",4893,"Liang2017",7252799,"2017-5-22 21:06")
MASUKKAN KE DALAM nama peribadi,maklumat peribadi jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangcifabiaoshijian,shangcihuodongshijian,zuihoufangwen,zhuceshijian,7bieshijian,) 93 , 0,0,0,0,0,0,0,0,0,0,0,0,"","","2100-01-01 12:00","2100-01-01 12: 00 ","2100-01-01 12:00","2004-1-3 19:28",0,"2100-01-01 12:00",0)
MASUKKAN KE DALAM pelawat (nama pemilik,owneruid,nama pelawat , visitoruid,masa lawatan) NILAI ("龙乐",4894,"Liang2017",7252799,"2017-5-22 21:06")
(1062, "Pendua masukan '4894-7252799-2017-2799-2017-05:-2017-05:-2017 06:00' untuk kunci 'PRIMER'")
MASUKKAN KE DALAM maklumat peribadi (nama pemilik,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,shengbiao,shijingchu,shengbiao,shijing shangcihuodongshijian ,zuihoufangwen,zhuceshijian,zaixianshijian,shegri,xingbie) NILAI("龙乐",4894,0,0,0,0,0,0,0,0,0,0,0,0,"","" , "2100-01-01 12:00","2100-01-01 12:00","2100-01-01 12:00","2004-1-3 20:21",0,"2100- 01 -01 12:00",0)
.....

Jadi saya tetapkan max_thread kepada 10, dan hasilnya adalah seperti berikut:

MASUKKAN KE DALAM pelawat (nama pemilik,owneruid,nama pelawat,visitoruid,masa lawatan) NILAI ("xiao61",4889,"Liang2017",7252799,"2017-5-22 21:06")

(2006, 'Pelayan MySQL telah hilang')

MASUKKAN KE DALAM maklumat peribadi (nama pemilik,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangjishangcifabiaoshuixohijiances,shangshifabiaoshuixanihoces,shangshifabiaosjijihodongces hi jian, shengri, xingbie ) NILAI("xiao61",4889,0,0,0,0,0,0,0,0,0,0,0,0,"","","2100-01-01 12:00" , "2100-01-01 12:00","2100-01-01 12:00","2004-1-3 15:56",0,"2100-01-01 12:00",0)

(2006, 'Pelayan MySQL telah hilang')

MASUKKAN KE DALAM pelawat (nama pemilik,owneruid,nama pelawat,visitoruid,masa lawatan) NILAI ("confused cool bear",4897,"Liang2017",7252799,"2017-5-22 21:06")

(2006, 'Pelayan MySQL telah hilang')

MASUKKAN KE DALAM maklumat peribadi (nama pemilik,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangjishangcifabiaoshuixohijiances,shangshifabiaoshuixanihoces,shangshifabiaosjijihodongces hai jian, shengri, xingbie ) NILAI("beruang sejuk keliru",4897,611,0,1655,0,0,2,0,0,0,34,0,0,"","","2007-3-27 00: 37","2007-3-27 00:37","2007-3-27 00:37","2004-1-3 21:08",0,"2100-01-01 12:00",1 )

(2006, 'Pelayan MySQL telah hilang')
......

Anda dapat melihat bahawa 2006 keluar, dan kemudian saya menetapkan max_thread kepada 30, dan hasilnya adalah seperti berikut:

Itu sahaja, adakah ia cukup terperinci, apa lagi yang anda perlukan?