Here you can set the rate/burst parameters on the web ui console to adjust the speed. Rate is the number of crawls per second, and burst is the number of concurrencies. The default is 1/3, so it is relatively slow. I still don’t know enough about this tool
I have never used a framework to write crawlers, but when I write crawlers myself, the more complex the model (such as thread control, thread status monitoring), the lower the efficiency when writing concurrently.
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
Here you can set the rate/burst parameters on the web ui console to adjust the speed. Rate is the number of crawls per second, and burst is the number of concurrencies. The default is 1/3, so it is relatively slow. I still don’t know enough about this tool
I have never used a framework to write crawlers, but when I write crawlers myself, the more complex the model (such as thread control, thread status monitoring), the lower the efficiency when writing concurrently.