What is the problem with Queue thread in Go's crawler Colly?

Go crawler Colly's request queue and thread concurrency: In-depth discussion

When using the Colly crawler library of Go, it is crucial to understand its request queue and thread concurrency mechanism. This article analyzes the interaction between the number of queue threads in Colly and the request delay, and answers "The question of Queue threads in Go crawler Colly?".

We use an example to illustrate: set the queue thread count to 2, use q, _ := queue.New(2, storage) to create a queue, and add three requests. To observe the effect, set the Collector delay to 5 seconds. Intuitively, both requests should be issued almost at the same time and returned after 5 seconds; the third request is executed after 10 seconds.

However, the actual results are different:

1. Two requests are created.
2. After 5 seconds, the first request returns.
3. The third request is created.
4. After another 5 seconds, the second request returns.
5. After another 5 seconds, the third request returns.

This shows that when Colly's Collector processes the request, it will consider the overall situation of the queue, but the delay of the request itself will affect the actual execution time. The number of queue threads limits the number of concurrent requests, but if the request is set, the delay will override the concurrent limit effect of the number of threads. Each request will be delayed by another 5 seconds after the previous request is completed, rather than being processed in real parallel.

Colly's OnRequest callback function is fired when the request is created, not when the request is issued. It is mainly used for preprocessing before the request issuance, rather than controlling the time of the request issuance. The actual request issuance time is determined by the delay setting of the Collector.

Therefore, when the request is set to delay, the number of threads in the Colly queue has little impact on concurrency, and the order and time of the request are mainly controlled by the delay setting of the Collector. This helps to have a clearer understanding of Colly's queue mechanism and concurrency control.

The above is the detailed content of What is the problem with Queue thread in Go's crawler Colly?. For more information, please follow other related articles on the PHP Chinese website!