How to Limit Concurrent Goroutine Execution in Go?

Limiting Concurrent Goroutine Execution

Consider a scenario where you have a list of URLs to process and want to limit the number of concurrent goroutines running. For instance, if you have 30 URLs, you may only want 10 goroutines working in parallel.

The provided code attempts to use a buffered channel of size parallel to limit the number of goroutines running. However, this approach doesn't seem to block after processing all the URLs. A more effective way to achieve this concurrency limitation is to create a specified number of worker goroutines and feed them URLs through a dedicated channel.

Here's an improved version of the code:

<code class="go">parallel := flag.Int("parallel", 10, "max parallel requests allowed")
flag.Parse()
urls := flag.Args()

// Create a channel to hold URLs that workers will consume
workerURLChan := make(chan string)

// Start a goroutine to feed URLs to the workers
go func() {
    for _, u := range flag.Args() {
        workerURLChan <- u
    }

    // Once all URLs have been distributed, close the channel, which will cause workers to exit
    close(workerURLChan)
}()

var wg sync.WaitGroup
client := rest.Client{}

results := make(chan string)

// Start the specified number of worker goroutines
for i := 0; i < *parallel; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()

        // Workers repeatedly fetch URLs from the channel until it is closed
        for url := range workerURLChan {
            worker(url, client, results)
        }
    }()
}

// Close the results channel when all workers have completed, which will allow the main goroutine to exit
go func() {
    wg.Wait()
    close(results)
}()

// Receive and print results from the worker goroutines
for res := range results {
    fmt.Println(res)
}</code>

In this updated code, we create a worker goroutine for each allowed concurrent execution, and these workers fetch URLs from the dedicated channel. Once all URLs have been distributed, the workerURLChan is closed, which triggers the workers to exit when their current URL is completed. This mechanism effectively limits the number of concurrent goroutines running.

The above is the detailed content of How to Limit Concurrent Goroutine Execution in Go?. For more information, please follow other related articles on the PHP Chinese website!