How to Efficiently Manage Memory Consumption When Using `Promise.all` with a Large Number of Promises?

Promise.all Memory Consumption: A Comprehensive Solution

In this case, the issue arises from having a substantial number (58k) of promises ready to be resolved. As each promise awaits its fulfillment, the associated async functions and result data accumulate, eventually leading to excessive memory consumption.

Efficient Promise Management

To optimize memory usage, avoid holding a large number of promises and their data in memory simultaneously. Instead, limit the number of concurrent operations (promises) to a reasonable value (referred to as "X"). For example, if the rate limiter allows 20 requests per second, setting X to 5 or 10 could be a suitable compromise.

Implementing Concurrency Control

There are multiple approaches to enforcing concurrency control:

1. Promise.map() with Concurrency Option:
Bluebird's Promise.map() provides a concurrency option that allows specifying the maximum number of concurrent operations. This simplifies the process of managing concurrency.

2. Manual Promise Management:
Alternatively, you can implement custom code to manage concurrency. The example below demonstrates this approach:

function mapConcurrent(items, maxConcurrent, fn) {
    // Limit concurrent operations to `maxConcurrent`
    let inFlightCntr = 0;

    return new Promise((resolve, reject) => {
        function runNext() {
            let i = index;
            let prom = fn(items[index], index++).then(...);
            inFlightCntr++;
            prom.finally(() => { inFlightCntr--; });
            prom.then(...); // Run resolve/reject based on returned promise
        }

        function run() {
            // Process items while count is below the limit
            while (inFlightCntr < maxConcurrent && index < items.length) {
                runNext();
            }

            // Resolve or reject parent promise based on completion
        }

        run();
    });
}

3. Placeholder Replacement:

If you do not require the resolved data, you can hasten its garbage collection by replacing it with a simple placeholder, such as a number:

const p = backgroundScheduler.getClanProfile(clanTags[i], true).then(data => {
    return 0;     // Replace resolved value to promote garbage collection
});
promiseArray.push(p);

The above is the detailed content of How to Efficiently Manage Memory Consumption When Using `Promise.all` with a Large Number of Promises?. For more information, please follow other related articles on the PHP Chinese website!