How to efficiently obtain more than 100 user lists through GitHub API (pagination tutorial)

This tutorial aims to solve the default 100 user limit issue encountered when using the GitHub API to get a list of users. We will introduce two main paging strategies in detail: utilizing the built-in paginate method of the Octokit library to implement automated paging, and manually implementing circular paging logic based on the since parameter. The article will provide clear code examples and highlight considerations for choosing appropriate methods in different scenarios, especially clarifying that the /users endpoint does not directly support username string filtering.

Overview of GitHub API Pagination Mechanism

When using the GitHub REST API to obtain a resource list, such as obtaining a user list (GET /users), the API usually sets a default upper limit on the number of returns, a common one is 100 records per page. This means that if you need to get more than 100 users, you must use paging mechanism.

For the GET /users endpoint, the GitHub API provides a parameter called since. This parameter accepts a user ID, and the API will return all users with an ID greater than the user ID, thereby achieving the purpose of obtaining subsequent data "starting from a certain point". By continuously updating the since parameter to the ID of the last user in the results on the previous page, we can gradually traverse the entire user list.

It is important to note that the GET /users endpoint returns users in ascending order of user IDs. It does not provide the function of fuzzy matching or filtering directly through username strings. If your goal is to search for users based on their names containing a specific string, you should use the GET /search/users endpoint, which supports more complex queries via the q parameter, such as q=username_substring in:login. This tutorial will focus on troubleshooting pagination issues with the GET /users endpoint.

Use octokit.paginate for efficient pagination (recommended)

For JavaScript/TypeScript projects that use the octokit library to interact with the GitHub API, the most recommended and convenient way to paginate is to use its built-in paginate method. The octokit.paginate method can automatically handle the logic of subsequent requests, including updating the since parameter, looping requests until all data is obtained, and aggregating all results into an array for return. This greatly simplifies writing pagination code.

The following is sample code using the octokit.paginate method:

 import { Octokit } from "octokit";

// Initialize the Octokit instance. If authentication is required, you can pass in the auth token here.
const octokit = new Octokit({
  // auth: 'YOUR_GITHUB_TOKEN' // If you need to access private data or increase the rate limit, please provide the authentication token
});

async function getAllGitHubUsers() {
  try {
    console.log("Getting all GitHub users through octokit.paginate...");

    // Use the paginate method to get all users // The per_page parameter specifies the number returned per page, the maximum is 100
    const allUsers = await octokit.paginate("GET /users", {
      per_page: 100, // Request 100 users per page headers: {
        "X-GitHub-Api-Version": "2022-11-28", // Specify API version},
    });

    console.log(`Successfully obtained ${allUsers.length} GitHub users.`);
    // console.log(allUsers); // Print all user data return allUsers;

  } catch (error) {
    console.error("An error occurred while obtaining the GitHub user:", error);
    throw error;
  }
}

// Call the function getAllGitHubUsers();

Code analysis:

• new Octokit({}): Create an Octokit instance. Depending on your needs, you can pass in an authentication token (auth) to increase the API rate limit or access protected resources.
• octokit.paginate("GET /users", { ... }): This is the core call. The first parameter is the API endpoint path. The second parameter is an object used to specify request parameters, such as per_page (number fetched per page, maximum 100) and headers (set API version, etc.).
• The paginate method will automatically process requests for all pages and accumulate all results into an array and return them. When the list length returned by the API is less than per_page, paginate will consider that the end of the list has been reached and stop the request.

Manually implement paging logic

If you are not using the Octokit library, or wish to have more fine-grained control over the paging process, you can manually implement the looping paging logic. This approach involves calling the API repeatedly in a loop, updating the since parameter with each request, until all data is obtained.

The following is the simulation code to manually implement the paging logic:

 // This is a simulation implementation to illustrate the logic of manual paging // The actual production environment should be replaced with a real API request library (such as axios, fetch, etc.)

async function getAllGitHubUsersManually() {
  let allUsers = [];
  let lastUserId = 0; //Initially start from the first user (ID is 0 or less)
  const perPage = 100; // Number of requests per page let hasMoreUsers = true;

  console.log("Manually retrieving all GitHub users...");

  while (hasMoreUsers) {
    try {
      // Simulate API request // In actual application, this would be `await octokit.request('GET /users', { since: lastUserId, per_page: perPage, ... })`
      // Or `await fetch(`https://api.github.com/users?since=${lastUserId}&per_page=${perPage}`, { ... })`
      const response = await mockApiRequest(lastUserId, perPage); // Mock function const currentUsers = response.data.users; // Assume the data structure returned by the API if (currentUsers && currentUsers.length > 0) {
        allUsers = allUsers.concat(currentUsers);
        //Update lastUserId to the ID of the last user in the current batch
        lastUserId = currentUsers[currentUsers.length - 1].id;

        // If the number of users returned is less than perPage, it means the end of the list has been reached if (currentUsers.length  user.id > sinceId);
  if (startIndex === -1) {
    return { data: { users: [] } }; // No more users}

  const usersSlice = mockUsers.slice(startIndex, startIndex perPage);
  return { data: { users: usersSlice } };
}

// Call the function getAllGitHubUsersManually();

Code analysis:

• allUsers: used to store all obtained user data.
• lastUserId: records the ID of the last user in the previous request, used for the since parameter of the next request. The initial value is 0, which means starting from the user with the smallest ID.
• while (hasMoreUsers): Loop condition to continue the request as long as there may be more users.
• In each loop, construct an API request with since and per_page parameters.
• Add the user data returned by the current request to the allUsers array.
• Update lastUserId to the ID of the last user in the current batch.
• if (currentUsers.length

Things to note

1. API rate limiting (Rate Limiting): GitHub API has strict limits on request frequency. Frequent requests may result in your IP being temporarily banned. Be especially careful when using manual paging to ensure appropriate delays between requests, or use authentication tokens to increase the rate limit. octokit.paginate generally manages these limitations better.
2. Error handling: In actual applications, complete error handling must be performed on API requests, such as network errors, error status codes returned by the API, etc.
3. The difference between GET /users and GET /search/users: As mentioned earlier, GET /users is used to get the user list in order of ID.

The above is the detailed content of How to efficiently obtain more than 100 user lists through GitHub API (pagination tutorial). For more information, please follow other related articles on the PHP Chinese website!