I recently encountered a scenario where I needed to download extensive logs from MongoDB Atlas Triggers locally. Currently, there are three ways to download logs from Atlas:
However, the GUI and CLI options have limitations regarding the volume of logs that can be downloaded, particularly a cap of 10,000 logs.
Using the GUI
With the GUI, users can filter logs by date, type, user ID, or request ID, but the limit is set to 10,000 logs when it comes to downloading.
Using the CLI
With the CLI, we can run a command like:
appservices logs list --project 5e208aa2d5ec1375ecd5*** --app triggers_realmapp-**** --type=trigger --start="2024-10-15T00:00:00.000+0000" -o log.logs
However, this also has the same download limit of 10,000 logs.
To overcome these download limitations, the App Services Admin API provides a way to access logs with pagination. By implementing pagination, users can fetch logs beyond the default 10K limit.
Detailed instructions on using pagination with the API can be found in MongoDB’s documentation: Get Paginated Logs.
To streamline this, I developed a script that automatically fetches logs using pagination. This script is available in a public repository here: Atlas App Logs Aggregator.
The script only uses the GET endpoint and aggregates logs into a file without modifying any data.
appservices logs list --project 5e208aa2d5ec1375ecd5*** --app triggers_realmapp-**** --type=trigger --start="2024-10-15T00:00:00.000+0000" -o log.logs
python3 -m venv venv source venv/bin/activate # On Windows use `venv\Scripts\activate`
pip install -r requirements.txt
With optional parameters
python main.py <project_id> <app_id> <public_api_key> <private_api_key> --start_date 2024-10-05T14:30:00.000Z --end_date 2024-10-06T14:30:00.000Z --type TRIGGER_FAILURE,SCHEMA_GENERATION
If start_date and end_date are not provided, the script will default start_date to the last 24 hours from the current time.
The --filter option allows you to filter logs by key-value pairs. This option accepts multiple key-value pairs separated by spaces. Each key-value pair should be in the format key=value.
The key-value pair must be the values returned by the endpoint. This way it will use them to filter and only keep those that match. For example, for a "type": "SCHEDULED_TRIGGER", the response key-values will be similar to:
python main.py <project_id> <app_id> <public_api_key> <private_api_key> --start_date 2024-10-05T14:30:00.000Z --type TRIGGER_FAILURE,SCHEMA_GENERATION --user_id 671d2e2010733ecbaa2bab8f --filter event_subscription_name=getUnpausedClustersMetrics
We can use any of this in the --filter option (e.g., --filter event_subscription_name=getUnpausedClustersMetrics)
The script supports logging to both the console and a log file. By default, log files are stored in the logs folder. The log file name includes a timestamp to ensure uniqueness for each run.
--verbose: When this flag is used, the log level is set to DEBUG, providing detailed logging information. Without this flag, the log level is set to INFO.
Log files are stored in the logs folder. Each log file is named with a timestamp to ensure that logs from different runs do not overwrite each other.
appservices logs list --project 5e208aa2d5ec1375ecd5*** --app triggers_realmapp-**** --type=trigger --start="2024-10-15T00:00:00.000+0000" -o log.logs
Please note: This repo is released for use "AS IS" without any warranties of any kind, including, but not limited to their installation, use, or performance. We disclaim any and all warranties, either express or implied, including but not limited to any warranty of noninfringement, merchantability, and/ or fitness for a particular purpose. We do not warrant that the technology will meet your requirements, that the operation thereof will be uninterrupted or error-free, or that any errors will be corrected.
Any use of these scripts and tools is at your own risk. There is no guarantee that they have been through thorough testing in a comparable environment and we are not responsible for any damage or data loss incurred with their use.
You are responsible for reviewing and testing any scripts you run thoroughly before use in any non-testing environment.
The above is the detailed content of Automating MongoDB Atlas Trigger Log Downloads Beyond the GUI and CLI Limitations. For more information, please follow other related articles on the PHP Chinese website!