How to use AWS Glue crawler with Amazon Athena

What is AWS Glue?

AWS Glue is a serverless hosting service that allows you to discover, prepare, move, and integrate data from multiple sources. As a data integration service, AWS Glue allows you to centrally manage data locations without managing infrastructure.

What is an AWS Glue crawler?

Glue crawler is an automated data discovery tool that scans data automatically classifies, groups and catalogs the data in it. It then creates a new table or updates an existing table directory in your AWS Glue data.

What is a Glue data directory?

The AWS Glue data directory is an index, schema, and runtime metrics of data locations. You need this information to create and monitor your Extract, Transform, and Load (ETL) jobs.

Why use Amazon Athena and AWS Glue?

Now that we've covered the basics of Amazon Athena, AWS Glue, and AWS Glue Crawlers, let's discuss them in a deeper way.

4 main Amazon Athena use cases

Amazon Athena provides a simplified and flexible method for analyzing petabytes of data where they are. For example, Athena can analyze data from Amazon Simple Storage Service (S3) or build application data lakes and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python.

Amazon Athena has four main use cases:

1. Run queries on S3, on-premises data centers, or other clouds

2. Prepare data for machine learning models

3. Simplify complex tasks such as anomaly detection, customer group analysis, and sales forecasting using machine learning models in SQL queries or Python

4. Perform multi-cloud analytics (such as querying data in Azure) Synapse Analytics and visualize the results with Amazon QuickSight)

3 key AWS Glue use cases

Now that we have introduced Amazon Athena, let’s talk about AWS Glue. You can use AWS Glue to do some different actions.

First, you can use the AWS Glue Data Integration Engine, which allows you to get data from several different sources. This includes Amazon S3, Amazon DynamoDB, and Amazon RDS, as well as databases EC2 (integrated with AWS Glue Studios) running on Amazon and AWS Glue for Ray, Python Shell, and Apache Spark.

Once the data is connected and filtered, it can be connected with locations where the data is loaded or created, and this list expands to places such as Amazon Redshift, data lakes, and data warehouses.

You can also use AWS Glue to run ETL jobs. These tasks allow you to isolate customer data, protect customer data rests in transmission and on-site, and access customer data requests only when responding to customer needs. When configuring an ETL job, all you need to do is provide the input data source and output data target cloud in the virtual private.

The last method of using AWS Glue is to quickly discover and search multiple AWS datasets through your data catalog without moving data. After data cataloging, it can be used immediately to search and query spectrum using Amazon Athena, Amazon EMR, and Amazon Redshift.

Getting Started with AWS Glue: How to Get Data from AWS Glue to Amazon Athena

So, how do I get data from AWS Glue into Amazon Athena? Please follow these steps:

1. First upload the data to the data source. The most popular option is the S3 bucket, but DynamoDB tables and Amazon RedShift are also options.

2. Select your data source and create a classifier if necessary. The classifier reads the data and generates a pattern (if satisfied) to identify the format. You can create custom classifiers to view different data types.

3. Create a crawler.

4. Set the name of the crawler, then select your data source and add any custom classifiers to make sure that AWS Glue recognizes the data correctly.

5. Set up the Identity and Access Management (IAM) role to ensure that the crawler runs the process correctly.

6. Creates a database that will save the dataset. Set the runtime and frequency of the crawler to keep your data up to date.

7. Run the crawler. This process can take a while, depending on how big the dataset is. After the crawler runs successfully, you will view changes to the tables in the database.

Now that you have finished this process, you can jump to Amazon Athena and run the query you need to filter the data and get the results you are looking for.