How to Read MySQL Tables as Spark DataFrames?

Integrating Apache Spark with MySQL for Database Table Reading

To connect Apache Spark with MySQL and leverage database tables as Spark dataframes, follow these steps:

1. Create a Spark Session:

   <code class="python">from pyspark.sql import SparkSession
   
   # Create a Spark session object
   spark = SparkSession.builder \
       .appName("Spark-MySQL-Integration") \
       .getOrCreate()</code>

   Copy after login

2. Instantiate a MySQL Connector:

   <code class="python">from pyspark.sql import DataFrameReader
   
   # Create a DataFrameReader object for MySQL connection
   jdbc_df_reader = DataFrameReader(spark)</code>

   Copy after login

3. Configure MySQL Connection Parameters:

   <code class="python"># Set MySQL connection parameters
   jdbc_params = {
       "url": "jdbc:mysql://localhost:3306/my_db",
       "driver": "com.mysql.jdbc.Driver",
       "dbtable": "my_table",
       "user": "root",
       "password": "password"
   }</code>

   Copy after login

4. Read Database Table:

   <code class="python"># Read the MySQL table as a Spark dataframe
   dataframe_mysql = jdbc_df_reader.format("jdbc") \
       .options(**jdbc_params) \
       .load()
   
   # Print the dataframe schema
   dataframe_mysql.printSchema()</code>

   Copy after login

This approach demonstrates how to integrate Apache Spark with MySQL, allowing you to access database tables as Spark dataframes.

The above is the detailed content of How to Read MySQL Tables as Spark DataFrames?. For more information, please follow other related articles on the PHP Chinese website!