How to use the pandas library for data manipulation in Python?

Pandas is a powerful Python library for data manipulation and analysis using Series and DataFrame structures. 1. Import and load data from CSV, Excel, JSON, or create manually with pd.DataFrame(). 2. Explore data using head(), tail(), info(), describe(), shape, and columns. 3. Select and filter data by column, row index (loc/iloc), or conditions with boolean indexing. 4. Handle missing values using isnull(), dropna(), or fillna() with mean or custom values. 5. Add columns with conditional logic or remove/rename them using drop() and rename(). 6. Transform data using apply(), map(), replace(), and astype() for cleaning and type conversion. 7. Sort data with sort_values() and group by categories using groupby() with aggregation functions like mean() or count(). 8. Merge DataFrames with merge() or concatenate with concat() along rows or columns. 9. Save processed data to CSV, Excel, or JSON using to_csv(), to_excel(), or to_json() with index=False if needed. Mastering loc, groupby, and apply enables efficient handling of most real-world data tasks, making pandas essential for data workflows.

Pandas is one of the most powerful and widely used libraries in Python for data manipulation and analysis. It provides easy-to-use data structures like Series (1D) and DataFrame (2D), along with a wide range of functions to clean, filter, transform, and analyze data efficiently.

Here’s a practical guide on how to use pandas for common data manipulation tasks:

1. Importing and Loading Data

Start by importing pandas and loading data from common formats like CSV, Excel, or JSON.

import pandas as pd

# Load data from a CSV file
df = pd.read_csv('data.csv')

# Load from Excel (requires openpyxl)
df = pd.read_excel('data.xlsx')

# Load from JSON
df = pd.read_json('data.json')

You can also create a DataFrame manually:

data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'City': ['New York', 'Los Angeles', 'Chicago']
}
df = pd.DataFrame(data)

2. Exploring the Data

Before manipulating, inspect your data to understand its structure.

# Display first 5 rows
df.head()

# Last 3 rows
df.tail(3)

# General info: data types, missing values, memory usage
df.info()

# Summary statistics for numeric columns
df.describe()

# Shape of the DataFrame (rows, columns)
df.shape

# Column names
df.columns

3. Selecting and Filtering Data

Access specific parts of the DataFrame using labels, positions, or conditions.

# Select a single column
df['Name']

# Select multiple columns
df[['Name', 'Age']]

# Select rows by index
df.loc[0]           # by label
df.iloc[0]          # by position

# Filter rows based on condition
df[df['Age'] > 28]

# Multiple conditions (use & for AND, | for OR, parentheses required)
df[(df['Age'] > 25) & (df['City'] == 'Chicago')]

4. Handling Missing Data

Real-world data often has missing values (NaN). Pandas provides tools to manage them.

# Check for missing values
df.isnull()

# Count missing values per column
df.isnull().sum()

# Drop rows with any missing values
df.dropna()

# Drop columns with more than 50% missing
df.dropna(thresh=len(df)*0.5, axis=1)

# Fill missing values
df['Age'].fillna(df['Age'].mean(), inplace=True)
df['City'].fillna('Unknown', inplace=True)

5. Adding and Removing Columns

Modify the structure of your DataFrame as needed.

# Add a new column
df['Senior'] = df['Age'] > 30

# Remove a column
df.drop('Senior', axis=1, inplace=True)

# Rename columns
df.rename(columns={'Name': 'Full Name'}, inplace=True)

6. Data Transformation

Apply functions to transform or clean data.

# Apply a function to a column
df['Age'] = df['Age'].apply(lambda x: x   1)  # Increment age

# Map values (e.g., replace city names)
df['City'] = df['City'].map({'New York': 'NYC', 'Los Angeles': 'LA'})

# Replace specific values
df.replace({'NYC': 'New York City'}, inplace=True)

# Convert data types
df['Age'] = df['Age'].astype(int)

7. Sorting and Grouping

Organize and summarize data effectively.

# Sort by one or more columns
df.sort_values('Age', ascending=False)

# Sort by multiple columns
df.sort_values(['City', 'Age'], ascending=[True, False])

# Group data and aggregate
grouped = df.groupby('City')['Age'].mean()  # Average age by city
grouped = df.groupby('City').agg({'Age': 'mean', 'Name': 'count'})  # Multiple stats

8. Merging and Concatenating Data

Combine multiple DataFrames, similar to SQL joins.

# Concatenate vertically (stack rows)
df_combined = pd.concat([df1, df2], axis=0)

# Concatenate horizontally (add columns)
df_combined = pd.concat([df1, df2], axis=1)

# Merge on a key (like SQL join)
merged = pd.merge(df1, df2, on='Name', how='inner')  # inner, left, right, outer

9. Saving Data

After manipulation, export the result.

# Save to CSV
df.to_csv('cleaned_data.csv', index=False)

# Save to Excel
df.to_excel('cleaned_data.xlsx', index=False)

# Save to JSON
df.to_json('cleaned_data.json', orient='records')

Pandas makes data manipulation intuitive and efficient. Start with small datasets to practice these operations, and gradually apply them to real-world problems. The key is to become familiar with indexing, filtering, and aggregation patterns — they form the backbone of most data workflows.

Basically, once you get comfortable with loc, groupby, and apply, you can handle most day-to-day data tasks.

The above is the detailed content of How to use the pandas library for data manipulation in Python?. For more information, please follow other related articles on the PHP Chinese website!