Python Web刮擦與登錄示例

使用requests.Session()發送POST請求並維護會話以登錄網站；2. 通過BeautifulSoup解析登錄頁面獲取隱藏字段如CSRF令牌；3. 構造包含用戶名、密碼和令牌的登錄數據並提交；4. 驗證登錄是否成功，檢查響應內容中的“Logout”或“Dashboard”等標誌；5. 登錄成功後用同一會話訪問受保護頁面並抓取所需內容；6. 對於JavaScript動態渲染的頁面應改用Selenium模擬瀏覽器操作；7. 始終遵守網站的robots.txt和使用條款，避免生產環境硬編碼憑證，推薦使用環境變量存儲敏感信息，最終確保scraping 行為合法合規。

If you need to scrape a website that requires login, you'll typically need to send a POST request with your credentials first, maintain the session, and then access the protected pages. Here's a practical example using Python's requests and BeautifulSoup libraries to log in and scrape a page behind authentication.

We'll use a dummy login form structure (like many real sites) and show how to handle it.

✅ 1. Required Libraries

Install the needed packages if you haven't:

 pip install requests beautifulsoup4

✅ 2. Example: Login and Scrape a Page

 import requests
from bs4 import BeautifulSoup

# Step 1: Start a session
session = requests.Session()

# Step 2: URL of the login page (example)
login_url = 'https://httpbin.org/post' # Placeholder - replace with actual login URL
target_url = 'https://example.com/dashboard' # Page you want to scrape after login

# Step 3: Get login page (to extract hidden form fields like CSRF tokens if needed)
login_page = session.get('https://example.com/login')
soup = BeautifulSoup(login_page.content, 'html.parser')

# Optional: Extract hidden inputs (eg, CSRF token)
csrf_token = soup.find('input', {'name': 'csrf_token'})['value'] # Adjust name as needed

# Step 4: Prepare login payload
payload = {
    'username': 'your_username',
    'password': 'your_password',
    'csrf_token': csrf_token # Include if present
}

# Step 5: Submit login form
response = session.post('https://example.com/login', data=payload)

# Step 6: Check if login was successful
if "Logout" in response.text or "Dashboard" in response.text:
    print("✅ Login successful")
else:
    print("❌ Login failed")
    print(response.status_code)
    print(response.text[:500]) # Debug output
    exit()

# Step 7: Scrape a protected page
protected_page = session.get(target_url)
soup = BeautifulSoup(protected_page.content, 'html.parser')

# Example: Extract page title or specific content
print("Page Title:", soup.title.string)
# Or scrape data
data = soup.find_all('div', class_='content') # Adjust selector
for item in data:
    print(item.get_text(strip=True))

✅ 3. Key Points to Remember

• Session Persistence : Use requests.Session() to keep cookies and stay logged in.
• Inspect the Login Form : Use browser DevTools (F12) to:
  • Find the correct login URL (form's action attribute)
  • Check input field names (eg, username , email , password , csrf_token )
• CSRF & Hidden Fields : Many sites require tokens — always check for hidden inputs.
• HTTPS & Security : Never hardcode credentials in production. Use environment variables:

   import os
  username = os.getenv('LOGIN_USER')
  password = os.getenv('LOGIN_PASS')

• Respect robots.txt and Terms of Service — scraping may be prohibited.

✅ 4. Real-World Example (Generic Pattern)

 import requests
from bs4 import BeautifulSoup
import os

session = requests.Session()

# Load credentials
USER = os.getenv('USERNAME', 'test@example.com')
PASS = os.getenv('PASSWORD', 'secret')

# Fetch login page
resp = session.get('https://example.com/login')
soup = BeautifulSoup(resp.text, 'html.parser')

# Extract CSRF token
token = soup.find('input', {'name': 'authenticity_token'})['value']

# Login data
data = {
    'authenticity_token': token,
    'user[email]': USER,
    'user[password]': PASS,
    'commit': 'Log in'
}

# Post to login
r = session.post('https://example.com/sessions', data=data)

# Now scrape
dashboard = session.get('https://example.com/my-account')

✅ 5. Alternative: Use Selenium for JavaScript-heavy Sites

If the login is handled by JavaScript (eg, React, Vue), use Selenium :

 from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com/login")

driver.find_element(By.NAME, "username").send_keys("your_user")
driver.find_element(By.NAME, "password").send_keys("your_pass")
driver.find_element(By.XPATH, "//button[@type='submit']").click()

# Wait and go to target page
driver.implicitly_wait(5)
driver.get("https://example.com/profile")

print(driver.page_source)
driver.quit()

Basically, for simple forms: requests Session works great. For dynamic sites: go with Selenium . Always test on a small scale and check the site's policies.

以上是Python Web刮擦與登錄示例的詳細內容。更多資訊請關注PHP中文網其他相關文章！