For a fun web scraping project, I want to collect NHL data from ttps://www.nhl.com/stats/teams.
There is a clickable Excel export tag and I can find it usingselenium
andbs4
.
Unfortunately, things end here: I can't seem to access the data since there is nohref
attribute.
I got what I wanted by simulating a mouse click usingpynput
, but I want to know:
What could I have done differently? If it feels awkward.
-> Labels with export icons can be found here:
a class="styles__ExportIcon-sc-16o6kz0-0 dIDMgQ"
-> This is my code
`import pynput from pynput.mouse import Button, Controller import time from bs4 import BeautifulSoup from selenium import webdriver driver = webdriver.Chrome(executable_path = 'somepath\chromedriver.exe') URL = 'https://www.nhl.com/stats/teams' driver.get(URL) html = driver.page_source # DOM with JavaScript execution complete soup = BeautifulSoup(html) body = soup.find('body') print(body.prettify()) mouse = Controller() time.sleep(5) # Sleep for 5 seconds until page is loaded mouse.position = (1204, 669) # thats where the icon is on my screen mouse.click(Button.left, 1) # executes download`
There is no
href
attribute, and the download is triggered through JS. When usingselenium
find your element and use.click()
to download the file:Use the
css selector here
to get theof direct children
or by ending with # The class starting with ##styles__ExportIconselect it directly:
Example You may need to deal with the onetrust banner, so click on it first and then download the table.