A complete guide to Web Scraping with Selenium & Python in 2024

Looking to extract data from a webpage?

Head over to Nanonets website scraper, Add the URL and click "Scrape," and download the webpage text as a file instantly. Try it for free now.‌

What is Selenium Web Scraping?

Web scraping is the process of extracting data from websites. It is a powerful technique that revolutionizes data collection and analysis. With vast online data, web scraping has become an essential tool for businesses and individuals.

Selenium is an open-source web development tool used to automate web browsing functions. It was developed in 2004 and is mainly used to automatically test websites and apps across various browsers, but it has now become a popular tool for web scraping. Selenium can be used with multiple programming languages, including Python, Java, and C#. It provides robust APIs for web page interaction, including navigating, clicking, typing, and scrolling.

Selenium web scraping refers to using the Selenium browser automation tool with Python to extract data from websites. Selenium allows developers to programmatically control a web browser programmatically, meaning they can interact with websites as if they were human users.

Why use Selenium and Python for web scraping?

Python is a popular programming language for web scraping because it has many libraries and frameworks that make it easy to extract data from websites.

Using Python and Selenium for web scraping offers several advantages over other web scraping techniques:

• Dynamic websites: Dynamic web pages are created using JavaScript or other scripting languages. These pages often contain visible elements once the page is fully loaded or when the user interacts with them. Selenium can interact with these elements, making it a powerful tool for scraping data from dynamic web pages.
• User interactions: Selenium can simulate user interactions like clicks, form submissions, and scrolling. This allows you to scrape websites that require user input, such as login forms.
• Debugging: Selenium can be run in debug mode, which allows you to step through the scraping process and see what the scraper is doing at each step. This is useful for troubleshooting when things go wrong.

Prerequisites for web scraping with Selenium:

Python 3 is installed on your system.

Selenium library installed. You can install it using pip with the following command:

pip install Selenium

WebDriver installed.

WebDriver is a separate executable that Selenium uses to control the browser. Here are the links I found to download WebDriver for the most popular browsers:

• Chrome: https://sites.google.com/a/chromium.org/chromedriver/downloads
• Firefox: https://github.com/mozilla/geckodriver/releases
• Edge: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/

Alternatively, and this is the easiest way, you can also install the WebDriver using a package manager like web driver-manager. This will automatically download and install the appropriate WebDriver for you. To install web driver-manager, you can use the following command:

pip install webdriver-manager

Extract complete text from webpage in seconds!

Head over to Nanonets website scraper, Add the URL and click "Scrape," and download the webpage text as a file instantly. Try it for free now.

‌

A step-by-step guide to Selenium web scraping

Step 1: Install and Imports

Before we begin, we have ensured that we have installed Selenium and an appropriate driver. We'll be using the Edge driver in this example.

from selenium import webdriver
from Selenium.webdriver.common.keys import Keys
from Selenium.webdriver.common.by import By

Step 2: Install and Access WebDriver

We can create a new instance of the Edge driver by running the following code:

driver = webdriver.Edge()

Step 3: Access Website Via Python

Next, we need to access the search engine's website. In this case, we'll be using Bing.

driver.get("https://www.bing.com")

Step 4: Locate Specific Information You’re Scraping

We want to extract the number of search results for a particular name. We can do this by locating the HTML element that contains the number of search results

results = driver.find_elements(By.XPATH, "//*[@id='b_tween']/span")

Step 5: Do it together

Now that we have all the pieces, we can combine them to extract the search results for a particular name.

try:
search_box = driver.find_element(By.NAME, "q")
search_box.clear()
search_box.send_keys("John Doe") # enter your name in the search box
search_box.submit() # submit the search
results = driver.find_elements(By.XPATH, "//*[@id='b_tween']/span")
for result in results:
text = result.text.split()[1] # extract the number of results
print(text)
# save it to a file
with open("results.txt", "w") as f:
f.write(text)
except Exception as e:
print(f"An error occurred: {e}")

Step 6: Store the data

Finally, we can store the extracted data in a text file.

with open("results.txt", "w") as f:

f.write(text)

Using a proxy with Selenium Wire

Selenium Wire is a library that extends Selenium's functionality by allowing you to inspect and modify HTTP requests and responses. For example It can also be used to configure a proxy for your Selenium WebDriver easily

Install Selenium Wire

pip install selenium-wire

Set up the proxy

from selenium import webdriver
from Selenium.webdriver.chrome.options import Options
from seleniumwire import webdriver as wiredriver
PROXY_HOST = 'your.proxy.host'
PROXY_PORT = 'your_proxy_port'
chrome_options = Options()
chrome_options.add_argument('--proxy-server=http://{}:{}'.format(PROXY_HOST, PROXY_PORT))
driver = wiredriver.Chrome(options=chrome_options)

Use Selenium Wire to inspect and modify requests.

for request in driver.requests:
if request.response:
print(request.url, request.response.status_code, request.response.headers['Content-Type'])

In the code above, we loop over all requests made by the WebDriver during the web scraping session. For each request, we check if a response was received and print the URL, status code, and content type of the response

Using Selenium to extract all titles from a webpage

Here's an example Python code that uses Selenium to scrape all the titles of a webpage:

from selenium import webdriver
# Initialize the webdriver
driver = webdriver.Chrome()
# Navigate to the webpage
driver.get("https://www.example.com")
# Find all the title elements on the page
title_elements = driver.find_elements_by_tag_name("title")
# Extract the text from each title element
titles = [title.text for title in title_elements]
# Print the list of titles
print(titles)
# Close the webdriver
driver.quit()

In this example, we first import the web driver module from Selenium, then initialize a new Chrome web driver instance. We navigate to the webpage we want to scrape, and then use the find_elements_by_tag_name method to find all the title elements on the page.

We then use a list comprehension to extract the text from each title element and store the resulting list of titles in a variable called titles. Finally, we print the list of titles and close the web driver instance.

Note that you'll need to have the Selenium and Chrome web driver packages installed in your Python environment for this code to work. You can install them using pip, like so:

pip install selenium chromedriver-binary

Also, make sure to update the URL in the driver. get a method to point to the webpage you want to scrape.

Conclusion

In conclusion, web scraping with Selenium is a powerful tool for extracting data from websites. It allows you to automate the process of collecting data and can save you significant time and effort. Using Selenium, you can interact with websites just like a human user and extract the data you need more efficiently.

Alternatively, you can use no-code tools like Nanonets’ website scraper tool to easily extract all text elements from HTML. It’s free to use completely.

Extract text from any webpage in just one click. Head over to Nanonets website scraper, Add the URL and click "Scrape," and download the webpage text as a file instantly. Try it for free now.

‌

FAQs: