Selenium with Sample output

Anix LynchAnix Lynch
31 min read

Table of contents

1. Introduction to Selenium

1.1 What is Selenium?

Selenium is a powerful web automation framework that allows interaction with web browsers programmatically. It is widely used for:

  • Web scraping dynamic content (JavaScript-rendered pages).

  • Testing and automating user workflows in web applications.

1.2 Key Features of Selenium

  • Cross-Browser Support: Works with Chrome, Firefox, Safari, Edge, etc.

  • Dynamic Content Scraping: Handles JavaScript-heavy websites by rendering them like a real browser.

  • Interaction with Web Elements: Allows clicking buttons, filling forms, and scrolling.

  • Built-in Wait Mechanism: Handles page load delays using implicit or explicit waits.

1.3 Use Cases for Web Scraping and Automation

  • Scraping Job Listings:

    • Extracting job data from dynamic websites like LinkedIn and Indeed.

  • Data Collection:

    • Collecting data from JavaScript-rendered web pages.

  • Automation:

    • Automating repetitive tasks, such as form submissions.

1.4 Setting Up Your Environment

Step 1: Install Selenium

pip install selenium

Step 2: Download a WebDriver

  • Selenium requires a driver to control browsers. Examples:

  • Ensure the WebDriver version matches your browser version.

Step 3: Verify Installation

from selenium import webdriver

# Initialize WebDriver
driver = webdriver.Chrome()  # Replace with Firefox() or other browsers
driver.get("https://example.com")
print(driver.title)
driver.quit()

Sample Output:

Example Domain

1.5 Running Your First Web Scraper

Objective: Open a browser, navigate to a website, and extract the page title.

from selenium import webdriver

# Set up the WebDriver
driver = webdriver.Chrome()  # Ensure chromedriver is in PATH

# Open the website
driver.get("https://example.com")

# Extract the title
page_title = driver.title
print(f"Page Title: {page_title}")

# Close the browser
driver.quit()

Sample Output:

Page Title: Example Domain

1.6 Headless Mode

Run Selenium without opening the browser for faster execution.

Code Example:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Enable headless mode
options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
print(driver.title)
driver.quit()

Sample Output:

Example Domain

2. Installing Selenium

2.1 Installing Selenium Using pip

Install Selenium using pip, which manages Python packages.

Command:

pip install selenium

Verification: Check the installation by importing Selenium in Python.

import selenium
print(selenium.__version__)

Sample Output:

4.10.0

2.2 Installing a WebDriver

Selenium uses WebDrivers to control browsers. Choose the WebDriver that matches your browser:

ChromeDriver (for Google Chrome)

  1. Find your Chrome version:

    • Open Chrome > Help > About Google Chrome.

    • Note the version (e.g., 118.0.5993.89).

  2. Download ChromeDriver:

  3. Extract and save the chromedriver executable to a known directory.

GeckoDriver (for Firefox)

  1. Find your Firefox version:

    • Open Firefox > Help > About Firefox.

    • Note the version.

  2. Download GeckoDriver:

  3. Extract and save the geckodriver executable to a known directory.

2.3 Configuring PATH for Drivers

Add the WebDriver executable to your system PATH for easy access.

On macOS or Linux

  1. Open the terminal.

  2. Edit the shell configuration file (~/.bashrc, ~/.zshrc, or ~/.bash_profile):

     nano ~/.zshrc

  3. Add the WebDriver directory to PATH:

     export PATH=$PATH:/path/to/driver

  4. Save and reload:

     source ~/.zshrc

On Windows

  1. Search for "Environment Variables" in the Start menu.

  2. Under "System Properties," click Environment Variables.

  3. Find the Path variable, edit it, and add the WebDriver directory.

2.4 Testing the Installation

Use Selenium to test the WebDriver.

Code Example:

from selenium import webdriver

# Initialize the WebDriver
driver = webdriver.Chrome()  # Replace with Firefox() if using GeckoDriver
driver.get("https://example.com")

# Print the title of the page
print(f"Page Title: {driver.title}")

# Close the browser
driver.quit()

Sample Output:

Page Title: Example Domain

Key Points

  1. Use pip install selenium to install Selenium.

  2. Download the WebDriver matching your browser version (e.g., ChromeDriver or GeckoDriver).

  3. Add the WebDriver to your PATH for convenience.

  4. Test the installation using a simple script.

3. Basic Selenium Concepts

3.1 Web Drivers and Their Role

What is a WebDriver?

  • A WebDriver is a browser automation tool that Selenium uses to control browsers programmatically.

  • It acts as a bridge between your Selenium script and the browser, enabling actions like clicking, scrolling, and navigating pages.

Examples of WebDrivers:

  • ChromeDriver: For Google Chrome

  • GeckoDriver: For Mozilla Firefox

  • EdgeDriver: For Microsoft Edge

  • SafariDriver: For Safari on macOS

How It Works:

  1. Your Selenium script sends commands to the WebDriver.

  2. The WebDriver interacts with the browser.

  3. The browser responds to the WebDriver, and the output is sent back to your script.

3.2 Anatomy of a Selenium Script

Every Selenium script follows a basic structure:

  1. Import Selenium: Import the Selenium library and the required WebDriver.

  2. Initialize the WebDriver: Create an instance of a WebDriver (e.g., ChromeDriver).

  3. Perform Actions: Open a webpage, interact with elements, and extract data.

  4. Close the Browser: Quit the WebDriver after execution.

Example Script:

from selenium import webdriver

# Step 1: Initialize WebDriver
driver = webdriver.Chrome()

# Step 2: Open a webpage
driver.get("https://example.com")

# Step 3: Perform actions
page_title = driver.title
print(f"Page Title: {page_title}")

# Step 4: Close the browser
driver.quit()

Sample Output:

Page Title: Example Domain

3.3 Differences Between Headless and Non-Headless Browsers

Non-Headless Browsers:

  • Run with a visible UI.

  • Useful for debugging and visually monitoring the scraping process.

  • Slower due to rendering overhead.

Headless Browsers:

  • Run without a visible UI.

  • Faster execution since rendering isn't required.

  • Useful for production environments where UI isn't needed.

How to Enable Headless Mode:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
print(driver.title)
driver.quit()

Sample Output:

Example Domain

3.4 Limitations of Selenium

  1. Performance:

    • Selenium is slower than APIs or headless scraping tools (like Scrapy or Playwright) because it interacts with the browser in real time.

  2. Scalability:

    • Selenium is resource-intensive and not ideal for large-scale scraping projects.

  3. JavaScript Rendering:

    • While Selenium handles JavaScript well, it is slower than dedicated frameworks like Playwright.

  4. Detection by Websites:

    • Selenium scripts can be detected by websites using anti-bot measures, such as:

      • Requiring CAPTCHAs.

      • Blocking Selenium-specific browser signatures.

  5. No Native API Support:

    • For websites with APIs, using direct API calls is faster and more efficient.

Key Takeaways

  1. WebDrivers:

    • WebDrivers are essential for controlling browsers.

    • Choose the WebDriver that matches your browser.

  2. Anatomy of a Script:

    • Initialize WebDriver, perform actions, and close the browser.

  3. Headless vs. Non-Headless:

    • Headless browsers are faster; non-headless browsers are better for debugging.

  4. Limitations:

    • Selenium is slower and less scalable for high-volume scraping.

4. Interacting with Web Pages

4.1 Finding Elements

Selenium provides multiple ways to locate elements on a webpage. Here’s how:

Finding by ID

Example:

element = driver.find_element("id", "search-box")

HTML:

<input id="search-box" type="text" placeholder="Search">

Finding by Class Name

Example:

element = driver.find_element("class name", "btn-primary")

HTML:

<button class="btn-primary">Submit</button>

Finding by Name

Example:

element = driver.find_element("name", "username")

HTML:

<input name="username" type="text">

Finding by Tag Name

Example:

elements = driver.find_elements("tag name", "a")

HTML:

<a href="/home">Home</a>
<a href="/about">About</a>

Output:

['<selenium.webdriver.remote.webelement.WebElement>', ...]

Finding by CSS Selector

Example:

element = driver.find_element("css selector", "div.container > input#search-box")

HTML:

<div class="container">
    <input id="search-box" type="text" placeholder="Search">
</div>

Finding by XPath

Example:

element = driver.find_element("xpath", "//div[@class='container']//input[@id='search-box']")

HTML:

<div class="container">
    <input id="search-box" type="text" placeholder="Search">
</div>

4.2 Performing Actions

Clicking Buttons

Example:

button = driver.find_element("class name", "btn-primary")
button.click()

HTML:

<button class="btn-primary">Submit</button>

Sending Text Inputs

Example:

search_box = driver.find_element("id", "search-box")
search_box.send_keys("Data Scientist jobs")

HTML:

<input id="search-box" type="text" placeholder="Search">

Handling Dropdowns

Using Select for <select> elements:

from selenium.webdriver.support.ui import Select

dropdown = Select(driver.find_element("id", "job-type"))
dropdown.select_by_visible_text("Full-time")

HTML:

<select id="job-type">
    <option value="ft">Full-time</option>
    <option value="pt">Part-time</option>
</select>

Submitting Forms

Example:

form = driver.find_element("tag name", "form")
form.submit()

HTML:

<form action="/search">
    <input id="search-box" type="text" placeholder="Search">
    <button type="submit">Go</button>
</form>

4.3 Full Example

Goal: Search for "Data Scientist jobs" and click the search button.

Code Example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

# Initialize WebDriver
driver = webdriver.Chrome()

# Open the webpage
driver.get("https://example-job-site.com")

# Find the search box and input text
search_box = driver.find_element(By.ID, "search-box")
search_box.send_keys("Data Scientist jobs")

# Click the search button
search_button = driver.find_element(By.CLASS_NAME, "search-button")
search_button.click()

# Extract and print the results
job_titles = driver.find_elements(By.CLASS_NAME, "job-title")
for job in job_titles:
    print(job.text)

# Close the browser
driver.quit()

Sample Output:

Data Scientist at TechCorp
Machine Learning Engineer at AI Corp
Deep Learning Researcher at DataTech

Key Takeaways

  1. Locators:

    • Use find_element or find_elements with locators like ID, class name, name, tag name, CSS selector, and XPath.

  2. Actions:

    • Automate clicks, text inputs, dropdown selections, and form submissions.

  3. Workflow:

    • Combine locators and actions to automate interactions with web pages.

5. Handling Dynamic Content in Selenium

5.1 Waiting for Elements

Dynamic pages often require waits to ensure that elements are fully loaded before interaction. Selenium offers several waiting mechanisms:

Implicit Waits

  • Global wait for all elements in the WebDriver instance.

  • Waits for a specified amount of time before throwing an error if an element is not found.

Code Example:

from selenium import webdriver

driver = webdriver.Chrome()
driver.implicitly_wait(10)  # Wait up to 10 seconds for elements
driver.get("https://example.com")

search_box = driver.find_element("id", "search-box")
print("Element found!")
driver.quit()

Explicit Waits

  • Waits for a specific condition to be met for a particular element.

Code Example:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")

# Wait for a specific element to become clickable
search_button = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.CLASS_NAME, "search-button"))
)
search_button.click()
driver.quit()

Fluent Waits

  • Polls for a condition at regular intervals while ignoring specific exceptions.

Code Example:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")

wait = WebDriverWait(driver, 10, poll_frequency=1, ignored_exceptions=[Exception])
search_box = wait.until(EC.presence_of_element_located((By.ID, "search-box")))
search_box.send_keys("Data Scientist jobs")
driver.quit()

5.2 Scrolling Pages Dynamically

Using JavaScript to Scroll

  • Scroll the page to a specific position.

Code Example:

driver = webdriver.Chrome()
driver.get("https://example.com")

# Scroll down by 500 pixels
driver.execute_script("window.scrollBy(0, 500);")

Scroll to the Bottom of the Page

Code Example:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

5.3 Handling Infinite Scrolling

For pages that load new content as you scroll, repeat scrolling and extract content dynamically.

Code Example:

import time

driver = webdriver.Chrome()
driver.get("https://example-infinite-scroll.com")

last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll to the bottom of the page
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait for new content to load
    time.sleep(2)

    # Calculate new scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

print("Finished scrolling!")
driver.quit()

5.4 Capturing JavaScript-Rendered Content

Selenium automatically renders JavaScript-heavy pages. Use waits to ensure the page is fully loaded before extracting content.

Code Example: Extract Job Listings

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example-javascript-jobs.com")

# Wait for job cards to load
job_cards = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.CLASS_NAME, "job-card"))
)

# Extract job details
for card in job_cards:
    title = card.find_element(By.CLASS_NAME, "job-title").text
    company = card.find_element(By.CLASS_NAME, "company-name").text
    print(f"Title: {title}, Company: {company}")

driver.quit()

5.5 Full Example: Handling Infinite Scroll and Extracting Data

Goal: Scrape job titles and companies from an infinite-scrolling job website.

Code Example:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example-infinite-scroll-jobs.com")

# Infinite scroll logic
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll to the bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)  # Wait for new content

    # Get new scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

# Extract job data
jobs = driver.find_elements(By.CLASS_NAME, "job-card")
for job in jobs:
    title = job.find_element(By.CLASS_NAME, "job-title").text
    company = job.find_element(By.CLASS_NAME, "company-name").text
    print(f"Title: {title}, Company: {company}")

driver.quit()

Sample Output:

Title: Data Scientist, Company: TechCorp
Title: AI Engineer, Company: AI Solutions
Title: Machine Learning Developer, Company: DataWorks

Key Takeaways

  1. Waiting Mechanisms:

    • Use implicit or explicit waits for dynamic elements.

  2. Scrolling:

    • Scroll programmatically using JavaScript for dynamic content.

  3. Infinite Scrolling:

    • Loop until all content is loaded.

  4. JavaScript Content:

    • Wait for elements to render before interacting.

6. Navigating Pages

6.1 Navigating to URLs

Open a Webpage

Use the get() method to navigate to a specified URL.

Code Example:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://example.com")
print(f"Navigated to: {driver.current_url}")
driver.quit()

Sample Output:

Navigated to: https://example.com

6.2 Using the Back and Forward Buttons

Selenium allows navigation through browser history.

Go Back

Code Example:

driver.get("https://example.com")
driver.get("https://example.com/about")
print(f"Current URL: {driver.current_url}")

driver.back()
print(f"After going back: {driver.current_url}")

Sample Output:

Current URL: https://example.com/about
After going back: https://example.com

Go Forward

Code Example:

driver.back()
driver.forward()
print(f"After going forward: {driver.current_url}")

Sample Output:

After going forward: https://example.com/about

6.3 Refreshing Pages

Reload the current page using refresh().

Code Example:

driver.get("https://example.com")
driver.refresh()
print("Page refreshed.")

Sample Output:

Page refreshed.

6.4 Extracting URLs and Page Titles

Get Current URL

Retrieve the current URL of the page using current_url.

Code Example:

print(f"Current URL: {driver.current_url}")

Sample Output:

Current URL: https://example.com

Get Page Title

Retrieve the title of the page using title.

Code Example:

print(f"Page Title: {driver.title}")

Sample Output:

Page Title: Example Domain

6.5 Full Example: Navigating and Extracting Details

Goal: Navigate between pages, refresh, and extract details.

Code Example:

from selenium import webdriver

driver = webdriver.Chrome()

# Navigate to homepage
driver.get("https://example.com")
print(f"Homepage URL: {driver.current_url}")
print(f"Homepage Title: {driver.title}")

# Navigate to About page
driver.get("https://example.com/about")
print(f"About Page URL: {driver.current_url}")
print(f"About Page Title: {driver.title}")

# Go back to the homepage
driver.back()
print(f"Back to Homepage: {driver.current_url}")

# Refresh the homepage
driver.refresh()
print("Refreshed the Homepage")

# Close the browser
driver.quit()

Sample Output:

Homepage URL: https://example.com
Homepage Title: Example Domain
About Page URL: https://example.com/about
About Page Title: About Example
Back to Homepage: https://example.com
Refreshed the Homepage

Key Takeaways

  1. Navigating URLs:

    • Use get() to load a page.

  2. Browser History:

    • Use back() and forward() to navigate between pages.

  3. Refreshing:

    • Use refresh() to reload the current page.

  4. Extracting Details:

    • Use current_url and title for page metadata.

Let me know when you're ready for the next section: Handling Popups and Alerts!

7. Handling Alerts, Pop-Ups, and Frames

7.1 Managing JavaScript Alerts

Accepting Alerts

Code Example:

from selenium import webdriver
from selenium.webdriver.common.alert import Alert

driver = webdriver.Chrome()
driver.get("https://example.com/alert")

# Simulate an alert
alert = Alert(driver)
alert.accept()
print("Alert accepted.")
driver.quit()

Sample Output:

Alert accepted.

Dismissing Alerts

Code Example:

alert.dismiss()
print("Alert dismissed.")

Sample Output:

Alert dismissed.

Sending Input to Prompts

Code Example:

alert.send_keys("Test Input")
alert.accept()
print("Prompt accepted with input.")

Sample Output:

Prompt accepted with input.

7.2 Switching Between Iframes

Switching to an Iframe

Code Example:

driver = webdriver.Chrome()
driver.get("https://example.com/iframe")

# Switch to iframe using ID
driver.switch_to.frame("iframeID")

# Interact with elements inside the iframe
iframe_element = driver.find_element("id", "inside-iframe-element")
print(f"Iframe Text: {iframe_element.text}")

# Switch back to the main content
driver.switch_to.default_content()
driver.quit()

HTML Example:

<iframe id="iframeID">
    <div id="inside-iframe-element">Iframe Content</div>
</iframe>

Sample Output:

Iframe Text: Iframe Content

Switching to Iframe Using WebElement

Code Example:

iframe = driver.find_element("tag name", "iframe")
driver.switch_to.frame(iframe)
print("Switched to iframe successfully.")

Sample Output:

Switched to iframe successfully.

7.3 Handling Browser Pop-Ups

Switching Between Windows

Code Example:

driver = webdriver.Chrome()
driver.get("https://example.com/main")

# Simulate opening a new window or pop-up
driver.execute_script("window.open('https://example.com/popup');")

# Get window handles
main_window = driver.current_window_handle
all_windows = driver.window_handles

# Switch to the pop-up
for window in all_windows:
    if window != main_window:
        driver.switch_to.window(window)
        print(f"Pop-up Title: {driver.title}")
        driver.close()

# Return to the main window
driver.switch_to.window(main_window)
print(f"Back to main window: {driver.title}")
driver.quit()

Sample Output:

Pop-up Title: Example Domain
Back to main window: Example Domain

7.4 Managing Modal Dialogs

Interacting with Modal Dialogs

Code Example:

driver = webdriver.Chrome()
driver.get("https://example.com/modal")

# Interact with modal dialog elements
modal = driver.find_element("id", "modal-dialog")
modal_text = modal.find_element("class name", "modal-text").text
print(f"Modal Text: {modal_text}")

# Close the modal
close_button = driver.find_element("class name", "close-modal")
close_button.click()
driver.quit()

HTML Example:

<div id="modal-dialog">
    <div class="modal-text">This is a modal dialog.</div>
    <button class="close-modal">Close</button>
</div>

Sample Output:

Modal Text: This is a modal dialog.

Full Example: Handling Alerts, Iframes, and Pop-Ups

Code Example:

from selenium import webdriver
from selenium.webdriver.common.alert import Alert

driver = webdriver.Chrome()
driver.get("https://example.com")

# Handle JavaScript alert
alert = Alert(driver)
alert.accept()
print("Alert handled.")

# Switch to iframe and interact
driver.switch_to.frame("iframeID")
iframe_text = driver.find_element("id", "inside-iframe-element").text
print(f"Iframe Text: {iframe_text}")
driver.switch_to.default_content()

# Open and handle pop-up
driver.execute_script("window.open('https://example.com/popup');")
main_window = driver.current_window_handle
for window in driver.window_handles:
    if window != main_window:
        driver.switch_to.window(window)
        print(f"Pop-up Title: {driver.title}")
        driver.close()
driver.switch_to.window(main_window)

driver.quit()

Sample Output:

Alert handled.
Iframe Text: Iframe Content
Pop-up Title: Example Domain

Key Takeaways

  1. Alerts:

    • Accept, dismiss, or send input to JavaScript alerts.

  2. Iframes:

    • Switch between iframes using ID, index, or WebElement.

  3. Pop-Ups:

    • Handle pop-ups by switching to the appropriate window.

  4. Modal Dialogs:

    • Locate and interact with modal elements.

8. Handling Cookies and Sessions

8.1 Viewing Cookies

You can retrieve cookies from the browser session using Selenium.

Code Example:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://example.com")

# Get all cookies
cookies = driver.get_cookies()
print("Cookies:", cookies)

driver.quit()

Sample Output:

Cookies: [{'name': 'session', 'value': 'abc123', 'domain': 'example.com', ...}]

8.2 Adding Cookies

You can manually add cookies to the browser session. This is useful for bypassing login or preloading specific states.

Code Example:

driver = webdriver.Chrome()
driver.get("https://example.com")

# Add a cookie
driver.add_cookie({"name": "test_cookie", "value": "test_value"})

# Verify the cookie
cookie = driver.get_cookie("test_cookie")
print("Added Cookie:", cookie)

driver.quit()

Sample Output:

Added Cookie: {'name': 'test_cookie', 'value': 'test_value', 'domain': 'example.com', ...}

8.3 Deleting Cookies

You can delete specific cookies or all cookies from the browser session.

Code Example:

driver = webdriver.Chrome()
driver.get("https://example.com")

# Add a test cookie
driver.add_cookie({"name": "test_cookie", "value": "test_value"})

# Delete the cookie
driver.delete_cookie("test_cookie")

# Verify deletion
cookie = driver.get_cookie("test_cookie")
print("Deleted Cookie:", cookie)

driver.quit()

Sample Output:

Deleted Cookie: None

Delete All Cookies

Code Example:

driver.delete_all_cookies()
print("All cookies deleted.")

Sample Output:

All cookies deleted.

8.4 Managing Sessions

Selenium allows you to manage and reuse browser sessions for stateful interactions.

Save and Reuse Cookies

You can save cookies from one session and load them in another.

Code Example: Save Cookies

import pickle

driver = webdriver.Chrome()
driver.get("https://example.com")

# Save cookies to a file
with open("cookies.pkl", "wb") as file:
    pickle.dump(driver.get_cookies(), file)

print("Cookies saved.")
driver.quit()

Code Example: Load Cookies

import pickle

driver = webdriver.Chrome()
driver.get("https://example.com")

# Load cookies from a file
with open("cookies.pkl", "rb") as file:
    cookies = pickle.load(file)

for cookie in cookies:
    driver.add_cookie(cookie)

# Refresh to apply cookies
driver.refresh()
print("Cookies loaded.")
driver.quit()

Sample Output:

Cookies saved.
Cookies loaded.

Maintain Session with Headers

If using APIs or requests with Selenium, pass session cookies to maintain state.

Code Example:

cookies = driver.get_cookies()
session_cookies = {cookie['name']: cookie['value'] for cookie in cookies}

# Use session cookies with requests
import requests
response = requests.get("https://example.com/api", cookies=session_cookies)
print("API Response:", response.text)

Full Example: Handling Cookies and Sessions

Goal: Save cookies from one session, load them in another, and validate functionality.

Code Example:

import pickle
from selenium import webdriver

# Step 1: Save Cookies
driver = webdriver.Chrome()
driver.get("https://example.com")
with open("cookies.pkl", "wb") as file:
    pickle.dump(driver.get_cookies(), file)
print("Cookies saved.")
driver.quit()

# Step 2: Load Cookies
driver = webdriver.Chrome()
driver.get("https://example.com")
with open("cookies.pkl", "rb") as file:
    cookies = pickle.load(file)
for cookie in cookies:
    driver.add_cookie(cookie)
driver.refresh()
print("Cookies loaded and session restored.")

driver.quit()

Sample Output:

Cookies saved.
Cookies loaded and session restored.

Key Takeaways

  1. Viewing Cookies:

    • Use get_cookies() to view all cookies or get_cookie() for specific cookies.

  2. Adding Cookies:

    • Use add_cookie() to inject cookies into the session.

  3. Deleting Cookies:

    • Use delete_cookie() or delete_all_cookies() to remove cookies.

  4. Managing Sessions:

    • Save and reuse cookies with Python's pickle module for session persistence.

9. File Uploads and Downloads

9.1 Automating File Uploads

Selenium interacts with <input type="file"> elements for file uploads by sending the file path to the input field.

Code Example: File Upload

HTML Example:

<input type="file" id="upload" />

Selenium Code:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com/upload")

# Locate the file input element
file_input = driver.find_element(By.ID, "upload")

# Provide the file path to upload
file_input.send_keys("/path/to/your/file.txt")

print("File uploaded successfully.")
driver.quit()

Sample Output:

File uploaded successfully.

9.2 Configuring File Downloads with Selenium

Selenium can configure browsers to handle file downloads automatically. This avoids interaction with download dialogs.

Configuring Chrome for Downloads

Code Example:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Configure Chrome options for downloading
options = Options()
options.add_experimental_option("prefs", {
    "download.default_directory": "/path/to/download/directory",  # Set download location
    "download.prompt_for_download": False,  # Disable download prompts
    "safebrowsing.enabled": True,  # Enable safe browsing
})

driver = webdriver.Chrome(options=options)
driver.get("https://example.com/download")

# Simulate file download by clicking the download button
download_button = driver.find_element("id", "download-button")
download_button.click()

print("File downloaded successfully.")
driver.quit()

Sample Output:

File downloaded successfully.

9.3 Handling File Paths Dynamically

Use Python modules like os or pathlib to handle file paths across different systems dynamically.

Generating File Paths Dynamically

Code Example:

import os

# Create a dynamic file path
current_dir = os.getcwd()  # Current working directory
file_path = os.path.join(current_dir, "downloads", "file.txt")

print(f"Dynamic File Path: {file_path}")

Sample Output:

Dynamic File Path: /Users/username/projects/downloads/file.txt

Full Example: File Upload and Download

Goal: Automate file upload and download, handling paths dynamically.

Code Example:

import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

# Set up file paths dynamically
current_dir = os.getcwd()
file_to_upload = os.path.join(current_dir, "test_files", "upload_file.txt")
download_dir = os.path.join(current_dir, "downloads")

# Configure Chrome options for downloading
options = Options()
options.add_experimental_option("prefs", {
    "download.default_directory": download_dir,
    "download.prompt_for_download": False,
    "safebrowsing.enabled": True,
})

driver = webdriver.Chrome(options=options)

# Automate file upload
driver.get("https://example.com/upload")
upload_input = driver.find_element(By.ID, "upload")
upload_input.send_keys(file_to_upload)
print("File uploaded successfully.")

# Automate file download
driver.get("https://example.com/download")
download_button = driver.find_element(By.ID, "download-button")
download_button.click()
print(f"File downloaded to: {download_dir}")

driver.quit()

Key Takeaways

  1. File Uploads:

    • Use send_keys() with <input type="file"> for automating uploads.

  2. File Downloads:

    • Configure browser preferences to set download directories and suppress dialogs.

  3. Dynamic Paths:

    • Use Python modules like os or pathlib for cross-platform file path handling.

10. Advanced Topics

10.1 Using Headless Browsers

Headless browsers run without a graphical interface, making them faster and suitable for automated tasks.

Code Example: Headless Chrome

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Set up headless mode
options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get("https://example.com")

print(f"Page Title: {driver.title}")
driver.quit()

Sample Output:

Page Title: Example Domain

10.2 Executing JavaScript in Selenium

Selenium allows you to execute custom JavaScript on web pages using execute_script().

Example 1: Scroll the Page

driver = webdriver.Chrome()
driver.get("https://example.com")

# Scroll down by 1000 pixels
driver.execute_script("window.scrollBy(0, 1000);")
print("Scrolled down.")
driver.quit()

Sample Output:

Scrolled down.

Example 2: Retrieve Element Properties

driver = webdriver.Chrome()
driver.get("https://example.com")

# Get the inner text of an element
inner_text = driver.execute_script("return document.querySelector('h1').innerText;")
print(f"Inner Text: {inner_text}")

driver.quit()

Sample Output:

Inner Text: Example Domain

10.3 Taking Screenshots

Selenium can capture screenshots of the current page or specific elements.

Capture Full Page Screenshot

driver = webdriver.Chrome()
driver.get("https://example.com")

# Save screenshot to file
driver.save_screenshot("screenshot.png")
print("Screenshot saved as screenshot.png.")
driver.quit()

Sample Output:

Screenshot saved as screenshot.png.

Capture Element Screenshot

element = driver.find_element("tag name", "h1")
element.screenshot("element_screenshot.png")
print("Element screenshot saved.")

Sample Output:

Element screenshot saved.

10.4 Capturing Network Traffic

To capture network traffic, third-party tools like BrowserMob Proxy or Selenium Wire are required. Below is an example using Selenium Wire.

Install Selenium Wire

pip install selenium-wire

Capture Network Requests

Code Example:

from seleniumwire import webdriver

# Start Selenium Wire WebDriver
driver = webdriver.Chrome()

driver.get("https://example.com")

# Access network requests
for request in driver.requests:
    if request.response:
        print(f"URL: {request.url}")
        print(f"Status Code: {request.response.status_code}")
        print(f"Response Body: {request.response.body.decode('utf-8', errors='ignore')}")

driver.quit()

Sample Output:

URL: https://example.com
Status Code: 200
Response Body: <!doctype html><html>...

Full Example: Combining Advanced Techniques

Goal: Use headless mode, execute JavaScript, capture screenshots, and log network requests.

Code Example:

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options

# Configure headless mode
options = Options()
options.headless = True

# Start WebDriver with Selenium Wire
driver = webdriver.Chrome(options=options)

# Navigate to the page
driver.get("https://example.com")

# Execute JavaScript to scroll
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
print("Page scrolled.")

# Capture a screenshot
driver.save_screenshot("full_page.png")
print("Screenshot saved.")

# Log network traffic
for request in driver.requests:
    if request.response:
        print(f"URL: {request.url}")
        print(f"Status Code: {request.response.status_code}")

driver.quit()

Sample Output:

Page scrolled.
Screenshot saved.
URL: https://example.com
Status Code: 200

Key Takeaways

  1. Headless Browsers:

    • Use headless mode for faster and invisible execution.

  2. Executing JavaScript:

    • Use execute_script() to run custom JavaScript.

  3. Taking Screenshots:

    • Capture full page or specific elements for debugging or reporting.

  4. Capturing Network Traffic:

    • Use tools like Selenium Wire or BrowserMob Proxy to log HTTP requests and responses.

11. Testing with Selenium

11.1 Writing Test Scripts with Selenium

Selenium can automate functional testing by simulating user actions on web applications.

Basic Test Script

Write a script to verify that the correct page loads and specific elements exist.

Code Example:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://example.com")

# Test: Verify the page title
assert "Example Domain" in driver.title

# Test: Check if the heading exists
heading = driver.find_element(By.TAG_NAME, "h1")
assert heading.text == "Example Domain", "Heading text does not match"

print("All tests passed!")
driver.quit()

Sample Output:

All tests passed!

11.2 Using Selenium with pytest

pytest is a powerful testing framework that integrates well with Selenium.

Install pytest

pip install pytest

Write Tests with pytest

Save the following script as test_example.py.

Code Example:

import pytest
from selenium import webdriver
from selenium.webdriver.common.by import By

@pytest.fixture
def driver():
    driver = webdriver.Chrome()
    yield driver
    driver.quit()

def test_page_title(driver):
    driver.get("https://example.com")
    assert "Example Domain" in driver.title

def test_heading_text(driver):
    driver.get("https://example.com")
    heading = driver.find_element(By.TAG_NAME, "h1")
    assert heading.text == "Example Domain"

Run pytest

pytest test_example.py

Sample Output:

============================= test session starts ==============================
collected 2 items

test_example.py ..                                                     [100%]

============================== 2 passed in 3.42s ===============================

11.3 Using Selenium with unittest

unittest is a built-in Python testing framework that supports Selenium testing.

Write Tests with unittest

Code Example:

import unittest
from selenium import webdriver
from selenium.webdriver.common.by import By

class TestExampleDotCom(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Chrome()

    def tearDown(self):
        self.driver.quit()

    def test_page_title(self):
        self.driver.get("https://example.com")
        self.assertIn("Example Domain", self.driver.title)

    def test_heading_text(self):
        self.driver.get("https://example.com")
        heading = self.driver.find_element(By.TAG_NAME, "h1")
        self.assertEqual(heading.text, "Example Domain")

if __name__ == "__main__":
    unittest.main()

Run unittest

python test_example.py

Sample Output:

..
----------------------------------------------------------------------
Ran 2 tests in 2.648s

OK

11.4 Generating Reports for Tests

Testing frameworks like pytest support plugins for generating detailed reports.

Generate Reports with pytest

Install the pytest HTML report plugin:

pip install pytest-html

Run tests with report generation:

pytest test_example.py --html=report.html

Generate Reports with unittest

Use the HTMLTestRunner library to create HTML reports for unittest.

Install HTMLTestRunner:

pip install html-testRunner

Code Example:

import unittest
from selenium import webdriver
from selenium.webdriver.common.by import By
import HtmlTestRunner

class TestExampleDotCom(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Chrome()

    def tearDown(self):
        self.driver.quit()

    def test_page_title(self):
        self.driver.get("https://example.com")
        self.assertIn("Example Domain", self.driver.title)

    def test_heading_text(self):
        self.driver.get("https://example.com")
        heading = self.driver.find_element(By.TAG_NAME, "h1")
        self.assertEqual(heading.text, "Example Domain")

if __name__ == "__main__":
    unittest.main(testRunner=HtmlTestRunner.HTMLTestRunner(output="reports"))

Run unittest

python test_example.py

This generates an HTML report in the reports directory.

Key Takeaways

  1. Basic Testing:

    • Write simple scripts to verify titles, elements, and functionality.

  2. pytest:

    • Use pytest for powerful and flexible testing with fixtures.

  3. unittest:

    • Utilize Python's built-in framework for structured test cases.

  4. Reports:

    • Generate reports with pytest-html or HTMLTestRunner.

12. Handling Captchas

12.1 Identifying Captchas

What are Captchas?

Captchas are security measures designed to differentiate between humans and bots. They are often used to prevent automated access to websites.

Common Types of Captchas:

  1. Text-based: Enter text from a distorted image.

  2. Image-based: Select images that match a specific criterion.

  3. Invisible Captchas (reCAPTCHA v3): Use behavioral analysis to detect bots.

  4. hCaptcha: Image-based captcha similar to reCAPTCHA.

  5. Math Captchas: Solve a simple math problem.

12.2 Integrating Third-Party Services

Using 2Captcha

2Captcha is a service that solves captchas by outsourcing them to human workers.

Step 1: Install Required Library

pip install requests

Step 2: Obtain an API Key

  • Sign up at 2Captcha.

  • Retrieve your API key from the dashboard.

Step 3: Solve Captchas

Code Example:

import requests

# Replace with your API key
API_KEY = "your_2captcha_api_key"

def solve_captcha(image_file_path):
    with open(image_file_path, "rb") as image_file:
        # Send captcha to 2Captcha
        response = requests.post(
            "http://2captcha.com/in.php",
            files={"file": image_file},
            data={"key": API_KEY, "method": "post"},
        )
        captcha_id = response.text.split("|")[1]

        # Wait for the result
        while True:
            result = requests.get(
                f"http://2captcha.com/res.php?key={API_KEY}&action=get&id={captcha_id}"
            )
            if "CAPCHA_NOT_READY" not in result.text:
                return result.text.split("|")[1]

captcha_solution = solve_captcha("captcha_image.png")
print(f"Captcha Solution: {captcha_solution}")

Using AntiCaptcha

AntiCaptcha is another popular service for solving captchas.

Step 1: Install the AntiCaptcha SDK

pip install anticaptchaofficial

Step 2: Solve reCAPTCHA

Code Example:

from anticaptchaofficial.recaptchav2proxyless import *

solver = recaptchaV2Proxyless()
solver.set_verbose(1)
solver.set_key("your_anticaptcha_api_key")
solver.set_website_url("https://example.com")
solver.set_website_key("site_key_from_website")

captcha_solution = solver.solve_and_return_solution()
if captcha_solution != 0:
    print(f"Captcha Solved: {captcha_solution}")
else:
    print(f"Error: {solver.error_code}")

12.3 Alternatives to Bypass Captchas

1. Avoid Triggering Captchas

  • Use human-like delays:

    • Introduce random pauses between actions.

    • Avoid rapid requests.

  • Rotate IP addresses:

    • Use proxy servers or VPNs.

  • Use different user agents:

    • Randomize browser headers to mimic real users.

2. Use Behavioral Detection

  • Invisible captchas like reCAPTCHA v3 rely on analyzing mouse movement and behavior.

  • Use browser automation tools like Playwright to simulate human-like behavior.

Code Example: Adding Delays in Selenium

from selenium import webdriver
from time import sleep
import random

driver = webdriver.Chrome()
driver.get("https://example.com")

# Simulate human-like typing
search_box = driver.find_element("id", "search-box")
for char in "Data Scientist jobs":
    search_box.send_keys(char)
    sleep(random.uniform(0.2, 0.5))

print("Human-like interaction complete.")
driver.quit()

3. Use Alternative APIs

Some websites offer APIs to fetch data without dealing with captchas.

4. Pre-solve Captchas Manually

  • If captchas are rare, consider solving them manually once and reusing session cookies.

Code Example: Save Cookies

import pickle

# Save cookies after solving captcha
with open("cookies.pkl", "wb") as file:
    pickle.dump(driver.get_cookies(), file)

# Load cookies in subsequent sessions
with open("cookies.pkl", "rb") as file:
    cookies = pickle.load(file)
    for cookie in cookies:
        driver.add_cookie(cookie)
driver.refresh()

Full Example: Integrating Captcha Solvers

Goal: Solve a reCAPTCHA on a website using 2Captcha.

Code Example:

import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

API_KEY = "your_2captcha_api_key"

def solve_recaptcha(api_key, site_key, url):
    # Send captcha solving request
    response = requests.post(
        "http://2captcha.com/in.php",
        data={
            "key": api_key,
            "method": "userrecaptcha",
            "googlekey": site_key,
            "pageurl": url,
        },
    )
    captcha_id = response.text.split("|")[1]

    # Wait for the solution
    while True:
        result = requests.get(
            f"http://2captcha.com/res.php?key={api_key}&action=get&id={captcha_id}"
        )
        if "CAPCHA_NOT_READY" not in result.text:
            return result.text.split("|")[1]
        sleep(5)

# Set up Selenium
driver = webdriver.Chrome()
driver.get("https://example.com")

# Solve reCAPTCHA
site_key = "site_key_from_website"
url = driver.current_url
captcha_solution = solve_recaptcha(API_KEY, site_key, url)

# Inject the captcha solution into the form
driver.execute_script(
    f'document.getElementById("g-recaptcha-response").innerHTML="{captcha_solution}";'
)
driver.find_element(By.ID, "submit").click()

print("Captcha solved and form submitted.")
driver.quit()

Sample Output:

Captcha solved and form submitted.

Key Takeaways

  1. Third-Party Services:

    • Use services like 2Captcha or AntiCaptcha for automated solving.

  2. Bypassing Tips:

    • Mimic human behavior, rotate IPs, and randomize headers to avoid captchas.

  3. Manual Solutions:

    • Solve captchas once and reuse session cookies to bypass them later.

  4. Alternative APIs:

    • Leverage site-provided APIs to avoid captchas entirely.

13. Common Challenges and Solutions

13.1 Debugging Common Selenium Errors

1. NoSuchElementException

Occurs when Selenium cannot find an element.

Possible Causes:

  • Incorrect locator.

  • Element is not rendered yet.

Solution:

  • Verify the locator in the browser's developer tools.

  • Use waits to ensure the element is loaded.

Code Example:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")

try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "search-box"))
    )
    print("Element found!")
except:
    print("Element not found.")
driver.quit()

2. StaleElementReferenceException

Occurs when the element is no longer attached to the DOM.

Possible Causes:

  • The page reloaded or the DOM changed after locating the element.

Solution:

  • Re-locate the element after the DOM update.

Code Example:

try:
    element = driver.find_element(By.ID, "search-box")
    element.click()  # Perform an action
except StaleElementReferenceException:
    print("Element became stale. Re-locating...")
    element = driver.find_element(By.ID, "search-box")
    element.click()

3. TimeoutException

Occurs when an element is not found within the specified wait time.

Possible Causes:

  • Element takes longer to load.

  • Incorrect wait condition.

Solution:

  • Increase the wait time or verify the condition.

Code Example:

from selenium.webdriver.support import expected_conditions as EC

try:
    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.ID, "dynamic-element"))
    )
    print("Element loaded.")
except TimeoutException:
    print("Element loading timed out.")

4. ElementClickInterceptedException

Occurs when another element overlays the target element.

Possible Causes:

  • Modals, pop-ups, or other elements obscure the target element.

Solution:

  • Scroll to the element or handle the overlay first.

Code Example:

from selenium.webdriver.common.action_chains import ActionChains

element = driver.find_element(By.ID, "button")
driver.execute_script("arguments[0].scrollIntoView();", element)  # Scroll to the element
ActionChains(driver).move_to_element(element).click().perform()  # Click on the element

13.2 Dealing with Stale Element Exceptions

Re-Locate Elements Dynamically

Re-fetch elements whenever the DOM changes.

Code Example:

while True:
    try:
        element = driver.find_element(By.ID, "dynamic-element")
        element.click()
        break
    except StaleElementReferenceException:
        print("Re-locating element...")

13.3 Avoiding Detection

Websites often block bots using detection techniques. Here’s how to avoid it:

1. Rotate User Agents

Changing the browser’s user agent makes your scraper appear as different devices or browsers.

Code Example:

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36")
driver = webdriver.Chrome(options=options)

2. Manage Headers

Set custom headers to mimic real browser behavior.

Code Example:

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

capabilities = DesiredCapabilities.CHROME.copy()
capabilities["goog:loggingPrefs"] = {"performance": "ALL"}
capabilities["browserName"] = "chrome"
capabilities["acceptSslCerts"] = True
driver = webdriver.Chrome(desired_capabilities=capabilities)

3. Rotate Proxies

Using proxies reduces the chance of IP bans.

Install a Proxy Manager:

pip install selenium-proxy

Code Example:

from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType

proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = "http://username:password@proxy_ip:port"
proxy.ssl_proxy = "http://username:password@proxy_ip:port"

capabilities = webdriver.DesiredCapabilities.CHROME.copy()
proxy.add_to_capabilities(capabilities)

driver = webdriver.Chrome(desired_capabilities=capabilities)

4. Introduce Random Delays

Introduce random pauses between actions to mimic human behavior.

Code Example:

import time
import random

time.sleep(random.uniform(2, 5))  # Wait for a random time between 2 to 5 seconds

5. Avoid Repetitive Patterns

  • Randomize navigation paths.

  • Vary the order of actions.

Full Example: Avoiding Detection with User Agents, Headers, and Proxies

Code Example:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.proxy import Proxy, ProxyType
import time
import random

# Set up options
options = Options()
options.add_argument("user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36")

# Configure proxy
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = "http://username:password@proxy_ip:port"

capabilities = webdriver.DesiredCapabilities.CHROME.copy()
proxy.add_to_capabilities(capabilities)

# Start driver with options
driver = webdriver.Chrome(options=options, desired_capabilities=capabilities)

# Navigate to page
driver.get("https://example.com")

# Simulate human-like actions
time.sleep(random.uniform(2, 5))
search_box = driver.find_element("id", "search-box")
search_box.send_keys("Data Scientist jobs")

time.sleep(random.uniform(1, 3))
search_button = driver.find_element("id", "search-button")
search_button.click()

print("Search completed!")
driver.quit()

Sample Output:

Search completed!

Key Takeaways

  1. Debugging Errors:

    • Use waits and re-fetch elements to handle StaleElementReferenceException.

  2. Avoid Detection:

    • Rotate user agents, use proxies, and randomize delays.

  3. Headers:

    • Mimic real browser headers to blend in with legitimate traffic.

14. Deploying Selenium Scripts

14.1 Scheduling Scripts with cron or APScheduler

Using cron on Linux/Mac

cron is a tool for scheduling tasks on Unix-based systems.

Steps to Schedule a Selenium Script

  1. Make Your Script Executable: Add the shebang line (#!/usr/bin/env python3) to the top of your script.

    Example Script (selenium_script.py):

     #!/usr/bin/env python3
 from selenium import webdriver

 driver = webdriver.Chrome()
 driver.get("https://example.com")
 print(driver.title)
 driver.quit()

  2. Grant Execution Permission:

     chmod +x selenium_script.py

  3. Edit the crontab File: Open crontab:

     crontab -e

  4. Add a Cron Job: Schedule the script to run at a specific interval. For example, run every day at 8 AM:

     0 8 * * * /path/to/selenium_script.py

Using APScheduler in Python

APScheduler allows you to schedule tasks directly in Python.

Install APScheduler:

pip install apscheduler

Schedule Selenium with APScheduler:

Code Example:

from apscheduler.schedulers.blocking import BlockingScheduler
from selenium import webdriver

def run_selenium_script():
    driver = webdriver.Chrome()
    driver.get("https://example.com")
    print(f"Page Title: {driver.title}")
    driver.quit()

scheduler = BlockingScheduler()
scheduler.add_job(run_selenium_script, 'cron', hour=8, minute=0)  # Run daily at 8:00 AM

print("Scheduler started. Press Ctrl+C to exit.")
scheduler.start()

14.2 Running Scripts on Cloud Platforms

Using AWS

  1. Launch an EC2 Instance:

    • Choose an AMI with Python installed.

    • Install Selenium and ChromeDriver on the instance.

Commands:

    sudo apt update
    sudo apt install python3-pip
    pip3 install selenium
    sudo apt install chromium-browser

  1. Transfer Your Script: Upload the Selenium script to the instance using scp:

     scp selenium_script.py ubuntu@<your_instance_ip>:~

  2. Run Your Script:

     python3 selenium_script.py

  3. Schedule with cron: Follow the steps in 14.1 to schedule your script on the EC2 instance.

Using Google Cloud Platform (GCP)

  1. Create a Compute Engine VM:

    • Choose a machine type and OS.

    • Install Python, Selenium, and ChromeDriver.

  2. Transfer and Run Your Script: Same steps as AWS.

  3. Schedule Using cron: Configure crontab as shown earlier.

Using Heroku

  1. Set Up a Heroku App:

    • Install the Heroku CLI:

        curl https://cli-assets.heroku.com/install.sh | sh

    • Log in and create an app:

        heroku login
  heroku create <your_app_name>

  2. Prepare Your Project:

    • Add a Procfile to specify the command to run your script:

        worker: python selenium_script.py

    • Add a requirements.txt file for dependencies:

        selenium

  3. Deploy the App:

     git init
 git add .
 git commit -m "Deploy Selenium script"
 heroku git:remote -a <your_app_name>
 git push heroku master

  4. Run and Schedule: Use Heroku's scheduler add-on to run the script at specific times.

14.3 Using Docker for Selenium

Why Use Docker?

  • Ensures consistency across environments.

  • Easily integrates with CI/CD pipelines.

Step 1: Install Docker

Follow the official installation guide.

Step 2: Create a Dockerfile

Example Dockerfile:

FROM python:3.9-slim

# Install Chrome and ChromeDriver
RUN apt-get update && apt-get install -y \
    chromium \
    chromium-driver

# Install Python dependencies
COPY requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

# Copy Selenium script
COPY selenium_script.py /app/selenium_script.py

WORKDIR /app
CMD ["python", "selenium_script.py"]

requirements.txt:

selenium

Step 3: Build and Run the Docker Image

  1. Build the image:

     docker build -t selenium-script .

  2. Run the container:

     docker run selenium-script

Full Example: Deploying with Docker

Goal: Automate a Selenium script in a Docker container and run it on schedule.

Code Example

selenium_script.py:

from selenium import webdriver

def run():
    driver = webdriver.Chrome()
    driver.get("https://example.com")
    print(f"Page Title: {driver.title}")
    driver.quit()

if __name__ == "__main__":
    run()

Dockerfile:

FROM python:3.9-slim

RUN apt-get update && apt-get install -y \
    chromium \
    chromium-driver

COPY requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt

COPY selenium_script.py /app/selenium_script.py

WORKDIR /app
CMD ["python", "selenium_script.py"]

Build and Run:

docker build -t selenium-script .
docker run selenium-script

Sample Output:

Page Title: Example Domain

Key Takeaways

  1. Scheduling:

    • Use cron or APScheduler for periodic execution.

  2. Cloud Platforms:

    • AWS, GCP, and Heroku provide scalable environments for running scripts.

  3. Docker:

    • Use Docker for consistent, portable deployments.
0
Subscribe to my newsletter

Read articles from Anix Lynch directly inside your inbox. Subscribe to the newsletter, and don't miss out.

seleniumwebscraping

Written by

Anix Lynch
Anix Lynch