Deep Dive into Selenium: Exploring Link Management and Web Interactions

In this blog we will cover essential Selenium concepts and different requirements from counting links on a webpage to handling broken links and more with practical examples. Let’s get started!

We will use the Practice Website for handling different scenarios.

Requirement 1: Counting Links on a Webpage

The first task is to count the number of links on the practice page. Links on a webpage are HTML elements with the <a> tag (anchor tag). Here’s how to do it:

Steps to Count Links

1. Find All Links: Use driver.findElements(By.tagName("a")) to locate all <a> tags on the page. The findElements method returns a list of web elements.

2. Get the Count: Store the list in a variable and use the .size() method to get the total number of links.

3. Print the Result: Output the count to the console with System.out.println.

Code Example

driver.get("https://rahulshettyacademy.com/AutomationPractice/");
List<WebElement> links = driver.findElements(By.tagName("a"));
System.out.println("Total links on the page: " + links.size());

Result

When the script runs, it navigates to the practice page and prints the total number of links. In this case, the page has 26 links.

Requirement 2: Counting Links in a Specific Section

Next, the requirement focuses on counting links in a specific section of the page. Let’s consider the footer section on the Practice Website. This introduces the concept of limiting the WebDriver’s scope to a particular area of the webpage.

Why Limit the Scope?

By default, driver.findElements searches the entire page. To focus on the footer, you need to narrow down the WebDriver’s scope to that section.

Steps to Count Footer Links

1. Locate the Footer Section: Inspect the footer in the browser’s developer tools to find its unique identifier (e.g., an id or class). In this case, the footer has an id attribute.

2. Create a WebElement for the Footer: Use driver.findElement(By.id("footer-id")) to select the footer section.

3. Find Links in the Footer: Use the footer WebElement to search for <a> tags within it: footerElement.findElements(By.tagName("a")).

4. Get the Count: Print the size of the resulting list.

Code Example

// counting the number of links in the footer section of the page
WebElement footer = driver.findElement(By.id("gf-BIG")); // limiting webdriver scope
System.out.println(footer.findElements(By.tagName("a")).size());

Result

The script finds 20 links in the footer section, meaning 6 links are outside the footer (since the total was 26). This demonstrates how to work with a subset of a webpage, a crucial skill for precise automation.

Requirement 3: Counting Links in Specific Column of a Section

Now, let’s get more specific by counting links in the first column of the footer, which has four columns. This task builds on the previous concept of limiting scope but goes one level deeper.

Steps to Count First Column Links

1. Start with the Footer WebElement: Use the footer WebElement from the previous step.

2. Locate the First Column: Inspect the footer to find the HTML structure. The first column is inside a <ul> tag within a <td> element in a table.

3. Write an XPath: Use an XPath like //table/tbody/tr/td[1]/ul to pinpoint the first column’s <ul> tag.

4. Create a WebElement for the Column: Use footer.findElement(By.xpath("//table/tbody/tr/td[1]/ul")).

5. Find Links in the Column: Use columnElement.findElements(By.tagName("a")) to get the links.

6. Print the Count: Output the size of the list.

Code Example

// counting links in a particular column in footer
WebElement column = footer.findElement(By.xpath("//table/tbody/tr/td[1]/ul"));
List<WebElement> columnLinks = column.findElements(By.tagName("a"));
System.out.println("Links in first column: " + columnLinks.size());

Result

The first column has 5 links (e.g., “Link 1”, “Link 2”, etc.). This task shows how to navigate complex HTML structures using XPath and limit the WebDriver’s scope multiple times.

Requirement 4: Clicking Links and Validating Navigation

The next requirement is to click each link in the first column and verify if the linked pages open correctly. This introduces dynamic link handling and window management.

Challenge: Dynamic Links

Links on a webpage can change (e.g., new products added to an e-commerce site). Hardcoding XPaths for each link is inefficient and error-prone. Instead, we’ll dynamically iterate through the links.

Steps to Click Links

1. Get the List of Links: Use the column WebElement to get all <a> tags: column.findElements(By.tagName("a")).

2. Iterate Through Links: Use a for loop to click each link, starting from index 1 to skip any non-functional links (e.g., a coupon link that doesn’t navigate).

3. Click Each Link: Use link.click() to open the linked page.

Initial Code (Problematic)

List<WebElement> links = column.findElements(By.tagName("a"));
for (int i = 1; i < links.size(); i++) {
    links.get(i).click();
}

Problem: Stale Element Exception

When you click a link, the browser navigates to a new page, making the original list of links “stale” (no longer valid). This causes a StaleElementReferenceException when trying to click the next link.

Solution 1: Navigate Back (Inefficient)

One way to fix this is to navigate back to the original page after each click using driver.navigate().back(). However, this is slow, especially for many links (e.g., 100 links could take 30 seconds per link).

Solution 2: Open Links in New Tabs (Optimized)

A better approach is to open each link in a new tab using the Control + Click keyboard action. This avoids navigation issues and is faster.

Steps for Optimized Solution

1. Simulate Control + Click: Use Selenium’s Keys class to send a Control + Enter key combination to each link, opening it in a new tab.

2. Iterate Through Links: Loop through the links and apply the key combination.

3. Get Window Handles: Use driver.getWindowHandles() to get all open tabs.

4. Switch to Each Tab: Iterate through the window handles, switch to each tab, and get its title.

5. Print Titles: Output the title of each page to verify it opened correctly.

Code Example

String clickOnLinkTab = Keys.chord(Keys.CONTROL, Keys.ENTER);
List<WebElement> links = column.findElements(By.tagName("a"));
for (int i = 1; i < links.size(); i++) {
    links.get(i).sendKeys(clickOnLinkTab);
}
Set<String> windowHandles = driver.getWindowHandles();
Iterator<String> iterator = windowHandles.iterator();
while (iterator.hasNext()) {
    driver.switchTo().window(iterator.next());
    System.out.println("Tab title: " + driver.getTitle());
}

Result

The script opens four links (REST API, SoapUI, Appium, JMeter) in separate tabs and prints their titles. This approach is efficient, as it avoids navigating back and forth, and works even if the number of links changes.

Why This is Smart

• Dynamic Handling: The script works regardless of the number of links (4, 10, or 100).

• No Hardcoding: It uses the <a> tag instead of specific link text or XPaths.

Requirement 5: Automating a Calendar UI

The next task is to automate a calendar UI on the practice page. The goal is to select a specific date (e.g., June 15, 2027) and verify the selection using assertions.

Understanding the Calendar

The calendar UI allows you to:

• Click the date input field to open the calendar.

• Click the navigation label to switch between months and years.

• Select a year, month, and date.

Input Parameters

The script takes three variables:

• monthNumber: 6 (June)

• date: 15

• year: 2027

Steps to Automate the Calendar

1. Navigate to the Page: Load the page using driver.get("https://rahulshettyacademy.com/seleniumPractise/#/offers").

2. Open the Calendar: Click the date input field using driver.findElement(By.cssSelector(".react-date-picker__inputGroup")).

3. Select the Year:

   • Click the navigation label twice to show the year view: driver.findElement(By.cssSelector(".react-calendar__navigation__label")).

   • Use an XPath to select the year dynamically: //button[text()='2027'].

4. Select the Month:

   • The months are listed as a grid, but the month number (6) corresponds to June. Since the list is 0-indexed, use monthNumber - 1 (i.e., 5 for June).

   • Get all month elements with driver.findElements(By.cssSelector(".react-calendar__month-view__months__month")) and select the 5th index.

5. Select the Date:

   • Use an XPath to select the date: //abbr[text()='15'].

6. Verify the Selection: Extract the selected date and compare it with the input parameters.

Code Example

String monthNumber = "6";
String date = "15";
String year = "2027";

driver.get("https://rahulshettyacademy.com/seleniumPractise/#/offers");
driver.findElement(By.cssSelector(".react-date-picker__inputGroup")).click();
driver.findElement(By.cssSelector(".react-calendar__navigation__label")).click();
driver.findElement(By.cssSelector(".react-calendar__navigation__label")).click();
driver.findElement(By.xpath("//button[text()='" + year + "']")).click();
List<WebElement> months = driver.findElements(By.cssSelector(".react-calendar__month-view__months__month"));
months.get(Integer.parseInt(monthNumber) - 1).click();
driver.findElement(By.xpath("//abbr[text()='" + date + "']")).click();

Challenge: Extracting the Selected Date

The selected date (e.g., “6/15/2027”) isn’t directly available as text in the input field. Instead, it’s split across three <input> elements for month, day, and year, each with a value attribute.

Steps to Extract and Verify

1. Get the Input Elements: Use driver.findElements(By.cssSelector(".react-date-picker__inputGroup__input")) to get the three input fields.

2. Extract Values: Iterate through the list and get the value attribute of each input (e.g., “6”, “15”, “2027”).

3. Store in an Array: Create an expected array with the input parameters: String[] expected = {monthNumber, date, year}.

4. Assert Equality: Compare each actual value with the expected value using Assert.assertEquals.

Code Example

List<WebElement> actualList = driver.findElements(By.cssSelector(".react-date-picker__inputGroup__input"));
String[] expectedList = {monthNumber, date, year};
for (int i = 0; i < actualList.size(); i++) {
    String actualValue = actualList.get(i).getAttribute("value");
    Assert.assertEquals(actualValue, expectedList[i], "Date component mismatch");
}

Result

The script selects June 15, 2027, extracts the values (6, 15, 2027), and verifies they match the input parameters. If any component mismatches, the assertion fails, ensuring the calendar works correctly.

Handling SSL Certificates

Some websites use HTTP instead of HTTPS, triggering SSL certificate warnings (e.g., “Your connection is not private”). Selenium can bypass these warnings using ChromeOptions.

Steps to Bypass SSL Warnings

1. Create ChromeOptions: Initialize a ChromeOptions object.

2. Accept Insecure Certificates: Set options.setAcceptInsecureCerts(true) to automatically accept SSL warnings.

3. Pass to ChromeDriver: Pass the options to the Chrome driver.

Code Example

ChromeOptions options = new ChromeOptions();
options.setAcceptInsecureCerts(true);
WebDriver driver = new ChromeDriver(options);
driver.get("https://expired.badssl.com/");

Result

The browser bypasses the SSL warning and loads the website. This works for other browsers too (e.g., FirefoxOptions, EdgeOptions).

Other ChromeOptions Uses

• Add Extensions: Use options.addExtensions(new File("path/to/extension.crx")) to load browser extensions.

• Set Proxy: Use options.setCapability("proxy", proxyObject) to configure proxy settings.

• Block Pop-ups: Use options.setExperimentalOption("prefs", Map.of("profile.default_content_setting_values.notifications", 2)) to disable pop-ups.

• Set Download Directory: Use options.setExperimentalOption("prefs", Map.of("download.default_directory", "path")) to specify where files download.

These options are handy for customizing browser behavior.

Managing Browser Settings: Maximizing and Deleting Cookies

Selenium provides methods to manage browser settings like maximizing the window and handling cookies.

Maximizing the Browser

To ensure the browser opens in full-screen mode:

driver.manage().window().maximize();

Deleting Cookies

Cookies store user data (e.g., session IDs). To delete them:

• All Cookies: driver.manage().deleteAllCookies();

• Specific Cookie: driver.manage().deleteCookieNamed("cookieName");

Use Case: Session Cookie Deletion

In some scenarios, deleting a session cookie should log the user out. You can automate this:

1. Delete the session cookie: driver.manage().deleteCookieNamed("sessionKey");.

2. Click a link on the page.

3. Verify the browser redirects to the login page.

Code Example

driver.manage().window().maximize();
driver.manage().deleteAllCookies();
driver.get("https://google.com");

Result

The browser opens maximized, and all cookies are deleted, ensuring a fresh session. These methods work across all browsers (Chrome, Firefox, Edge).

Taking Screenshots

Capturing screenshots is useful for debugging or reporting test results. Selenium’s TakesScreenshot interface makes this easy.

Steps to Take a Screenshot

1. Cast WebDriver to TakesScreenshot: Convert the WebDriver to a screenshot-capable object.

2. Capture the Screenshot: Use getScreenshotAs(OutputType.FILE) to save the screenshot as a file.

3. Copy to Local Machine: Use FileHandler.copy to save the screenshot to a specified path.

Code Example

import org.openqa.selenium.io.FileHandler;
driver.get("https://www.google.com/");
TakesScreenshot ts = (TakesScreenshot) driver;
File source = ts.getScreenshotAs(OutputType.FILE);
FileHandler.copy(src, new File("D:\\screenshot.png"));

Important Note

• Avoid C Drive: Windows may deny access to the C drive due to admin restrictions. Save screenshots in another directory or your user directory (e.g., C:/Users/YourName/).

Result

The script captures a screenshot of the current page (e.g., Google) and saves it as screenshot.png in the specified directory.

Handling Broken Links

The final topic is identifying broken links - links that lead to invalid URLs (e.g., 404 errors). This requires combining Selenium with Java’s networking capabilities.

What is a Broken Link?

A broken link is an <a> tag with an href attribute pointing to a URL that returns an error (status code ≥ 400, like 404 or 500). For example, clicking the “Broken Link” on the Practice Page returns a 404 error.

Why Automate Broken Link Detection?

Manually clicking each link to check if it works is time-consuming, especially on large websites with hundreds or thousands of links (e.g., e-commerce sites). Automation allows us to scan all links efficiently and report any that are broken.

Steps to Identify Broken Links

1. Set Up SSL Configuration (For Testing):

   • Some websites use HTTP or have invalid SSL certificates, causing verification issues.

   • Configure an SSLContext to trust all certificates, bypassing SSL errors.

     Note: This is insecure and should only be used in non-production environments.

   • In production, use a proper trust store for secure SSL handling.

2. Configure HttpClient:

   • Create an HttpClient to send HTTP requests. It’s configured to:

     • Use the custom SSL context (for testing).

     • Follow redirects automatically (e.g., 301/302 redirects).

3. Extract Links with Selenium:

   • Use Selenium to navigate to the practice page and find all <a> tags within the footer section (identified by li[class='gf-li'] a).

   • Store the links in a list of WebElement objects.

4. Handle Relative URLs:

   • Some links have relative URLs (e.g., /page instead of https://site.com/page).

   • Resolve these by combining them with the base URL of the page using Java’s URI class.

5. Check Each URL:

   • For each link, extract the href attribute using getDomAttribute("href").

   • Send an HTTP HEAD request (which checks the URL without downloading the page content) using HttpClient.

   • Get the HTTP status code (e.g., 200 for success, 404 for not found).

   • If the status code is ≥ 400, the link is considered broken.

Code Example (Initial Version)

TrustManager[] trustAllCerts = new TrustManager[] {
    new X509TrustManager() {
        // Return null to indicate no specific trusted issuers
        public X509Certificate[] getAcceptedIssuers() {
            return null;
        }
        // Skip client certificate validation
        public void checkClientTrusted(X509Certificate[] certs, String authType) {}
        // Skip server certificate validation
        public void checkServerTrusted(X509Certificate[] certs, String authType) {}
    }
};

// Initialize an SSLContext with the trust-all configuration
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new SecureRandom());

// --- HTTP Client Setup ---
// Create an HttpClient instance for sending HTTP requests
// Configure it to use the custom SSL context (for testing) and follow redirects
HttpClient client = HttpClient.newBuilder()
    .sslContext(sc) // Remove this line in production; use a proper trust store instead
    .followRedirects(HttpClient.Redirect.NORMAL) // Follow HTTP redirects automatically
    .build();

// --- Selenium WebDriver Setup ---
// Initialize ChromeDriver to interact with the Chrome browser
WebDriver driver = new ChromeDriver();
// Navigate to the target webpage for link extraction
driver.get("https://rahulshettyacademy.com/AutomationPractice");

// --- Link Extraction ---
// Find all anchor elements (<a>) within list items with class 'gf-li'
// These elements contain the URLs to be checked
List < WebElement > links = driver.findElements(By.cssSelector("li[class='gf-li'] a"));
for (WebElement link: links) {
    String url = link.getAttribute("href");
    HttpURLConnection conn = (HttpURLConnection) new URL(url).openConnection();
    conn.setRequestMethod("HEAD");
    conn.connect();
    int responseCode = conn.getResponseCode();
    System.out.println("Link: " + link.getText() + ", Status: " + responseCode);
    if (responseCode >= 400) {
        Assert.assertTrue(false, "Link with text '" + link.getText() + "' is broken with code " + responseCode);
    }
}

Problem: Hard Assertions

The script uses a hard assertion (Assert.assertTrue), which stops execution as soon as a broken link is found (e.g., “Broken Link” with code 404). This prevents checking the remaining links.

Solution: Soft Assertions

To check all links and report all failures, use TestNG’s SoftAssert:

1. Add TestNG JAR: Download from mvnrepository.com and add to your project.

2. Create SoftAssert Object: Initialize SoftAssert softAssert = new SoftAssert();.

3. Use Soft Assertions: Replace Assert.assertTrue with softAssert.assertTrue.

4. Assert All: Call softAssert.assertAll() after the loop to report all failures.

Optimized Code Example

// --- Soft Assertions ---
// Initialize SoftAssert to collect all assertion failures and report them at the end
SoftAssert a = new SoftAssert();

// --- Link Validation Loop ---
// Iterate through each link element to check if the associated URL is valid
for (WebElement link: links) {
    // Extract the 'href' attribute from the link element
    String url = link.getDomAttribute("href");
    // Log the URL being checked for debugging
    System.out.println("Checking URL: " + url);

    // --- Handle Relative URLs ---
    // Check if the URL is relative (does not start with 'http')
    if (!url.startsWith("http")) {
        // Get the base URL of the current page
        URI baseUri = new URI(driver.getCurrentUrl());
        // Resolve the relative URL against the base URL to form an absolute URL
        url = baseUri.resolve(url).toString();
    }

    // --- HTTP Request and Validation ---
    try {
        // Create a URI object from the URL string
        URI uri = new URI(url);
        // Build an HTTP HEAD request to check the URL without downloading the body
        HttpRequest request = HttpRequest.newBuilder()
            .uri(uri) // Set the target URI
            .method("HEAD", HttpRequest.BodyPublishers.noBody()) // Use HEAD method
            .build();
        // Send the request and get the response
        HttpResponse < Void > response = client.send(request, HttpResponse.BodyHandlers.discarding());
        // Extract the HTTP status code from the response
        int respCode = response.statusCode();
        // Log the response code for debugging
        System.out.println("Response Code: " + respCode);
        // Assert that the status code is less than 400 (indicating a successful request)
        // If the status code is 400 or higher, mark the link as broken
        a.assertTrue(respCode < 400, "The link with Text " + link.getText() + " is broken with code " + respCode);
    } catch (IOException | InterruptedException e) {
        // Handle exceptions (e.g., network errors, SSL issues) during the HTTP request
        System.out.println("Error checking URL: " + url + " - " + e.getMessage());
        // Record the failure in the soft assertion
        a.assertTrue(false, "Error for " + link.getText() + ": " + e.getMessage());
    }
}

// --- Final Assertion Check ---
// Execute all soft assertions and throw an exception if any failed
a.assertAll();

Result

The script scans all 16 footer links, identifies the “Broken Link” with a 404 status code, and reports the failure without stopping. The output shows:

Checking URL: https://rahulshettyacademy.com/brokenlink
Response Code: 404
Assertion failed: The link with Text 'Broken Link' is broken with code 404

Why This is Powerful

• Comprehensive Testing: Checks all links, even after finding a failure.

• Clear Reporting: Logs the text and status code of broken links.

• Scalability: Works for any number of links, making it ideal for large websites like e-commerce platforms.

Important Notes

• SSL Configuration: The trust-all SSL setup is for testing only. In production, use a proper trust store to verify certificates securely.

• TestNG Dependency: You need the TestNG JAR (mvnrepository.com/artifact/org.testng/testng) for SoftAssert. Add it to your project’s build path.

• Java 11+: The code requires Java 11 or later for HttpClient. Ensure your environment is compatible.

Key Takeaways

This tutorial covers a wide range of Selenium concepts, each building on the previous one. Here’s the summary:

1. Counting Links: Use findElements to locate and count <a> tags on a page or section.

2. Limiting Scope: Narrow the WebDriver’s focus to specific sections (e.g., footer, column) for precise automation.

3. Dynamic Link Handling: Iterate through links dynamically to handle changing content, using keyboard actions for efficiency.

4. Window Handling: Manage multiple tabs with getWindowHandles and switch between them to verify page titles.

5. Calendar Automation: Navigate complex UI components like calendars using dynamic locators and assertions.

6. Browser Customization: Use ChromeOptions to handle SSL issues, add extensions, set proxies, and more.

7. Browser Management: Maximize windows and manage cookies for consistent test conditions.

8. Screenshots: Capture and save screenshots for debugging or reporting.

9. Broken Links: Combine Selenium and Java’s HttpClient to identify invalid URLs, using soft assertions for comprehensive testing.

Conclusion

Selenium WebDriver is a versatile tool for automating web interactions, and this article has equipped you with practical skills to tackle common automation tasks. From counting links to handling broken URLs, you now have a solid foundation to build on. Keep practicing, explore the practice page, and try these scripts on other websites to deepen your understanding.

Happy automating, and see you in the next blog!

Check out the complete code repository below: