20 BeautifulSoup concepts with Before-and-After Examples

Anix LynchAnix Lynch
11 min read

Table of contents

1. Creating a Soup Object (BeautifulSoup) 🍜

Boilerplate Code:

from bs4 import BeautifulSoup

Use Case: Create a soup object to parse HTML or XML data. 🍜

Goal: Load and process an HTML or XML document for web scraping. 🎯

Sample Code:

# Load HTML data
html_data = "<html><body><h1>Hello World!</h1></body></html>"

# Create a soup object
soup = BeautifulSoup(html_data, "html.parser")

Before Example:
You have raw HTML or XML but no way to process or extract its content. 🤔

Data: "<html><body><h1>Hello World!</h1></body></html>"

After Example:
With BeautifulSoup, you can now parse and manipulate the HTML document! 🍜

Output: A soup object that allows you to navigate and extract information.

Challenge: 🌟 Try parsing an HTML file from a webpage and print out the <title> tag.

2. Finding Elements by Tag Name (find) 🔍

Boilerplate Code:

soup.find('tag_name')

Use Case: Use find to retrieve the first element that matches a given tag. 🔍

Goal: Locate a specific tag in the HTML or XML document. 🎯

Sample Code:

# Find the first h1 tag
h1_tag = soup.find('h1')
print(h1_tag.text)  # Output: "Hello World!"

Before Example:
You have an HTML document but don’t know how to find specific elements. 🤔

Data: "<h1>Hello World!</h1>"

After Example:
With find, you can easily extract the first matching element! 🔍

Output: "Hello World!"

Challenge: 🌟 Try using find to locate other tags like <p> or <div>.

3. Finding Multiple Elements (find_all) 📝

Boilerplate Code:

soup.find_all('tag_name')

Use Case: Use find_all to retrieve a list of all elements that match a given tag. 📝

Goal: Extract all occurrences of a specific tag. 🎯

Sample Code:

# Find all paragraph tags
paragraphs = soup.find_all('p')
for p in paragraphs:
    print(p.text)

Before Example:
You need to extract multiple elements but find only gives you the first one. 🤔

Data: "<p>Paragraph 1</p><p>Paragraph 2</p>"

After Example:
With find_all, you can extract all matching elements! 📝

Output: ["Paragraph 1", "Paragraph 2"]

Challenge: 🌟 Try using find_all to extract all <a> tags and print their href attributes.

4. Extracting Text Content (get_text) 🗣️

Boilerplate Code:

soup.get_text()

Use Case: Use get_text to retrieve all text content from an element or document. 🗣️

Goal: Extract only the text from an HTML or XML element. 🎯

Sample Code:

# Get all text from the document
text_content = soup.get_text()
print(text_content)

Before Example:
You want to extract the text but HTML tags are in the way. 🤔

Data: "<p>Hello <b>World</b>!</p>"

After Example:
With get_text, you extract only the plain text! 🗣️

Output: "Hello World!"

Challenge: 🌟 Try using get_text() on different sections of the HTML document.

5. Navigating the DOM (soup.tag) 🧭

Boilerplate Code:

soup.tag

Use Case: Use the dot notation to directly access a tag in the document. 🧭

Goal: Quickly navigate the document’s structure using the tag names. 🎯

Sample Code:

# Access the body tag
body = soup.body
print(body)

Before Example:
You need to locate specific parts of the document but don’t want to call find repeatedly. 🤔

Data: "<body>...</body>"

After Example:
With dot notation, navigating the document becomes quick and easy! 🧭

Output: The content inside the `<body>` tag.

Challenge: 🌟 Try using soup.title to extract the <title> tag content.

6. Extracting Attributes (tag['attribute']) 🏷️

Boilerplate Code:

tag['attribute']

Use Case: Extract attributes (like href, src) from HTML elements. 🏷️

Goal: Retrieve specific attributes from tags, such as URLs from anchor tags. 🎯

Sample Code:

# Extract href attribute from an anchor tag
link = soup.find('a')
print(link['href'])

Before Example:
You need to get the URL from an anchor tag, but all you have is the tag itself. 🤔

Data: "<a href='https://example.com'>Example</a>"

After Example:
With tag['attribute'], you can easily extract the href value! 🏷️

Output: "https://example.com"

Challenge: 🌟 Try extracting the src attribute from an <img> tag.

7. Finding by Class or ID (find class_/id) 🏷️

Boilerplate Code:

soup.find('tag', class_='class_name')
soup.find('tag', id='id_name')

Use Case: Use class_ or id to find elements by their CSS class or ID. 🏷️

Goal: Locate elements based on their class or ID attributes. 🎯

Sample Code:

# Find element by class name
element = soup.find('div', class_='my-class')
print(element)

# Find element by ID
element = soup.find('div', id='my-id')
print(element)

Before Example:
You have elements with specific classes or IDs but don’t know how to locate them. 🤔

Data: "<div class='my-class'>...</div>"

After Example:
With class_ or id, you can directly find elements by their attributes! 🏷️

Output: The element with the matching class or ID.

Challenge: 🌟 Try using find_all to locate multiple elements with the same class.

8. Modifying HTML Content (tag.string) 🛠️

Boilerplate Code:

tag.string = "New Content"

Use Case: Modify the content of an HTML element using tag.string. 🛠️

Goal: Change the text inside an HTML tag. 🎯

Sample Code:

# Modify the content of an h1 tag
h1_tag = soup.find('h1')
h1_tag.string = "New Heading"
print(h1_tag)

Before Example:
You want to change the content of an HTML tag but don’t know how to edit it. 🤔

Data: "<h1>Hello World!</h1>"

After Example:
With tag.string, you can change the text inside the tag! 🛠️

Output: "<h1>New Heading</h1>"

Challenge: 🌟 Try modifying multiple elements in the document by looping through them.

9. Inserting New Tags (new_tag) 🏗️

Boilerplate Code:

new_tag = soup.new_tag("tag_name")

Use Case: Use new_tag to create and insert new HTML elements dynamically. 🏗️

Goal: Add new elements to the document for manipulation or enhancement. 🎯

Sample Code:

# Create a new tag
new_tag = soup.new_tag("p")
new_tag.string = "This is a new paragraph."

# Append it to the body
soup.body.append(new_tag)
print(soup.body)

Before Example:
You want to add new content but don’t know how to create new HTML elements. 🤔

Data: "<body>...</body>"

**After

Example**:
With new_tag, you can dynamically insert new elements! 🏗️

Output: A new paragraph added to the body.

Challenge: 🌟 Try inserting multiple tags dynamically at different places in the document.

10. Removing Tags (decompose) 🧹

Boilerplate Code:

tag.decompose()

Use Case: Remove tags and their contents from the document using decompose. 🧹

Goal: Clean up unwanted tags or elements from the HTML. 🎯

Sample Code:

# Find and remove an h1 tag
h1_tag = soup.find('h1')
h1_tag.decompose()
print(soup)

Before Example:
You want to remove an element but don’t know how to delete it. 🤔

Data: "<h1>Hello World!</h1>"

After Example:
With decompose, the element and its contents are completely removed! 🧹

Output: The `<h1>` tag is removed from the document.

Challenge: 🌟 Try using decompose to remove multiple tags or entire sections of the document.

11. Finding by CSS Selectors (select) 🕵️‍♂️

Boilerplate Code:

soup.select('css_selector')

Use Case: Use select to find elements using CSS selectors (like .class, #id, tag). 🕵️‍♂️

Goal: Locate elements based on complex CSS selectors. 🎯

Sample Code:

# Find elements by CSS selector
elements = soup.select('div.my-class')
for element in elements:
    print(element)

Before Example:
You need to find elements using CSS-style selectors. 🤔

Data: "<div class='my-class'>...</div>"

After Example:
With select, you can target elements using flexible and complex selectors! 🕵️‍♂️

Output: Elements found using the `.my-class` CSS selector.

Challenge: 🌟 Try using more advanced CSS selectors like div > p or ul li:first-child.

12. Extracting Tag Names (name) 🔠

Boilerplate Code:

tag.name

Use Case: Extract the tag name from an element using the name attribute. 🔠

Goal: Identify the type of element (e.g., h1, p, div). 🎯

Sample Code:

# Extract tag name
tag = soup.find('h1')
print(tag.name)  # Output: "h1"

Before Example:
You want to check what kind of element you're dealing with but don't know its tag name. 🤔

Data: "<h1>Hello World!</h1>"

After Example:
With tag.name, you can extract and confirm the element type! 🔠

Output: "h1"

Challenge: 🌟 Try printing the names of all elements within a specific tag (like <div>).

13. Navigating Parent Elements (parent) 🔄

Boilerplate Code:

tag.parent

Use Case: Access the parent element of a tag using tag.parent. 🔄

Goal: Navigate to the parent element of a given tag. 🎯

Sample Code:

# Find parent of an h1 tag
h1_tag = soup.find('h1')
print(h1_tag.parent)

Before Example:
You want to access the container element (parent) of a specific tag. 🤔

Data: "<body><h1>Hello World!</h1></body>"

After Example:
With parent, you can move up the DOM to the parent tag! 🔄

Output: The parent `<body>` element.

Challenge: 🌟 Try accessing the grandparent by chaining .parent.parent.

14. Navigating Sibling Elements (next_sibling / previous_sibling) ↔️

Boilerplate Code:

tag.next_sibling
tag.previous_sibling

Use Case: Use next_sibling and previous_sibling to navigate between sibling elements. ↔️

Goal: Access the next or previous sibling element of a tag. 🎯

Sample Code:

# Find next sibling of an h1 tag
h1_tag = soup.find('h1')
print(h1_tag.next_sibling)

Before Example:
You want to move between elements on the same level in the DOM (siblings). 🤔

Data: "<h1>Hello World!</h1><p>This is a paragraph.</p>"

After Example:
With next_sibling, you can move to the next sibling element in the DOM! ↔️

Output: The `<p>` element following the `<h1>`.

Challenge: 🌟 Try looping through all siblings of a tag.

15. Navigating Child Elements (children) 🧑‍🤝‍🧑

Boilerplate Code:

tag.children

Use Case: Access the children (direct descendants) of a tag using tag.children. 🧑‍🤝‍🧑

Goal: Iterate over all child elements of a given tag. 🎯

Sample Code:

# Loop through child elements of the body tag
body = soup.body
for child in body.children:
    print(child)

Before Example:
You want to access all direct child elements of a parent tag. 🤔

Data: "<body><h1>Hello</h1><p>World</p></body>"

After Example:
With children, you can easily loop through and extract all child tags! 🧑‍🤝‍🧑

Output: The `<h1>` and `<p>` elements inside the body.

Challenge: 🌟 Try using .descendants to access all descendants, including nested ones.

16. Accessing Tag Attributes (attrs) 🏷️

Boilerplate Code:

tag.attrs

Use Case: Use attrs to get all attributes of an HTML tag. 🏷️

Goal: Retrieve a dictionary of all attributes associated with a tag. 🎯

Sample Code:

# Get all attributes of an anchor tag
anchor = soup.find('a')
print(anchor.attrs)  # Output: {'href': 'https://example.com'}

Before Example:
You need to access all attributes of a tag, but you only know one. 🤔

Data: "<a href='https://example.com' title='Example Link'></a>"

After Example:
With attrs, you get a dictionary of all the attributes associated with the tag! 🏷️

Output: {'href': 'https://example.com', 'title': 'Example Link'}

Challenge: 🌟 Try modifying an attribute by directly editing tag.attrs['attribute'].

17. Searching with Multiple Criteria (find/find_all with filters) 🔎

Boilerplate Code:

soup.find('tag', {'attribute': 'value'})

Use Case: Use find or find_all with filters to search based on multiple criteria. 🔎

Goal: Locate elements that match specific tags and attributes. 🎯

Sample Code:

# Find div with specific class and id
div = soup.find('div', {'class': 'my-class', 'id': 'my-id'})
print(div)

Before Example:
You need to find elements that match both tag name and attributes. 🤔

Data: "<div class='my-class' id='my-id'>...</div>"

After Example:
With filters, you can locate elements that meet multiple conditions! 🔎

Output: The `<div>` tag with matching class and id.

Challenge: 🌟 Try combining class_, id, and other attributes for more complex searches.

18. Using Lambda for Custom Filters (find_all) 🧑‍💻

Boilerplate Code:

soup.find_all(lambda tag: some_condition)

Use Case: Use a lambda function in find_all to apply custom search filters. 🧑‍💻

Goal: Apply custom logic to filter elements based on non-standard conditions. 🎯

Sample Code:

# Find all tags with more than one attribute
tags = soup.find_all(lambda tag: len(tag.attrs) > 1)
for tag in tags:
    print(tag)

Before Example:
You need a custom search condition that standard filters can't handle. 🤔

Data: Multiple elements with varying attributes.

After Example:
With lambda filters, you can apply any custom condition to find elements! 🧑‍💻

Output: All tags with more than one attribute.

Challenge: 🌟 Try using lambda to find all tags with specific text content or custom attribute logic.

19. Searching for Specific Text (string/text) 📝

Boilerplate Code:

soup.find_all(string="specific text")

Use Case: Use string to find elements that contain specific text. 📝

Goal: Search for tags based on their text content. 🎯

Sample Code:

# Find all tags containing the specific text "Hello"
tags = soup.find_all(string="Hello")
print(tags)

Before Example:
You need to find elements that contain a specific string of text. 🤔

Data: "<p>Hello</p><p>World</p>"

After Example:
With string, you can locate elements based on their text content! 📝

Output: A

 list of elements containing the text "Hello".

Challenge: 🌟 Try searching for partial matches or case-insensitive text.

20. Modifying the DOM Tree (insert_before/insert_after) 🛠️

Boilerplate Code:

tag.insert_before(new_tag)
tag.insert_after(new_tag)

Use Case: Use insert_before or insert_after to insert elements into the DOM tree. 🛠️

Goal: Dynamically insert new elements before or after existing ones. 🎯

Sample Code:

# Create a new paragraph tag
new_paragraph = soup.new_tag("p")
new_paragraph.string = "This is a new paragraph."

# Insert the new paragraph after the h1 tag
h1_tag = soup.find('h1')
h1_tag.insert_after(new_paragraph)

Before Example:
You want to add new content in specific positions in the DOM. 🤔

Data: "<h1>Hello World!</h1>"

After Example:
With insert_before and insert_after, you can add new elements dynamically! 🛠️

Output: A new paragraph inserted after the `<h1>` tag.

Challenge: 🌟 Try inserting multiple elements at different locations in the DOM.

0
Subscribe to my newsletter

Read articles from Anix Lynch directly inside your inbox. Subscribe to the newsletter, and don't miss out.

BeautifulSoupweb scrapingPythonData Science

Written by

Anix Lynch
Anix Lynch