Hey everyone! Today, I dove into web scraping, and I’m excited to share my experience. I built a project that collects company data from a real website, processes it, and saves it in a CSV file — all using Python!

🧐 Why Web Scraping?

Web scraping is a powerful tool for automating data collection from websites. Whether you’re gathering business leads, tracking product prices, or analyzing trends, scraping saves time and enables you to turn unstructured web data into actionable insights.

🔧 Tools I Used

I used the following Python libraries:

✅ requests – To fetch web pages.
✅ BeautifulSoup – To parse and extract data from HTML.
✅ pandas – To structure the data and save it into a CSV file.

📖 What I Did

1️⃣ Sent HTTP requests to the website to get HTML content of pages containing US company information.
2️⃣ Used BeautifulSoup to parse the HTML and extract company names, addresses, and phone numbers.
3️⃣ Collected all the scraped data into a pandas DataFrame.
4️⃣ Exported the cleaned data to a CSV file named united states companies.csv.

📂 The Files

My project includes: CHECK MY GITHUB(https://github.com/philkasumbi)

real site webscrapping.ipynb – A Jupyter Notebook with all the scraping, parsing, and exporting steps.
united states companies.csv – The structured dataset containing the scraped company details.

🚨 Key Takeaways

✅ Web scraping requires understanding the structure of the website’s HTML.
✅ BeautifulSoup makes it easy to locate and extract specific elements like <div>, <span>, or table rows.
✅ pandas is super helpful for cleaning and saving the scraped data.
✅ Always check a site’s robots.txt and terms of service before scraping!

📌 Challenges I Faced

🔹 Figuring out the right CSS selectors to get exactly the data I needed.
🔹 Handling pages with inconsistent formatting.
🔹 Making sure the data was cleaned before exporting it.

🚀 What I Learned Today: Web Scraping Real-World Data with Python