Introduction

Have you ever wanted to automate weather data collection and store it securely in the cloud? In this post, I'll walk you through a simple Python project that fetches weather data from the OpenWeather API and saves it to an AWS S3 bucket. This project demonstrates API integration, cloud storage, and automation—essential skills for any data enthusiast or cloud engineer.

Project Overview

Core Concepts I will be Covering:

1. Fetching Data with API – Using OpenWeather API to get live weather updates.

2. Storing Data in AWS S3 – Automatically saving the data in a cloud-based storage solution.

3. Automation – Running the script to automatically create S3 bucket and for multiple cities and handling debugging and errors gracefully.

Architecrture

Project Setup

Step 1: Prerequisites

Before we start this project, ensure you have the following:

• Python 3+ installed

• An active AWS account (the project is free tier so don’t worry)

• OpenWeather API key (Get one from OpenWeather)

• Boto3 and Requests libraries (Install with pip install boto3 requests python-dotenv)

• A .env file to store API keys securely

Step 2: Writing the Python Script

Here’s the breakdown of our Python script:

• Step 2.1: Setting Up the Environment and Dependencies

  Before we start implementing the code, we need to:
  - Import necessary libraries (os, json, boto3, requests, etc.)
  - Load environment variables securely using dotenv

•     import os
      import json
      import boto3
      import requests
      from datetime import datetime
      from dotenv import load_dotenv
  
      # Load environment variables
      load_dotenv()
  

• Step 2.2: Initializing the Weather Dashboard Class

• - Retrieves API keys and bucket name from environment variables
  - Creates an S3 client for interacting with AWS storage

•     class WeatherDashboard:
          def __init__(self):
              self.api_key = os.getenv('OPENWEATHER_API_KEY')
              self.bucket_name = os.getenv('AWS_BUCKET_NAME')
              self.s3_client = boto3.client('s3', region_name='us-east-1')
  

• Step 2.3: Create an S3 bucket (check if it already exists).

• This function checks if the S3 bucket already exists using head_bucket() and creates a new bucket if needed

•     def create_bucket_if_not_exists(self):
              """Create S3 bucket if it doesn't exist"""
              try:
                  self.s3_client.head_bucket(Bucket=self.bucket_name)
                  print(f"Bucket {self.bucket_name} exists")
              except:
                  print(f"Creating bucket {self.bucket_name}")
              try:
                  self.s3_client.create_bucket(Bucket=self.bucket_name)
                  print(f"Successfully created bucket {self.bucket_name}")
              except Exception as e:
                  print(f"Error creating bucket: {e}")
  

• Step 2.4: Fetching Weather Data from OpenWeather API

  This function retrieves real-time weather data for a given city using OpenWeather API.

•     def fetch_weather(self, city):
              """Fetch weather data from OpenWeather API"""
              base_url = "http://api.openweathermap.org/data/2.5/weather"
              params = {
                  "q": city,
                  "appid": self.api_key,
                  "units": "imperial"
              }
  
              try:
                  response = requests.get(base_url, params=params)
                  response.raise_for_status()
                  return response.json()
              except requests.exceptions.RequestException as e:
                  print(f"Error fetching weather data: {e}")
                  return None
  

• Step 2.5: Saving Weather Data to AWS S3

  This function stores the fetched weather data in an AWS S3 bucket as a JSON file. It also generates a timestamped filename for organized storage and tracking

•     def save_to_s3(self, weather_data, city):
              """Save weather data to S3 bucket"""
              if not weather_data:
                  return False
  
              timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
              file_name = f"weather-data/{city}-{timestamp}.json"
  
              try:
                  weather_data['timestamp'] = timestamp
                  self.s3_client.put_object(
                      Bucket=self.bucket_name,
                      Key=file_name,
                      Body=json.dumps(weather_data),
                      ContentType='application/json'
                  )
                  print(f"Successfully saved data for {city} to S3")
                  return True
              except Exception as e:
                  print(f"Error saving to S3: {e}")
                  return False
  

• Step 2.6: Running the Weather Data Pipeline

  This function implements the process of fetching and storing the weather data for multiple cities. (The cities can be modified accordingly)

•     def main():
          dashboard = WeatherDashboard()
  
          # Create bucket if needed
          dashboard.create_bucket_if_not_exists()
  
          cities = ["Lagos", "Seattle", "New York"]
  
          for city in cities:
              print(f"\nFetching weather for {city}...")
              weather_data = dashboard.fetch_weather(city)
              if weather_data:
                  temp = weather_data['main']['temp']
                  feels_like = weather_data['main']['feels_like']
                  humidity = weather_data['main']['humidity']
                  description = weather_data['weather'][0]['description']
  
                  print(f"Temperature: {temp}°F")
                  print(f"Feels like: {feels_like}°F")
                  print(f"Humidity: {humidity}%")
                  print(f"Conditions: {description}")
  
                  # Save to S3
                  success = dashboard.save_to_s3(weather_data, city)
                  if success:
                      print(f"Weather data for {city} saved to S3!")
              else:
                  print(f"Failed to fetch weather data for {city}")
  
      if __name__ == "__main__":
          main()
  

Step 3 : Running the Python Script

•   OPENWEATHER_API_KEY=your_openweather_api_key
    AWS_BUCKET_NAME=your_s3_bucket_name
  

SUCCESS!!

Data fetched successfully

•   mac@macs-MacBook-Pro weather-dashboard-demo % python3 src/weather_dashboard.py
    Creating bucket weather-dashboard-152
    Successfully created bucket weather-dashboard-152
  
    Fetching weather for Lagos...
    Temperature: 88.63°F
    Feels like: 98.04°F
    Humidity: 63%
    Conditions: overcast clouds
    Successfully saved data for Lagos to S3
    Weather data for Lagos saved to S3!
  
    Fetching weather for Seattle...
    Temperature: 47.43°F
    Feels like: 44.08°F
    Humidity: 84%
    Conditions: overcast clouds
    Successfully saved data for Seattle to S3
    Weather data for Seattle saved to S3!
  
    Fetching weather for New York...
    Temperature: 61.36°F
    Feels like: 61.09°F
    Humidity: 83%
    Conditions: overcast clouds
    Successfully saved data for New York to S3
    Weather data for New York saved to S3!
  

S3 Bucket created Successfully

Conclusions

This was a comprehensive project that took some time to implement. I was able to refresh my skills by building and debugging gradually to perfect it.

I still need to read more on:

• how to configure an API gateway to route requests from its root url to Lambda

Till Next time✨