Building an NBA Data Lake project

Introduction

This is the Day 3 of DevopsAllStarsChallenge

This project automates the creation of a data lake for NBA analytics using AWS services. A Python script integrates Amazon S3, AWS Glue, and AWS Athena, and sets up the infrastructure needed to store and query the NBA-related data.

Architectural Diagram

Prerequisites

SportsData - Create an account and get an API Key

An IAM credentials are needed for the AWS CLI and to connect to the AWS services from the terminal.

Project Setup

1. Get the API KEY from SportsData above

2. Create the file directory as shown below

3. Go to the GitHub account, copy the content of setup_nba_data_lake.py

4. The .env file should have the following:

SPORTS_DATA_API_KEY=your_sportsdata_api_key #YOUR API KEY
NBA_ENDPOINT=https://api.sportsdata.io/v3/nba/scores/json/Players

5. We are working with VS code, hence, the AWS CLI should be configured in the terminal. Once the AWS CLI is installed, the IAM should be configured in the terminal using the

aws configure

6. Add the Access key and the Secret Access Key. You should have something as shown below using,

    aws configure list
   

   

7. Create the virtual environment to run the Python program. The requirements.txt file should be in the root of the directory.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

8. The .gitignore file should have these files. This will ensure that the API Key is not exposed on Git and the dependencies are not added to the GitHub repository.

.env
.venv

Run the Script

S3

Glue

Before you can query Athena, you must configure the S3 bucket with Athena.

Athena result

Delete the resources to avoid unnecessary charges from AWS. This can be done by running the delete_aws_resources.py script.

Thank you for reading. Please follow, like, comment, and share for a wider reach and more DevOps-related articles.