This project aims to automate the creation of a data lake for NBA analytics using AWS services. By integrating Amazon S3, AWS Glue, and Amazon Athena, we can set up the infrastructure needed to store and query NBA-related data efficiently.

Project Overview

The project consists of two main scripts:

1. nba_datalake.py: Running this script performs the following actions:

• Creates an Amazon S3 bucket to store raw and processed data.
• Uploads NBA data (JSON format) to the S3 bucket.
• Creates an AWS Glue database and an external table for querying the data.
• Configures Amazon Athena for querying data stored in the S3 bucket.

2. delete_resources.py: Running this script performs the following actions:

• Deletes the Amazon S3 bucket and its contents.
• Deletes the AWS Glue database and associated tables.
• Deletes Athena query results stored in the S3 bucket.

Prerequisites

Before running the scripts, ensure you have the following:

• AWS account with appropriate permissions.
• AWS CLI configured with your credentials.
• Python 3.x or higher installed.
• Required Python packages installed (see requirements.txt file).

Setup

Step 1: Clone the Repository

First, clone the repository to your local machine:

git clone https://github.com/Peterkinywa/NBADataLake
cd NBADataLake

Step 2: Install Required Python Packages

Install the necessary Python packages using pip:

pip install -r requirements.txt

Step 3: Create a .env File

Create a .env file in the src directory with your API key and endpoint:

cd src
touch .env

Add the following content to the .env file:

SPORTS_DATA_API_KEY=<your_sportsdata_api_key>
NBA_ENDPOINT=https://api.sportsdata.io/v3/nba/scores/json/Players

Usage

Setting Up the Data Lake

Run the nba_datalake.py script to set up the data lake:

python src/nba_datalake.py

This script will:

1. Create an S3 Bucket: An S3 bucket will be created to store raw and processed NBA data.
2. Create a Glue Database: An AWS Glue database will be created to catalog the data.
3. Fetch NBA Data: NBA data will be fetched from the Sportsdata.io API.
4. Upload Data to S3: The fetched data will be uploaded to the S3 bucket.
5. Create a Glue Table: An external table will be created in AWS Glue to query the data.
6. Configure Athena: Amazon Athena will be configured to query the data stored in the S3 bucket.

Querying Data with Athena

After setting up the data lake, you can query the data using Amazon Athena. Here are the steps:

1. Go to the Amazon Athena console.
2. Select the database created by the script.
3. Write and execute SQL queries to analyze the NBA data.

Deleting the Data Lake Resources

Run the delete_resources.py script to delete the data lake resources:

python delete_resources.py

This script will:

1. Delete the S3 Bucket: The S3 bucket and its contents will be deleted.
2. Delete the Glue Database: The AWS Glue database and associated tables will be deleted.
3. Delete Athena Query Results: Any Athena query results stored in the S3 bucket will be deleted.

IAM Policy

The iam_role_policy.json file contains the IAM policy required for the scripts to execute successfully.

Conclusion

This project demonstrates how to automate the creation of a data lake for NBA analytics using AWS services. By leveraging Amazon S3, AWS Glue, and Amazon Athena, we can store and query large datasets. Check out the full project on GitHub and try it out for yourself!

GitHub Repository: NBA Data Lake on AWS

#AWS #DevOps

Building a Weather Dashboard using AWS S3 and OpenWeather API Integration

Project Visualization

For this project, we developed a weather dashboard that fetches real-time weather data for multiple cities through an external API integration (OpenWeather API). This data is automatically stored in an AWS S3 bucket and displayed to users.

Features

• Fetches weather data for multiple cities from OpenWeather.
• Stores weather data in an AWS S3 bucket.
• Displays temperature, humidity, and weather conditions.

Technical Architecture

• Language: Python 3.13.1
• Cloud Provider: AWS (S3)
• External API: OpenWeather API Link
• Dependencies:

boto3 (AWS SDK Python used to interact with AWS services like S3)
python-dotenv (Manages our environment variables)
requests (We are able to make HTTP requests to the OpenWeather API)

Project Structure

weather-dashboard/
  src/
    __init__.py
    weather_dashboard.py
  tests
  .env
  .gitignore
  requirements.txt

Setup Instructions

1. Clone the repository:

git clone https://github.com/Peterkinywa/weather-dashboard-demo

2. Navigate to the project directory:

cd weather-dashboard

3. Install dependencies:

pip install -r requirements.txt

4. Configure environment variables (.env):

OPENWEATHER_API_KEY=your_api_key
AWS_BUCKET_NAME=your_unique_bucket_name

5. Configure AWS credentials:

aws configure
- AWS Access Key ID
- AWS Secret Access Key
- Default region name [e.g., us-east-1]
- Default output format [e.g., json]

6. Run the application:

python src/weather_dashboard.py

7. View the fetched weather data in your terminal, displaying temperature, humidity, and weather conditions.

8. Navigate to your AWS Console and check your AWS S3 Bucket for weather data files stored in the bucket.

9. Empty your bucket and delete your S3 bucket to avoid incurring any unbudgeted costs.

Happy Developing!

Project Overview

This project is an event-driven alert system that sends real-time NBA game day score notifications to subscribed users via Email.

Features

• Fetches live NBA game info using the external NBA API.
• Transforms the info into a human-readable format and publishes to an Amazon SNS Topic.
• Sends game notifications to subscribers through Email.
• Automates regular updates and invocations using Amazon EventBridge.

Prerequisites

• Free account with subscription and API Key at sportsdata.io
• Personal AWS account (Free Tier)
• Basic understanding of AWS and Python

Technical Architecture

Technologies

• Cloud Provider: AWS
• Core AWS Services: SNS, Lambda, EventBridge
• External API: NBA Game API (SportsData.io)
• Programming Language: Python 3.x
• IAM Security:
• Least privilege policies for Lambda, SNS, and EventBridge.

Project Structure

bash

game-day-notifications/
├── src/
│ ├── nba_notifications.py # Main Lambda function code
├── policies/
│ ├── nba_sns_policy.json # SNS publishing permissions
│ ├── nba_eventbridge_policy.json # EventBridge to Lambda permissions
│ └── nba_lambda_policy.json # Lambda execution role permissions
├── .gitignore
└── README.md # Project documentation

Setup Instructions

Clone the Repository

git clone https://github.com/Peterkinywa/game-day-notifications
cd game-day-notifications

Create an SNS Topic

1. Open AWS Management Console, navigate to SNS, and create a topic (e.g., nba_topic).

Add Subscriptions to the SNS Topic

1. Select the topic, click “Create subscription”, and enter a valid email address.
2. Confirm the subscription via the email link.

Create the SNS Publish Policy

1. In IAM, go to Policies → Create Policy.
2. Paste the JSON policy from nba_sns_policy.json, replacing REGION and ACCOUNT_ID.
3. Name the policy (e.g., nba_sns_policy.json) and create it.

Create an IAM Role for Lambda

1. In IAM, go to Roles → Create Role.
2. Select AWS Service → Lambda.
3. Attach nba_sns_policy and AWSLambdaBasicExecutionRole.
4. Name the role (e.g., nba_role) and create it.
5. Save the role ARN.

Deploy the Lambda Function

1. In Lambda, click “Create Function”.
2. Name it (e.g., nba_notifications), choose Python 3.13, and assign nba_role.
3. Copy code from src/nba_notifications.py into the editor and deploy.
4. Add environment variables: NBA_API_KEY and SNS_TOPIC_ARN.
5. Create the function.

Set Up Automation with EventBridge

1. In EventBridge, create a rule with Event Source: Schedule.
2. Set the cron schedule (e.g., cron (0/10 8-23 * * ? *)).
3. Select the Lambda function (nba_notifications) as the target.

Test the System

1. Create a test event in Lambda and run it.

2. Verify SMS notifications are sent to subscribers.

Here’s my GitHub Repository https://lnkd.in/dPuQjSVQ

Here’s my LinkedIn post https://www.linkedin.com/posts/peterkinywamutua_devopsallstarschallenge-aws-activity-7283431069360533504-oIt_?utm_source=share&utm_medium=member_desktop

This project was designed with security in mind, following the principle of least privilege for IAM roles.

#DevOps #AWS