This project demonstrates the deployment of the ELK Stack on an AWS EC2 instance and the ingestion of Flaws.cloud CloudTrail logs into the ELK Stack for threat detection.

💡

Side Note: When you click on the link in each step, you will be redirected to Notion pages where you can follow the detailed steps to set up your own EC2 instance for the ELK stack and Investigate threats or events from Flaws.cloud CloudTrail logs.

Step 1: Setting up ELK Stack On AWS EC2

This step provides detailed instructions on deploying the ELK Stack on an AWS EC2 instance running Ubuntu 20.04 LTS, using a t2.medium instance type."

Step 2: Initial Configuration for ELK Setup

This step demonstrates detailed instructions on deploying the ELK (Elasticsearch, Logstash, and Kibana) login page for security purposes.

Step 3: Ingesting Flaws.cloud CloudTrail Logs to ELK - Guide

The Logs ingested to ELK are the flaws.cloud AWS CloudTrail dataset to identify unusual patterns and detect potential security threats.

CloudTrail Log Dataset Summary

The Log Dataset used in this lab is the Public dataset of Cloudtrail logs from flaws.cloud which can be accessed Here.

You can Read through the blog post Here. md5: 4f0481c6be700ffc7610dbbf8cee5578 Size: ~240MB of log data

💡

Unfortunately, the File upload restriction to ELK is 100MB, so the FlawsReadyToIngest.json file can’t be uploaded. So, therefore, there is a need to split the data into multi-files for easy upload. A Python program has been created to separate the FlawsReadyToIngest.json into chunks of 50MB files.

Flaws.cloud cloud-trail dataset cleaning

🔥 Python Code to Convert Multiple CloudTrail Json files to a single Json File. - Helper Script aws-cloudtrail2sof-elk.py from the SOF-ELK repository.

🔥 Python Code to Split a Single JSON File into chunks of JSON files of 50MB - Split_json.py

The Pyhton Scripts can be access via my GitHub page.

Ingesting 50MB chunks of CloudTrail Log Files to ELK

Step 4: Access the Logs and investigate cloud threats

Threat detection🕵️‍♂️ With Flaws.cloud CloudTrail Logs using ELK Stack

Table of contents