Amazon Kinesis Overview

Amazon Kinesis is a fully managed service for real-time data streaming at scale. It enables you to collect, process, and analyze streaming data efficiently.

Key Features

1. Real-time Data Processing – Allows ingestion and processing of data with low latency.

2. Scalability – Seamlessly scales to handle gigabytes of data per second.

3. Multiple Data Stream Options – Offers Kinesis Data Streams, Kinesis Firehose, Kinesis Data Analytics, and Kinesis Video Streams.

4. Durability & Reliability – Stores data for up to 7 days (default: 24 hours) with replication across multiple Availability Zones.

5. Integration with AWS Services – Works with Lambda, S3, Redshift, EMR, and other AWS services.

6. Serverless Architecture – No need to manage servers; pay only for the data processed.

7. Security – Supports encryption (KMS), IAM-based access control, and VPC connectivity.

How It Works

1. Data Ingestion – Data is continuously ingested from sources like IoT devices, applications, logs, and social media.

2. Processing – The data flows through Kinesis Streams, where consumer applications process and analyze it.

3. Delivery – Processed data is stored in S3, Redshift, or a custom destination for further analytics.

Use Case Example

📌 Real-time Log Analytics

• A company collects logs from multiple web applications using Kinesis Data Streams.

• AWS Lambda processes the data to detect anomalies.

• The processed logs are stored in Amazon S3 for historical analysis with Amazon Athena.

Topics for AWS DVA-C02

• Kinesis Data Streams (Shards, Producers, Consumers, Retention Periods)

• Kinesis Firehose (Destination Options, Data Transformation with Lambda)

• Kinesis Data Analytics (SQL Queries, Real-time Analytics, Window Functions)

• Security (IAM Policies, Encryption, PrivateLink, VPC Endpoints)

• Comparison with Other AWS Services (SNS, SQS, EventBridge, MSK)