Learning AWS Day by Day — Day 68 — Amazon Kinesis Data Firehose

Exploring AWS !!

Day 68

Amazon Kinesis Data Firehose

- Managed AWS service for delivering real-time streaming data.
- Delivery destinations include S3, Amazon Redshift, Amazon OpenSearch Service, Amazon OpenSearch Serverless, Splunk, and any custom HTTP endpoint.
- No need to write or manage applications, just need to configure data producers to send data using Firehose and your data will be delivered to the desired destination.
- You can also configure it to transform your data before delivering it.

Terms:
- Firehose stream: underlying entity of Data Firehose, Firehose stream needs to be created and data is been sent to stream in order to create a Data Firehose.
- record: data of interest that is been sent to the stream, and can be up-to 1000 kb.
- data producer: producers send records to Data Firehose, like let’s say any web server that sends log data to streams is a producer.
- buffer size and buffer interval: Firehose buffers incoming data for a certain time or size before delivering to destinations.

Data Flow If the destination is S3, the streaming data is delivered to S3 buckets, if data transformation is enabled, then you can backup source data to another bucket.

Data Flow — Destination: S3

Data Security Data protection is the main motive and highest priority of AWS.
- Data Firehose encrypts data in transit using TLS.
- Data stored in interim storage during processing, Firehose encrypts data using KMS and verifies integrity using checksum.
- In case of sensitive data, enable server-side encryption