Kinesis is a fully managed, scalable, and resilient streaming service. It is designed to ingest a large amount of data in real-time from many producers.
- Streaming Data: online purchase, stock prices, social network data, geospatial data, or game data
- Use Kinesis (rather than SQS) when you need many producers/consumers.
- Kinesis consists of a couple of services:
- Video Streams
- Data Streams
- Data Firehose
- Data Analytics.
Kinesis Data Firehose
Kinesis Firehose is a fully managed service that loads streaming data into data stores (S3) and analytics tools (Redshift, Elasticsearch, or Splunk).
- Kinesis has shards to store data for some time (data persistence), but Firehose does not save the data.
- Firehose can optionally invoke an AWS Lambda function to transform incoming data before delivering it to destinations. But Lambda functions can not be a destination. The destinations are storage/analytic services.
- Kinesis Firehose is used when:
- collect streaming data and deliver to the destination quickly
- processing is optional, and data retention is not important
- e.g.) capturing data from IOT devices and stream into a data lake
Kinesis Video Streams
Kinesis Video Streams securely stream video (audio or images) from connected devices to AWS for analytics, machine learning, and other processing.
Use Case: Amber Alert System
Kinesis Data Analytics
Kinesis Data Analytics analyzes streaming data and gains actionable insights in real-time.
- You can provide the SQL command against the input stream, and the result is passed to the destination in real-time.
- When to use:
- Run SQL queries on streaming data and provides insight on your data.
- Create metrics, dashboard, notification, and alarms
- e.g.) send real-time alarms when certain metrics reach the predefined threshold
Use Case: Click System Analysis
