[AWS] Kinesis

Kinesis is a fully managed, scalable, and resilient streaming service. It is designed to ingest a large amount of data in real-time from many producers.

  • Streaming Data: online purchase, stock prices, social network data, geospatial data, or game data
  • Use Kinesis (rather than SQS) when you need many producers/consumers.
  • Kinesis consists of a couple of services:
    • Video Streams
    • Data Streams
    • Data Firehose
    • Data Analytics.

Kinesis Data Firehose

Kinesis Firehose is a fully managed service that loads streaming data into data stores (S3) and analytics tools (Redshift, Elasticsearch, or Splunk).

  • Kinesis has shards to store data for some time (data persistence), but Firehose does not save the data.
  • Firehose can optionally invoke an AWS Lambda function to transform incoming data before delivering it to destinations. But Lambda functions can not be a destination. The destinations are storage/analytic services.
  • Kinesis Firehose is used when:
    • collect streaming data and deliver to the destination quickly
    • processing is optional, and data retention is not important
    • e.g.) capturing data from IOT devices and stream into a data lake

Kinesis Video Streams

Kinesis Video Streams securely stream video (audio or images) from connected devices to AWS for analytics, machine learning, and other processing.

Use Case: Amber Alert System

Kinesis Data Analytics

Kinesis Data Analytics analyzes streaming data and gains actionable insights in real-time.

  • You can provide the SQL command against the input stream, and the result is passed to the destination in real-time.
  • When to use:
    • Run SQL queries on streaming data and provides insight on your data.
    • Create metrics, dashboard, notification, and alarms
    • e.g.) send real-time alarms when certain metrics reach the predefined threshold

Use Case: Click System Analysis

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s