Kinesis Data Streams (KDS) collects and processes a large amount of incoming data from an unlimited number of producers.
https://aws.amazon.com/kinesis
- Amazon Kinesis Data Streams is a scalable and durable real-time data streaming service.
KDS Features
- KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.
- The data collected is available in milliseconds to enable real-time analytics.
Use Cases
- Producers supply data to Kinesis, e.g., any IoT (Internet of Things) devices.
- Consumers are any entity that can consume the data.
- Kinesis Data Streams are used:
- Real-time analytics or feed data into other services in real-time with data retention.
- e.g.) analyze logs continuously or run real-time analytics on click system data
Data Streams
- By default, Lambda invokes your function as soon as records are available in the stream. Lambda can process up to 10 batches in each shard simultaneously. If you increase the number of concurrent batches per Shard, Lambda still ensures in-order processing at the partition-key level.
- Transient Data Store:
- Streams are deleted based on their rolling retention window.
- Retention Period
- 24-hour default
- ~ 7 days: resolve potential downstream data losses
- ~ 365 days: reprocess data, back-fill data stores, and auditing
- Kinesis Data Stream provides an ordering of records.
Kinesis Shards
- Kinesis Shards
- Shards are the capacity of a Kinesis Stream.
- Each shard has the same partition key, but the data are divided by the sequence number.
- Allows streams to scale. A stream starts with at least 1 shard. Shards can be added or removed from streams.
- Per Shard
- 5 read transactions per second, maximum 2 MB of consumption (read) capacity per second
- 1000 write records per second, up to maximum 1 MB per second of ingestion
- Kinesis Data Record
- Data record is the basic entity.
- Each shard consists of a sequence of data records.
- Data records are composed of a sequence number, a partition key, and a data blob. Data blob can be up to 1MB.
- Resharding
- Resharding enables you to increase or decrease the number of shards in a stream in order to adapt to changes in the rate of data flowing through the stream.
- Typically, each shard is processed by a single worker and has one corresponding record processor. All record processors run in parallel within a process.
- When you do the resharding, you can use Auto Scaling to automatically scale your instances.
Error Handling
The “PutRecords” API call to write multiple data records into a Kinesis data stream in a single call.
An unsuccessfully processed record includes ErrorCode and ErrorMessage values. The “ProvisionedThroughputExceededException” indicates that the request rate for the stream is too high, or the requested data is too large for the available throughput.
- Reduce the frequency or size of your requests
- Use an error retry and exponential backoff mechanism
- Distribute read and write operations as evenly as possible across all of the shards in Data Streams
- Finally, you can reshard your stream to increase the number of shards in the stream
Interacting with Kinesis Data Streams
- Kinesis Producer Library (KPL) passes data to Kinesis Data Stream.
- KPL provides the efficient abstraction layer for ingesting data with automatic retry and better performance
- Kinesis Client Library (KCL) delivers all records for a given partition key to the same record processor, making it easier to build multiple applications reading from the same Kinesis data stream.
- Kinesis API (AWS SDK) is used to interact with Kinesis Data Stream through love level API operations such as (PutRecord or GetRecords).
Working with Lambda
Kinesis Data Stream can be used as an event source for Lambda.
Kinesis Data Stream
- The number of Shards
Lambda
- Function batch size
- Concurrent batches per shard
- Batch window
Scriptorium 