[AWS] Amazon SQS

Queues are used to implement loosely coupled systems by allowing component-to-component communication using messages.

Amazon Simple Queue Service (Amazon SQS) is a fully managed, highly available message queue service to create distributed/decoupled architecture.

---

Amazon Simple Queue Service (SQS)

A queue is a temporary repository for messages that are waiting for processing. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. It allows asynchronous processing.

Messages are added to a queue and retrieved via polling. (Pull-based).

1. One resource writes a message to a queue.
2. The queue keeps the message for a while.
3. And then, another resource will retrieve it from the queue.

Short Polling

• Default – WaitTimeSeconds setting is 0
• Available messages are returned right away, might return 0 messages, cause a lot of API calls.

Long Polling

• Wait for messages for a given interval (WaitTimeSeconds setting – 1 ~ 20 seconds)
• Reduces API requests and will have fewer empty API responses

Configuration

• When a message is polled, it is hidden and can be deleted when the processing is completed. Otherwise, after a VisibilityTimeout period, the message will return to the queue (auto return).
• Queues can be configured with a maxReceiveCount.

Bad messages

• Dead-Letter Queues: used to deal with a malformed message from consumers
• To delete a message, you need the queue URL and the receipt handle.

Access and Security

• SQS can encrypt message data using KMS.
• To grant access to SQS queues to another AWS account, SQS resource-based policies can be used.
• Lambda functions can be a good choice to handle requests based on messages – Scaling and fast response.

---

SQS Settings

• Delivery Delay
  • The behavior is controlled by the “DelaySeconds” setting.
  • 0 (default) up to 15 minutes
  • A message is hidden during the specified period.
• Message Size
  • up to 256 KB text of any format
• Encryption
  • Encrypted in transit by default
  • AT-Rest Encryption
    • You have an option to enable.
    • SQS provides the SSE (Server-side encryption) using SQS-owned encryption (SSE-SQS) as a default.
• Message Retention
  • The behavior is controlled by the “MessageRetentionPeriod ” setting.
  • 1 minute up to 14 days (4 days default)
• Long vs. Short
  • Short Polling is the default, but you should use long polling in most cases.
  • You can control it by setting the “Receive Message Wait Time” (WaitTimeSeconds) setting. (0~20 seconds)
    • 0: short polling
    • ~ 20: long polling
• Queue Depth
  • Can be used for auto-scaling
• Visibility Timeout
  • Once the message is pulled from the queue, the message becomes hidden for a while so that it cannot be accessed twice.
  • The behavior is controlled by the “VisibilityTimeout” setting.
  • up to 12 hours (30 seconds default)

---

Types of SQS Queues

SQS offers two types of message queues.

Standard Queues

• Guaranteed delivery of each message at least once with best-effort ordering (ordering is not guaranteed, and there might be duplicate messages) and at-least-once delivery.
• It supports a nearly unlimited throughput.
• It supports multiple producers and multiple consumers.

FIFO queues

• Guarantee that messages are sent and processed exactly once, in the exact order that they are sent.
• It supports multiple producers but only supports multiple consumers through group IDs.
• Messages are processed in order with respect to the group.
• Limited throughput, 300 messages/sec.

---

Dead-Letter Queue

Messages in the SQS can be back when it is not processed after the “VisibilityTimeout“. When you fail to process the message a couple of times, you can move the message to the temporary queue before removing it completely.

• You need to create the dead-letter queue before creating the main queue.
• When you create a main queue, set the Dead-Letter queue with the maximum number of tries.

---

Working with Lambda

SQS and Lambda can be used to create a serverless application. Here is what you need to configure:

SQS

• Select the queue type
  • Standard (preferred) or FIFO (only if required)
• Set the Visibility Timeout
  • The visibility timeout must be a couple of times (six times as recommended) more the the function timeout.
• Set the Dead-letter Queue

Lambda Function

• Batch Size
  • higher for fast workloads and lower for long workloads
• Timeout
• Handle duplicate messages
  • Design for idempotency

---

SQS Extended Client Library

You can manage large SQL messages using S3 through SQS Extended Client Library.

• It is supported only for Java SDK.
• Specify whether messages are always stored in Amazon S3 or only when the size of a message exceeds 256 KB
• Send a message to an S3 bucket
• Retrieve the message object from an S3 bucket
• Delete the message object from an S3 bucket