Design distributed messaging queue system like SQS

Learning about SQS

SQS is a fully managed distributed queue system which comes in two flavours:

1. Standard Queue:

   1. Unlimited throughput -> no restrictions
   2. At-least once delivery -> occasionally deliver more than once so application code must handle it through Idempotency.
   3. Best effort ordering -> messages may come occasionally out of order.
   4. Usage:
      1. Decouple and process background jobs
      2. Batch message and doing heavy processing

2. FIFO Queue:

   1. Limited throughput -> as maintaining FIFO order would bound system to use a single node to server (can't fully scalable like standard queue) everything hence limited throughput bound by that system bandwidth.
   2. Exactly once processing
   3. Ordering is FIFO
   4. Usage:
      1. User entered commands needs to be processed in order
      2. Prevent a student from enrolling in a course if event of admission hasn't reached.

Features to know:

1. Payload size: Each message is counted as single for payload size at max 64KB. One message can have upto 256KB but billed for 4 messages.
2. Batching support: Each batch can have at-most 10 messages and billed as single message.
3. Long polling: If manual consumer like EC2, ECS then prefer long polling. For lambda, it is handled by AWS internally.
4. Fair queue: A high performant tenant doesn't impact throughput/latency of normal tenant.
5. Message retention: upto 14 days
6. Message locking/visibility timeout: Once a consumer consumes a message it becomes in-visible for other consumers till visibility timeout.
7. DLQ support: When retries have exhausted or other issues are there.