AWS SQS/SNS¶

Overview¶

AWS SQS (Simple Queue Service) and SNS (Simple Notification Service) are fully managed messaging services provided by Amazon Web Services. SQS provides reliable, highly-scalable message queuing, while SNS offers publish-subscribe messaging for distributed applications and microservices.

Data Model¶

Core Concepts¶

graph TB
    subgraph "AWS SNS (Pub/Sub)"
        TOPIC[SNS Topic]
        SUBS[Subscriptions]
    end

    subgraph "AWS SQS (Queuing)"
        QUEUE[SQS Queue]
        DLQ[Dead Letter Queue]
    end

    subgraph "Publishers"
        APP[Applications]
        LAMBDA[Lambda Functions]
    end

    subgraph "Subscribers"
        SQS[SQS Queues]
        EMAIL[Email]
        SMS[SMS]
        HTTP[HTTP Endpoints]
        LAMBDA2[Lambda Functions]
    end

    APP --> TOPIC
    LAMBDA --> TOPIC

    TOPIC --> SUBS
    SUBS --> SQS
    SUBS --> EMAIL
    SUBS --> SMS
    SUBS --> HTTP
    SUBS --> LAMBDA2

    APP --> QUEUE
    QUEUE --> DLQ

Service Models¶

• SQS Standard: At-least-once delivery, high throughput
• SQS FIFO: Exactly-once processing, message ordering
• SNS: Fan-out messaging to multiple subscribers
• SNS FIFO: Ordered message delivery to FIFO SQS queues

Message Format¶

{
  "Records": [
    {
      "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
      "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
      "body": {
        "orderId": "order-123",
        "customerId": "cust-456",
        "timestamp": "2025-01-11T16:56:59Z",
        "amount": 99.99,
        "items": [
          {
            "productId": "prod-789",
            "quantity": 2
          }
        ]
      },
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1641916619000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1641916619000"
      },
      "messageAttributes": {
        "correlationId": {
          "stringValue": "req-456",
          "dataType": "String"
        }
      }
    }
  ]
}

Architecture Overview¶

SQS Architecture¶

graph TB
    subgraph "AWS SQS Service"
        subgraph "Standard Queue"
            SQ[Standard Queue]
            MSG[Message Store]
        end

        subgraph "FIFO Queue"
            FQ[FIFO Queue]
            ORDER[Message Ordering]
        end

        subgraph "Features"
            VIS[Visibility Timeout]
            DLQ[Dead Letter Queue]
            BATCH[Message Batching]
        end
    end

    subgraph "Producers"
        PROD[Applications]
        LAMBDA[Lambda Functions]
        EC2[EC2 Instances]
    end

    subgraph "Consumers"
        CONS[Applications]
        LAMBDA2[Lambda Functions]
        ECS[ECS Tasks]
    end

    PROD --> SQ
    LAMBDA --> SQ
    EC2 --> FQ

    SQ --> MSG
    FQ --> ORDER

    SQ --> VIS
    FQ --> DLQ

    MSG --> CONS
    ORDER --> LAMBDA2
    VIS --> ECS

SNS Architecture¶

graph TB
    subgraph "AWS SNS Service"
        subgraph "Topics"
            TOPIC[SNS Topic]
            FIFO[FIFO Topic]
        end

        subgraph "Delivery"
            FANOUT[Fan-out Delivery]
            FILTER[Message Filtering]
            RETRY[Retry Logic]
        end
    end

    subgraph "Publishers"
        APP[Applications]
        CLOUDWATCH[CloudWatch]
        S3[S3 Events]
    end

    subgraph "Subscribers"
        SQS[SQS Queues]
        EMAIL[Email]
        SMS[SMS]
        HTTP[HTTP/HTTPS]
        LAMBDA[Lambda]
        MOBILE[Mobile Push]
    end

    APP --> TOPIC
    CLOUDWATCH --> TOPIC
    S3 --> FIFO

    TOPIC --> FANOUT
    FIFO --> FILTER

    FANOUT --> SQS
    FANOUT --> EMAIL
    FANOUT --> SMS
    FILTER --> HTTP
    FILTER --> LAMBDA
    FILTER --> MOBILE

    FANOUT --> RETRY

Target Operating Model (TOM)¶

Without High Availability¶

Basic Setup¶

Resource Requirements¶

Configuration Example¶

# CloudFormation template for basic setup
Resources:
  OrderQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: order-processing-queue
      MessageRetentionPeriod: 1209600  # 14 days
      VisibilityTimeoutSeconds: 300
      ReceiveMessageWaitTimeSeconds: 20

  OrderTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: order-notifications
      DisplayName: Order Processing Notifications

  QueueSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrderTopic
      Protocol: sqs
      Endpoint: !GetAtt OrderQueue.Arn

With High Availability¶

Enterprise Setup¶

Resource Requirements¶

Deployment Architecture¶

graph TB
    subgraph "Primary Region (us-east-1)"
        subgraph "SQS Queues"
            SQ1[Order Queue]
            SQ2[Payment Queue]
            DLQ1[Dead Letter Queue]
        end

        subgraph "SNS Topics"
            SNS1[Order Topic]
            SNS2[Notification Topic]
        end

        subgraph "Lambda Functions"
            L1[Order Processor]
            L2[Notification Handler]
        end
    end

    subgraph "Secondary Region (us-west-2)"
        subgraph "SQS Queues"
            SQ3[Order Queue Replica]
            SQ4[Payment Queue Replica]
            DLQ2[Dead Letter Queue]
        end

        subgraph "SNS Topics"
            SNS3[Order Topic Replica]
            SNS4[Notification Topic Replica]
        end
    end

    subgraph "Applications"
        APP[Web Application]
        API[API Gateway]
        MOBILE[Mobile App]
    end

    subgraph "Monitoring"
        CW[CloudWatch]
        ALERT[CloudWatch Alarms]
        DASH[CloudWatch Dashboard]
    end

    APP --> API
    MOBILE --> API
    API --> SNS1

    SNS1 --> SQ1
    SNS1 --> SQ2
    SNS1 --> SNS2

    SQ1 --> L1
    SQ2 --> L2

    SQ1 --> DLQ1
    SQ2 --> DLQ1

    SNS1 -.-> SNS3
    SQ1 -.-> SQ3
    SQ2 -.-> SQ4

    L1 --> CW
    L2 --> CW
    CW --> ALERT
    CW --> DASH

HA Configuration¶

# CloudFormation template for HA setup
Resources:
  OrderQueueFIFO:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: order-processing-queue.fifo
      FifoQueue: true
      ContentBasedDeduplication: true
      MessageRetentionPeriod: 1209600
      VisibilityTimeoutSeconds: 300
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt OrderDLQ.Arn
        maxReceiveCount: 3

  OrderDLQ:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: order-processing-dlq.fifo
      FifoQueue: true
      MessageRetentionPeriod: 1209600

  OrderTopicFIFO:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: order-notifications.fifo
      FifoTopic: true
      ContentBasedDeduplication: true

  CrossRegionReplication:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrderTopicFIFO
      Protocol: sqs
      Endpoint: !Sub 'arn:aws:sqs:us-west-2:${AWS::AccountId}:order-processing-queue-replica.fifo'

Pros and Cons¶

Pros¶

Fully Managed¶

• No Infrastructure: AWS handles all infrastructure
• Auto Scaling: Automatically scales with demand
• High Availability: Built-in redundancy and failover
• Maintenance Free: No patching or updates required

Reliability & Durability¶

• 99.999999999%: Eleven 9's durability guarantee
• At-least-once: Standard queue delivery guarantee
• Exactly-once: FIFO queue processing guarantee
• Dead Letter Queues: Built-in error handling

Integration & Ecosystem¶

• AWS Services: Native integration with AWS services
• IAM Integration: Fine-grained access control
• CloudWatch: Built-in monitoring and alerting
• AWS SDK: Support for all major programming languages

Cost Effectiveness¶

• Pay-per-use: No upfront costs
• Free Tier: 1 million requests per month
• Predictable Pricing: Clear pricing model
• No Idle Costs: Pay only for what you use

Cons¶

Vendor Lock-in¶

• AWS Specific: Tied to AWS ecosystem
• Migration Complexity: Difficult to migrate to other platforms
• API Dependencies: Applications depend on AWS APIs
• Regional Limitations: Limited to AWS regions

Feature Limitations¶

• Message Size: 256KB maximum message size
• Throughput Limits: 300 msgs/sec for FIFO queues
• Retention Limits: 14 days maximum retention
• Limited Routing: Basic routing capabilities

Cost Considerations¶

• High Volume Costs: Can be expensive for high volumes
• Data Transfer: Charges for data transfer between regions
• Storage Costs: Additional costs for message storage
• Request Pricing: Per-request pricing model

Operational Constraints¶

• Limited Customization: Cannot customize underlying infrastructure
• AWS Outages: Dependent on AWS service availability
• Limited Monitoring: CloudWatch-only monitoring
• Compliance: May not meet specific compliance requirements

Best Practices¶

Production Deployment¶

1. Queue Configuration
2. Choose appropriate queue type (Standard vs FIFO)
3. Configure dead letter queues
4. Set appropriate visibility timeouts

5. Enable long polling

6. Security

7. Use IAM policies for access control
8. Enable encryption in transit and at rest
9. Use VPC endpoints for private access

10. Implement least privilege principles

11. Monitoring

12. Set up CloudWatch alarms
13. Monitor queue depth and age
14. Track delivery failures

15. Use AWS X-Ray for tracing

16. Cost Optimization

17. Use batching to reduce requests
18. Implement message deduplication
19. Monitor usage patterns
20. Use reserved capacity where applicable

Development Guidelines¶

1. Error Handling
2. Implement proper retry logic
3. Handle throttling gracefully
4. Use dead letter queues effectively

5. Design for idempotency

6. Performance Optimization

7. Use batch operations when possible
8. Implement connection pooling
9. Use long polling for consumers

10. Optimize message size

11. Message Design

12. Keep messages small and focused
13. Use message attributes effectively
14. Implement proper message formatting
15. Version your message schemas

When to Choose AWS SQS/SNS¶

Ideal Use Cases¶

• AWS-Native Applications: Applications built on AWS
• Serverless Architectures: Lambda-based applications
• Microservices: Service-to-service communication
• Event-Driven Systems: Decoupled architectures
• Notification Systems: Multi-channel notifications

Consider Alternatives When¶

• Multi-Cloud: Need for cloud-agnostic solutions
• High Throughput: Millions of messages per second
• Complex Routing: Advanced routing requirements
• Large Messages: Messages larger than 256KB
• Long Retention: Need for longer message retention