Amazon SQS

What does Amazon SQS do and what still runs your code?

The app sends a message to the queue. A worker, which is code running separately from the visitor’s page, takes it and makes the report. SQS stores and delivers the message. Your worker and file storage are separate.

Keep the queue private. Give the sending app only the permissions it needs to send to that queue. Give the worker its own permissions to receive and remove messages. Put a request ID in the message and let the worker look up what it needs.

Why can a message come back after the worker starts?

Receiving a message hides it from other workers for a limited time. That period is called the visibility timeout. The worker deletes the message after the result is safely saved. If it stops first or takes too long, the message can become available again.

Standard queues can deliver duplicates and may deliver out of order. FIFO queues order messages within a group and can prevent repeated sends with the same ID within five minutes. Your worker still needs to make repeats safe. Configure a separate dead-letter queue for messages that keep failing, and check whether the result already exists before retrying from it.

How would you check that one request makes one report?

On paper, draw a queue, a worker and a place to save a fictional report. Use no AWS account, live worker, storage bucket or real data.

1. 1
   Mark the first receipt

   Put one request ID in the queue. Draw the worker picking it up and the message becoming temporarily hidden.
2. 2
   Stop after the save

   Mark one saved file and one result record. Pretend the worker stops before deleting the queue message. When the message returns, mark that the next worker finds the existing result.
3. 3
   Mark a job that never finishes

   Draw a separate request that repeatedly fails before saving. Mark it for review in the failed-message queue and keep its app status separate from the successful one.

What should you count and keep outside SQS?

AWS lists one million SQS requests free each month. Sending, receiving, deleting and changing visibility count as requests. Billing counts payload in 64 KB units. Rates can depend on the queue type and region, and the worker and file storage have separate costs.

Use queue metrics to see waiting or failed work. Keep the app’s request and result records, completed files and permissions outside the queue. Before moving, decide how to handle waiting messages and confirm which reports are already complete.