9. Messaging & Async Processing

Question: How would you implement idempotent message processing from a queue?

Answer: Idempotency ensures that processing the same message multiple times has the same effect as processing it once. This can be achieved by using a unique identifier from the message (or a hash of its content) and checking a persistent store (like a database table or Redis) to see if this ID has been processed before.

Explanation: This is often implemented with the "outbox pattern." The business logic and the record of the processed message ID are committed together in a single atomic database transaction. This prevents a failure between processing the message and recording its completion, which would lead to reprocessing.

Question: What is a "dead-letter queue" (DLQ) and why is it important?

Answer: A dead-letter queue is a separate queue where messages are sent after they have failed processing a certain number of times. This prevents "poison messages" (malformed or problematic messages) from getting stuck in a retry loop and blocking the main queue.

Explanation: Implementing retries with exponential backoff and jitter is a standard pattern for handling transient failures. However, if a message repeatedly fails, it's likely a persistent issue. Moving it to a DLQ allows the main queue to continue processing while the failed messages can be inspected and handled manually or by a separate process later.

Question: What delivery guarantees do typical queues provide?

Answer: Most systems provide at-least-once delivery; exactly-once is rare and expensive; at-most-once risks loss.

Explanation: Design consumers to be idempotent. Use deduplication keys, transactional outbox/inbox, and make side effects safe to retry.

Question: How do you handle ordering and visibility timeouts?

Answer: Ordering is not guaranteed unless using partitioned/ordered queues. Respect visibility timeouts by finishing work or extending the lease before it expires.

Explanation: Shard by key to maintain per-key order when required. Use heartbeats to extend visibility during long processing.