Building a Scalable SQS Consumer in Go

In this post, the author shares their experience implementing a robust SQS consumer in Go to handle user registration events for Keycloak. The solution uses the fan-out/fan-in concurrency pattern to process messages efficiently without overwhelming system resources. The fan-out/fan-in pattern is perfect for this use case because it maintains a fixed pool of worker goroutines, distributes work evenly across workers, prevents resource exhaustion, and provides better control over concurrent operations. The implementation consists of three main components: Message Receiver, Worker Pool, and Message Channel. Key configuration parameters include MaxNumberOfMessages, WaitTimeSeconds, and VisibilityTimeout. The worker pool is where the fan-out pattern comes into play, starting 10 workers to process messages concurrently. Duplicate message handling is ensured using a sync.Map. Best practices and learnings include error handling, message cleanup, graceful shutdown, and monitoring. Performance considerations involve choosing the right worker count, batch size, and visibility timeout. Future improvements could include dynamic worker scaling, circuit breaker, metrics collection, dead letter queue handling, and retries. The fan-out/fan-in pattern provides an elegant solution for processing high-volume SQS messages in Go, maintaining a fixed worker pool to avoid resource exhaustion and ensure efficient message processing.