Sqs

Occasionally, you’ll do something that you think you’ve done dozens of times before and are then surprised it no longer works. While setting up a log delivery mechanism for Splunk, I had one of these experiences again. (Feel free to replace relearning with forgetting in the headline.) Splunk’s preferred method of ingesting log data from AWS is the SQS-based S3 input. In a nutshell, you ensure that all logs end up in an S3 bucket. That bucket is configured to send all object create events to an SNS topic (so that multiple systems can subscribe), to which an SQS queue is subscribed. Splunk subsequently consumes the object create events from the queue and ingests the corresponding objects from S3. ...