Add Functionality to Poll Logs from AWS CloudWatch and Send to SigNoz

[ravimahey]

As part of the setup process for SigNoz with AWS ECS, it has been observed that there is no built-in simple solution in the market to collect logs from AWS ECS while retaining the ECS logs in AWS CloudWatch. Many clients have requested a solution where they can keep their ECS logs in CloudWatch, but also send those logs to SigNoz for centralized observability.

Implement functionality that allows for:

• Polling logs from AWS CloudWatch.
• Forwarding these logs to SigNoz.
• Ensuring ECS logs are kept in CloudWatch, while also being exported to SigNoz for processing.

Current setup and its drawback

Currently, services like Firehose and Log Forwarder are utilized to send logs from the ECS sidecar. While this is a common practice, especially seen in tools like Datadog and other monitoring solutions, this setup has several limitations:

• Logs are forwarded but the configuration and setup complexity are often high.
• Logs are sent to the monitoring solution, but there's no straightforward mechanism to keep them in AWS CloudWatch and also forward them to SigNoz.
• Many clients require a simple, seamless way to keep logs within CloudWatch for debugging when service gets down, while also exporting them for analysis in SigNoz.