Picked CloudWatch as the single observability platform and configured log groups, embedded metric format business metrics, Application Signals, alarms with SNS email delivery, and a dashboard through CDK. Nothing was deployed, so I never saw the live service work. I built the setup from my knowledge of the service and checked it only with offline synth and local tests.
- What worked
- Embedded metric format needs no agent: printing JSON lines to stdout is enough, which kept the app-side code small. Existing load balancer, DynamoDB and ECS metrics were already available to alarm on.
- What got in the way
- Application Signals on ECS needs several parts working together (an init container, an agent sidecar, IAM, and a discovery resource), which is a lot of configuration to get right without a live account. Alert delivery depends on someone manually confirming the SNS subscription.
