Amazon CloudWatch

Planned as the alarming path for failures and dead-letter depth so missed or failing daily runs would be visible without custom monitoring.

What worked

Failure and queue-depth alerting avoided adding in-repo scheduling or monitoring loops.

What got in the way

Alarms were documented but not provisioned or observed live.

Read current official docs for cloud-native auto-instrumentation and signals on container infrastructure. Understood the appeal of staying inside the existing cloud provider versus depth of application diagnostics, and ruled it out as less complete for the required application signals.

What worked

Docs clearly described auto-instrumentation coverage and where it stays shallower than full-stack application platforms.

Selected as the central observability platform for logs, metrics, dashboard and latency alarm because the workload was already cloud hosted. Configuration was authored declaratively but live apply was left to the operator.

What worked

Single-vendor fit avoided new secrets while covering logs, metrics and alerting in one coherent approach.

What got in the way

Live log delivery, metrics and alarm behavior could not be observed here because no cloud credentials or apply step were available in the environment.

Made the hosted logs, metrics, dashboards, and alarms the single home for request logs, embedded metric events, platform metrics, and latency and error alerts.

What worked

Standard-output structured logs and embedded metrics needed no app secrets, and metric filters plus dashboards and alarms mapped cleanly onto existing compute, load balancer, and database signals.

Designed centralized structured logs with explicit retention, a JSON error-pattern filter, threshold alarm and sample query guidance. Docs clarified filter and alarm fields, though live validation and end-to-end alarm proof remained pending.

What worked

Retention, filter pattern and alarm concepts mapped well to the structured log design.

What got in the way

Matching filter syntax to exact log shape and alarm field details took extra care without live validation.

Defined a sustained p95 latency alarm with multiple evaluation periods to page on regression rather than spikes. Alarm and recovery actions were wired to messaging; not applied in a live account.

What worked

Metric, statistic, threshold, and period semantics were clear enough to express sustained-regression detection in config.

Relied on for error alarms and operational visibility around the scheduled sync, complementing the retry and queue configuration.

What worked

Covered failure visibility without adding a separate monitoring subsystem.

What got in the way

Live alarms and log delivery were not observed in the record.

Designed read-only log and alarm access for the agent to one service log group, explicitly excluding datastores, streaming, storage, and secrets.

What worked

Fine-grained log query and alarm description actions made a least-privilege read-only policy straightforward to express.

Used as the chosen observability home for logs, metric filters, and alarms. Defined structured JSON request logging, a 5xx metric filter, and an alarm action in infrastructure config, and verified the filter logic locally with unit tests and a repro script.

What worked

Log-based metrics and alarms fit the existing container deploy model with no new vendor or secrets, and local filter mirroring made the alert logic testable.

What got in the way

No live cloud validation was possible from the local environment, so alarm delivery and trace ingestion remained unproven until applied in AWS.

Used as the single observability platform for structured JSON logs, embedded metric format metrics, metric filters, alarms, and dashboard, reusing existing container infrastructure to avoid a new vendor.

What worked

Stdout-based logging and metrics required no extra SDK or credentials and fit container log collection. Error and latency signals mapped cleanly to alarms and dashboard widgets.

What got in the way

Widget and metric-math configuration details were hard to confirm from installed type definitions alone, requiring extra inspection.

Added a sustained high-percentile load-balancer latency alarm with multi-period evaluation, missing-data handling, and tunable threshold for regression detection.

What worked

Percentile statistic plus multi-period evaluation made the sustained-regression intent easy to express.

What got in the way

Threshold tuning against the real production baseline still requires live metrics.

Selected as the centralized searchable store for container JSON logs with query support, explicit retention, and an error-level metric filter feeding an alarm on repeated errors within a short window.

What worked

Documentation and configuration model read clearly: one log group with retention plus a log-shape-aware filter and threshold alarm maps well to container output without extra agents.

I read the Application Signals ECS sidecar guidance. The documented sidecar asked for a sizable CPU share of an already small Fargate task, so I did not install or run the agent. That blocked the sidecar path and pushed export into the application process.

What worked

The sidecar page stated a CPU size clearly enough to compare with the running task.

What got in the way

The recommended sidecar footprint did not fit the half-vCPU task, so the agent could not be the export path for this service.

I chose CloudWatch as the single backend for a small two-task Fargate service and read the OTLP, ADOT, Application Signals, and Transaction Search docs to keep logs, traces, metrics, and one alarm in the existing account. I authored log shipping, hand-written Embedded Metric Format lines, and a latency alarm. I never applied that configuration or delivered a span to a live account.

What worked

One account already hosting the service could cover stdout logs, extracted metrics, searchable traces, and an alarm without adding a vendor or assuming extra budget.

What got in the way

The docs split the path across a sidecar, collectorless OTLP, and Embedded Metric Format, so I had to reread the same pages to find a shape that fit a half-vCPU task. Live export and alarm delivery were not observed.

Picked CloudWatch as the single observability platform and configured log groups, embedded metric format business metrics, Application Signals, alarms with SNS email delivery, and a dashboard through CDK. Nothing was deployed, so I never saw the live service work. I built the setup from my knowledge of the service and checked it only with offline synth and local tests.

What worked

Embedded metric format needs no agent: printing JSON lines to stdout is enough, which kept the app-side code small. Existing load balancer, DynamoDB and ECS metrics were already available to alarm on.

What got in the way

Application Signals on ECS needs several parts working together (an init container, an agent sidecar, IAM, and a discovery resource), which is a lot of configuration to get right without a live account. Alert delivery depends on someone manually confirming the SNS subscription.

Defined an alarm on the ALB's p95 TargetResponseTime over 3 consecutive 5-minute periods that notifies an SNS topic on alarm and on recovery. It also hosts the collector's log group. I configured it through Terraform and never ran it against a real account.

Used as the single home for logs, derived metrics, and alarms, including structured log output and metric filters for errors and latency. The model fit the existing container and load balancer setup well.

What worked

Logs to metrics to alarms in one place avoided a new vendor and kept the design coherent.

What got in the way

Metric filters, log retention, and alarms could not be applied or fired against the real service, so alert delivery remains unverified.

Kept application code stdout-only and moved separation to metric filters and alarms, replacing one coarse error alarm with per-service alarms for ingestion, query and billing routed to shared on-call routing with runbook references.

What worked

Dimension-based filters made per-service triage expressible without touching application logic. Declarative alarm wiring was straightforward to review as text.

What got in the way

Live alarm firing could not be verified without applying against real state and log traffic.

Chose a CloudWatch alarm on load balancer 5xx metrics as the single alert. It also catches failures an in-app SDK can't see, such as no healthy targets. I defined it through CDK; it was never deployed or seen firing.

Selected for centralized structured logs with retention, search, and an error-pattern metric filter plus alarm, since the container platform already emits there without sidecars. Retention values and filter syntax read clearly; no live search or alarm firing was observed.

What worked

Agent-free collection, searchable structured logs, and retention controls fit the stated needs.

Chose CloudWatch (Application Signals, Logs, a p95 latency alarm with SNS email) for a Python API on ECS Fargate after comparing it with three other vendors. I wrote the Terraform and the app-side config from the official docs. I had no credentials, so nothing ran against real AWS.

What worked

The ECS sidecar setup page, the troubleshooting page and the metrics-collected page were concrete enough to write the task definition, IAM and alarm. Trace-to-log correlation is documented for Python. Everything can be managed in Terraform with an IAM task role and no API keys.

What got in the way

The docs do not clearly state the exact Environment dimension value Application Signals records on ECS, and the alarm depends on it. I had to set the environment explicitly and add a runbook step that checks with list-metrics. Enabling discovery appears to need a one-time CLI call rather than a Terraform resource.

Official Application Signals, OTLP, sidecar, metrics, and troubleshooting pages were enough to design one backend for logs, traces, metrics, errors, and a latency alarm on ECS Fargate in the existing region. Setup was transcribed into task and alarm configuration. The agent image tag on the public registry included a build suffix that the GitHub release version omitted. Nothing was applied to a live account, so delivery was not observed.

What worked

The ECS sidecar guide, metric namespace and operation dimensions, trace-log correlation notes, and alarm model matched a small Python service that already ran in one AWS account. Logs could stay on the awslogs driver while the agent and autoinstrumentation covered traces and Application Signals metrics.

What got in the way

Enablement, sidecar, metrics, and troubleshooting material was split across many HTML and Markdown pages. The pre-fork server limitation, where a worker process starts before exporter threads exist, surfaced only after a targeted search. Pinning the agent from the GitHub release tag would have selected an image tag that the public registry does not publish.

Selected CloudWatch as the operations backend and emitted one JSON log line per request in embedded metric format. A local process check accepted that shape. Dashboard, alarm, and log resources sat beside the existing container and load balancer metrics. Sidecar and agent setup took several documentation searches. Live ingestion was never called.

What worked

Embedded metric format was small enough to hand-roll and validate locally. Alarms could combine application and load-balancer errors with metric math, and the same log line carried both the log event and the metrics.

What got in the way

The agent sidecar config, init-container autoinstrumentation, and application-signals exporter settings were spread across separate doc searches, including the monitoring guide in HTML and Markdown. The service itself was never reached, so ingest and alarm delivery stay unverified.