AWS X-Ray

Used for distributed trace context propagation and trace backend alongside the metrics and logs platform. Propagator setup was resolved locally; remote trace delivery awaits deployment.

What worked

Propagator and exporter concepts mapped cleanly to the existing request flow once configured.

What got in the way

End to end trace viewing was not possible without a live collector and credentials, so exporter behavior beyond initialization was not observed.

Selected as the single production trace backend via a collector sidecar forwarding OTLP to managed tracing. Configuration was authored but never applied or validated in a live account.

What worked

Documentation read clearly for sidecar plus managed backend as the single production destination.

Used for request and dependency traces alongside logs and metrics. Added lightweight best-effort trace emission that stays off locally and activates when a daemon address is configured, with spans around database, cache, payment, and email paths.

What worked

Sidecar-based tracing avoided new application dependencies and kept local development behavior unchanged while preserving trace continuity headers.

Selected as the single production trace backend for collector-exported spans so request and database timing can be inspected without adding another vendor.

What worked

Configuration surface for sending collector output to traces was small and clear.

What got in the way

Actual trace visibility was not observable without a live deployment.

Imported and exercised the tracing SDK to create request segments in async middleware. Core recording worked, but context propagation needed substantial debugging and custom handling.

What worked

Core segment creation and emission concepts were usable once context handling was stabilized.

What got in the way

Async context handling was difficult. Segments were intermittently missing, sampling behavior was confusing, and several documented integration patterns did not fit the async middleware model without custom lifecycle handling.

I chose X-Ray as the trace backend and added a task role with permission to write to it. The ALB trace header propagated and its trace ID showed up in the logs during the local run. No traces were sent to the real service.

What worked

X-Ray accepts W3C-style trace IDs through the OpenTelemetry AWS extension, so the standard OpenTelemetry path works.

Chose it as the single trace backend because the project was already fully on AWS and it needs no new vendor or secret, just an IAM write permission on the task role. Trace ID format compatibility was handled with the OpenTelemetry AWS ID generator. I didn't verify real export.

Designed the tracing side of the solution around this trace service, with guarded middleware that stays silent when no daemon is configured. Configuration intent was clear, but live delivery was unverified.

What worked

The daemon-based model made failure-guarded local behavior easy to reason about.

What got in the way

No live trace backend was reachable in the environment, so end to end trace delivery and sampling could not be confirmed.

Selected as the single production trace backend with OTLP via a collector sidecar. Docs review supported the collector to X-Ray path and regional data handling, but integration was config-only with no live account run.

What worked

Documentation clearly described the collector to backend path and kept the design to one production destination.

What got in the way

No live validation was possible in the task; span delivery to the backend and end-to-end trace visibility remain unverified.

Set X-Ray as the trace store behind the collector sidecar and attached the standard daemon write policy on the task role. The synthesized template included that managed policy. The integration that was checked was the policy plus collector settings. No X-Ray API was called.

What worked

The managed write policy was a clear, recognizable permission and showed up on the task role in the asserted template.

What got in the way

There was no direct trace API in the app, so span export depended entirely on the sidecar configuration. Trace acceptance was not observed.

Wired traces to the regional X-Ray endpoint and enabled Transaction Search in infrastructure code so spans would be stored for investigation. Documentation for Transaction Search and the collectorless exporter was readable. The installed infrastructure library had no generated resource for that setting. No span was accepted by the live service.

What worked

Transaction Search docs explained how spans become searchable, and local instrumentation produced X-Ray-shaped trace ids on ordinary requests while leaving health checks alone. Those ids also appeared on the matching request log line.

What got in the way

The generated infrastructure class for Transaction Search was absent from the installed library release, so the setting had to be declared as a raw resource. Permissions for the traces endpoint were not obvious from one page. A local export targeted a closed port, so indexing and search were never observed.

Configured X-Ray as the trace backend via an OpenTelemetry collector sidecar, with an IAM task role granting segment writes and sampling access. Used X-Ray-format trace IDs in app logs for correlation. Never exercised against the live service.

What worked

Required IAM permissions were small and clear. X-Ray-compatible trace IDs could be generated in-app so logs and traces link up.

What got in the way

Could not verify that traces actually appear in X-Ray; cost at full sampling had to be left as a tunable variable.

I read the traces endpoint and Transaction Search docs and configured export only when the task points at the regional traces endpoint, so a request log trace id can be opened in search. Indexing was declared in the deployment configuration. No span was delivered to a live account.

What worked

The OTLP traces endpoint and Transaction Search gave a single place to open a failed or slow request from the id already written on the log line.

What got in the way

Enablement was spread across endpoint docs, segment-destination notes, and provider resources, and I could not confirm that a span arrived.

X-Ray was the trace store behind the chosen Application Signals view. Correlation guidance was read and trace identifiers were planned onto structured log lines via the collector path. No trace was sent and the X-Ray console was never opened.

What worked

The correlation guidance made it clear that trace and span identifiers on log lines are what join a request log to its trace in the same console.

What got in the way

Standalone X-Ray setup, sampling, and service-map behavior were not covered by the page that was used. Trace export was only designed, so latency and completeness of traces are unknown.

Turned on the X-Ray remote sampler through the Node autoinstrumentation. It called a daemon on localhost port 2000, which this Fargate service does not run, logged a connection error, and fell back to a default rule. The task was switched to a local ratio sampler so tasks would not poll a missing daemon. Trace export still targets the cloud trace endpoint with signed OTLP.

What worked

When the daemon was unreachable the process kept serving and fell back to a default sample rate instead of exiting. Trace ids were still created and written on the request logs.

What got in the way

Central sampling rules are unavailable without the daemon the defaults expect, and a sidecar would spend memory on a small task. The repeated localhost errors look like application failures. Docs and the sampler client make that daemon the normal path, so a daemon-free Fargate task needs a different sampler to stay quiet.

X-Ray docs were used to decide how Application Signals records traces. The default 5 percent sampler would drop most requests, so the design records every trace for a low-volume API, relying on the free monthly trace allowance. The sampling rule was declared in infrastructure code and not applied.

What worked

Pricing docs made full recording look affordable at this volume, and a sampling rule can keep every trace for one service so a latency alarm can be tied to the slow request.

What got in the way

The default sampler is low enough to miss the slow calls that matter during a latency incident, and that default is easy to leave in place. No traces were sent to a real account.

Read the official ECS X-Ray collector sample and used it as the trace half of the sidecar config, with the X-Ray trace id added to JSON logs. Application, database, cache, and outbound calls were wired for export through that pipeline. No segment was sent to X-Ray.

What worked

The ECS sample showed a concrete trace pipeline, including batching, that could be merged with the metrics and logs configuration.

X-Ray was selected as the only trace backend. The collector config uses the X-Ray exporter, and the task role is limited to putting trace segments and telemetry records. No segments were sent and the service API was not called.

What worked

The exporter and the two trace-write IAM actions were specific enough to point the existing Fargate task at one trace store and to separate that permission from the application role.

Tracing stayed on the default X-Ray sampling rule so request metrics still include every call while traces remain sampled. Matching those traces to JSON logs meant checking the X-Ray identifier format against the W3C identifiers Application Signals now emits. I never opened a trace or called the X-Ray API.

What worked

Default sampling was spelled out clearly enough to keep full latency and fault counts without increasing trace volume.

What got in the way

Correlation notes still mix the older X-Ray trace id shape with 32-character W3C ids, so the log field format was not obvious from one page.

Chose X-Ray as the trace backend because it stays inside the existing cloud account and Terraform root with no new vendor. Configured an X-Ray-compatible ID generator and propagator in the app so load balancer trace headers join the trace, scoped IAM to the put/sampling actions, and added an X-Ray group. Not exercised against the live service.

What worked

Tight integration with the load balancer trace header and generous free tier made it an easy recommendation; the IAM surface needed is small.

What got in the way

Requires a non-default trace ID format, so the app must be configured with a specific ID generator or traces are silently rejected; this is an easy thing to miss.

Selected X-Ray as the backend because the stack was already AWS-native, and configured the IAM write policy, X-Ray-format trace IDs and the collector exporter with route indexing as an annotation. Not run against the live service in this task.

What worked

Fits a single-vendor, single-apply deployment story with no new credentials; the managed write-only IAM policy keeps the task role minimal.

What got in the way

Needed to remember non-obvious details such as trace ID format requirements and the dot-to-underscore conversion of indexed attribute names.

Targeted X-Ray as the trace backend via an OpenTelemetry collector sidecar, generating X-Ray-format trace ids from the frontend so a web failure can be looked up directly. Required knowing the X-Ray id format (epoch prefix plus random hex) and header precedence with the load balancer's own trace header. Frontend spans themselves are not exported since that app runs outside AWS. Not exercised against the live service.

What worked

OpenTelemetry integration path is well-trodden; id format is simple to generate server-side.

What got in the way

Ingesting traces from a non-AWS frontend host has no lightweight path, so only the trace id, not spans, bridges the gap.

Targeted X-Ray as the trace store through the collector's exporter, using the X-Ray id generator in the app so trace ids are valid and log lines carry the same id. Worked through how the load balancer's trace header interacts with propagation and concluded W3C propagation was sufficient. Not exercised against the live service.

What worked

Requiring only an id-generator change on the app side, with the collector handling format conversion, kept the application vendor-neutral.

What got in the way

The constraints around trace id format and the load balancer header lacking parent/sampled fields are subtle and poorly surfaced; I had to read propagator source to be confident the default setup would not drop all traces.