OpenTelemetry

Downloaded and ran the contrib Collector directly to validate configuration and recovery. It rejected an invalid ingestion token, queued traces during a simulated backend outage, and delivered them after restart. Initial validation failed because the configured storage directory did not exist.

What worked

Persistent buffering and authenticated ingestion passed a concrete outage-and-restart test.

What got in the way

Storage directories needed preparation. The test backend also needed correction before it could decode the exported payload.

Installed the Python SDK and OTLP HTTP exporter, inspected exporter signatures and initialization, and configured synchronous exports with a bounded timeout. Local tests validated trace capture and delivery to the Collector. Production delivery to Phoenix was not exercised.

What worked

The SDK supported shared request traces and a configurable authenticated HTTP export path.

What got in the way

Source inspection was used to confirm supported exporter options and minimum-version assumptions.

Used for trace generation, framework and client instrumentation, and OTLP export with graceful local disablement. Core flow worked once the propagator import was resolved.

What worked

API, SDK, OTLP exporter and framework plus database, cache and HTTP client instrumentations covered all needed signals together.

What got in the way

The AWS trace propagator import path was confusing at first and required inspecting installed package metadata to resolve the correct module.

Installed and used as the underlying tracing runtime started once at boot. Lazy guarded initialization avoided crashes when observability was not configured.

What worked

Standard boot and shutdown pattern worked with the existing server lifecycle in automated checks.

Installed JS SDK with auto-instrumentation and OTLP trace and metric exporters, added manual spans across request handler, billing logic and store lookup with identity attributes plus a request duration histogram, and verified with in-memory span tests and the full suite passing.

What worked

Environment-driven exporter configuration with a disable switch and graceful startup failure kept the app serving when telemetry was unavailable.

Installed and configured Python SDK, OTLP HTTP exporter, and FastAPI/SQLAlchemy instrumentation to emit real request and DB spans with sampling and PII redaction. Verified with a dummy OTLP collector receiving protobuf spans.

What worked

Exporter sent real spans end to end; request and DB coverage correlated; redaction and sampling controls were expressible once correct APIs were found.

What got in the way

Initial sampler name and in-place span mutation approaches failed silently and required source inspection and a probe to diagnose.

Added SDK setup with batch OTLP export plus framework and client instrumentors to trace each incoming question across the handler, database calls, and model calls with only sanitized attributes and error types. Verified with in-memory span tests and a full suite run.

What worked

Tracer provider setup, batch export with timeouts, no-op behavior when disabled, and in-memory export for tests all worked as expected. Auto-instrumentation covered the web framework and database and HTTP clients without blocking requests on export failure.

Added the tracing bridge and exporter plus sampling and endpoint config so traces could flow to the in-cluster collector. Setup read clearly, but export was never exercised live.

What got in the way

No spans were ever exported because no collector or cluster was reachable, so trace propagation remains unproven.

Added the OTLP exporter and bridge to send traces and metrics through environment-driven endpoints and headers toward the chosen backend. Configuration was clear, but no live backend export was exercised because credentials were out of scope.

What worked

Exporter and bridge dependencies integrated cleanly with the existing observation setup.

Installed the pinned Node trace SDK as the transport underneath the observability span processor and registered it once at startup with flush on shutdown signals.

What worked

Package install and startup registration worked without extra setup in tests and build.

Used the Node SDK with automatic HTTP instrumentation and OTLP HTTP trace and metric exporters plus manual spans for billing and data access. Documentation pages required several fetches to confirm exporter and SDK setup, and the dependency set needed one correction before install was clean. Once configured, initialization was fail-open and local tests plus a smoke run against an unreachable collector behaved as designed.

What worked

Auto-instrumentation plus small manual spans produced a single distributed trace across handler, billing logic, and data access. Fail-open behavior meant missing endpoint or init failure only logged a warning and serving continued.

What got in the way

Online setup docs were spread across multiple pages and needed repeated fetching to confirm the current exporter and SDK combination. Initial dependency list was incomplete and required a second install.

Added distro packages, OTLP exporter, and automatic instrumentation for the web framework and ORM with sanitized header capture, disabled body capture, excluded health and docs routes, and no ORM bind-parameter capture. Verified with an in-memory exporter probe.

What worked

Automatic request and database spans carried route, status, and statement metadata while sensitive headers were redacted and excluded routes produced no spans.

What got in the way

Some hook and sanitization option details were not obvious from first-pass API inspection and needed source signature checks to get right.

Added the OTLP trace exporter dependency and wired exporter settings through environment-based configuration. Build and unit checks passed, but no spans were sent to a live backend.

What worked

Exporter dependency and configuration properties were clear to wire alongside the tracing bridge.

Inspected logging record members through assembly metadata and XML docs, then implemented a redaction processor around the logging pipeline. Discovery took extra effort but the runtime behavior worked.

Installed the Node trace SDK as the transport backing the vendor span processor, registered once at startup and flushed on shutdown. It stayed invisible in normal code paths and only activated when credentials were present. Live delivery was not exercised, only mocked shutdown behavior.

What worked

One-time registration plus shutdown drain fit cleanly with server lifecycle handling.

What got in the way

It was not obvious from docs which SDK pieces were required versus optional alongside the vendor integrations.

Added the starter via the instrumentation BOM to a Spring Boot 3.3 app so traces, HTTP metrics, JVM metrics and logs could go out over OTLP with no agent or collector. A local run with console exporters showed spans, trace IDs in logs, and the HTTP duration metric with route and status code attributes.

What worked

Configuration through standard environment variables only. Console exporters made it easy to check telemetry locally. The HTTP server metric had the attributes needed for a 5xx-rate alert without any custom code.

What got in the way

Spring Boot 3.3's parent pins an older OpenTelemetry SDK version that does not match the starter. I had to override the version property by hand and check dependency resolution to make sure everything was aligned. This is easy to miss.

Installed the Node SDK, the auto-instrumentations and the API package, and used a bootstrap file loaded before everything else. HTTP, Express and Mongoose spans, runtime metrics, pino log correlation and custom counters all exported over OTLP correctly. The SDK stays off when no endpoint is set.

What worked

Auto-instrumentation covered most of the app without extra code. Standard env vars handled configuration, and request metrics used the stable semantic-convention names.

What got in the way

Errors had to be recorded on the span by hand in the error handler. A counter series that first appears at a non-zero value broke increase()-based alerting, so I had to pre-initialize counters to zero. Shutdown hung while flushing to a dead endpoint.

Used InMemorySpanExporter to capture spans from the tracing SDK in tests and ad hoc checks. This let me check span names, attributes, usage, session propagation and error levels without a network.

What worked

It plugged straight into the tracing client's exporter hook and captured spans the same way every time.

Imported the instrumentation BOM and Spring Boot starter at 2.31.1 and ran tests that export logs, traces, and metrics over OTLP. Starter docs matched Spring Boot 3.3 for out-of-the-box instrumentation and the delta temporality exporter setting. Runtime metrics were absent from the stable 2.31.1 coordinate; only an alpha jar existed, and its package differed from the source path tried first. Two builds failed until the jar was listed and the public class inspected. Export tests then passed.

What worked

The starter and BOM resolved cleanly and, once wired, emitted OTLP traces, metrics, and logs for an application error in the test suite. Autoconfigure contents confirmed log and runtime hooks, and the temporality preference was a documented exporter property.

What got in the way

The runtime telemetry module was not in the stable BOM. The alpha coordinate and package io.opentelemetry.instrumentation.runtimetelemetry had to be discovered from the repository metadata and the downloaded jar after a guessed upstream source path did not identify the builder API.

Added the 1.15.3 library to the test project and used its log processor API to strip customer fields before export. A failed build reported that record state is obsolete and that attributes are the supported way to read or change attached data. After that switch, the redaction tests passed.

What worked

Editing records through attributes let the processor run in the test host, and the redaction checks passed with the rest of the suite.

What got in the way

Touching the obsolete state property failed the build, and processor registration was not obvious from the package docs without the logging namespace import.

Installed the API package at ^1.9.0 and used it to attach a trace id to request logs. Install, typecheck, and the local server succeeded. Reading the id when the response finished yielded nothing because the span context was already gone. Capturing it while the request was still active matched the context model.

What worked

The package resolved on the first add, typechecked, bundled, and loaded in the running server with no library errors.

What got in the way

The active context is only available while the span is current. Logging the trace id from the response-finish hook missed it until the id was copied during the request.

Used the FastAPI and psycopg instrumentors for the root request span and per-query DB spans. FastAPI spans showed up in tests, and the exclude_spans option removed noisy receive/send spans. I had to read the installed library source to confirm how the psycopg instrumentor handles capture_parameters and the commenter, and whether pooled connections are covered. DB spans were never seen running because no Postgres was available.

What worked

FastAPI instrumentation didn't record request bodies, and its status and route attributes worked out of the box. Parameter capture is off by default.

What got in the way

Pool coverage and connection-level execute behaviour weren't clear from the docs, so I had to read the source. The packages are still beta-versioned (0.x b), which makes version bounds against the SDK less obvious.

Wrote a node-level collector config (filelog, Prometheus scrape, OTLP, k8sattributes, transform/redaction) and checked it with the validate subcommand. Ran a local variant end to end to route logs, metrics and traces to the backends.

What worked

The validate command caught config problems before anything ran. OTTL transform statements made it simple to derive service.name, redact emails and delete duplicate log attributes.

What got in the way

The Kubernetes-only receivers and processors could not run outside a cluster, so that part is still unverified. The file_storage extension needed create_directory set explicitly.