Grafana Tempo

Wrote a minimal single-binary Tempo config with OTLP ingestion and local storage for the compose stack and wired it as a Grafana datasource. Not booted in this environment.

What worked

Native OTLP receiver means no protocol translation in the collector; minimal config for a local stack is short.

Authored Tempo config as the trace backend behind Alloy OTLP export, including a local gRPC port shift so it would not collide with the collector. Config was never executed, and newer Tempo fields were only guessed at.

What worked

OTLP ingest plus Grafana as the query UI fit the one-pipeline design without a proprietary agent.

What got in the way

Port overlap with the collector required a manual remap, and production validity of the Tempo 2.x config was untested.

Configured a self-hosted trace backend from official docs, including receiver TLS and a port offset so it would not collide with the collector.

What worked

Docs were enough to give traces a dedicated store in the same family as logs and metrics.

What got in the way

Receiver TLS and port overlap needed extra care, and the backend was never started to confirm ingest.

Added Tempo as the trace backend receiving OTLP from the collector, with local and Kubernetes configs and emptyDir storage paths. The service was never started, so ingest and query were not observed.

What worked

Tempo fitted the OTLP export path and Grafana datasource story without introducing a separate tracing vendor.

What got in the way

Storage mount paths in the cluster manifests needed a correction. Runtime ingest was not exercised.

Wrote a single-binary configuration with local storage for the reproducible local stack, receiving traces from the collector rather than from the application directly.

What worked

A single-binary mode with filesystem storage makes a trace backend genuinely cheap to stand up for a reproduction, and OTLP ingestion meant no application-side awareness of it at all.

What got in the way

The configuration surface is large and the examples vary a lot between versions, so picking a minimal correct config is more guesswork than it should be. Nothing was verifiable offline beyond YAML shape.

Configured trace storage from the official Helm chart values and service templates. The HTTP query port in the first wiring did not match the port the chart actually listens on, which would have broken Grafana links. Metrics-generator nesting also needed a correction. Tempo was never run.

What worked

Fetching the chart’s values and service templates eventually showed OTLP receiver ports and the TCP service layout clearly enough to finish GitOps values.

What got in the way

Initial setup used the wrong HTTP listen port for Grafana queries. Metrics-generator configuration was easy to flatten incorrectly relative to the chart schema. Helper templates fetched for naming were incomplete.

Configured a minimal Tempo service with local-disk trace storage and a retention window sized for a single small server's limited disk space. Config was written but never started or verified in this sandbox.

Configured local single-binary storage to receive traces from the collector. Like the log store, needed a root-user workaround for expected volume permission issues on first run. Config was syntax-checked but never executed.

Configured Tempo as self-hosted trace storage with persistent local data and seven-day retention. Its runtime behavior was not assessed because the container stack could not be launched in the environment.

What worked

Tempo fit the requirement to retain sensitive prompt and response traces on infrastructure controlled by the application owner.

What got in the way

Collector-to-Tempo compatibility, persistence, and querying could not be validated at runtime without the container tooling.

Configured Tempo as the trace backend receiving spans over OTLP from the collector, paired with Grafana's trace query editor feature toggle.

What worked

Required minimal configuration beyond the OTLP receiver block and storage path, since sane defaults covered the rest.

What got in the way

Never started, so trace ingestion and querying were never actually exercised.

Chose it as the self-hosted trace backend and wrote a single-binary local config with OTLP ingest and local block storage, wired behind the collector. Configured only; not run here.

What worked

Single-binary mode is a genuinely low-friction way to get trace storage running on your own hardware, and it speaks OTLP natively so no translation layer was needed.

What got in the way

The config schema is large and has moved across releases, so writing one without consulting the matching version's reference is risky. Retention defaults are short and are easy to overlook, which matters here because trace retention is effectively customer-data retention — I had to call that out explicitly rather than rely on the default.

Wrote a Tempo configuration to receive OTLP traces from the app and a matching Docker Compose service. The config required a revision pass to get right, and was never actually run.

What worked

OTLP receiver configuration was well-documented enough to produce a plausible config on the first couple of attempts.

What got in the way

Needed a second pass to correct the config file, suggesting the initial structure wasn't fully right the first time.

Wrote a tempo.yaml config to receive OTLP traces from the collector and serve them back to Grafana; validated for YAML correctness only, never started.

Configured Tempo as the OTLP-native trace backend (tempo.yaml plus a docker-compose service exposing the OTLP ports) so the already-installed OpenTelemetry SDK could export spans directly without extra client libraries. Never run live to confirm ingestion.

Wrote a Tempo config and added it as a docker-compose service to store traces, pinning a current release version found via a search of its releases page, but never ran it since Docker wasn't available.

What worked

The config format matched expectations on the first attempt and passed YAML validation cleanly.

Added as the trace storage backend fed by the collector, and wired into the dashboarding tool with trace-to-log linking. The config needed a second pass to get right, and like the rest of the stack was only statically validated, never actually run.

Wrote a minimal tempo.yaml so the local stack could receive and store traces forwarded by the OTel Collector, but never started the service to verify trace ingestion.

Configured Tempo as the internal OTLP trace destination and provisioned it for correlation in Grafana.

What worked

The application instrumentation initialized with the configured tracing modules and the service YAML was parsed.

What got in the way

End-to-end trace export could not be tested because the container runtime was unavailable.

Wrote a tempo.yaml configuring the OTLP receiver and local storage to pair with Grafana as the trace backend. Config was syntax-validated but never actually run because Docker was not available in this environment.

Wrote a tempo.yaml to receive and store traces exported by the OTel Collector, as the tracing backend for the stack.

What worked

Basic [redacted:basic] config was quick to draft.

What got in the way

Never run against a live collector or verified a real trace landed, since Docker wasn't present in the sandbox.

Configured a trace backend to receive OTLP traces over HTTP from the instrumented service; validated only for YAML syntax since the service was never actually run in this environment.