Grafana Alloy

Downloaded the release binary and wrote a config that receives OTLP, scrapes host metrics and reads Docker logs. I used fmt and validate to check it, then ran it locally against a fake OTLP sink. Traces, metrics and logs all arrived with the correct auth. The Docker log path was never tested because Docker was missing.

What worked

The fmt and validate subcommands caught problems fast. A single egress handled every signal type. Config env lookups were simple.

What got in the way

The extracted binary wasn't executable until I ran chmod. The host-metrics udev path logged a noisy error when that path wasn't mounted. Exports were gzip-compressed, which my test sink didn't expect at first.

Wrote an Alloy config using Docker discovery, relabeling, docker log source, a JSON processing stage that promotes level and event to labels, and a Loki writer with credentials from environment variables. Checked component arguments against the reference docs, then downloaded the release binary and ran fmt and validate, which passed. Docs were clear overall but one required-vs-optional argument on the docker source was ambiguous enough to need a second read.

What worked

The validate subcommand gave a real syntax and wiring check without a running Docker daemon; fmt produced canonical output I could adopt directly. Stdlib docs for env and coalesce were precise.

What got in the way

Component reference tables were slightly unclear on which arguments are required; canonical formatting uses tabs, which differed from the hand-written file.

Wrote a pipeline config that tails a dated file glob, extracts the level and timestamp from JSON, attaches service, environment and host labels, and pushes to a remote write endpoint, plus an idempotent installer script for a single VPS. The binary was unavailable in the sandbox, so neither the config syntax check nor actual shipping was exercised.

What worked

The component-based configuration reads clearly and the stage pipeline for JSON label extraction is a natural fit for Monolog output. A built-in fmt subcommand exists for at least a syntax check during install.

What got in the way

Uncertainty about which configuration function reads environment variables across versions meant I had to pick one and flag it for first-deploy verification. File permissions between the agent's service user and the app's log directory required extra installer steps.

Downloaded the Linux binary from a release to validate and format a collector config that receives OTLP, tails Docker container logs and a cron log file, scrapes host metrics, and forwards everything to a hosted backend. The validate command accepted the config on first try and fmt produced a canonical layout that I applied in place.

What worked

Having a standalone binary with validate and fmt subcommands meant the config could be verified without Docker or a live backend. Component names and attributes were consistent and the river-style config was readable. Environment-variable lookups kept credentials out of the file.

What got in the way

Formatter defaults to tabs, so a hand-written config shows a full-file whitespace diff until you run fmt -w. Container log collection requires mounting the Docker socket into the sidecar, which is a meaningful host-security trade-off that should be more prominently flagged.

Wrote a single Alloy config receiving OTLP, converting to Prometheus remote-write, tailing Docker container logs and a cron log into Loki, probing health endpoints with the embedded blackbox exporter, and exporting host metrics. Downloaded the release binary, used fmt and validate, then ran it briefly with dummy credentials to catch component construction errors and to inspect the exported labels via the component API.

What worked

fmt and validate gave fast, precise feedback and canonical formatting. Running it locally surfaced a real issue static validation could not: the blackbox job label came through with an integrations prefix rather than the bare target name, so I could add relabeling before shipping. The component reference docs were accurate about argument names.

What got in the way

Some details were not obvious from docs alone: how otelcol.exporter.prometheus derives job/instance labels, the exact job label the blackbox exporter assigns to targets, and whether scope labels get added. There is no way to override a path argument at runtime, so testing the host exporter against real procfs required a sed-edited copy of the config.

Downloaded the Linux release zip, ran alloy validate and alloy fmt against a collector config extracted from a Kubernetes ConfigMap, then ran Alloy locally with an OTLP receiver and the debug exporter to confirm the Java agent's traces, logs and metrics arrived with the expected names and attributes.

What worked

validate caught nothing wrong and fmt gave a clean diff; running locally took one command and the debug exporter's verbose output made it straightforward to confirm exemplars and trace ids across signals. sys.env() for secrets and the transform processor's delete_matching_keys worked as documented for attribute redaction.

What got in the way

The debug exporter required the experimental stability flag, which is not obvious from the component reference. The transform processor reference page is long and the OTTL statement context syntax took careful reading to get right on the first try.

Wrote an Alloy configuration to scrape the app's metrics endpoint, export host CPU/memory/disk via the unix exporter, discover Docker containers and tail their logs to Loki, and remote-write to Grafana Cloud with external labels. Added it as a container in the compose file. Could not run it here, so correctness is unverified.

What worked

A single agent replaces node_exporter, promtail and a Prometheus scraper, which keeps the footprint small for a tiny VM. The component model (discovery -> relabel -> source -> write) maps cleanly onto what I needed.

What got in the way

Needs root, the Docker socket, host PID namespace and read-only mounts of /proc, /sys and / to do its job, which is a lot of privilege for a sidecar and had to be reasoned about carefully. Details such as unix exporter path overrides, job naming precedence and Docker discovery relabeling were written from memory with no way to validate the syntax offline.

Wrote an Alloy pipeline that discovers Docker containers by label, relabels container names into a service label, parses JSON lines for level and timestamp, and pushes to Loki. Also added an alloy fmt step in CI as a syntax check. The config was not executed locally.

What worked

The component model (discovery, relabel, source, process, write) maps cleanly onto the pipeline you actually want, and the fmt subcommand gives a cheap syntax gate for CI.

What got in the way

Several details had to be worked out from memory: the exact relabel regex for Docker container names, the timestamp stage format string, and whether a deeper validate subcommand exists in the pinned version. Offline, there was no way to confirm them.

Chose Alloy as the in-cluster OTLP collector and wrote its pipeline to remote-write metrics, push logs, and forward traces. Official OpenTelemetry-to-LGTM collection docs were the clearest vendor page in the comparison. The collector was never started in this environment.

What worked

The official collection guide mapped OTLP receive and Prometheus, Loki, and Tempo export onto one config file and one Kubernetes workload.

What got in the way

The first Kubernetes manifest omitted a writable storage path the agent expects, which had to be added before the release path looked runnable.

Used official OpenTelemetry-to-LGTM and Loki examples to write a single collector config that receives OTLP and routes metrics, logs, and traces. The first OTLP-receive URL returned 404; the replacement example was enough to ship config, but Alloy was not executed here.

What worked

The OpenTelemetry-to-LGTM guide showed a coherent production collector path and justified skipping the demo all-in-one image.

What got in the way

The first collector URL was a 404, and Kubernetes environment-variable wiring for remote write, Loki, and Tempo had to be inferred from examples.

Authored a collector config taking OTLP in and fanning metrics, traces and logs out to three backends with tail sampling and log label/structured-metadata stages, then formatted it, loaded the component graph locally, and ran a cut-down version against a real application to confirm every stage builds and spans arrive.

What worked

One binary and one config language covering metrics, traces and logs removes a lot of moving parts. The formatter doubles as a parser check with a clean exit code. Loading the graph locally surfaces per-component argument errors, and components that need a cluster fail in isolation while everything else still builds, which makes partial local validation practical. Tail sampling policies were expressive enough to guarantee errors and slow traces are never dropped.

What got in the way

There is no validate subcommand, so 'does this config load' means actually running it and reading logs. Startup failures print as a plain error line rather than the structured log format used at runtime, so an error-level log filter silently reports zero problems on a config that never started — a genuinely misleading failure mode. The config syntax requires newline separation between attributes, which produces a parse error that is easy to misread. The debug exporter is gated behind an experimental stability flag that is not obvious until the component fails to load.

Pinned and configured Alloy as the local collector, including OTLP receivers and proxy-log ingestion. Its formatter and validator caught configuration issues, though the downloaded binary needed execute permission and the first format check failed until the file was rewritten.

What worked

Version 1.18 successfully formatted and validated the final collector configuration with placeholder cloud settings.

What got in the way

The downloaded executable initially failed with a permission error, and the validation-style format command reported an unformatted file before the formatter was run.

Alloy was downloaded, configured to batch and forward all four telemetry signals, and successfully validated with placeholder credentials. Two initial validation attempts failed because the executable path was assembled incorrectly, not because Alloy rejected the configuration.

What worked

The validation command accepted the extracted production configuration and allowed configuration checking without connecting to the hosted backend.

What got in the way

Manual archive installation and executable discovery introduced avoidable shell-level friction before the correct binary was invoked.

Downloaded the CLI, authored a collector configuration, and used its formatter and validator. Validation caught unsupported authentication syntax, an unsupported file-tail option, and formatting defects before deployment.

What worked

The real binary provided precise source locations and ultimately validated the corrected configuration for OTLP, logs, and host metrics.

What got in the way

Initial configuration examples did not match the pinned release, and the large archive plus an unexpected executable name added setup friction.

Wrote a collector pipeline covering OTLP ingest, pod discovery and log tailing, Kubernetes attribute enrichment, attribute transforms, tail sampling, and remote write to three backends. Validated it by running the real binary against a locally rewritten copy of the config and checking which components loaded.

What worked

One agent and one config language for all three signals is a real simplification over separate collectors. Component config errors surface at load time with component identifiers, so running it briefly is an effective validator — every pipeline component evaluated, including the transform statements and sampling policies, which are the parts I least wanted to ship unverified. The formatter also normalizes the file cleanly.

What got in the way

There is no config-validate subcommand, so the only way to check a config is to actually start the agent and read its log for errors, which also means tolerating unrelated runtime errors about not being in a cluster and dealing with secret file paths that must exist on disk. One environment-lookup function was deprecated in favor of a namespaced one, which only showed up as a warning buried in startup output.

Rendered and validated a collector pipeline for OTLP signals and container logs. Its validator caught malformed object syntax and an unavailable component, though reaching a valid configuration required several correction cycles.

What worked

The local validator checked the actual rendered collector configuration and accepted the final pipeline, including native trace and span identifiers on logs.

What got in the way

The downloaded executable initially lacked execute permission, and configuration errors around field separators and component availability required manual diagnosis.

Alloy was configured to select only the target service's journal records, add production labels, and send logs to Loki over TLS. Its validator accepted the final configuration.

What worked

The configuration expressed journal discovery, service filtering, labels, credentials, and Loki forwarding in one validated pipeline.

What got in the way

The downloaded binary was not executable on the first validation attempt. Adding execute permission resolved the issue, after which validation succeeded.

Created a journal-to-Loki pipeline and used Alloy's formatter and validator with environment-supplied credentials. Validation passed after fixing executable permissions and applying Alloy's canonical formatting.

What worked

The native formatter and validator caught configuration presentation issues and confirmed the final environment-driven collector configuration locally.

What got in the way

The downloaded executable initially lacked execute permission, and the first full check failed because the configuration did not match Alloy's canonical indentation.

Authored a River-syntax Alloy config (OTLP receiver, batch processor, OTLP HTTP exporter with basic auth) and wired it into docker-compose as a new service, cross-checking exact component and function syntax against official docs before writing the file.

What got in the way

Could not actually run or lint the config because no container runtime was available in the environment, so the final syntax is backed by documentation cross-checks rather than a real validation run.

Read several component reference pages (OTLP receiver/exporter, basic auth, file-match, delta-to-cumulative processor) to hand-author an Alloy pipeline config and systemd install notes; never installed or ran Alloy itself since there was no server access in this task.

What worked

Component-level reference docs were precise enough to get non-obvious details right on the first attempt, such as the nested client_auth block for basic auth and the delta-to-cumulative conversion needed before data reaches Mimir.

What got in the way

Finding the systemd environment-file path and default config location took an extra web search since it wasn't on the component reference pages first consulted.

Wrote a new Alloy config to receive OTLP traces/metrics from the app, tail container logs, scrape host metrics, and forward everything to Grafana Cloud; downloaded the real binary to validate and smoke-run the config before deploying it as a docker-compose sidecar.

What worked

The validate and run subcommands caught issues immediately, and a dummy-credential smoke test confirmed the whole pipeline wired together correctly, down to the one expected failure from a host mount path that only exists inside the real container.

What got in the way

Some component syntax, like the stdlib env function, was unclear from search results alone because of documentation redirects, requiring direct page fetches to confirm the current syntax.

Wrote an Alloy config to tail the app container's Docker logs via a mounted docker.sock and forward them to Loki, choosing it over the deprecated Promtail. The newer Alloy configuration language was less familiar, but the config was authored and validated as syntactically correct; never executed in this sandbox.

Configured Alloy to discover containers and forward their JSON logs to Loki, consulting the component documentation to refine the pipeline and labels.

What worked

The published component reference was useful for resolving the configuration structure.

What got in the way

The agent could not be launched to validate discovery or log forwarding because the container runtime was absent.