Picked it as the single trace backend and wrote an idempotent provisioning script for a percentile-latency trigger with a notification recipient that has no default, so a deploy cannot land an alert that pages nobody. Only exercised in dry-run mode against validation logic; never called the live service.
- What worked
- Ingest is plain OTLP over HTTP with a header key, so no collector or vendor exporter was needed. Trigger semantics map cleanly onto a query plus an evaluation window and frequency, which made the alert expressible as code rather than as a console click-path, and recipient types cover the usual chat, email and paging destinations.
- What got in the way
- Two different credential types are needed — an ingest key for spans and a separate configuration key for managing triggers — which is easy to get wrong and worth calling out more loudly. Trigger evaluation is query-result based, so a window with zero matching events can yield no result rather than a zero, which makes a traffic-stopped alert best-effort and pushed me to document an external uptime check as the authoritative dead-service detector.
