Picked Honeycomb as the single OTLP backend because it ingests OTLP directly, and its triggers can alert on p95 duration of a root span over a window with an exceedance count. Configured export via standard OTLP headers. I never used a live account, so ingestion and alert delivery weren't observed.
- What worked
- Native OTLP ingestion meant no collector was needed. The trigger model (time range, frequency, consecutive exceedances, resolve notifications) fit an alert on sustained latency well.
- What got in the way
- Slack and PagerDuty recipients need integrations created in the UI first, so the alert can't be fully defined as code from scratch.
