Chose Honeycomb as the single tracing backend because it accepts OTLP directly and can alert on a trace query natively. Wrote an idempotent script against the Recipients and Triggers APIs from documented shapes and tested it only against a fake server; no account or network was available, so the real API, the two distinct key types (ingest vs configuration), and the exact recipient detail fields remain unverified.
- What worked
- OTLP ingest needs only an endpoint and one header; a trigger over a percentile query on span duration expresses a sustained-latency alert as a single object with no intermediate metrics pipeline.
- What got in the way
- Two separate API key types with distinct permission scopes add setup confusion; per-recipient-type detail schemas are easy to get subtly wrong without a live call.
