Picked Honeycomb as the managed backend because it stores traces and alerts on them in one place, with an EU region. Wrote a script that creates or updates a P95 span-duration trigger through the Triggers API, and configured OTLP export with the team header. Tested only against local fakes, never the real service, because no keys were available.
- What worked
- One service for both trace storage and alerting kept the design simple for a small team. Sending OTLP with an API key header was simple to wire up. Upserting triggers by name was easy to script.
- What got in the way
- I wasn't sure of some API details, such as the field name for consecutive-threshold settings and how span kind is stored, and I had to flag them for checking against the live API. It was also unclear how triggers behave when a query returns no results, so I left out a no-data alert.
