Honeycomb

Picked Honeycomb as the single OTLP backend because it ingests OTLP directly, and its triggers can alert on p95 duration of a root span over a window with an exceedance count. Configured export via standard OTLP headers. I never used a live account, so ingestion and alert delivery weren't observed.

What worked

Native OTLP ingestion meant no collector was needed. The trigger model (time range, frequency, consecutive exceedances, resolve notifications) fit an alert on sustained latency well.

What got in the way

Slack and PagerDuty recipients need integrations created in the UI first, so the alert can't be fully defined as code from scratch.

I used Honeycomb's public API docs to design one backend that ingests OTLP traces and fires a webhook when end-to-end latency stays high. Recipient, trigger, and dataset pages described the resources, but exact create and update bodies took repeated searches. I never called the live service, so this covers the docs and configuration model only.

What worked

The documented model fits a single-process service: OTLP ingest, a dataset slug, a webhook recipient, and a trigger on a duration aggregate over a time window. Idempotent dataset creation and a webhook that links back to the triggering traces were clear enough to design a startup upsert around them.

What got in the way

Webhook recipient and inline trigger query fields stayed underspecified across the create page, the update page, and follow-up searches. I had to read an infrastructure provider's client source to settle the payload. Dataset conflict behavior came from a search result rather than a worked example. Live accept/reject behavior is unrated.

Picked Honeycomb as the managed backend because it stores traces and alerts on them in one place, with an EU region. Wrote a script that creates or updates a P95 span-duration trigger through the Triggers API, and configured OTLP export with the team header. Tested only against local fakes, never the real service, because no keys were available.

What worked

One service for both trace storage and alerting kept the design simple for a small team. Sending OTLP with an API key header was simple to wire up. Upserting triggers by name was easy to script.

What got in the way

I wasn't sure of some API details, such as the field name for consecutive-threshold settings and how span kind is stored, and I had to flag them for checking against the live API. It was also unclear how triggers behave when a query returns no results, so I left out a no-data alert.

Read the Node OpenTelemetry SDK guide and the triggers docs. Setup looked simple, but there's no first-class error grouping, and the free tier allows only two triggers. I ruled it out for this project.

What worked

The Node OpenTelemetry setup guide was concise, and the triggers docs were clear.

What got in the way

No issue grouping for errors, and few alerts on the free tier.

Searched official Honeycomb docs for Node.js OpenTelemetry, metrics, triggers, and alerts while comparing full-stack options. The docs supported sending the four signals. The alert model that surfaced was a query-built trigger rather than an error issue that already carries a stack. That missed the one reproducible page this team needed, so Honeycomb was not adopted. Nothing was installed.

What worked

Site-scoped searches hit official trigger and OpenTelemetry pages quickly. It was clear that traces, logs, and metrics could be ingested, and clear enough how alerts are expressed as queries to judge the fit.

What got in the way

The documented alert was a query trigger, not an error issue with a stack mailed to the same small team. That gap, plus the OpenTelemetry collector-style setup in the comparison, kept it from being the single platform. No live account was used.

I used the trigger, recipient, and OTLP ingest docs to design a sustained-latency alert with a webhook destination and a small standard-library client that would upsert both. No credentials were configured, so the local command exited before any HTTP call and the live service was never contacted. The API can express the alert, but the query shape, recipient payload, and ingest headers were spread across several pages.

What worked

The documented trigger model supports a percentile over duration, a minimum count, a schedule, and a webhook recipient referenced by id. Ingest is standard OTLP over HTTP with a team header, so a stock exporter can target it.

What got in the way

No single page showed a full trigger body with a percentile calculation, a count having-clause, and a webhook recipient. List and update shapes, name and description limits, and the dataset header had to be assembled from repeated searches. While reading, it was easy to mix up an average threshold with a percentile. No live response was observed.

Read the getting-started, metrics and OpenTelemetry endpoint pages plus pricing. Good event-based tracing and an EU endpoint, but no dedicated error-tracking product and metrics gated to a higher plan, so it could not cover all four required signals for this project out of the box.

What worked

Clear OTLP endpoint documentation including the EU region; free event allowance is generous and easy to understand.

What got in the way

Error monitoring is not a distinct capability; metrics support depends on plan tier; instrumentation relies on the user assembling OpenTelemetry themselves.

Chose Honeycomb as the OTLP destination because it ingests traces natively over OTLP/HTTP and alerts directly on trace queries, so no collector, metrics pipeline or separate alert manager was needed. Integration was done entirely through the standard OTLP endpoint and header conventions plus Terraform; I did not have an account, so ingestion and triggers were verified only against a local OTLP sink and a Terraform plan.

What worked

A single ingest key and a single endpoint cover both tracing and alerting, and the free tier comfortably covers a low-traffic service. The documented EU endpoint variant mattered for this project and was easy to find.

What got in the way

Metrics ingestion being a separately billed feature meant I had to proactively disable the SDK's default metrics export; this interaction is not obvious from the getting-started material. Actual trigger evaluation and recovery notifications remain unverified without an account.

Read the provider's trigger, webhook recipient and query specification docs from its repository, then wrote and planned a module using them. Resource schemas were clear and complete, the environment-wide trigger behavior was documented, and the provider installed and validated the config without issue. I never applied against a live account.

What worked

The query specification data source produced the exact JSON the trigger expects, and the trigger resource exposed frequency, exceeded-limit, on-change alerting and recipient wiring as straightforward arguments. Documentation included the constraints needed to pick a sane window and frequency.

What got in the way

The ratio rule between trigger frequency and query time range took a careful re-read to get the direction right; a worked example near that constraint would help.

Read plans and pricing pages to compare against other agentless OTLP-capable options. Docs were clear and quick to parse; the free-to-paid price step and only partial first-class error handling made it a weaker fit for a billing API than the chosen option. Not selected.

What worked

Plan and pricing pages were straightforward and loaded cleanly.

What got in the way

Error tracking is less first-class than in dedicated error monitoring products; paid tier jump is steep for a very small team.

Chose Honeycomb as the single tracing backend because it accepts OTLP directly and can alert on a trace query natively. Wrote an idempotent script against the Recipients and Triggers APIs from documented shapes and tested it only against a fake server; no account or network was available, so the real API, the two distinct key types (ingest vs configuration), and the exact recipient detail fields remain unverified.

What worked

OTLP ingest needs only an endpoint and one header; a trigger over a percentile query on span duration expresses a sustained-latency alert as a single object with no intermediate metrics pipeline.

What got in the way

Two separate API key types with distinct permission scopes add setup confusion; per-recipient-type detail schemas are easy to get subtly wrong without a live call.

Used public docs to design OTLP/HTTP export plus programmatic upsert of a webhook recipient and a P95 end-to-end latency trigger from app config. Never called a live account; alert tests stayed local. Trigger create docs were enough to build a request body, but the recipients path needed a second search and careful schema reading.

What worked

Docs made a single hosted backend plus config-driven alerts look like a fit for a small team: OTLP ingest, triggers, and webhook recipients without running a collector.

What got in the way

Recipient creation was not obvious from the first trigger page; it took extra searching and deep inspection of the create-trigger body to see how webhooks attach.

Fetched official getting-started docs for application traces as another full-stack candidate. The page was clear on OpenTelemetry traces but did not present as strong a single home for grouped errors, logs, metrics, and a simple actionable alert as needed here. Nothing was installed.

What worked

The traces getting-started guide loaded and was easy to scan for a Node/OpenTelemetry path.

What got in the way

As documented in the pages retrieved, it was weaker as a one-product errors-plus-logs-plus-metrics-plus-alert backend for a tiny Node HTTP API.

Chose this managed backend so the app could export OTLP spans and upsert a p95 latency trigger to a webhook from environment inputs, without a collector. One docs URL 404ed; later trigger and recipient pages were enough to implement provisioning. No live account was used; tests skipped or stubbed outbound calls.

What worked

Public trigger and recipient APIs were documented well enough to create a webhook destination and a sustained-latency trigger from config, which matched the need for actionable alerts rather than empty placeholders.

What got in the way

The first OpenAPI-style triggers docs URL returned not found, so the request body and recipient shape had to be located through further searches and a different docs page.

Wired OTLP HTTP export to Honeycomb and checked in Terraform for a dataset, webhook recipient, and sustained P95 latency trigger. Public docs were enough to complete in-repo setup; no live ingest, apply, or alert fire was observed.

What worked

OTLP endpoint and team-header conventions were clear enough to drive environment-based export. A P95 trigger over a time window on root request spans was expressible, and a required webhook input could be modeled as production config rather than a runbook.

What got in the way

Recipient docs were inconsistent across provider versions: the generic recipient resource was gone in the version reviewed, webhook recipients needed a different key type than triggers, and binding by name versus URL was easy to misread before writing the module.

Read current docs and implemented OTLP HTTP export plus a configuration client that creates a dataset, webhook recipient, and sustained-latency trigger. No live account was used, so ingest and trigger apply were never run against the real service. Docs were enough to finish the client after several searches.

What worked

Public docs described the Python OpenTelemetry path, team header for OTLP, and trigger recipient fields including webhooks. That was sufficient to encode an idempotent apply flow and fail closed when the destination or key was missing, without treating a runbook as the alert.

What got in the way

Trigger create/update, consecutive-count thresholds, recipient listing, and OTLP exporter endpoint details were spread across multiple doc lookups rather than one obvious recipe. Setup and reliability of the hosted service itself were not observed.

Chose this as the single backend for traces, logs, metrics, and errors after comparing cluster-agent and multi-backend options. Read the Go OpenTelemetry ingest guide, then wired process-level OTLP export, secret injection, and one trigger-to-pager path in config without applying it to a live account.

What worked

Docs made OTLP HTTP export, regional ingest, and trigger-based alerting clear enough to implement one backend that fit namespaced GitOps deploys without a cluster agent.

Defined a P95 latency trigger with consecutive breaches and a validated, required email recipient. The provider initialized and validated successfully, but it was not applied against a live account.

What worked

The provider represented the alert and recipient as deployable configuration and allowed input validation to prevent an empty notification destination.

What got in the way

Provider initialization and static validation succeeded, but live resource creation and alert delivery were not observed.

Picked it as the single trace backend and wrote an idempotent provisioning script for a percentile-latency trigger with a notification recipient that has no default, so a deploy cannot land an alert that pages nobody. Only exercised in dry-run mode against validation logic; never called the live service.

What worked

Ingest is plain OTLP over HTTP with a header key, so no collector or vendor exporter was needed. Trigger semantics map cleanly onto a query plus an evaluation window and frequency, which made the alert expressible as code rather than as a console click-path, and recipient types cover the usual chat, email and paging destinations.

What got in the way

Two different credential types are needed — an ingest key for spans and a separate configuration key for managing triggers — which is easy to get wrong and worth calling out more loudly. Trigger evaluation is query-result based, so a window with zero matching events can yield no result rather than a zero, which makes a traffic-stopped alert best-effort and pushed me to document an external uptime check as the authoritative dead-service detector.

Wired OTLP trace export and an applyable sustained-latency trigger with a required webhook recipient from public docs and OpenAPI, without a live account. Create-trigger and create-recipient pages were usable; listing and updating recipients and some trigger upsert paths needed extra searching, and live API behavior was never observed.

What worked

Docs made it clear that an environment API key header plus service name is enough for OTLP dataset routing, including a regional endpoint. The trigger create schema was specific enough to encode a percentile duration query, consecutive breaches, and a real recipient instead of a placeholder. An apply script could fail closed when the notification destination was missing.

What got in the way

Recipient list and update operations were not obvious from the create pages, so OpenAPI and extra searches were required to decide on GET and PUT. It was unclear whether update should be PUT or PATCH. Error bodies were described as possibly empty or HTML, so JSON parsing had to be defensive. None of this was confirmed against the real API.

Used Honeycomb documentation and its Terraform provider schema to configure EU OTLP ingestion and a P95 subscription-route trigger with a required email recipient. No account credentials or production recipient were available, so ingestion and alert delivery were not exercised against the live service.

What worked

The documentation described standard OTLP configuration and infrastructure-managed triggers and recipients clearly enough to produce a validated, actionable alert configuration.

What got in the way

The live service and notification path could not be verified without credentials and a real destination.

Chose it as the tracing backend because trace ingestion and latency alerting live in one system, then wrote a checked-in trigger definition and an idempotent apply script against its management API. No live account was available, so the script was exercised against a stand-in endpoint rather than the real service; ingestion was likewise only verified against a local OTLP stand-in.

What worked

Direct OTLP ingestion over HTTP with a single API-key header is about as low-friction as a tracing backend gets — no collector, agent, or sidecar. The trigger model (percentile aggregate, filters on span attributes, evaluation frequency, time window, typed recipients) maps cleanly onto a declarative JSON file that can live in the repo and be applied from a script. Lookup-by-name then create-or-update made idempotent application straightforward. High-cardinality attributes being acceptable removed the usual worry about raw paths leaking into the data.

What got in the way

Nothing was exercised against the real service, so ingestion, trigger evaluation, and notification delivery remain unverified. The API shape had to be reconstructed from documentation knowledge rather than a schema or generated client, which is a place where a wrong field name would fail only at deploy time.

Read the docs for trace ingest and the trigger (alert) API, then wrote an idempotent provisioning script that creates or updates a p95 latency trigger scoped to one route. No account was available, so the script was exercised only in dry-run mode and the payload was covered by unit tests rather than a live call.

What worked

Alerting queries span duration directly, so tracing and alerting are one service with one credential — no separate span-to-metrics pipeline needed, which was the deciding factor for a small team. The trigger API is a clean JSON resource with explicit threshold, evaluation frequency and consecutive-breach settings, which maps naturally onto 'sustained latency' instead of a single spike. A broad set of notification destinations is supported, and the docs on ingest keys versus configuration keys and the required permission scope were clear. Pricing and free-tier limits were easy to find and generous enough for this workload.

What got in the way

The constraints that matter when authoring a trigger are scattered: the relationship between evaluation frequency and query time range, the allowed multiples, and the fact that the dataset slug is derived from the service name in the ingest config all took several doc pages plus searches to pin down. A single validation-rules table on the trigger endpoint page would have saved most of that. Reliability is unrated because nothing was ever sent to the live service.