PagerDuty

Recommended as a read-only investigator plus fix-PR proposer for ingestion, query, and billing incidents, scoped to service logs and code with no raw event payloads or secrets and human-approved releases.

What worked

Mapped to the existing alert and service-log boundary on paper and supported a narrow read-only role plus explicit payload and credential exclusions.

What got in the way

No live connection, trial run, or service API call appears in the record, so actual detection quality, setup effort, and operational reliability could not be observed.

Selected as the AI SRE for read-only investigation plus PR-proposed fixes across ingestion, query and billing, with human approval before release. Wired existing alert routing to its event intake and scoped its access to sanitized telemetry and read-only repo access, preserving tenant isolation.

What worked

Conceptual fit was clear: read-only diagnosis with fix proposals through normal review matched the isolation and release constraints. Configuration surface was small and variable-driven.

What got in the way

No live account or end-to-end incident run was available, so alert delivery and fix-PR behavior could not be observed. Setup depended on team-supplied endpoint and remote state.

Wrote a small client for the oncalls endpoint, keyed by schedule IDs and using token auth. I tested it only against a local fake server; there was no live account, and real schedule IDs are still placeholders.

I added a read-only client for schedules and open incidents using GET requests and a shared token header, following the published remote-server example. Unit tests checked the on-call path. I never called the live API.

What worked

The on-call request path was easy to pin down, and read versus write operations were distinct enough to expose only reads.

Added a read-only client for the current primary and secondary on-call responders, with the token taken from the secret files. Unknown rotation names are rejected in process before a request is built. Coverage was local fixtures, and API documentation was not opened.

What worked

The required operation is a single read, so a small HTTP client and fixtures could define the primary and secondary fields the gateway returns.

What got in the way

Live response schemas, error bodies, and rate limits were not observed, so the client still rests on an assumed payload.

Evaluated Advance AIOps grouping and Automation Actions runner, compared to alternative observability vendors. Integrated via Terraform service, CloudWatch inbound integration and automation runner with variable-gated SNS subscription.

What worked

Intelligent grouping and inbound CloudWatch integration fit existing log routing without moving data; Terraform resources model service lifecycle clearly.

What got in the way

Registry docs lagged provider schema for runner; endpoint and runner field names required live schema lookup.

Targeted the CloudWatch integration endpoint as an HTTPS subscription destination, with the integration URL taken as a sensitive, regex-validated required input so it is supplied from the secrets pipeline rather than committed. Sending both alarm and OK actions means incidents auto-resolve. No account was available, so the integration was configured but not exercised.

What worked

The CloudWatch integration accepts alarm state changes directly over an SNS HTTPS subscription and auto-resolves on OK, so no glue code was needed.

What got in the way

The integration URL embeds a secret key, so it has to be handled as a secret end to end; there is no way to reference the integration by a non-secret identifier from infrastructure code.

Modeled the Datadog-side integration and a service object so monitor messages can address the on-call service by handle, with the service key and subdomain as required, validated inputs. The integration still needs a one-time manual creation step in the PagerDuty UI before apply, which I documented in the setup order. Nothing was tested end to end.

What worked

The service-key model maps cleanly onto Terraform variables, making the paging destination mandatory rather than optional.

What got in the way

A manual UI step to enable the integration remains unavoidable; I could not confirm an incident actually opens without credentials.

Wrote a small relay that converts error-tracker webhooks into Events API v2 trigger calls with per-issue dedup keys and a drill-mode severity, targeting the EU regional endpoint. Covered by unit tests with a fake server; not exercised against the live API.

What worked

Events v2 payload is simple and dedup_key maps naturally onto an issue identifier. Routing keys fit cleanly into a per-service secret store path.

What got in the way

Regional endpoint selection is easy to get wrong silently; it had to be a deliberate configuration choice rather than a default.

Configured Events API v2 receivers in Alertmanager with per-team routing keys read from injected files and severity mapping. No live account, so delivery was not tested; the regional API URL choice remains an open question for the operators.

What worked

The Alertmanager integration needs only a routing key per service, which fits a one-receiver-per-rotation model well.

What got in the way

Could not verify event delivery or severity mapping end to end without credentials.

Configured an Alertmanager PagerDuty receiver using an Events API v2 routing key supplied via a Kubernetes secret, with summary and description templated from alert annotations. Nothing was sent to the real service. The integration surface is small and well understood; the only configuration subtlety was that EU-hosted accounts need a different endpoint URL, which I exposed as an optional value.

What worked

A single routing key is all the receiver needs, so the destination can be a required chart input.

Configured Alertmanager PagerDuty receivers per team using routing-key files mounted from a secrets manager, and documented the out-of-band steps (urgency rules, support hours) an operator must complete. No live integration key was available, so delivery was not exercised.

What worked

The Alertmanager receiver model maps cleanly onto one service per rotation, and file-based routing keys avoid putting secrets in manifests.

What got in the way

End-to-end delivery could not be verified without real integration keys.

Treated the existing pager service as the destination for one actionable alert and configured it as a contact point from the observability side. The official integration page that was fetched did not load. No pager account or event API was exercised.

What worked

It was obvious that reuse of current rotations was better than introducing a second paging product, and a contact-point block could be written from prior knowledge of the integration.

What got in the way

The vendor integration document that was requested never loaded, so routing-key and URL field requirements stayed uncertain.

Wrote a read-only client that resolves who is currently on call for a team and returns the escalation path, querying the on-calls endpoint filtered by escalation policy. Covered the unconfigured and selection paths in tests; never called the live service.

What worked

The on-calls resource is a direct fit for the question being asked — filter by policy, get current responders — so the client stayed small and obviously read-only.

What got in the way

Everything hangs off opaque escalation-policy identifiers, so a team-name-to-policy mapping has to be maintained outside the API as configuration, and a stale mapping fails quietly with an empty result rather than an error. Rotation metadata such as paging windows is not usefully derivable, so it ended up duplicated from internal documentation with a staleness warning.

Designed Terraform configuration to connect a Datadog 5xx monitor to a PagerDuty service using an externally supplied routing key. The configuration was validated locally, but the integration required prior account activation and was not exercised against the live service.

What worked

The service-routing model provided a clear operator destination for the selected production failure signal.

What got in the way

No credentials or enabled integration were available, so incident creation and delivery reliability were not assessed.

Wired latency SLO alerts to the existing on-call path through hosted contact points and required routing-key inputs, rejecting empty keys. Nothing was sent to a live PagerDuty account.

What worked

Contact-point plus required-key design met the rule that destinations are production-configured and actionable rather than documented for later.

What got in the way

No event was delivered, so routing-key validity and page behavior were not observed.

Configured PagerDuty as the required destination for repeated monitor failures through Checkly's alert-channel construct. The integration-key model was clear enough to validate configuration locally, but the real service was not exercised because production credentials were unavailable.

What worked

A single required service integration key provided a direct, actionable paging destination and could be enforced before deployment.

What got in the way

End-to-end delivery and recovery notifications could not be verified without a live PagerDuty integration key.

Reused the already chosen pager as the destination for a gateway error-rate alert instead of adding a second on-call path. Wired it through the observability provisioner and a staging reproduction note. No page was sent or confirmed.

What worked

Existing rotations and services made the operator path obvious. The integration was only a contact target plus a runbook, not a new paging product.

What got in the way

Keys and live delivery were not exercised, so it is unknown whether the alert would actually reach an on-call service. Contact-point naming in the provisioner needed a dedicated target rather than a catch-all default.

Chose the vendor's own MCP server as the only real source for who is currently holding a pager, and configured it as a gateway upstream with a scoped read-only credential from a secrets manager. Configuration only; it was never launched.

What worked

Having a first-party server for the service removed the need to wrap an API by hand, and a documented read-only posture fit a deployment whose whole premise was look-but-don't-touch. It cleanly solved the part of the problem that static documentation could not: a rotation name in a config file is not a person, and only the live scheduling system knows the answer.

What got in the way

Same unverified gaps as the other upstreams: transport mode and health endpoint were not confirmable from documentation, and the exact tool names needed for the gateway allowlist had to be taken on faith. The allowlist at least fails closed, so a wrong name hides a tool rather than exposing one.

Wired production alert contact points so a gateway 5xx burn pages the existing on-call rotation, with a local webhook standing in for that path. No PagerDuty account or API was used; integration was Grafana contact-point YAML only.

What worked

Treating paging as a Grafana contact point mapped cleanly onto rotations already described in the repo. The local webhook substitute kept the alert reproducible without a live pager.

What got in the way

Delivery, dedup, and acknowledgment were never exercised against the real paging service.

Fitted latency alerts to the on-call path already documented in the repo by requiring integration keys as inputs and creating provider-backed contact points. No live Events API call was made; a helper script refused to proceed when keys were absent.

What worked

Service-level integration keys were a clear, automatable destination. Treating them as mandatory inputs avoided empty or manual-only notification config.

What got in the way

Paging was not exercised against a real account, so delivery, dedup, and rotation behavior were not observed.

Pointed the new monitor at an existing paging integration instead of adding another on-call backend. No account, API, or test page was exercised.

What worked

A notification handle on the monitor was enough to reuse the current paging path for one high-severity edge condition.

What got in the way

Live integration setup still sits outside the repo, so delivery and handle validity were not confirmed.

Configured it as the paging destination in an alert router, using the standard integration receiver with a routing key read from a secret file. No account or key was available, so nothing was ever delivered and the choice was explicitly flagged to the team as an assumption to confirm.

What worked

It is a first-class, well-known receiver type in the alert router, so the config block is small and the required inputs are obvious: a routing key plus the alert payload mapping.

What got in the way

Cannot be validated without a real integration key, so correctness beyond config shape is unknowable offline. Nothing in the project indicated which paging vendor the team actually uses, making this the least evidence-backed part of the work.