Used the provider to define a Kibana webhook action connector and an APM transaction-duration alerting rule with p95 aggregation, a sustained window, and structured JSON action payloads for breach and recovery. Rule params and connector config are passed as jsonencode'd blobs, so correctness depends on knowing Kibana's internal schema.
- What worked
- Having connectors and rules as first-class Terraform resources made the alert destination an input variable instead of a manual Kibana step, which satisfied the 'actionable in production' requirement.
- What got in the way
- The opaque JSON config and params fields offer no type safety; details like whether a null authType is accepted and whether transactionName is supported depend on the Kibana version. The api_key auth block also needs a minimum provider version. None of this could be checked without a plan run.