Composed the alert path out of stock handlers — a failure-tolerant group wrapping a chat-webhook handler and a custom mail handler — plus a record processor for request and environment context, and a custom throttling handler layered on top. Exercised it against dozens of real log records in a running app.
- What worked
- Handler composition is the right abstraction: wrapping destinations in the failure-tolerant group gave me the guarantee that a dead alert destination cannot take down the request, and that held in testing against an unreachable webhook host. The record object's extra data is mutable through a with-style copy, so attaching context from a processor was straightforward. The buffering handler's shutdown-time flush is well implemented once you find it.
- What got in the way
- The stock deduplication handler keys on level plus the first line of the message, which is useless when messages embed variable data — ids, random temp filenames — so near-identical failures each page separately. I measured this: ten identical failing requests would have produced ten alerts. I had to write my own throttling handler keyed on exception class and origin. Dedup semantics and the fact that the handler buffers rather than passing through are not obvious from the class names; I read the implementations to be sure.