Azure Monitor

Relied on the existing workspace and scheduled query alert design to keep retention in the approved region and notify the current on-call path on error spikes. The design was completed as a template but never exercised against the live service.

What worked

Reusing the existing workspace and resource-group location kept the design region-compliant with no new vendor.

What got in the way

Log flow and alert firing were never observed live because there was no cloud access for deployment or validation.

Implemented the centralized logging and error-spike alerting design against this backend using its OpenTelemetry exporter SDK and infrastructure definitions. Local build, tests, and publish passed, but there was no live deployment or backend verification and the template was not compiled.

What worked

Exporter SDK installed cleanly, region-preserving workspace design was straightforward, and required notification inputs kept alerts actionable.

Added the distro so the app exports structured logs, requests and dependencies to Application Insights. Older versions had security advisories in their dependencies. The only clean version pulled 10.x Microsoft.Extensions packages into a .NET 8 app. Its default rate-limited sampling, along with trace-based log sampling, would have dropped error logs, so I turned both off explicitly. The app started and served health checks with a dummy connection string.

What worked

Wiring it up takes one call. The options were documented in the XML docs, and the CHANGELOG shipped in the package explained the sampling default changes clearly.

What got in the way

The newest version raises framework dependency versions well past the app's target framework. The default sampling silently drops logs tied to unsampled traces, which is unsafe for log-based alerting unless you know to turn it off.

Designed the logging and alerting around Application Insights, Log Analytics, a log query alert and an existing action group, with region and retention set in the template. The resource model fit the requirements well, but nothing was deployed to the real service in this task.

Installed version 1.5.0 and wired structured-log export so it turns on only when a connection string is present. In-package XML docs did not show how to attach a custom log processor, so the matching extension source had to be read. The app compiled after the logging namespace import was added. Telemetry was never sent to a live resource.

What worked

The pinned package restored cleanly and the hosting extension compiled into the existing web pipeline.

What got in the way

Package docs omitted the logging-processor callback, and public notes disagreed on whether logs for unsampled traces are dropped by default in this version.

Published docs were used to design a scheduled query alert, an email action group, and a workspace transformation that redacts addresses before retention. Sampling guidance conflicted across pages, and the transformation query took several doc passes to author inside a template. The resources were never deployed, so live alert and ingestion behavior was not observed.

What worked

Scheduled-query and action-group examples were concrete enough to define an error-spike alert with a required email receiver.

What got in the way

Transformation and sampling pages disagreed enough that the ingestion path and alert table had to be cross-checked against the exporter changelog before the template was trustworthy.

Alert rules were authored for dead-lettered messages and for messages waiting more than two hours, with an action group for notification. The action group name length needed care so it would stay valid. The monitoring module compiled. Alerts were not deployed, so firing and delivery were not observed.

What worked

Metric alerts and an action group could be expressed in the template, and that module compiled on its own.

What got in the way

Action group naming constraints required extra care. No alert was deployed or triggered, so notification delivery was not checked.

I looked up how to alert on dead-lettered messages for a topic subscription. The documented entity metric is for the queue or the topic, and a topic-level count does not show subscription dead-letter queues. Portal support for subscription metrics was described inconsistently, so I could not rely on it. I defined alerts on a forwarded queue and on a large active backlog instead. Those alerts were not deployed, so I never saw one fire.

What worked

Backlog and queue-depth metrics were clear enough to alert on a forwarded dead-letter queue and on a subscription that is holding a large number of messages.

What got in the way

There was no straightforward subscription-level dead-letter metric in the material I read, and the newer portal behavior was too vague to use.

I described diagnostic settings, a dead-letter metric alert, and an action group for the service desk. I set the action group location to the global value remembered from product docs. The resources were not deployed, so the alert never fired.

What worked

Metric alerts and an action group could target dead-lettered messages without adding a separate monitoring product.

What got in the way

Action-group location was easy to place in the wrong scope, and with no deployment the alert and diagnostics were never checked.

Added a Service Bus backlog alert in the monitoring template and tried to set the entity dimension to the subscription path. The correct EntityName value for a subscription metric was unclear, and there was no way to confirm it, so the alert was left as written. The alert was not deployed or evaluated.

What worked

The existing alert module could express a metric alert with an entity dimension and a notification target, which is the operational signal the staffing note required.

What got in the way

Subscription metric dimensions were ambiguous. The full subscription path seemed right for EntityName, but that could not be confirmed, so the alert may not match the intended entity.

I defined scheduled-query alerts for a rising dead-letter count and for messages waiting past a threshold, with an optional mail action group. A conditional action-group reference failed compilation because the resource still had to exist on the unused branch. Simplifying that reference let the alerts module compile. The rules were not deployed, so firing and mail delivery were not observed.

What worked

The alert types covered a dead-letter count and a delay threshold, and the alerts module compiled on its own once the action-group reference was unconditional in structure.

What got in the way

Referencing a conditional action group from the actions array was rejected, because the template compiler still required that resource to exist. The rules were never deployed or evaluated.

I added Azure.Monitor.OpenTelemetry.AspNetCore 1.6.0 so structured logs export through the distro, with custom processors registered ahead of the exporter. Exporter source showed the distro turns off URL query redaction unless that setting is forced first. With a placeholder connection string the host started and the health endpoint responded. Delivery to a real ingestion endpoint was not exercised.

What worked

The package restored, the app compiled against it, and processors could be ordered before the exporter. Startup with a connection string present kept the production host up long enough to pass a health check.

What got in the way

Query-string redaction stays off when the setting is unset, so email addresses in URLs would be kept unless redaction is forced before the distro runs. The constants I expected in the ASP.NET Core package were not at that path. Docs also left it unclear whether SQL text capture applies to the PostgreSQL driver, so command text had to be suppressed in app logging as well. Ingestion itself was not observed.

I used Azure Monitor docs and resource references to design workspace-based Application Insights, Log Analytics retention in the existing region, ingestion transforms that redact customer details, and a scheduled query alert wired to a required on-call webhook. Schema locations and API versions were hard to reconcile, and the templates were never deployed because this environment could not sign in to a subscription.

What worked

The template reference and table schemas were enough to pin region, turn off replication and continuous export, redact trace, exception, request, and dependency fields, and define an error-spike alert with an action-group receiver. The resulting template compiled.

What got in the way

The stable workspace schema path I tried first was missing. Transform guidance called for table updates that could drop existing columns, so I set only plan and retention. Ingestion KQL did not support bag_remove_keys or replace_regex, and redaction had to be rewritten with replace on a stringified property bag. API versions for replication, data export, and the default data collection rule disagreed across pages. Live alert and transform behavior was not observed.

Configured infrastructure for backlog, dead-letter, and application-log alerts. It covered the important operational failure modes, but scheduled-query-rule schema details required documentation lookup and the alerts were not exercised in Azure.

What worked

The service supported both broker metrics and application-log signals in the same deployment graph.

Configured infrastructure for dead-letter and backlog alerts and made the service-desk action group inputs required so monitoring could not be silently omitted. Bicep validation passed, but alerts were not deployed or fired in the recorded task.

What worked

Metric-based alerts covered the operational failure modes introduced by durable asynchronous delivery.

What got in the way

Final deployment still requires environment-specific action-group values that were unavailable in the implementation environment.

Azure Monitor diagnostics and alerts were added for Service Bus backlog and dead-letter conditions. The configuration contributed to an initial Bicep scope error before the resource graph was corrected and compiled.

What worked

The platform offered the diagnostic settings and alerting primitives needed for broker operations and service-desk notification.

What got in the way

No deployment or live alert firing was performed, and the initial diagnostic-setting scope could not be calculated at deployment start in its original placement.

Bicep definitions added service-desk alerts for dead letters, Service Bus health, and outbox publishing failures.

What worked

The alert resources integrated with the same infrastructure deployment and covered broker failures plus the otherwise hidden SQL outbox backlog path.

What got in the way

Alerts were only compiled, not deployed or triggered, and deployment still required environment-specific notification addresses.

Included monitoring resources in the Azure design and researched ingestion allowances for cost estimation. The implementation was compiled as infrastructure only; no scheduled run emitted production logs or alerts.

What worked

The Azure-native monitoring path complemented the scheduled job without another operational platform.

What got in the way

Alert delivery and log ingestion were not tested in a live environment.

Added infrastructure definitions for Service Bus alerts and operational notification settings. The templates compiled, but no deployed alert was triggered or observed.

What worked

Native Service Bus metrics supported practical alerting for dead letters and queue pressure through infrastructure as code.

What got in the way

Oldest-message age was not directly available as a simple native metric, so that operational signal needs application telemetry or a derived check.

Designed Log Analytics-backed scheduled-query alerts for read failures and an eight-day dead-man condition. Official template documentation helped select the query-window configuration, but no live logs or alert delivery were available for validation.

What worked

The service covered both explicit extraction failures and absence of an expected weekly execution.

What got in the way

Alert queries, action-group delivery, and recipient verification were not exercised in Azure.

Added a dead-letter depth metric alert in the existing monitoring templates so operations would be notified when a subscription stopped completing. The alert was not deployed or fired.

What worked

Metric alert shape and workspace outputs were clear enough to attach a dead-letter signal without a live check.

Declared an action group routed to a support rota plus metric alerts for dead-lettered messages and sustained backlog, and a scheduled log query rule to catch relay failures from application traces. Authored as infrastructure only; never deployed or fired, so none of its behaviour was observed.

What worked

Broker-side metrics for dead letters and active message counts were available as first-class alert targets, so the two most important conditions needed no custom instrumentation. Routing everything through a single action group kept the alert definitions short.

What got in the way

The most meaningful backlog signal for this design lives in the application database, not in broker metrics, so it could not be expressed as a metric alert at all and needed a log query rule instead. That means two different alert mechanisms, with different authoring shapes and different evaluation semantics, for what is conceptually one dashboard.

Added infrastructure definitions for backlog, dead-letter, outbox-age, and ERP-uncertainty alerting. Compilation succeeded, but live metrics and notification delivery were not exercised.

What worked

The metric-alert model covered the operational failure modes identified for durable asynchronous processing.

What got in the way

Deployment remained dependent on environment-specific action-group pipeline variables that were not available in the recorded task.