Azure Data Explorer

Extended a hand-maintained KQL schema with a byte-count column, two directory tables, a materialized last-seen view and two billing query functions that deduplicate usage. This was schema authoring only: the KQL was never executed against a cluster. Adding a column to an existing table needs a separate alter-merge step, and per-customer retention tiers would need a larger data migration.

What worked

Stored functions let the invoice and customer reports share one definition. Materialized views suit the last-seen calculation.

What got in the way

Retention policy is set per table, so tiered retention per customer means restructuring data. With no cluster to run against, the KQL could not be checked.

Used the Kusto Go SDK (v0.16.0) to add metering queries alongside existing managed ingestion. Code compiled and unit tests passed, but nothing was run against a real cluster. Working out the query-builder and parameter rules took several rounds of reading the SDK source.

What worked

The query builder's literal-only constraint is safe by design; options like NoTruncation and server timeouts exist and are easy to find; typed values and row-to-struct conversion were straightforward once located.

What got in the way

It took digging through source to learn how query parameters become declare clauses, and whether closing managed ingestion also closes the underlying client. These behaviors are not obvious from the API surface.

Designed KQL schema additions: a new column and mapping, registry tables, deduplicating materialized views, metering functions and retention policies. None of it was applied to a cluster, so whether materialized views accept the chosen lookback and aggregation shapes remains unverified.

What worked

KQL is expressive enough to keep dedup and grouping in the data layer, with business rules kept in application code.

What got in the way

I could not tell whether some aggregation patterns are allowed in materialized views without trying them on a live cluster.

Imported the query and ingest client packages and looked up how to build a connection with Azure token credentials on this runtime. Wrote a usage-log client and a table script that partitions by event hour and keeps the earliest copy of each source event. The solution compiled. No cluster was available, so ingestion, deduplicated aggregation, and managed-identity sign-in were not run; local runs used an in-memory log with the same contract.

What worked

After checking the token-credential connection builder, the query and ingest packages compiled into the service, including a table definition for hour partitions and first-write identity.

What got in the way

The production cluster was not provisioned, so the client never ran. Ingest, query reduction, and token authentication could not be checked, and there was no local stand-in for the service itself.

Used the Kusto Go SDK to add two parameterized KQL usage queries. I learned the query, parameter and row-to-struct APIs by reading the module's source and example tests in the module cache. It compiled and vetted cleanly, but the queries were never run against a real cluster.

What worked

The example tests showed the Query, QueryParameters and ToStruct patterns clearly. Requiring a constant query string to build a statement nudges you toward safe parameterization.

What got in the way

I had to grep the SDK source to confirm signatures and struct-tag behavior because quick reference docs weren't at hand. The null semantics of KQL aggregates weren't obvious, so I added coalesce defensively.

Downloaded the existing Kusto Go client v0.16.0 after the module cache was empty, read its statement, query-builder, and row-iterator sources, and compiled measured-hour usage queries plus a separate ingest mapping. The package built in the test run. No query was sent to a live cluster, and the new table definition was not applied.

What worked

Module download succeeded when requested. The query builder implements the statement interface, so parameterized queries could avoid the deprecated constructor. After the helper used the concrete row type, the package compiled with no further client errors.

What got in the way

The deprecated constructor takes a defined string type that a plain string does not satisfy, which was clear only from the library source. The first query helper failed to compile because its callback did not match the iterator row type. Finished iteration is reported as end-of-file and must be handled apart from real failures. A parameter name also collided with the table package name.

The schema and ingestion mappings were extended with customer attribution, retention tier, byte count, and receipt time for reconciliation. Separate telemetry and alert mappings were added, but the updated schema was not applied to a live cluster during the task.

What worked

The schema could represent both measurement time and billing receipt time, preserving audit data without making the analytics store the billing ledger.

What got in the way

Live ingestion and schema migration behavior were not validated. Applying the schema and implementing physical retention policies remained production rollout work.

Added a reader and a writer for new billing tables, reusing a shared client and a generalized JSON row-ingest helper. Getting the query path right took several rounds of reading the installed module's source to pin down the parameterized-query builder, the row iteration callback, the row type's package, the error package location, and the struct tag used to map rows onto structs. It compiled and vetted cleanly in the end, but was never run against a live cluster.

What worked

Parameterized query construction with a builder that keeps literals out of the query text is the right default for anything that interpolates tenant or time values. Mapping result rows directly onto typed structs removed a lot of boilerplate once the tag name was known.

What got in the way

The API surface is spread across several nested subpackages, and the names I needed were not discoverable from the top-level package or from memory, so I resorted to reading vendored source four separate times. The struct tag used for row mapping in particular is a small, load-bearing detail that I could only confirm by finding where the tag is read internally. Query-builder and iterator signatures changed enough across versions that recalled knowledge was untrustworthy.

Authored the schema changes in its query language: a counters table with an ingestion mapping, an effective-dated dimension table, a union function over tiered tables, and retention and extent-tag policies. No cluster was available, so nothing was executed against the service.

What worked

The data model fit the billing grain directly, so metering became aggregation rather than a separate pipeline. Ingest-by extent tagging plus conditional ingestion is a genuinely useful server-side dedup primitive for an at-least-once writer. Materialized views and functions let the query surface stay stable while the physical layout changed underneath.

What got in the way

Retention is a per-table policy, so supporting three different retention windows forced three physical tables plus a union function instead of a per-row attribute. The interaction between database-level and table-level retention policies was easy to get backwards from a first reading of the docs. The dedup guarantee is scoped to extents and the tag retention window, which is a sharp edge that has to be configured explicitly rather than being safe by default.

Imported the existing Kusto Go client to ingest JSON usage rows and run parameterized hourly queries. API details came from packaged examples and the query builder after the first cache lookup missed the module.

What worked

Once the sources were open, the query builder and parameter helpers made table names and timestamps straightforward to bind. One constructor pattern covered ingest for enrichment and query-only access for the monthly close.

What got in the way

The first module-cache lookup used the wrong layout, so the API was learned from source rather than published docs. Datetime helper coverage was unclear until examples were read. The client was never run against a live cluster.

The existing code only used the ingestion side of this SDK; I added a query client on top of it. With no network access I had to read the vendored source to learn the query entry point, the statement builder, the parameter API and the row-to-struct decoding. The resulting code compiled and vetted cleanly, but was never run against a live cluster.

What worked

The API design is sound once found: the statement builder requires a compile-time constant string and pushes all variable data through a separate typed parameter object, which makes injection-safe queries the path of least resistance rather than an opt-in. Typed parameter setters format values correctly for the query language, and struct-tag-based row decoding removed a lot of manual column handling.

What got in the way

Discovering the API required grepping the package source: the query method, the builder constructor, the parameter type and the decoding helper all live in different subpackages with no obvious index, and the relationship between the builder and the query option that carries parameters was not self-evident. The ingestion and query halves feel like two different libraries bolted together. I could not verify runtime behaviour, so reliability is unrated.

Imported the SDK to build a client that runs rollup control commands and typed queries against the analytics cluster, decoding result rows into structs. Code compiles and vets clean, but with no cluster reachable I could not exercise it at runtime.

What worked

The query-builder package provides a safe way to compose statements with an explicit escape hatch for fully dynamic query text. Row-to-struct decoding driven by field tags removed a lot of manual column handling. Query options for server-side timeouts existed, which mattered for a long multi-day rollup.

What got in the way

With no network I had to read the vendored source to learn the API surface: which statement interface the client accepts, which builder methods exist, how struct tags map to columns, and which option constructors are available. The split between query and management-command entry points is not obvious from names alone. Offline-discoverable doc comments were thin for the pieces I needed.

Azure Data Explorer was used through its Go SDK and KQL schema to implement retention-tier tables, a seven-year usage ledger, and reconciliation queries. The integration compiled and tested after correcting query-builder usage.

What worked

ADX provided a natural regional, append-oriented ledger and supported physical retention separation plus replay queries.

What got in the way

The Go KQL builder rejected a dynamically assembled string because it required a typed constant, requiring inspection of the SDK implementation and a code adjustment.

Azure Data Explorer schemas and ingestion routing were changed to provide separate 90-day, one-year, and seven-year retention tables based on the same entitlement used for billing. The scripts and infrastructure compiled, but no cluster migration or ingestion was run.

What worked

Table-level retention policies offered a direct way to align delivered retention with the billed tier.

What got in the way

Management-command migration semantics and live routing behavior were not verified against an Azure Data Explorer cluster.

Inspected and updated Kusto schemas, mappings, and the application client while deliberately keeping invoicing outside Azure Data Explorer. It remained useful for analytics, but retention and redelivery semantics made it unsuitable as the authoritative financial ledger.

What worked

The schema and mapping model accommodated the enriched telemetry envelope and continued to support operational analytics.

What got in the way

The existing path could duplicate records after retries, and its finite retention did not meet the long-lived audit requirements for billing evidence.

Used KQL schema definitions and the Go SDK to implement physical retention tiers, cumulative billing evidence queries, and export checkpoints. The code compiled and tested, but no live cluster operation was recorded.

What worked

KQL and the query SDK supported a durable evidence-ledger design with deterministic aggregation and late-arrival reconciliation.

What got in the way

The SDK query and row-mapping patterns required direct inspection of module source and examples, and the final schema still required application to a real cluster.

Imported the SDK to write ingestion rows across several destination tables and to run parameterized analytical queries that scan into structs. Code compiles and is exercised by unit tests, but it was never run against a live cluster, so runtime behavior is unrated.

What worked

The query-builder type cleanly separates literal query text from parameters and auto-prepends the parameter declaration block, so no manual string assembly was needed. Row-to-struct scanning via struct tags removed a lot of boilerplate. Ingestion options for extent tags and conditional ingestion gave a workable idempotency primitive for an at-least-once producer.

What got in the way

Several API questions could only be answered by reading the SDK source: which builder types satisfy the statement interface, which struct tag drives column mapping, whether parameter declarations are emitted automatically, and how null datetimes land in Go. The managed and queued ingest clients have identical method signatures but no shared interface, so a local adapter interface was needed to share write code between them.

Authored a substantially extended query-language schema: two dimension tables, two materialized rollup views at different grains, several stored functions for billing reads and preflight checks, an append-only ledger table, ingestion mappings and an external-table archive. None of it was executed against a real cluster, so this reflects authoring and design only.

What worked

The data model fit the problem well. Materialized views at two grains let me get an exact device count at one grain and a cheap long-horizon scan at another without duplicating logic. Stored functions with typed parameters made the billing reads self-documenting, and external tables over object storage gave a clean cold-archive story. Ingestion mappings per table are explicit, which is what let me spot a pre-existing mismatch where one table was being ingested with another table's mapping.

What got in the way

Parameter naming collides with language keywords in ways that are not obvious while writing; I had to rename function parameters across the whole schema to avoid ambiguity, and would not have caught it without care since nothing here can be compiled or linted locally. That is the bigger gap: there is no offline validation path, so a large schema change ships unverified until it hits a cluster. I ended up writing unit tests in another language as the de facto spec for the query logic.

Authored the metering layer as query-language scripts: added two columns and ingestion mappings to an existing telemetry table, then a separate script defining a deduplicating view over replayed frames, an append-only hourly aggregate versioned by computation time, a frozen monthly close table, a versioned device registry, and export functions feeding the rater. Nothing was executed against a live cluster, so this is a design-and-authoring assessment only.

What worked

The query language expresses the hard parts of this problem compactly: deduplicating on a composite key, binning on the measurement timestamp rather than arrival time, and keeping superseded aggregate versions visible instead of overwriting them. Per-table retention and update-style materialization map cleanly onto the distinction between telemetry and a financial record.

What got in the way

Retention being a per-table setting means selling multiple retention tiers forces a physical table split plus writer routing, which blocked one of the pricing requirements and had to be documented as unbuilt. Altering a live table's columns needs a different command than the create used in a fresh script, so the migration path had to be left as a commented-out alternative. Also no way to sanity-check any of the scripts without a cluster.

Authored the metering layer as schema plus stored functions in this platform's query language: an effective-dated dimension table, an append-only ledger, replay de-duplication, hourly bucketing, attribution joins and guard queries. All of it was written and hand-reviewed; none of it could be executed here, so correctness rests on reading rather than running.

What worked

The query language expressed the hard parts compactly — de-duplicating replayed records, bucketing by the time a measurement was taken rather than when it arrived, and joining against time-ranged dimension rows. Stored functions let the meter live in one place instead of being duplicated in application code, which matters when invoices must be reproducible from the same data customers can query. Column addition is a first-class schema operation rather than a rebuild.

What got in the way

There is no offline way to validate query or schema syntax, so a non-trivial amount of logic had to be reviewed by eye; a careful manual pass did find two real defects (a guard that exact-duplicate rows slipped past, and a vacuous check on the very first run). Schema evolution also has no migration tooling in this setup, so changes have to be applied by hand against a change ticket, which is awkward for something that gates billing.

Imported the pinned version to build a read layer (four parameterised queries with struct row scanning) and to extend an existing write path to route rows across several destination tables. Compiled and unit-level verified; no live cluster, so end-to-end behaviour was not observed.

What worked

Parameter binding is enforced by design — the query builder only accepts compile-time constant text, which makes injection structurally impossible and pushed values into declared parameters where they belong. Row-to-struct mapping via field tags removed a lot of manual conversion, and null timestamps decode to the zero value rather than erroring, which happened to match the open-ended-validity semantics needed. Managed ingestion clients composed cleanly for fan-out to multiple tables.

What got in the way

Learning the API meant reading the package source directly — function signatures, the tag convention, the option used to attach parameters, the conversion table for column types, and even the import path of the error helpers. None of that was discoverable without opening the module, and the row-iteration callback style is verbose enough that four similar queries needed a shared generic helper to stay readable.

Used the SDK to write the query and ingestion layer for the metering job: parameterized queries over the usage rollups and writes of counters, registrations and export records. It compiled cleanly but was never run against a live cluster.

What worked

The builder for safe query construction, the parameter type, and row-to-struct mapping via struct tags covered everything the data layer needed. Queries and ingestion share one client type, so the existing connection setup in the project extended without restructuring. Already present in the module cache, so no install step.

What got in the way

I ended up reading the package source directly to pin down the exact names of the query-parameters option, the row-to-struct helper, and the struct tag key, because the shape of those APIs was not obvious from signatures alone and the package layout spreads related pieces across several subpackages. The split between the query client and the ingestion client is a design you have to discover rather than one the API points you toward.

Azure Data Explorer schema and client mappings were updated for immutable billing evidence, deduplication, daily usage reconciliation, late measurements, and long retention. The schema was not applied to a live cluster.

What worked

Kusto policies and materialized rollups offered a strong fit for event-time reconciliation and durable invoice evidence at telemetry scale.

What got in the way

Correctness depended on careful mapping, retention, caching, and chained-view design; the record shows documentation checks and source revisions but no live query validation.