BenchmarkDotNet

Added BenchmarkDotNet to a new console project, wrote benchmarks with a baseline method, built a ManualConfig with a memory diagnoser, JSON exporter, custom artifacts path and validators, and consumed the Summary programmatically to compare time ratios and allocated bytes against a committed budget. Measurements were stable (about 1% spread across three runs) and allocation counts were exact, which made a blocking gate feasible. A full run took roughly a minute locally.

What worked

Statistics, allocation data and baseline flags were all reachable from the Summary object, so the comparison logic could live in C# with no second scripting language. Per-op allocation figures were byte-exact and deterministic, and the JIT optimizations validator correctly refused to run a Debug build once added.

What got in the way

ManualConfig.CreateEmpty() ships with no validators at all, so a Debug build initially ran to completion and only failed by coincidence; the need to add JitOptimizationsValidator explicitly is easy to miss. Namespaces for columns and the JSON exporter were not obvious and caused a compile round-trip. ReturnValueValidator assumes all benchmarks in a class return the same value, which does not fit a baseline-vs-target layout. Culture-sensitive number formatting had to be pinned manually when rendering the report.

Used BenchmarkDotNet to benchmark HTTP endpoints of an ASP.NET Core app in-process, with a memory diagnoser and a custom config (short warmup/iteration counts, in-process toolchain), then read the Summary programmatically to compare medians and allocated bytes against a committed baseline JSON. Measurements were stable (StdDev under 3% locally) and the gate correctly caught an injected N+1 and a CPU stall. Getting the public API right for 0.15.x took trial and error.

What worked

Statistics and allocation numbers were consistent run-to-run, MemoryDiagnoser made allocations a deterministic tripwire, and a programmatic config plus reading the Summary object made it straightforward to build a custom gate without external services. Runs completed in about a minute, acceptable for a PR check.

What got in the way

Several API locations had shifted in the 0.15 line (CPU info moved off HostEnvironmentInfo.CpuInfo, column providers in a different namespace, the Descriptor description not publicly accessible) and I had to resort to inspecting the assembly to find the right members. The display-name API wraps descriptions containing spaces in quotes, which needed stripping. Whether reported statistics are already normalized by OperationsPerInvoke was not obvious and I initially double-divided.

Added BenchmarkDotNet to measure an HTTP endpoint hosted in-process, with the InProcess toolchain, MemoryDiagnoser and JSON/markdown exporters feeding a custom ratio-based gate. Warmup, outlier handling and allocation reporting worked as advertised; allocation counts were stable to a tiny fraction of a percent across runs, which made a second gating signal possible. Configuring the toolchain, validators and exporters programmatically took some care but was well-structured.

What worked

MemoryDiagnoser gave near-deterministic allocation figures; exporters produced review-friendly evidence files; in-process toolchain avoided a separate host process and worked with dotnet run.

What got in the way

Programmatic config requires knowing about the optimizations validator interplay with Debug builds; nothing failed, but it needed deliberate handling.

Added BenchmarkDotNet as the harness for three benchmarks, configured a custom job (iteration counts, memory diagnoser, memory randomization) and the full JSON exporter, then parsed the per-iteration measurements for a statistical comparison. Smoke run, several control runs and slowdown runs all produced usable exports.

What worked

The full JSON export includes raw workload and overhead measurements per iteration, which made an external Mann-Whitney comparison straightforward. Config API for jobs, diagnosers and exporters was expressive; filter and artifacts command-line options worked as expected.

What got in the way

A type I expected under the BenchmarkDotNet namespace had moved to a Perfolizer namespace in this release, causing a compile error on first build. Nanosecond-scale benchmarks showed bimodal medians across process launches (likely alignment effects), which produced a false positive until I enabled memory randomization and batched work to microsecond scale; this is documented behavior but easy to trip over.

Installed 0.14.0 and used it in-process to time a seeded HTTP list path, write a committed median baseline, and fail when the median exceeded a 2x ceiling. It produced the statistics and HTML/JSON/CSV evidence needed for a blocking check, but job attributes, entry-point binding, and attribute parameters took several failed builds and one runtime crash to get right.

What worked

After a single in-process job was configured, Release runs reported median latency and allocations, exporters left review artifacts, and a control run stayed under the ceiling while an injected delay was rejected.

What got in the way

InProcess plus SimpleJob registered two jobs and crashed the gate with a single-element assertion. SimpleJob rejected an explicit unroll factor at compile time. The benchmark entry type collided with the web app entry type. Combining diagnoser attributes with a manual config was easy to get wrong.

Installed 0.14.0 and used an in-process run as the blocking CI measurement. After toolchain and fixture workarounds it produced stable JSON stats, a committed baseline, a passing control, and a failing intentional slowdown. Setup failures were noisy and once dumped core.

What worked

Full JSON export gave mean, median, deviation, and percentiles that a small comparer could gate on. Manual in-process config avoided out-of-process builds. Control means clustered near the baseline; a large delay exceeded the 4x limit.

What got in the way

The emit in-process toolchain invoked the benchmark on a different instance than global setup, so the HTTP client stayed null. Setup failures then crashed inside result logging with a large core dump. A duplicate markdown exporter warning appeared until extra exporters were removed. Static fixtures and the no-emit toolchain were required.

Installed 0.14.0 and used it as an in-process median-latency gate for a list HTTP path, with a committed baseline, a noise-tolerant threshold, and a same-job injected slowdown. Once the measured host stayed up, it produced medians, spread, and review artifacts that the control passed and the slowdown failed.

What worked

Statistical summaries were enough to pick headroom for runner noise and to keep a blocking pass/fail without a separate load-test service. Control and slowdown in one executable fit the existing dotnet CI path.

What got in the way

When the app under test died during setup, the run ended as missing statistics rather than a clear host error, so the first failures were easy to misread as a benchmark-library problem. A few result-export details needed a quick surface check.

Added a focused benchmark with bounded warmups and measurements, JSON export, and allocation reporting. It produced useful stable results, but generated-build overhead was significant and a failed workload could otherwise leave the process with a successful exit.

What worked

Warmup, repeat measurement, JSON output, and memory metrics supported a realistic blocking regression gate.

What got in the way

The initial provider failure produced no useful measurements without reliably failing the process, so the runner needed an explicit failed-report check. Total benchmark execution also included noticeable generated-build overhead.

Installed and ran BenchmarkDotNet 0.15.8 for benchmark diagnostics and reports. Blocked execution and generated useful statistics, but method blocking introduced order bias and some failed runs produced no usable statistics, so the gate decision needed a separate alternating paired sample set.

What worked

Its warmups, repeated measurements, outlier reporting, ratio output, and Markdown artifacts gave useful diagnostics and made intentional slowdowns obvious.

What got in the way

Executing base and candidate methods in blocks let the second process benefit from warm state. Initial endpoint failures yielded NA statistics, and full runs took roughly a minute, so BenchmarkDotNet results alone were not robust enough for the blocking decision.

Reviewed regression and statistical-test guidance while selecting an approach. It informed the comparison, but was not chosen because the target path needed full HTTP, serialization, EF Core, and SQL coverage.

What worked

The documented statistical comparison concepts helped frame threshold and confidence requirements for the custom gate.

What got in the way

For this project, an in-process microbenchmark did not directly provide the desired application-level A/B coverage across two revisions.

BenchmarkDotNet provided benchmark discovery, measurement artifacts, statistics, and memory data for the gate design. A JSON probe ran successfully and confirmed the export shape consumed by the custom comparator, but the real PostgreSQL timing scenarios could not run in the available workspace.

What worked

The benchmark cases compiled, were discovered, and produced machine-readable full JSON with workload measurements and allocation data. The same benchmark source also built against a separate baseline checkout.

What got in the way

End-to-end database timing remained unassessed because the workspace had no usable PostgreSQL environment.

BenchmarkDotNet 0.15.8 provided repeatable median timing, raw measurements, and allocation data for two deterministic premium calculations. Its outputs enabled both acceptance and rejection proofs.

What worked

The library produced stable unchanged-control ratios and clearly detected the final intentional slowdown. Exporters and memory diagnostics supplied the evidence needed by the comparator and CI artifacts.

What got in the way

The exporter namespace was not initially obvious, and launching an absolute project path from another repository caused BenchmarkDotNet to rediscover the wrong project context. Setting the worktree as the working directory resolved this.

Added BenchmarkDotNet to measure six base, candidate, and control HTTP cases and export evidence for a statistical gate. It completed a live HTTP smoke run, but setup required correcting an invalid sealed benchmark class and distinguishing dry-job settings from the configured production job.

What worked

It executed all benchmark cases, emitted detailed timing warnings, exported artifacts, exposed useful command-line help, and supplied measurements that the custom gate could validate conservatively.

What got in the way

The first smoke run failed because benchmark methods were declared in a sealed class. A later dry run produced only one iteration and therefore an intentionally inconclusive gate result; the configuration displayed alongside the error also made the effective job settings initially confusing.

BenchmarkDotNet supplied warmup, isolated launches, adaptive measurements, statistics, allocation diagnostics, and JSON reports for two PostgreSQL-backed scenarios. It ultimately produced sufficiently precise control data for the gate proof.

What worked

Multiple launches and confidence data supported a low-noise paired comparison, and the final runs clearly separated controls from intentional slowdowns.

What got in the way

A sealed benchmark class was silently excluded except for a discovery message, and a failed benchmark produced null statistics that caused the first report reader to fail. Multimodal-distribution warnings also required interpretation.

Used it as the measurement engine for a blocking latency gate on one HTTP read path: allocation tracking first, then raw per-request latency samples feeding a paired A/B comparison between two builds. It produced the samples and allocation numbers needed, and a full six-round comparison ran in about a minute.

What worked

Allocation measurement was exact and reproducible across runs, which made it trivially gateable. Setup and teardown hooks fit an in-process server fixture cleanly, and the generated result files doubled as review evidence without extra work.

What got in the way

Getting raw per-iteration samples rather than only aggregate statistics required a non-obvious configuration combination (single invocation per iteration, unrolling disabled, explicit warmup and iteration counts); this was the least discoverable part of the API. Short default configurations reported error margins larger than the mean, which looks alarming until you realise it is just the sample count. Startup overhead per invocation is noticeable when you invoke it many times in a loop.

Built a two-benchmark harness (a guarded endpoint plus a machine-reference control) and drove the pass/fail gate off the ratio of their medians, read programmatically from the in-process summary objects. It supplied warmup, per-iteration sampling, outlier handling and JSON/Markdown export for free, which is exactly the statistical machinery a hand-rolled timer lacks. Measurements were stable enough to separate a clean control from an intentional ~30% slowdown with clear margin on both sides.

What worked

The programmatic API exposes per-benchmark statistics and reports directly, so computing a derived metric and emitting a custom verdict artifact took no parsing of console output. The built-in validator that refuses to produce results from a non-optimized build is the right default for a gate. Exporters made preserving evidence for reviewers trivial.

What got in the way

Discovering the right types took trial and error: the statistics object is a reference type rather than a struct (so nullable-value handling was wrong on the first try) and the default column providers live in a namespace I had to guess at. Each full run cost roughly 40 seconds, which is fine for CI but made the repeated-sampling work needed to derive a defensible threshold slow. The documentation is oriented toward console reporting rather than toward consuming results in code.

Consulted official material while evaluating statistical comparison options, then selected an HTTP and PostgreSQL approach that better matched the production request path.

BenchmarkDotNet supplied warmups, repeated measurements, confidence information, outlier handling, allocation diagnostics, and machine-readable artifacts. An early near-zero control produced no median, so the harness had to be changed to benchmark meaningful relational work.

What worked

Paired control and candidate measurements produced stable evidence and clearly separated the unchanged case from a deliberate 50 ms regression.

What got in the way

When the initial SQLite control performed effectively no measurable work, the result lacked a usable median and the custom gate threw an exception. Meaningful setup and workload selection were essential.

Installed and ran BenchmarkDotNet to measure optimized baseline and candidate implementations with repeated launches and warmups. It distinguished unchanged controls from a deliberate slowdown and supplied the samples used by the blocking decision.

What worked

The completed run accepted two unchanged comparisons near parity and clearly rejected the injected slowdown, providing strong end-to-end evidence for the gate.

What got in the way

Setup initially exposed framework dependency conflicts, and roughly one-second measured iterations made the six-method proof take long enough to require explicit pipeline timeout planning.

BenchmarkDotNet provided the benchmark configuration, discovery, execution model, and measurement data needed for the proposed gate. Two initially assumed APIs were absent in the installed release, requiring assembly metadata inspection and code changes. Discovery worked, but real database timings could not be run locally.

What worked

The framework supported dedicated benchmark classes, async workloads, setup hooks, memory diagnostics, and a separate executable harness suitable for CI.

What got in the way

The expected JsonExporter symbol and GcStats.BytesAllocatedPerOperation member were unavailable in this version, so the report extraction approach had to be revised.

Built a five-benchmark suite over a CPU-bound calculation path and a JSON serialization path, drove it programmatically so I could read the result summaries in-process, and compared them against a committed baseline file to produce a pass/fail gate. It produced stable, byte-exact allocation numbers that made a workable gate possible on hardware where timings swung widely.

What worked

The attribute model and custom job configuration (warmup, iteration and launch counts, JSON export, custom artifact path) were easy to set up. Running it from code and consuming the summary objects directly avoided parsing exported files. The memory diagnoser was impressively deterministic: identical allocation byte counts across runs, which is exactly what a regression gate needs. It also correctly surfaced an injected allocation regression in the hot path.

What got in the way

Reading a metric out of the result summary by name is a guessing game: the obvious key for allocated bytes silently returned zero while the console table printed real kilobyte values, which would have left the gate quietly dead. I had to inspect the shipped assembly to find the real descriptor identity and then match defensively. Separately, launching the built executable from an unrelated working directory failed partway through because the generated toolchain project could not resolve references, and the failure looked like per-benchmark errors rather than an environment problem.

BenchmarkDotNet was documented, installed, imported, and run to measure an end-to-end service operation with allocation metrics. Its short job and JSON/Markdown exporters supported a conservative CI design where execution failures gate builds but noisy timing changes do not.

What worked

The benchmark completed with warmup and repeated measurements, and generated machine-readable and human-readable artifacts using the intended CI command.

What got in the way

Some command-line exporter and job behavior needed documentation searches and local validation before the exact invocation was trusted.

BenchmarkDotNet measured a production-shaped listing workload, reported latency, GC activity, allocations, statistical error, and outlier removal, and supplied results used to set explicit CI budgets.

What worked

The runner produced stable measurements near 68 ms and 26 MB, detailed diagnostics, and a machine-usable result for enforcing latency and allocation thresholds.

What got in the way

The first run exited as a failure because the provisional 35 ms and 5 MB budgets were much lower than the measured baseline. This was budget calibration rather than a benchmark engine failure.