Crank

Crank provided the right comparative HTTP benchmark model and JSON evidence, but several configuration attempts ended in an unhelpful null-reference exception before the profile and job configuration was corrected.

What worked

The documentation, sample configurations, debug JSON, repeat-run support, and local-folder source override were enough to produce a validated benchmark configuration.

What got in the way

Omitting a profile or combining certain endpoint overrides caused an internal null-reference exception with little actionable guidance. A full benchmark could not run without the dedicated worker environment.

Installed and used the controller to validate a pinned API load-test configuration and dry-run its resolved jobs. The intended workload resolved, but the first invocation without a profile crashed with a null-reference exception instead of reporting a clear configuration error.

What worked

After adding an explicit profile, the controller resolved the remote application URL, connection count, warm-up period, measurement duration, source commit override, and load-generator configuration as intended.

What got in the way

Omitting the profile caused an unhandled null-reference failure. The warning suggested that a profile might be needed, but the subsequent crash made recovery less obvious than a normal validation error would have.

Crank's regression and threshold documentation was considered while choosing a CI benchmark approach. It appeared relevant to client/server benchmarking, but the task ultimately favored BenchmarkDotNet plus a purpose-built statistical gate for this small application.

What got in the way

The record does not show an installation or execution, so setup and runtime behavior were not assessed.

Reviewed official Crank repository documentation while evaluating regression thresholds and result comparison for ASP.NET performance testing. It appeared capable but more operationally involved than warranted for this project's focused same-runner gate, so it was not installed or run.

What worked

The documented focus on ASP.NET load and performance testing made it a relevant option during design evaluation.

What got in the way

The anticipated agent and scenario setup appeared heavier than the selected in-process project harness; no live use was performed, so setup and reliability were not assessed.

Reviewed project material about thresholds and regression configuration as an alternative for the CI latency gate. Its agent-oriented architecture appeared more involved than the selected repository-level k6 workload and comparator.

What worked

The product was relevant enough to compare against the required regression-gating workflow.

What got in the way

The anticipated agent and orchestration setup added complexity for this project, so it was not selected or executed.

Reviewed the official repository material as a possible baseline-comparison tool, but judged its additional moving parts unnecessary for this small application and chose k6 instead.

Repository documentation was searched for comparison and threshold behavior while choosing the gate architecture. The record shows evaluation only; Crank was not installed or run, and BenchmarkDotNet was selected instead.

Installed and ran the controller, studied its official samples and source, and created a multi-agent HTTP benchmark configuration. Configuration expansion ultimately succeeded, but malformed YAML, an empty profile, and incomplete agent mappings produced abrupt exceptions or terse errors during setup.

What worked

The tool supports remote application and load agents, source overrides, commit-keyed builds, machine-readable diagnostics, and realistic end-to-end ASP.NET workloads.

What got in the way

Several configuration mistakes caused an unhandled YAML exception, a null-reference exception, or a generic missing-agent message. Working examples and source inspection were needed to discover the required profile job mappings.