PHPBench

Used as the selected in-repository benchmarking runner for one user-visible rendering path. Installed as a development dependency, configured a stable local run with bootstrap and fixed fixtures, established a reproducible baseline, and verified the gate accepted an unchanged control and rejected an intentional slowdown while preserving evidence for review.

What worked

Mode-based timing over repeated runs was stable enough for a noise-tolerant threshold on shared runners. No external service or extra infrastructure was needed, and results plus verdict artifacts were easy to preserve.

What got in the way

Runner configuration and result schema needed extra discovery to get bootstrap, paths, and statistics parsing right.

Installed PHPBench 1.7.0 and used the CLI plus PHP attributes to time a user-visible request against a same-run probe. Regression, configuration, and assertion docs were readable, but XML field names, the KDE mode definition, executor lifecycle, and retry controls only became clear from the library source. Local runs were stable enough to commit a baseline; unchanged runs stayed within a few percent and injected slowdowns failed a 30% gate.

What worked

The 1.4 constraint resolved to 1.7.0 and the first real local-executor run completed. Class-level iterations, revolutions, warmup, and before-methods applied to each subject. The XML dump exposed mode and relative standard deviation, and a progress logger named none was accepted. Spread was about 2% on the main path and under 5% on the probe, so a noise margin could be set from observed samples rather than guesswork.

What got in the way

There is no public attribute to cap retries, so a tight retry threshold can loop. Built-in assertions and environment baselines did not cover a committed ratio of two subjects with a fixed tolerance, so a separate script parsed the XML. The install also pulled an abandoned annotations package that stayed unused.

Added PHPBench as a dev dependency to benchmark a logged-in dashboard request in-process and compare it against a merge-base baseline. I used tag/ref baselines, expression assertions with a tolerance, XML dumps as evidence and a custom report. It ran in about 2.5 s with roughly 1% spread when the machine was idle. Gating on the minimum time accepted 6/6 noisy control runs and rejected 4/4 realistic regressions.

What worked

The tag/ref baseline comparison and the assertion language (min(), a +/- percent tolerance) handled the gate cleanly. Separate exit codes for a failed assertion and a runtime error let the script tell a regression from a broken harness. Its remote reflection reads attributes without needing PHPBench classes in the target tree, so one binary could benchmark both checkouts. Results were very stable.

What got in the way

I guessed a report option that doesn't exist in this version and had to switch to the built-in aggregate report. Tolerance syntax and how 'tolerated' results affect the exit status were only clear after reading the source. With progress output turned off, the assertion failure message never reached the saved output, so I had to switch to plain progress.

Installed PHPBench 1.7.0 to time an authenticated page against a same-run health check, then applied a separate ratio limit because stored baselines are absolute wall time. Guides covered assertions and CI, while file output, retry caps, and time units became clear only from the schema and runner source. After longer samples, local repeats stayed close to the median and an intentional slowdown failed the gate.

What worked

The pinned 1.7.0 release ran two subjects in one invocation and surfaced a subject fatal together with the underlying PHP exception. With progress on the error stream, a machine-readable report could be taken from standard output. Once each sample did more work, the mode was stable enough to compare the page with a health-check ruler in the same run.

What got in the way

Documented baselines store absolute duration, so they cannot travel across shared runners. No attribute caps retries, and an open-ended reject-and-retry loop can hang when noise stays outside the threshold. JSON defaults to standard output, a startup banner and progress share the streams, one progress line garbled a subject name, and figures under a nanosecond column name were on a microsecond scale. Attribute classes also sat on a shorter path than the namespace suggested. Built-in assertions went unused.

Installed a pinned release and built a local regression gate using stored baselines, repeated measurements and assertions. The unchanged control passed, and an intentionally slowed candidate caused a blocking failure.

What worked

Baseline comparisons, configurable warmups and iterations, statistical assertions and XML evidence supported a reviewable gate without an external service.

What got in the way

Implementation required inspecting package source to understand expressions and execution details. A tag containing hyphens was rejected; switching to underscores resolved it. Additional reporting checks were implemented to reject missing evidence.

Evaluated it as the obvious off-the-shelf engine for a PHP performance gate but did not install it. It offers statistical assertions and baseline storage, but its baselines are stored as UUID-keyed XML rather than something a reviewer can diff in a merge request, it has no notion of counting SQL statements, and it would pull a sizeable dependency tree into a project with a strict third-party policy. Those gaps, not quality, led me to a small in-repo harness instead.

What worked

Mature statistical model and retry/assertion features that a hand-rolled harness has to reinvent.

What got in the way

Baseline format is not diff-reviewable, there is no built-in query-count metric, and the dependency footprint is heavy for a locked-down project.

Installed phpbench 1.x as a dev dependency and built a benchmark that drives a real HTTP request through the framework kernel, then used a dumped XML suite from a reference build as the baseline (--file) with a tolerance assertion on the mode, a custom side-by-side report and an HTML output. Per-iteration subprocess isolation, warm-up, retry threshold and the +/- tolerance syntax all worked exactly as intended; A/A noise stayed under 10% and a real slowdown was rejected clearly. Getting there required reading the library source rather than docs for several details.

What worked

Baseline comparison via a dumped suite file needs no storage setup. Expression-language assertions with percentage tolerance are concise and produce a clear tolerated/failed marker. Reports are fully configurable (columns, include_baseline, HTML output path overridable per run) and the JSON schema made it easy to check config keys. Environment samplers recorded in the XML dump are useful evidence. Measurements were stable run after run.

What got in the way

The package ships no offline docs; I had to grep the source to learn how --file works for run, the tolerance operator semantics, report generator names and attribute class names. The default aggregate report shows null min/max columns and a misleading memory-peak diff when a baseline is present (first vs max). The assertion-failure exit code (2) is not obviously documented and collided with my own sentinel. --tag silently implies storing results into a directory at the repo root until the storage path is reconfigured.

Installed PHPBench, read its storage and benchmark docs, and used it as the blocking CI gate on an authenticated dashboard request. Official docs were not enough: executor selection, before-class lifecycle, and autoloading required reading vendor source and several failed runs before a baseline, control pass, and slowdown reject all worked.

What worked

After setup, tagged XML storage, an absolute time assertion, and report dumps were enough to keep a reproducible baseline, accept an unchanged control, and fail an injected request-path delay with a non-zero exit. Iteration noise on the control was low.

What got in the way

Config and class attributes did not reliably select the local executor, so before-class setup first ran remotely and broke. Lifecycle methods had to be static. The benchmark class was invisible until autoload-dev was added. A relative threshold would have been too tight for shared CI, and a slowdown run took much longer than the measured work because of retries.

Used PHPBench to store a control result, compare a deliberately slowed candidate, enforce a percentage assertion, and generate benchmark output. The final proof accepted the control and rejected the slowdown with a nonzero exit code.

What worked

Stored baselines, comparison assertions, process isolation, and machine-readable output supported the blocking gate without a custom timing engine.

What got in the way

Setup required several corrections: tags reject hyphens, warmup cannot be zero, report names depend on the loaded configuration, and paths become relative to the selected working directory unless made absolute.

PHPBench supplied stored baselines, comparative assertions, warmups, iterations, reports, and meaningful exit codes. The unchanged control passed and an injected slowdown failed with exit code 2, proving the blocking contract.

What worked

Baseline storage and the relative assertion accurately distinguished a stable control from a roughly fivefold slowdown, with a clear numerical failure report.

What got in the way

Configuration discovery took trial and error: an inline value was mistaken for a config filename, tags rejected hyphens, and a database failure surfaced through a verbose remote-executor error.

Installed and configured repeated benchmark runs, generated machine-readable results, and used those results to prove that an unchanged control passed while an intentional delay failed the gate.

What worked

The runner, attributes, dump output, debug executor, and repeated probe measurements supplied the data needed for a deterministic blocking comparator.

What got in the way

Expected example configuration paths were absent, so command help and installed source code were needed to clarify attributes and retry behavior.

Installed it as a dev dependency to benchmark one HTTP request path of a PHP web app and drive a blocking CI check. Wrote a benchmark class with attribute-based warmup/revs/iterations plus a retry-until-stable deviation threshold, pointed it at a custom bootstrap, and ran it repeatedly to characterise noise. Runs finished in about two seconds with roughly one percent relative standard deviation, which was tight enough to build a ratio-based gate on top of.

What worked

The measurement science is built in: warmup, revolutions vs iterations, robust aggregate statistics and an automatic retry when dispersion is too wide. Attribute-based configuration on the benchmark class is readable, a custom bootstrap file is easy to wire through the JSON config, and the autoloading of a separate benchmark namespace worked without fuss. Results were highly repeatable across ten orchestrated runs.

What got in the way

Getting machine-readable results into my own orchestrator meant dumping raw XML and parsing per-iteration attributes by hand; a simple JSON summary report would have saved a step. I had to scan the run command's help output to discover which of the baseline/store/tag/assert options actually fit a same-job A/B comparison rather than a committed baseline.

Created a warmed application benchmark, stored a baseline, compared revisions with a 15% assertion, and verified both an unchanged pass and a deliberate slowdown failure. It ultimately provided the required blocking exit behavior and useful aggregate reports.

What worked

Stored references, statistical assertions, nonzero failure exits, and aggregate timing reports directly supported the regression gate. The intentional slowdown was detected decisively.

What got in the way

Tag names rejected hyphens, and checkout/bootstrap isolation was not obvious. Several early runs failed through indirect remote-executor messages when the database driver, fixture assertion, or Composer autoloader setup was wrong.

Used it as the measurement engine for a latency gate on one authenticated page: a bench class with warmup/revs/iterations attributes booting a real application kernel, plus an XML result dump that a wrapper script parsed into a normalised ratio. Measurement quality was good — within-run relative standard deviation stayed at 1-2% once the harness was correct, and it caught an intentionally injected slowdown at roughly +21% while passing unchanged code.

What worked

Attribute-based configuration of warmup, revisions, iterations and retry threshold is concise and expressive. Machine-readable XML dump with per-iteration timings made it easy to compute my own medians instead of fighting the built-in reports. Startup cost was low — a full suite ran in a couple of seconds, cheap enough to repeat several times per gate invocation.

What got in the way

Config keys were hard to discover from docs; I ended up reading the bundled JSON schema shipped in the package to find valid options. One run produced no output at all and a non-zero exit until I re-ran at high verbosity to see the real error — a silent failure mode. The runner executes all iterations of one subject and then all of the next, so a normalisation ratio between two subjects compares non-adjacent time windows; on a contended machine that produced a false regression and I had to build pairing into a wrapper script.

PHPBench provided stored comparisons, percentage assertions, XML evidence, and a nonzero exit that made the regression check blocking. Setup required debugging tag restrictions and an expression API mismatch with the documentation consulted.

What worked

The final gate accepted an unchanged control and rejected a deliberately slowed candidate with exit code 2. Its CLI assertions directly supplied the required CI failure behavior without an advisory wrapper.

What got in the way

A hyphenated tag was rejected, and an initial assertion expression failed because the result structure did not contain the documented-looking mode key. Local package source inspection was needed to find the compatible expression.

PHPBench provided isolated child-process execution, stored reference runs, aggregate reports, retry thresholds, and regression assertions. The controlled proof accepted an unchanged workload and rejected a deliberate slowdown with a nonzero exit.

What worked

Its reference comparison and assertion exit status directly supported a blocking same-runner gate, and the command help and installed source clarified configuration details.

What got in the way

Tag syntax rejected a hyphenated value with only a terse parse error; switching to an accepted tag resolved it. The full database workload remained untested locally because of missing PHP database drivers.

PHPBench measured a representative controller-and-template path, enforced a statistical latency assertion, emitted reviewable XML and HTML evidence, accepted the control, and rejected an injected slowdown with a nonzero exit.

What worked

Attributes, configurable iterations and warmup, assertions, report generation, and process exit status provided everything needed for a stable blocking CI gate.

What got in the way

Initial setup failed because a class-level hook had to be static, then because framework user refresh required an identifier. Both errors required inspecting package source and revising the fixture setup.

Added PHPBench and used it to benchmark the authenticated dashboard for ten iterations, dump machine-readable results, compare revisions, and demonstrate both a passing control and a rejected slowdown.

What worked

It produced stable measurements and XML dumps; the unchanged control passed and the intentional slowdown was clearly detected with a non-zero gate result.

What got in the way

The first run failed because generated tag names contained hyphens, while PHPBench accepts only alphanumeric characters, periods, and underscores. The runner was corrected and subsequent runs succeeded.

Added PHPBench and used stored references, assertions, warmups, repetitions, retry thresholds, and reports to implement the comparator. Synthetic checks ultimately proved an unchanged control passes and an intentional delay fails, though reference storage and bootstrap setup took several attempts.

What worked

Its baseline assertions produced a nonzero exit for the intentional slowdown and accepted the unchanged control, which maps cleanly to a blocking CI gate.

What got in the way

Initial runs failed because the database driver was unavailable, and early synthetic attempts produced a confusing missing-reference error until storage and bootstrap usage were corrected.

PHPBench measured an isolated, warmed template-rendering workload, emitted XML evidence, enforced a latency assertion, accepted six unchanged controls, and rejected an intentional slowdown with a nonzero exit.

What worked

Once configured, repeated measurements were exceptionally stable, and the assertion and exit status provided exactly the blocking CI behavior required.

What got in the way

An initial custom report referenced a generator that was not registered. Configuration and bundled reference material required exploration before a valid minimal setup was found.

Used it as the measurement engine for a blocking latency gate on one web request path: process-per-iteration runs with warmup, a fixed fixture, stored runs tagged per side, and an A/B comparison of a baseline checkout against the current one. Measurements were impressively stable (around 1% run-to-run spread on the mode), which is exactly what made a tolerant threshold defensible, and the regression case was separated from the control by a wide margin.

What worked

Process isolation per iteration, warmup support, and the KDE mode statistic gave a low-noise signal with very little tuning. Storing runs under a tag and re-reporting a comparison between two tagged suites fit the A/B design naturally. Small dependency footprint with permissive licenses, and attribute-based benchmark definitions are read by name only, so a benchmark file can be executed against a checkout that doesn't have the tool installed.

What got in the way

I had to read the vendor source repeatedly to answer basic questions: which report generators and output renderers are registered, what the valid configuration keys are, and how to pass environment variables to child processes. The retry-on-variance setting looked safe but the retry limit is never applied anywhere in the code, so the loop is effectively unbounded — a hang risk on a noisy shared runner, and I dropped the feature because of it. The bundled HTML output pulls CSS from a public CDN, which makes it useless as an artifact in a network-restricted environment.

PHPBench 1.7 supplied stored baselines, comparison reports, retry thresholds, warmups, and failure assertions needed for the CI gate. Debug runs validated the final baseline expression, but finding the correct metric syntax required inspecting package source after an initially plausible expression failed.

What worked

The debug executor allowed the benchmark configuration and stored-baseline assertion flow to be exercised without a live database. The final relative regression assertion ran successfully.

What got in the way

A zero warmup was rejected, and the first assertion used an invalid metric path that produced a low-level missing-key error. The report configuration also emitted a deprecation warning for an option scheduled for removal in version 2.0.

Installed it as a dev dependency and built a micro-benchmark around one authenticated page-render path, then drove it from a shell wrapper that measures a base revision and the current revision on the same machine and asserts on the delta. Measurements were remarkably stable: ~2.6% worst-case spread across a dozen repeated comparisons on unchanged code, and barely moved under heavy CPU contention. It cleanly rejected an intentional ~10% slowdown.

What worked

Attribute-based benchmark declaration is concise. The built-in retry/variance logic re-runs iterations until relative standard deviation converges, which did most of the stability work for free. Tagged runs plus reference comparison gave a straightforward baseline workflow, and the on-disk result store is a plain nested XML tree that is trivial to copy between two checkouts. Reports are readable enough to parse from a script and to keep as CI evidence. Runs were fast (a couple of seconds for a full configured run).

What got in the way

The biggest trap: an assertion that compares against a baseline passes trivially when no reference is supplied, because the variant is compared to itself. A gate built naively on this is a silent no-op, so the wrapper has to fail hard when the baseline run is absent. Documentation for the assertion expression language, the tolerance syntax and the available statistical functions was thin; I ended up reading library source to confirm syntax and the result parameter keys. The JSON config also rejects unknown keys, so comment-style keys break the run.