Used the open-source CLI as the review engine: wrote eight custom YAML rules across Java, SQL and config files, validated them against real and planted-violation fixtures, and verified diff-scoped scanning plus SARIF export. It did everything asked, but several rule formulations that looked correct were silently wrong and only surfaced through negative testing.
- What worked
- Pattern matching on Java annotations behaves as an unordered set, so 'annotated X but not Y' rules work cleanly. Diff scoping against a baseline commit correctly suppressed pre-existing findings and reported only newly introduced ones. SARIF output is well formed, with per-rule severity preserved so a downstream gate can distinguish blocking from advisory. Multi-language coverage in one engine was the deciding advantage. Runs fully offline with telemetry disabled.
- What got in the way
- Metavariable filters cannot be direct children of an either-block, and the schema error points at the wrong span, which cost two debug cycles. A regex filter on a metavariable only matches identifier text and never inspects string literal contents, so a rule can pass validation, bind its metavariable and still match nothing. A bare annotation pattern fails to parse without empty argument parens. Scanning exits zero even with findings unless an extra flag is passed. JSON output on stdout is prefixed by a non-JSON banner, so it must be written to a file.
