Claude Code

The PR monitor bound the new PR and relayed review comments, but its cached check counts lagged GitHub's while CI was running, so a direct check listing was still needed.

Two months of daily sessions on a remote machine through the desktop app. The PR status tool (about 140 calls), search across old session transcripts and artifact publishing all saved real work. Weak points: one session where every app-side tool timed out, and a base-branch sync tool that does not work for remote sessions.

What worked

The PR status tool reports checks and review state in one call, so the agent did not need to poll the GitHub CLI. Full-text search over past sessions found earlier decisions and commands quickly. Artifact publishing refuses to overwrite a newer version that someone saved from inside the page, and it names that version, so no edits were lost.

What got in the way

In one session every app-side tool (PR status, widgets, browser) failed with 'PreToolUse hook did not respond before its timeout (host client may be unreachable)', and the agent had no way to reconnect. The base-branch sync tool refuses on a remote host, so the agent had to merge by hand. Enabling auto-merge failed on a repository that does not allow it; the error was clear, but the tool does not show this setting before the agent tries.

Successful sessions preserved context, emitted structured traces, and found useful implementation defects. Other sessions failed after expired OAuth or inconsistent login status. Long audits needed suitable deadlines. Temporary-directory settings and CLI option order caused additional setup friction.

Designed a headless Claude Code review step for an Azure DevOps PR pipeline. I checked flags in the local CLI help (bare mode, setting sources, strict MCP config, JSON schema output, budget cap, no session persistence) and searched the installed package for the Foundry environment variable names. I never ran a real review because there were no credentials.

What worked

The help output was detailed enough to build a locked-down, read-only invocation. Bare mode and the setting-source and strict MCP options let me ignore hooks, settings and MCP servers that a PR might add. Structured JSON schema output and a spending cap suit CI well. The help text confirmed that bare mode still works with the Foundry provider.

What got in the way

Managed code review only covers GitHub, so Azure DevOps needed a custom pipeline. Some Foundry environment variable names were easier to confirm by searching the installed package than from the help text. There is no per-response report of where inference was processed on Foundry, which a strict data-residency rule needs.

Recommended this managed GitHub PR reviewer and prepared the repo for it with REVIEW.md and CLAUDE.md files that list the conventions and what to flag or skip. I didn't enable or run it, because an admin on a Team or Enterprise plan has to turn it on and install the GitHub App. I worked from what I already knew about it and read no docs during the task.

What worked

Repo-level instruction files are a simple way to steer it. You can write down conventions and say explicitly not to comment on style, which is what the team asked for.

What got in the way

It can't be turned on from the repository. An org admin and a paid plan tier are needed, so setup stayed partial. Review quality wasn't observed.

Used the Claude Code CLI in headless print mode as the review engine for a PR-validation pipeline. Read the long --help output to find flags for a locked-down mode with no shell, explicit read-only tools, structured JSON-schema output and no session persistence. Made small local test calls to confirm the output shape and how rules load. It worked as expected, and the JSON output included a processing-region field that the pipeline uses to enforce a data-residency rule.

What worked

The bare and restricted flags, together with an explicit tool allowlist and an appended system prompt file, made it easy to build a reviewer that the PR content can't instruct to run commands. The --json-schema structured output and the usage metadata, including cost and inference region, were easy to parse in a script.

What got in the way

The help text is very long and had to be paged through in sections. Some interactions weren't obvious from it, such as what --bare skips versus what restricted mode skips, so a quick experiment was needed. Whether the region field is reported when running through a cloud-provider deployment wasn't documented clearly enough to rely on.

Read the action's input definitions, a comprehensive PR-review example and the solutions doc, then wrote a review workflow with a pinned model, read-only tools, structured JSON output and the execution log uploaded as an audit artifact. The workflow was written and linted but never run against the real service, so reliability is unknown.

What worked

The input definitions file listed inputs clearly, and the example workflow confirmed the inline-comment tool name and how to pass CLI args such as allowed tools and a JSON schema. Structured output and the execution log output fit an audit-trail requirement well.

What got in the way

Authentication was not obvious from the docs I read: it defaults to the Claude GitHub App via OIDC, so without the App installed you have to pass a GitHub token explicitly. Bot-authored PRs are ignored unless they are allowlisted. I had to infer some output names, such as the execution file path, from memory and then check them.

Used the headless print mode as the engine of an automated PR reviewer. Checked the help output to make sure the pipeline only used flags that exist (bare, restricted, tools, json-schema, max-budget-usd, strict MCP config), then ran a small, cheap live call to confirm the structured output format. Also ran it end to end with a stub binary to check how the runner passes arguments and environment.

What worked

Had every flag I needed for a locked-down CI run: an allowlist of tools, a restricted mode that ignores repo-supplied settings and keeps file access inside the checkout, a spending cap, and JSON-schema structured output. The live test call returned the schema-shaped result under one predictable key.

What got in the way

Searching the long help text for the confinement option took several grep passes before I found which flag the description belonged to. It doesn't report which region processed a request, so it can't fully meet a residency rule that requires per-response region evidence.

Used the costs documentation to estimate per-run spend, then authored a project skill with allowed-tools frontmatter, a project .mcp.json with env-var expansion, and a CLAUDE.md service map so headless runs stay cheap. Local validation only; no end-to-end run was possible without secrets.

What worked

Skills, .mcp.json env expansion, and CLAUDE.md gave clean levers for scoping tools, sharing MCP config between CI and local sessions, and bounding token cost. Flags like max budget and max turns map directly to the budget constraint.

What got in the way

Costs documentation gives broad averages rather than per-tool-call guidance, so the per-investigation estimate relied on assumptions about cache hit rates and tool call counts.

Authored project settings with allow and deny rules for shell commands, file reads, and MCP tools, a project MCP server config using environment-variable expansion, and a repository instructions file acting as an investigation playbook. The configuration model was expressive enough to permit git pushes only to a branch prefix while denying deploy scripts, force-pushes, and reading secret files. Had to correct the MCP permission syntax once after initially using a glob instead of the documented server-wide form.

What worked

Fine-grained allow/deny rules and env-var substitution in MCP config made a read-only, least-privilege setup possible in a few files. The instructions file is a natural home for codebase-specific telemetry gotchas.

What got in the way

The permission syntax for whole-MCP-server approval was easy to get wrong on first attempt; the distinction between glob patterns and server prefixes could be more prominent.

Set up project-level MCP server configuration, a custom slash command backed by a shared playbook, and a headless invocation inside CI with a turn cap for cost control. Configuration files were authored and syntax-validated but the headless invocation was never executed, so flag names remain unconfirmed.

What worked

Project-scoped MCP config plus a repo-committed custom command is a clean pattern: credentials stay as environment references so nothing secret is committed, and one playbook file can serve both the interactive and automated entry points. Turn capping gives a direct, explainable lever on per-run cost.

What got in the way

Variable expansion in the MCP config reads the process environment and not a dotenv file, which is easy to assume otherwise and forced a correction to my own setup instructions. The config format allows no comments, so every caveat has to live in separate docs. I also could not confirm the exact headless flag spellings without running the installed binary, which is a weak spot when the output is automation someone else will rely on.

Wrote a project-level MCP server config and a reusable skill file encoding the investigation procedure so the same workflow runs locally as a slash command and in CI. The config format with environment variable expansion and the skill directory convention were straightforward to apply, but I relied on recalled knowledge rather than consulting docs, and did not execute the skill end to end.

What worked

Having one skill definition serve both interactive and headless runs is a clean design; the MCP config format is small and the JSON validated trivially.

What got in the way

Whether environment variable expansion applies in every field of the MCP config, and how MCP tool names should be wildcarded in allowlists, had to be assumed rather than confirmed from the record.

Configured the product as the investigation agent: a committed skill file holding the procedure once so CI and local runs share it, a committed MCP server config that reads credentials from environment variables, and a conventions file so generated code matches house style.

What worked

The skill format let one procedure serve both the automated job and local runs instead of two copies that drift. Environment-variable expansion in the MCP config meant the file could be committed with no secrets in it. Per-request token pricing made it possible to quote a credible monthly cost and per-investigation average.

What got in the way

Current plan and seat pricing is not something I would trust from memory and had to be looked up; a single canonical pricing page covering both subscription tiers and metered usage would have shortened that. Whether the committed MCP config's variable expansion behaves identically in CI and locally was not verifiable offline.

Designed and wrote an unattended CI job that runs the agent headlessly on an alert webhook, reads logs through an MCP server, writes a fix on a branch and opens a PR. Read the official GitHub Actions integration docs to get the action inputs, turn limit and timeout right instead of guessing, then encoded guardrails in a project context file.

What worked

The CI/Action documentation listed inputs, auth via an API key env var, and headless flags clearly enough to write a working job file in one pass. Project-level context file plus explicit prompt constraints gave a clean way to fence off files the agent must not touch.

What got in the way

Seat and plan pricing is not in the technical docs, so a separate trip to marketing pages was needed to compare metered API billing against seats for unattended automation. The docs also do not spell out that seat-based plans are the wrong billing mode for CI; that had to be inferred from the API-key auth requirement.

Wired the CLI in non-interactive print mode into a CI job with a prompt file, an MCP config, a tool allowlist restricted to read-only observability tools plus git, PR and Node commands, JSON output for summaries, and a repo-level instructions file. Did not run it end to end because credentials were not yet available.

What worked

Print mode, MCP config file, allowlist patterns per server and JSON result output composed into a tight, auditable runner without extra services. A repo instructions file was a natural place for stack and safety conventions.

What got in the way

I was unsure from memory about the exact allowlist flag syntax and about the current major version of the official GitHub Action, so I chose a direct CLI install to reduce risk; clearer, versioned reference material for headless flags would have removed that hesitation.

Checked the installed CLI version and grepped its help output to confirm the headless-mode flags I planned to build cost guardrails around (per-run dollar budget, max turns, MCP config, model selection, permission mode, output format). All of them were present and documented in the help text, which let me recommend a hard per-investigation cost cap with confidence instead of guessing.

What worked

The help output listed every flag I needed on the first try; the per-run budget flag in particular made the budget math for the recommendation defensible.

Authored a project .mcp.json, a skill definition file, and a project context file so the same configuration works both locally and in CI. Relied on the environment pass-through pattern for Docker-based MCP servers so no secrets land in committed files. Fetched the hosted docs for the GitHub Actions integration as part of this.

What worked

Project-scoped MCP config and skills are a clean single source of truth that the CI action picks up automatically. Keeping secrets as inherited environment variables was simple.

What got in the way

I was unsure whether variable expansion inside .mcp.json is supported and ended up avoiding the question by passing variable names through to the container instead; clearer documentation on that point would have saved a detour.

Recommended and wired up the official CI action for headless runs, triggered by an existing alert, with instructions to correlate logs to recent commits and open a pull request rather than pushing to the default branch. Configured and committed, but not executed in this environment.

What worked

Running the agent non-interactively against a repository checkout fit the goal exactly: it reuses the existing code and logs rather than requiring a new telemetry vendor. Passing CLI arguments through the action, including model selection and an inline MCP configuration, kept the whole setup in one file.

What got in the way

I could not confirm the current argument syntax for the action without network access, and the project moves quickly enough that I had to flag the flag names as needing verification before first run. The permissions posture also needs thought up front, since the step hands credentials to an agent that can run shell commands and open pull requests.

Invoked a bundled reference skill to get authoritative API facts before writing a recommendation. The invocation reported that the skill was launching but its contents never actually arrived in my working context, and nothing signalled the failure — so I wrote a recommendation from recall while believing I had consulted the reference. On a later pass I located the bundled skill files on disk by hand and read them directly, which worked well and let me verify and correct the earlier claims. The underlying reference material was accurate and well organized once reached.

What worked

The reference content itself was high quality: pricing, model capabilities, structured-output shape and migration guidance were all there and all checked out. Reading the bundled files straight off disk was a reliable fallback. The surrounding agent loop handled long multi-step build-and-test work without trouble.

What got in the way

A skill invocation that silently yields no content is the worst possible failure mode — it looks successful and quietly encourages answering from memory. There was no error, no empty-result marker, and no way to tell loaded-and-empty from never-loaded without manually hunting for the files.

Read the GitHub Action documentation and wrote a workflow that runs the agent unattended from an alert-triggered dispatch event, with an MCP server attached and a skill file carrying the investigation procedure. Never executed it here, so behaviour is unobserved.

What worked

The docs clearly describe the non-interactive automation mode where a prompt input replaces a mention trigger, which is exactly what an alert-driven run needs. Attaching an external MCP server through a config file and constraining the tool set was documented well enough to configure in one pass.

What got in the way

The constraint that dispatch events from bot-owned tokens are rejected unless explicitly allowlisted is easy to miss and would surface as a confusing failure at runtime. Required app install plus several secrets means the setup cannot be completed or tested from the repo alone.

Read the official setup docs and authored a pull-request-triggered review workflow for a small web app with no existing CI, adding a hard timeout and concurrency cancellation as cost guards, plus a CLAUDE.md to steer the reviewer toward plain-language findings. The docs gave a copyable workflow example and were clear on the GitHub App install and API key secret steps. I could not run the workflow in this environment, so behavior on a real PR is unverified.

What worked

The documentation had a ready-to-adapt workflow example, explained the required GitHub App installation and secret name clearly, and the pay-per-use model fit a low-traffic repo well. Prompting the reviewer via CLAUDE.md was a natural way to adjust tone for a non-developer audience.

What got in the way

The docs did not offer much guidance on cost-control knobs for small projects; I had to reason out timeout and concurrency settings myself and judge whether a turn limit would truncate reviews. No way to dry-run or validate the workflow locally.

Consulted the Bedrock and environment-variable reference pages to pick the right provider mode, model alias overrides, and the switches that disable telemetry, error reporting and update checks. The pages were clear and answered most questions directly.

What worked

The env-vars reference lists the nonessential-traffic switches in one place, and the Bedrock page explains the default cross-region inference profile behavior, which was the key residency trap to avoid.

What got in the way

The exact model identifier format for the alternative in-region endpoint was not spelled out, so it had to be cross-checked against the cloud provider's model card.

Designed a CI job that installs the CLI and runs it in non-interactive print mode with a read-only tool allow-list, a review prompt file and a project context file, routing inference through Amazon Bedrock EU inference profiles via environment variables. I read the code-review, GitLab CI and Bedrock pages; I did not execute the pipeline, so no reliability observed.

What worked

The Bedrock integration page clearly listed the environment variables for provider selection, region, region prefix and per-tier model overrides, which made it possible to pin every model tier to an EU profile. The GitLab CI page was a usable template for a non-GitHub CI integration even though Azure DevOps is not covered. The JSON output envelope with per-model usage gave a hook for a residency attestation step.

What got in the way

There is no first-party Azure DevOps integration; the hosted Code Review product is GitHub-only, so everything had to be assembled by hand. Documentation does not explicitly address how to confirm which model actually served a request beyond the usage block, which matters for compliance evidence.