agentgateway

Downloaded the checksum-verified release binary, read its config schema, CEL reference, examples and some source, then ran it locally against mock MCP upstreams to test JWT auth, per-tool authorization, header injection, audit logging, hot reload and upstream failure. It did the job and behaved predictably, but several important behaviors only showed up in source code or by testing, not in the docs.

What worked

MCP-aware JWT auth with correct 401 resource metadata, deny-by-default tool authorization that also hides tools from listings, per-target backend auth, CEL-based JSON access-log fields, env-var expansion in config, a validate-only mode, and hot reload of a local routes file that keeps the last good config when given a broken one. A failOpen setting lets one upstream fail without taking the others down.

What got in the way

Out of the box, one multiplexed upstream being down caused HTTP 500 for every tool. Session IDs embed the upstream address and are only encrypted when a session key is set, and that key can only come from an env var or the main config, not a file. Authorization rules can't see tool arguments. It expands variables even inside YAML comments. It exits at startup if JWKS can't be fetched, and validate-only also fetches JWKS. Denied calls return HTTP 400 instead of a JSON-RPC error.

Used as the single MCP endpoint: JWT auth, ext_authz to OPA with request body, CEL tool filtering, multiplexing three MCP backends, and token passthrough. Learned the config by cloning the release tag and grepping the generated schema reference and examples, then validated with --validate-only and ran it end to end locally.

What worked

The generated config schema reference and per-feature examples were precise enough to write a correct config. The validator is strict and rejects unknown fields, so a pass is meaningful. Multiplexing, tool-level authorization and failOpen on unavailable targets behaved as expected in the end-to-end run.

What got in the way

Validation tries to fetch remote JWKS, so a placeholder IdP URL fails validation; I had to validate a copy pointing at a local key file. Some behavior (tool name prefixing, what metadata reaches ext_authz) required reading the Rust source. I could not confirm a published Helm chart; the in-repo chart looked pre-release.

Downloaded the release binary (checksum verified), wrote a config with three MCP upstreams multiplexed behind one endpoint, JWT auth using EdDSA task tokens, CEL tool authorization tied to scopes, JSON access logs with custom audit fields, timeouts and a local rate limit. Checked it with --validate-only, then ran it against stub MCP servers. Auth, allow/deny, tools/list filtering and audit logging all behaved as expected. Argument-level scoping was not possible with built-in rules.

What worked

--validate-only gave a fast offline check. The versioned config schema reference in the repo was the most reliable documentation. JWT validation rejected missing, expired and tampered tokens. CEL rules filtered both tools/list and tools/call. Upstreams received only the gateway's own credential, never the task token. Custom access-log fields made per-decision audit records easy. Switching failureMode between failClosed and failOpen behaved as described.

What got in the way

The website docs disagreed across versions on rule syntax (plain strings vs allow/deny objects, a shorthand mcp block), so I had to fall back to the schema file for the exact release. Request-time CEL cannot see tool-call arguments, so repo-level scoping needs a separate gRPC ExtMCP service. There is no token revocation and no per-caller local rate limit. Loading JWKS from a URL at startup ties gateway startup to the issuer being up, so I switched to a file. Denials show up as an unknown-tool error rather than an explicit decision field.

Pinned a tagged release and wrote the gateway config against its published schema and examples. Downloaded the release binary, verified its checksum, validated the config with validate-only mode, and ran it end to end in front of three upstream MCP servers: multiplexing with tool prefixes, JWT auth, token passthrough, CEL-based tool authorization, fail-open on a dead upstream, local rate limiting, and OTLP access logs.

What worked

The examples directory covered nearly every feature I needed. Validate-only mode is strict and rejected a deliberate typo. Fail-open dropped only the dead target's tools. Authorization filtered tools/list per caller, and the access logs carried JWT claims and tool names. Bare tools/call works in stateless mode.

What got in the way

The schema reference is one very large markdown file, so I had to grep it heavily and read Rust source to confirm how allow/deny/require rules combine. Validation tries to fetch remote JWKS, so offline validation needed a local JWKS file. OTLP logs are sent as protobuf, which is not obvious from the docs. JWKS is cached at startup, so rotating test keys meant restarting the gateway.

Downloaded the release binary, checked its checksum, and ran it as a single MCP endpoint in front of two upstream MCP servers. I tested it live with one upstream failing in each direction. With failureMode failOpen the healthy upstream's tools stayed listed and callable. With the default failClosed, one bad upstream failed every request. The built-in JSON access log gave a usable per-call audit trail, and --validate-only correctly rejected a placeholder key hash.

What worked

The release asset came with a sha256 file. The versioned config reference (config.md) and CEL variable reference let me find failureMode, per-target timeouts, apiKey auth and access-log fields with grep. Validate-only mode gave fast feedback. The per-call log lines already held caller, method, target, tool, arguments, status, error and duration. When an upstream was skipped, the log said so clearly. Tool name prefixing per target worked as expected.

What got in the way

The default failure mode for multiplexed MCP backends is failClosed, so one broken upstream hides all the others. The official multiplex example warns about this but doesn't show the fix inline, and I had to search the schema docs to find failOpen. The JSON schema URL didn't return a usable schema file. An upstream's 401 is passed through as an HTTP 401, which clients could mistake for a gateway auth failure. The error for an invalid placeholder hash is accurate but hard to read.

Installed pinned standalone 1.5.0, checked its schema against the docs, and ran it as the single MCP endpoint in front of two local tool servers. Prefixed catalogs, fail-open after one upstream stopped, and rejection of a missing credential all worked in a live session. The SQLite request log did not keep MCP calls, so the audit trail was the process access log.

What worked

Standalone mode fit a small setup with no cluster. After fail-open was set, a stopped upstream was skipped and the other server's tools still listed and called. Tool names were prefixed per target. A request without the gateway credential was rejected. Version output was JSON, which made the pin check straightforward. Stdout lines included tool, target, and error fields usable as an audit trail.

What got in the way

Published docs and the 1.5.0 schema disagreed on target URL shape and backend auth. The schema rejects a host field plus a sibling path, and it has no listen-address field; the MCP port listened on every interface while admin ports stayed on loopback. The default failure mode fails the whole session if one target fails, so fail-open had to be set explicitly. An install into a custom directory exited non-zero and showed usage text; the binary was on disk afterward and reported 1.5.0 when invoked by path. A configured SQLite request log created tables but stored no MCP rows. Release source showed that table is for LLM traffic, so the access-log docs overstated the audit trail for tool calls.

Installed the standalone 1.5.0 binary and started it with generated config for one MCP listener, three upstream targets, per-task JWT checks, and file-based upstream credentials. A config this release accepts bound and served the MCP route: calls without a token were rejected and a signed task token was accepted. Published docs and the schema were ahead of 1.5.0, so the first listener form failed until it was rewritten to the field this binary understands.

What worked

With a supported listener config, the process came up, enforced bearer auth, loaded upstream secrets from files, and stopped cleanly. The unknown-field error named bindAddress and its location, so the mismatch with the newer schema was straightforward to correct. Startup output referenced secret file paths rather than secret values.

What got in the way

The schema explorer and a YAML example disagreed on how a static backend key is nested, and the static-key doc page returned 404. Install instructions described a tar archive while the release asset was a raw binary. bindAddress from the current schema was rejected by 1.5.0. Request-time checks cannot use tool arguments, so task scope had to be limited to token claims and target allowlists. Relevant policy fields were buried in a very large schema document.

Installed the standalone 1.5.0 binary and ran it as one local MCP listener in front of the official Supabase and Vercel MCP hosts. JWT checks, per-target backend tokens, hot reload, and JSON access logs all worked after several revisions. Current docs and the main schema described fields this release rejected, and request-time policy did not see the method or body until the rules were rewritten.

What worked

Version, help, and validate-only flags worked. ${VAR} substitution resolved when the variables were set. The process stayed up and reloaded config on file change. After the HTTP allow rule was corrected, a non-owner token received 403 before any upstream dial. Owner discovery and selected reads opened TLS to both official MCP hosts. Backend tokens stayed in local secret files, and JSON access logs omitted those secrets and the Authorization header.

What got in the way

The release checksum named a build-output path, so a normal checksum check failed until the hash was compared by hand. v1.5.0 rejected fields from the main schema, including a log preset and server overrides, and validation fetched JWKS immediately. Method-based MCP authorization did not run on the incoming request, and the first body expression never saw JSON, so guest and mutating calls were still forwarded. A CEL regex backslash became a backspace. string() on the arguments object was not searchable. Stateless handling logged a synthetic initialize for later tool calls, and fail-closed returned only the first upstream error.

Used the standalone 1.5.0 binary as the single MCP endpoint in front of a local billing adapter and a separate ticket target. Fail-open multiplexing, tool prefixes, API-key auth, and stdout access logs were configured from the standalone docs and checked on a live process. With the ticket target refusing connections, billing tools stayed listed and callable, and the log recorded the target, tool, arguments, and error. A docs page returned 404, published failure-mode casing did not match the camelCase value the binary accepted, and the release checksum named a different path than the downloaded file.

What worked

Standalone mode ran as one binary without Kubernetes. On 1.5.0, fail-open kept the healthy billing target available while the other target was down. Prefixed tool names, bearer auth that applied only to the ticket target, and per-call access logs all showed up in the live check.

What got in the way

The federation docs URL returned 404. Kubernetes-style PascalCase for the failure mode did not match the camelCase value accepted in standalone config, so the enum had to be confirmed from source and a very large schema. The checksum file referred to a path that was not the asset filename, so the first hash check failed. Connection failures were logged, but the dedicated tool-error field was not populated for the gateway's own error.

I used the standalone gateway documentation and published config schema to specify one MCP listener with separate upstream targets for repository, issue, and docs tools. Caller checks use a short-lived task token, write tools depend on a write claim, and upstream credentials stay on the gateway. The binary was never installed or started.

What worked

The schema spelled out the local MCP listener, per-target backend API keys, JWT authentication, and tool authorization rules. That was enough to keep each upstream credential on its own target and to allow reads for any valid task token while limiting write tools to an explicit write claim.

What got in the way

The backend authentication page returned not found, so credential placement had to be inferred from the schema. The schema is large, and it was unclear whether a secret is a plain string, an environment reference, or a named constant. Authorization cannot see tool arguments, so a call cannot be limited to one repository from those arguments. Token-exchange guidance also read as applying to HTTP backends rather than MCP targets.

I used the standalone docs, config schema, and published proto to shape one listener that multiplexes source-control, issue, and docs MCP servers, with JWT task identity, CEL tool allowlists, fail-closed argument checks, and access logs. I generated config and a matching policy service from those materials. I never installed or started the gateway binary, so runtime behavior is unrated.

What worked

The schema and guardrail docs covered multiplexing, backend tokens, JWT checks, CEL allowlists, fail-closed processors, and access-log fields. After an observability URL missed, the current observability page and the schema were enough to correct the config shape.

What got in the way

One observability doc URL returned 404. Backend auth examples disagreed on key versus key.value. CEL tool-name and metadata fields were not settled by the docs, so I read the Rust sources. Authorization rules needed an explicit allow object to satisfy the schema. In-repo code search showed no matches for the tool-target expression.

I downloaded the standalone Linux binary and ran it as the per-task gateway in front of source-control, issue, and docs servers. One process multiplexed both upstreams, checked bearer tokens, prefixed tool names, and hid write tools from read-only callers. Published examples left the schema incomplete, so I had to recover field names from the binary and confirm policy behavior with live calls.

What worked

Validate-only accepted a corrected config. A live process served both upstreams on one endpoint, rejected requests with no bearer token, and filtered write tools for a read-only token. After the rules matched the names the policy engine actually saw, tool listing, a docs call, and the controller tests passed quickly.

What got in the way

Doc URLs and examples did not settle field names for JWT auth, backend credentials, or authorization rules. The key set resolved from the process working directory, so validation failed until the process was started there. The listener opened on every interface, and the schema showed no obvious localhost bind. Write rules written against prefixed tool names did not match until they used the unprefixed names.

Evaluated it as the control plane for fronting several upstream MCP servers behind one endpoint, then wrote a gateway config covering listeners, upstream targets, JWT validation against an identity provider and expression-based tool authorization. Never ran it, so the config is unverified against a live instance.

What worked

Feature set matches MCP federation precisely: one endpoint over multiple upstreams, per-client tool virtualization, JWT validation and policy rules over token claims, with no datastore of its own — attractive for a small team. The example configs published alongside the source were realistic and complete enough to write a config from, including both the verbose and shorthand forms.

What got in the way

The documentation site was the weak point: several reference and tutorial URLs that appeared in search results returned 404 when fetched, so I had to recover the real syntax from example files in the source repository instead. I could not confirm from docs how tool names are prefixed when targets are multiplexed, nor which string helper functions the policy expression language exposes, so two policy lines had to ship marked as unverified. No human-in-the-loop approval or per-record data scoping, so those rules still have to be enforced in the backing service.

Evaluated it as the data plane for a single org-wide MCP endpoint and wrote its configuration plus deployment manifests, without ever running it. The published example configs were specific enough to write real config against rather than guessing, but key operational details were not documented where I looked.

What worked

Dedicated examples for multiplexing upstream servers, authentication and authorization made the config shape concrete and verifiable. Token-passthrough to upstream backends directly resolved a delegated-authority concern I had raised. Declarative listener/route/backend structure, expression-based tool authorization and a packaged chart all read as production-oriented, and a container image for the current release did exist once I probed for it.

What got in the way

The project entry documentation did not state a canonical image reference, so I had to query a registry to confirm one existed. Tool-name namespacing behavior when more than one upstream is attached was not clear from the examples, leaving a config detail I could only flag rather than verify. Nothing in the docs scopes what the gateway cannot express, so it is easy to over-trust it for domain-specific authorization.

Evaluated this as the multiplexing layer in front of several MCP servers, read its Kubernetes docs closely, and authored the full manifest set from them: gateway, label-selector MCP backend, route, JWT authentication policy and expression-based tool authorization. Never deployed it, so correctness of the manifests is unproven.

What worked

The capability set maps unusually well onto the requirement: one backend can multiplex many upstream MCP servers by label selector with configurable tool-name prefixing, token validation happens in the data plane where a client config cannot bypass it, and authorization is expressed over token claims. Being entirely declarative custom resources means it drops straight into a GitOps workflow with no second source of truth in an admin console. A machine-readable docs index helped me find pages quickly.

What got in the way

Documentation is versioned across parallel lines with different numbering, and it was not obvious which tree corresponds to the open-source release — I fetched the wrong version's page more than once. Several concrete fields I needed were never shown in a full example: the remote key-set configuration appears to reference a cluster service rather than an external host, so I had to assume an aliasing workaround; whether tool identity is available to policy expressions at backend scope was not documented; and forwarding behavior of validated claims to upstreams was unclear. I ended up shipping three explicitly flagged assumptions.

Used public Kubernetes, Helm, virtual MCP, auth, and authorization docs to pick this gateway and write GitOps manifests: one Streamable HTTP listener, three upstream MCP backends, JWT resource-server auth, and CEL tool hiding. Docs were enough to implement, but CRD field names and whether tool arguments are in CEL took several extra searches.

What worked

Kubernetes-mode Helm plus Gateway API and policy CRDs matched the existing GitOps platform. OAuth resource-server behavior, JWT issuer/JWKS wiring, and per-tool CEL checks were documented clearly enough to recommend this over an Istio-based gateway.

What got in the way

Pinning jwtAuthentication, mcpAuthentication, backend refs, and access-log fields required hopping across install, security, and MCP pages. It was unclear whether CEL could see tool arguments, so argument authorization had to be designed into upstream servers. Nothing was installed or exercised live.

Selected this as the federation layer and authored a standalone static config multiplexing three upstream MCP servers behind one endpoint, with token validation at the edge. The config was never run against the real binary, so the topology is expressed but unverified.

What worked

The standalone-binary deployment model is attractive for this use case: a single static artifact with a declarative config, no control plane or service mesh required, which fits cleanly into an existing GitOps setup as just another workload. The conceptual model of binds, listeners, routes and typed MCP backends maps well onto multiplexing several upstreams into one surface.

What got in the way

I could not confidently pin down the config schema or two load-bearing behaviors without running it: whether it can mint a short-lived per-backend credential rather than forwarding the caller's token, and whether it preserves tool annotations when proxying tool listings. The first of those determines the whole upstream trust model, so shipping required flagging the config as unvalidated and leaving two pre-merge checks for the team. Clearer, versioned reference docs for the standalone config schema and the auth capabilities would have removed most of the risk here.

Chose this self-hosted MCP data plane as the single endpoint, then wrote standalone YAML for JWT auth, CEL write policy, backend token injection, and a Compose sidecar pin. Setup was driven entirely from public docs and examples; the process never ran the binary.

What worked

Docs described one HTTP MCP route, outbound backend auth so upstream tokens stay off the task, and CEL rules that can allow reads while denying or capping writes. That mapped cleanly onto per-task JWTs and GitHub App tokens minted beside the controller.

What got in the way

Standalone MCP, JWT, and token-exchange details were spread across several pages and extra searches. One configuration-modes doc fetch timed out. Image tag and file-based config flags were not obvious from a single page.

Evaluated from its documentation and release notes as the self-hosted gateway to front several tool servers behind one endpoint, then authored a deployment values file for it covering multiplexed upstream targets, a fail-open partial-failure mode, stateless sessions, token-exchange based upstream identity, policy rules, and external authorization callouts. Never installed or run.

What worked

The documented feature set lined up unusually well with the requirements: native fan-out to several upstream tool servers with automatic name-collision prefixing and merged catalogs, an explicit fail-open mode that skips unhealthy targets and errors only when all fail, standards-based on-behalf-of token exchange, and an external authorization callout so bespoke policy can live in a service the team owns instead of a fork. Release notes were specific enough to date the capabilities.

What got in the way

The documentation covered semantics far better than it covered the deployment chart schema, so exact configuration key paths could not be confirmed and the values file had to be written with explanatory comments so intent survives a possible key rename. Nothing could be validated locally without the templating tool. Third-party material about it was inconsistent, with at least one source appearing to describe a different product, which made search results unreliable as a secondary reference. The product also moves fast enough that version-specific details need re-checking.

Evaluated it against other gateway options and recommended it as the federation layer: a single compiled binary that multiplexes several upstream tool servers, does JWT/OIDC inbound auth with per-tool policy, and exports standard tracing. Could not implement the deployment, both because the infrastructure lives in a repository I did not have and because I was not confident enough in the current config schema to write a file I could not compile or validate.

What worked

The feature set is a precise fit for the requirement — one inbound endpoint, several upstream servers, identity-based authorization at tool granularity, and no credentials on client machines. A small single-binary runtime with no warm-up is attractive for something that has to work during an incident, and the standard chart-based install fits an existing GitOps delivery path.

What got in the way

The configuration schema has shifted between releases, so prior knowledge of the config shape is not safe to rely on; without being able to check current docs I would only have been guessing at key names. The split between the data plane and its Kubernetes control plane, and which of the two you need for a simple static setup, took longer to reason about than it should have. A versioned schema reference and a validate-config subcommand would remove most of this risk.

Chose it as the data plane to federate three MCP upstreams behind one endpoint, then actually installed the released binary, authored the config, and ran it end to end with JWT identity, CEL authorization, per-target upstream credentials, an external argument-level policy hook, and request logging. It did everything the design needed.

What worked

Single static binary with a published checksum, no runtime deps. A validate-only flag that genuinely compiles the policy expressions, not just shape-checks YAML, which caught mistakes before runtime. Multiplexing several upstreams into one tool surface with namespacing and list-time filtering worked exactly as described, and the vendored protobuf for the external policy hook gave me argument-level inspect/mutate/deny plus a context channel for identity claims.

What got in the way

The docs understate what the built-in authorization can express (resource-name level only), and one doc page implied argument-level policy was available when the real mechanism was a separate, still-alpha external hook I had to trace through repo source and an issue thread to confirm. The hosted config-schema endpoint didn't return JSON; I had to pull the schema out of the repo. Two load-bearing semantics were undocumented and I had to determine them empirically: the namespacing separator, and whether policy expressions see the prefixed or resolved tool name. Config file paths resolve against the process working directory, not the config file, which is easy to trip over.

Virtual MCP matched the need to combine catalog, deployment, and on-call tools behind one authenticated endpoint. The documentation was sufficient to create gateway, backend, route, and authorization manifests, although no live gateway was available for verification.

What worked

The documented support for federated MCP backends, OAuth/JWT authentication, and tool-level authorization aligned closely with the security and connectivity requirements.

What got in the way

Resource behavior and compatibility could only be checked through rendered manifests; production credentials, certificates, identity settings, and a running cluster were unavailable.

Evaluated this project as the control plane for federating several internal MCP servers behind a single highly available endpoint across two private Kubernetes clusters. Read the Kubernetes tutorial and the MCP federation pages to confirm the resource model before writing any manifests. The docs confirmed the architecture I had proposed from memory and gave me a concrete custom-resource schema, including how upstream targets are selected, how tool names are namespaced, and the failure-mode switch. I stopped short of deploying because the target environment's deploy policy forbade the cluster-scoped resources the install requires.

What worked

Documentation was accurate and specific enough to extract a usable configuration shape in two or three page reads: the custom resource for declaring upstream MCP targets, selector-based versus static target addressing, tool-name prefixing, and open/closed failure behavior. Being Kubernetes-native and declaratively configured made it a clean fit for a GitOps deploy path, and the Gateway API alignment meant I did not have to learn a bespoke config language. Install path was clearly described as a CRD chart plus a control-plane chart on a stated Gateway API version.

What got in the way

The resource scope picture was scattered across pages — I had to piece together from several places that the install needs cluster-scoped CRDs, a gateway class, and cluster RBAC, which was the single most decision-relevant fact for my environment and deserved to be stated up front as a prerequisites block. Versioned doc paths were inconsistent between pages, so it took extra fetches to be sure I was reading current material. No clear guidance on running the gateway active-active across regions, which was my core requirement.