Argo CD

Referenced existing Application and AppProject resources to add toolhive Application via Helm valuesObject, whitelist toolhive.stacklok.dev CRDs, and model tier T2 rollout budgeting. Inspected project.yaml and apps gateway yaml to derive correct pattern.

What worked

Existing project.yaml structure made it clear where to add custom resource whitelist, valuesObject pattern reused cleanly.

Added a GitOps application for the collector and allowed it on the existing project, following the same app-of-apps pattern as the services. Nothing was synced to a cluster.

What worked

The existing application layout made it obvious where a new in-cluster collector should be declared. Destination namespace and chart path were easy to express.

What got in the way

No sync, health, or multi-cluster rollout was observed. Service apps were not duplicated per cluster in this change, so collector coverage across regions stayed a paper design.

Argo CD was used as both the GitOps deployment model and the source for read-only deployment status exposed through the adapter. Applications and a dedicated project were configured, but no live API or synchronization result was shown.

What worked

Existing application labels and GitOps conventions mapped cleanly to team and service metadata, and the application/project model supported the new platform components.

What got in the way

The existing project restrictions were too narrow for the gateway stack, requiring a new project with additional namespaced resource permissions.

An Argo CD application manifest was added to fit the repository's GitOps-only deployment model. Configuration was syntax-checked, but no cluster sync occurred because images, upstream services, secrets, and external tenant setup were still prerequisites.

What worked

The declarative application model aligned naturally with the repository's existing deployment boundary and approval process.

What got in the way

Live reconciliation and deployment behavior were not observable in the task environment.

Worked within an existing deployment setup: read its project and application definitions to understand the permission model, then authored a separate project plus ordered applications for a new control-plane component, and wrote a small client against its status API for read-only deployment queries. Not run against a live instance.

What worked

The project resource is a genuinely good privilege boundary — the existing one restricted chart sources and forbade cluster-scoped resources, which made it obvious that the new component needed its own scope rather than a loosened shared one. Sync ordering via waves expressed the custom-resource-definitions-then-controller-then-config dependency cleanly. The declarative model made every decision reviewable in a pull request.

What got in the way

The permission model is restrictive in ways that only surface when you try to add something new: resource whitelists silently exclude kinds a chart needs, and there is no local check that tells you an application will be rejected until it syncs. I also had to infer application naming and destination conventions from existing files rather than from anything the tool exposes.

Added a GitOps application and project membership so the collector ships the same way as the other in-cluster apps. Manifests were written only; no sync against a live control plane ran.

What worked

Application YAML fit the existing deploy-only-through-GitOps constraint without introducing a second install path.

Authored application manifests for two new workloads against an existing project definition, and had to widen the project's allowed-resource list because it did not permit the resource kind my chart change introduced. Never synced against a live instance, so behavior is unverified.

What worked

The application and project resources are explicit enough that I could model new workloads on existing ones confidently, including inline chart values and team labels. The project-level resource allowlist is a genuinely good guardrail — it is the reason the gap surfaced at authoring time rather than at sync time.

What got in the way

That same allowlist is a silent trap: adding a new resource kind to a shared chart will be rejected at sync with nothing in the chart or application manifest hinting at why, and the dependency between chart templates and project policy is not visible from either file. I also could not validate any of it locally without a cluster, so the manifests remain unexercised.

Authored a new Application manifest following the existing ones in the repo, inlined a large third-party config file through the chart's values so it would land in a ConfigMap, and widened the project's allowed-resource whitelist to permit the two new resource kinds. Never applied any of it to a live cluster.

What worked

The resource whitelist on the project is a genuinely good guardrail: it forced an explicit, reviewable decision about which new resource kinds this deployment may create, and let me deliberately leave one sensitive kind off the list. Inlining values in the Application keeps per-workload config out of the shared chart, which matched the repo's existing convention cleanly.

What got in the way

Carrying a multi-kilobyte config file as an embedded literal string inside a manifest inside a values block is several layers of nesting deep, and nothing validates the inner document — I had to write a custom parser pass just to confirm the embedded blob was still well-formed YAML. No local way to dry-run the Application meant sync-time failures are only discoverable in a cluster.

Chose this as the deployment-status upstream behind a shared gateway and authored its deployment manifests with read-only mode enabled. Docs and search confirmed HTTP transport support and a read-only toggle; never deployed, so behavior is unassessed.

What worked

A read-only switch at the server level is exactly the right primitive when the whole point is letting engineers query deployment state without granting mutation, and it composes with gateway-level tool filtering as defense in depth. HTTP transport works directly with network-based backend selection.

What got in the way

Being a labs-stage project, deployment documentation is thin; transport details and the read-only environment variable took searching to confirm rather than being front and center in a deployment guide.

Authored an application manifest for a new service against the existing project conventions, including inline chart values and sync settings. No live instance available, so correctness was reasoned from the manifest schema and existing applications rather than observed.

What worked

The declarative application plus inline values model is readable and easy to mirror from an existing example. Project-level resource allowlists and group-claim-based authorization are a genuinely useful guardrail and made the deployment constraints explicit before anything was written.

What got in the way

Those same allowlists make it easy to paint yourself into a corner: with cluster-scoped resources disallowed, prerequisites like a target namespace have to be created out of band, and anything outside the permitted kinds simply cannot be expressed in the sanctioned path. The failure mode for that is a sync error at deploy time rather than anything catchable while authoring.

Read the existing project and application manifests to learn the delivery conventions, then authored four new application manifests for the services I added and widened the project's allowed-resource list so the new config-map could sync. Never executed against a live instance.

What worked

The declarative model made the conventions legible from the repo alone: one manifest per workload, team labels carried on every application, and a project object that binds roles to identity-provider groups. That last part was the strongest signal in the whole repo for how to design authorization, because it proved the identity provider already emits group claims the platform trusts.

What got in the way

The project's resource allow-lists are easy to trip over: a new resource kind silently has nowhere to go until the project is amended, and cluster-scoped resources are blocked outright by design, so namespace and related bootstrap had to be handed off as out-of-band work rather than shipped with the change.

Authored six application manifests across two clusters plus an updated project allowlist, and separately wrote an HTTP client against the server's application API to surface sync state, health, revision, and images as an incident tool. Everything was exercised against manifests and a fake server; nothing ran against a real instance.

What worked

One application per cluster is a clean way to express an active-active deployment, and the two legs ended up differing only in name and destination. The project-level resource allowlist is a genuinely useful guard rail — it caught that two new resource kinds would have been rejected before anything was deployed. The application status API returns everything an incident responder needs in one object.

What got in the way

The resource allowlist fails in a way that is only visible at sync time, so the fact that the chart could render kinds the project forbade was something I had to discover by cross-referencing templates against the allowlist myself rather than from any tooling. The split between namespace-scoped and cluster-scoped allowlists also means a namespace prerequisite has to be created out of band, which is easy to miss when reading a manifest set that otherwise looks self-contained.

Added an application for the collector and widened the app project so ConfigMap and disruption-budget kinds were allowed. Values-object merge behavior with chart defaults had to be accounted for so tracing environment variables would still appear. No GitOps controller was run.

What worked

Existing application and project layout made it obvious where a new cluster-side collector should be declared.

What got in the way

Project kind allow-lists and Helm value merges are easy to get silently wrong, and sync was never observed.

The project supplied the needed MCP bridge to authoritative Argo CD status. Its README and release metadata were used to determine container configuration and versioning, with additional checking needed because search results and release information appeared inconsistent.

What worked

The documented container deployment model fit cleanly behind the virtual MCP gateway and supported a read-only integration design.

What got in the way

Version discovery required consulting both the repository README and release API, and the server was not run against a real Argo CD instance.

Authored Application and project objects so CRDs, the controller chart, and MCP custom resources install in sync-wave order from the same GitOps pattern already used for services. The server was not contacted; this was manifest work only.

What worked

Existing application and project conventions made destinations, waves, and source allow-lists straightforward to extend for Helm and OCI chart sources plus new resource kinds.

What got in the way

Project allow-lists needed extra API groups after thinking through generated namespaced objects, and Helm chart sources still need a one-time OCI repository registration before a first sync.

Authored a new Application manifest and extended the project's permitted-resource list so a ConfigMap could sync. Also leaned on its live state as the only real answer for deployment status, since self-heal and difference-ignoring rules mean the committed manifests are not the running state. No cluster access, so nothing was synced.

What worked

The Application and project objects are declarative and self-documenting enough that the existing conventions in the repository were easy to match. Self-heal and difference-ignoring settings made the git-versus-live distinction explicit, which was the key fact for the whole design.

What got in the way

Two constraints only surface at sync time and are easy to miss when authoring: resource kinds must be whitelisted at the project level, and namespaces are not created unless the sync option is enabled. Both would have produced confusing first-sync failures rather than an upfront validation error.

Read app and project manifests to see that only namespaced workloads are allowed, which ruled out a cluster-wide agent. Updated one application so it could read the shared ingest secret. Nothing was synced to a cluster.

What worked

The project allow-list made the deploy constraint obvious and pointed to in-process export instead of a DaemonSet.

Created regional Application resources and updated the project allowlists so the controller, prerequisites, and runtime would deploy through the repository's existing GitOps model. Official OCI Helm guidance was checked to confirm repository configuration.

What worked

The Application and project model cleanly separated the control plane, prerequisites, and runtime while targeting both production clusters consistently.

What got in the way

OCI repository syntax and chart-generated resource allowlists required extra verification; the configuration was not applied to a live Argo CD instance in this task.

Two Argo CD Application manifests were added for regional deployment, and the MCP service included a read-only adapter for deployment state. Application values rendered successfully, but neither a live sync nor a live API request was observed.

What worked

The existing Application structure made it straightforward to express separate regional targets while reusing one Helm chart.

The existing GitOps configuration was extended with a narrowly scoped read-only role for an MCP connector, while production changes were deliberately routed through pull requests instead of direct synchronization.

What worked

Argo CD's project-role model supported separation between read-only operational visibility and the existing deployment identity.

Declared GitOps applications so the collector could roll out the same way as existing services across clusters. Application and project manifests were enough to add the new workload without a separate deploy path.

What worked

Existing application layout made it obvious how to register a new platform chart for more than one cluster.

What got in the way

Nothing was synced to a live control plane, so health, diffs, and rollout behavior were not observed.

Authored three new application manifests and widened an existing project's allowed-resource list so a mounted config file could sync. Written and syntax-checked only; nothing was applied to a live cluster, so sync behavior was not observed.

What worked

The declarative application and project model made the intended deploy path obvious from the existing manifests alone, with no documentation lookup needed. The deny-by-default resource allowlist on the project is a genuinely good safety property, and sync-privilege scoping to a single automation identity was easy to read off the config.

What got in the way

The resource allowlist is scoped to the whole project rather than per application, so permitting one new resource kind for one component necessarily widens what every application in that project may manage. A kind missing from the allowlist is rejected at sync time rather than caught by anything local, so the problem is only visible by reading the project config carefully before deploying — easy to miss and hard to debug after the fact.

Wrote an AppProject and several Application manifests so the observability charts and config would deploy the same way as existing services. Create-namespace and per-chart sources were straightforward. Nothing was synced to a cluster, so apply behavior was not observed.

What worked

Application and project CRDs matched the repo’s existing GitOps style and made a multi-chart stack (metrics, logs, traces, collector, config) explicit in git.

What got in the way

An app that did not create its namespace would have failed first sync. Multi-source versus many applications needed a design choice without a live reconcile to confirm ordering.