development-gitops-argo-cd-3darch/SKILL.md

---
name: development-gitops-argo-cd
description: >
  Create, debug, and maintain GitOps repositories where ArgoCD syncs cluster state from a
  Copier-templated Git repo — app-of-apps Application manifests under cluster/argocd/, Helm
  values.yaml.jinja under cluster/resources/, Jinja2 conditional directory names used as
  feature flags, copier.yaml schemas, per-environment copier-answers.yaml — and connect the
  rendered repo to a live cluster. Use when adding an operator, database, or workload;
  editing sync policies, finalizers, or destination namespaces; wiring image address/tag
  through Copier variables into Helm values; toggling a feature flag for one environment;
  running copier update; registering the repository Secret, applying the AppProject and
  app-of-apps bootstrap, trusting the git host key, or granting SSO read access; or debugging
  an app stuck OutOfSync, a failed render, or blank lines that break rendered YAML — even
  when the user only names their repository (e.g. CogArchHubGitOps) instead of saying
  "GitOps" or "ArgoCD".
license: Proprietary
metadata:
  author: workspace-skills-code-agent
  version: "1.0"
  spec: agentskills.io/specification
  reference-implementation: CogArchHubGitOps
compatibility: Designed for Cline, Claude Code, GitHub Copilot, OpenAI Codex, and other compatible agent environments
---

# ArgoCD GitOps on a Copier Template Repository

Work rules for GitOps repositories that combine three layers: **Copier** renders the
repository from per-environment answer files, **ArgoCD** reads the rendered output and syncs
it, and **Helm** consumes the rendered `values.yaml` at sync time.

The rules here are repo-agnostic. `references/cogarchhubgitops-reference.md` records a
complete, concrete instance of every pattern (bands in use, flag names, global settings) —
read it when working in that repository, or as a worked example when standing up a new one.

Two distinct jobs live in this skill. **Authoring** — sections 1–10 — produces the repository:
templates, flags, values, all of it files in git. **Operating** — section 11 — hands a
rendered repository to a live cluster: repository credentials, bootstrap manifests, RBAC. The
first never touches a cluster; the second never edits a template. If you are reaching for
`kubectl`, you are in section 11.

## Scope routing

Pick the reference file that matches the change; do not load all of them.

| The change touches | Read |
|---|---|
| An `application.yaml.jinja` — sync policy, finalizers, destination, source path | `references/argocd-applications.md` |
| `copier.yaml`, any `.jinja` syntax, conditional directory names, the `render()` macro | `references/copier-templates.md` |
| A `values.yaml.jinja` — image wiring, `global` block, env vars, probes, replicas | `references/helm-values.md` |
| A `<env>/copier-answers.yaml` — feature flags, image tags, drift, `copier update` | `references/environments.md` |
| Adding a whole new resource across all three layers at once | `references/adding-a-resource.md` |
| Connecting a rendered repo to a cluster — repo Secret, bootstrap apply, host key, SSO RBAC | `references/connecting-a-repo.md` |
| Variables and flags for the connect operations | `references/connect-env-vars.md` |
| CogArchHubGitOps specifically — bands in use, flag names, global settings | `references/cogarchhubgitops-reference.md` |

## 1. Canonical repository shape

```text
<repo>/
├── copier.yaml                                  # variable schema + defaults
├── cluster/
│   ├── argocd/
│   │   ├── application.yaml.jinja               # root app-of-apps (000-root)
│   │   └── {{ 'NNN-name' if <flag> }}/
│   │       └── application.yaml.jinja           # one child Application per resource
│   └── resources/
│       └── {{ 'NNN-name' if <flag> }}/
│           └── values.yaml.jinja                # Helm values for that Application
└── <env>/
    └── copier-answers.yaml                      # per-environment answers
```

Every file under `cluster/` is a Copier template. There are no raw `.yaml` files there — a
manifest without the `.jinja` extension will be copied through unrendered and will reach
ArgoCD with literal `{{ … }}` in it.

## 2. The three-layer contract

Every managed resource spans three layers that must stay in sync:

```text
Layer 1 — copier.yaml
    declares the feature flag + every configuration variable
Layer 2 — cluster/argocd/{{ 'NNN-name' if flag }}/application.yaml.jinja
    tells ArgoCD where the chart and values live
Layer 3 — cluster/resources/{{ 'NNN-name' if flag }}/values.yaml.jinja
    supplies the rendered values.yaml to Helm at sync time
```

All three share the **same feature flag variable** and the **same conditional directory
expression**. Breaking that symmetry is the single most common failure in this repo shape:
if the two directory expressions differ, one side renders and the other is omitted, and
ArgoCD fails with a path-not-found error.

**Mirroring rule** — every `cluster/argocd/<dir>/application.yaml.jinja` must have a
`cluster/resources/<same-dir>/` sibling containing at least a `values.yaml.jinja` (for Helm
apps) or raw manifests. Same expression, character for character.

## 3. App-of-apps bootstrap

`cluster/argocd/application.yaml.jinja` is the **root application**, conventionally named
`000-root`. It watches `cluster/argocd/` recursively inside the rendered repository. When
ArgoCD reconciles the root, it discovers every `application.yaml` beneath it and creates
those child `Application` resources automatically. Adding a child app therefore requires no
change to the root — only a new conditional directory.

## 4. Numeric band convention

Every child directory and its ArgoCD `Application` name carry a three-digit prefix:

| Band | Domain | Examples |
|---|---|---|
| 000 | Root app-of-apps | `000-root` |
| 200–299 | Operators | ESO, CNPG, Redis operator |
| 300–399 | Cluster configuration | ClusterSecretStore, namespaces |
| 400–499 | Databases | PostgreSQL cluster, Redis cluster |
| 700–799 | Advantage / MCP workloads | MCP servers, AI agents |
| 800–899 | Platform applications | notification, integration, agent-service |

Sub-groups inside a band take the next free number in that band — e.g. `210-eso-operator-group`,
`213-eso-subscription`, `215-eso-configuration`. Choose the band **before** writing any file
and scan existing directories to confirm the number is free. Never reuse a number across
domains.

## 5. Conditional directories are the feature-flag mechanism

Directory names inside `cluster/` are Jinja2 expressions:

```text
{{ '210-eso-operator-group' if operators.eso_is_enabled }}
```

Copier evaluates the expression at render time. Truthy → the directory renders under that
name. Falsy → **the entire directory tree is omitted**, so neither the ArgoCD app nor its
resources are generated.

This is why manifests in this repo contain almost no Jinja2 logic. Do not use `{% if %}`
blocks to conditionally include whole YAML sections — gate the directory instead. Templates
substitute values; directories decide what exists.

## 6. Non-negotiable rules

These apply to every change regardless of which layer it touches:

- **Never hardcode `repoURL` or `targetRevision`** — always `{{ gitops.repo_url }}` and
  `{{ gitops.branch }}`.
- **Never hardcode image address or tag** in a values template — environments must deploy
  different versions of the same template.
- **Every `values.yaml.jinja` uses the `{% macro render() %}` blank-line-stripping footer.**
  YAML is whitespace-sensitive and unexpanded conditionals leave blank lines that break the
  parse. The pattern is mandatory, not stylistic — see `references/copier-templates.md`.
- **Use `.get()` chains with defaults** for any nested variable that may be absent in some
  environment: `{{ applications.get('svc', {}).get('replicas', 1) }}`. Direct dot access on
  an optional key raises `UndefinedError` and fails the render for every environment.
- **New feature flags default to `false`** in `copier.yaml` — resources are opt-in per
  environment.
- **No secrets in Git.** Not in `.jinja` templates, not in `values.yaml.jinja`, not in
  `copier-answers.yaml`. Use placeholders and inject at runtime via ExternalSecret /
  ClusterSecretStore.
- **Feature flags are YAML booleans** (`true`/`false`) in answer files. Helm string-booleans
  (`is_internal`, `maintenance`) are quoted strings (`"true"`/`"false"`). These are different
  things — see `references/environments.md`.

## 7. Adding a new resource — the short path

Full walkthrough with copy-paste templates: `references/adding-a-resource.md`.
Starting templates: `assets/example-cluster-resource/`.

1. Pick the band, the next free number, the kebab-case name, and the flag variable name.
2. Declare the flag (`type: bool`, `default: false`) and its config variables in `copier.yaml`.
3. Create `cluster/argocd/{{ 'NNN-name' if <flag> }}/application.yaml.jinja`.
4. Create `cluster/resources/{{ 'NNN-name' if <flag> }}/values.yaml.jinja` — identical expression.
5. Enable the flag and supply image/config values in the answer file of each target environment.
6. Render with the flag off **and** on, and validate both (section 8).

## 8. Validation

Never commit a template change without rendering it. A broken template blocks every
environment that renders from the repo, not just the one you were working on.

```bash
# Render with defaults — new resource must NOT appear (flag defaults to false)
copier copy . /tmp/gitops-off --overwrite --defaults
ls /tmp/gitops-off/cluster/argocd/ | grep "<NNN>"      # expect no output

# Render with the resource enabled
copier copy . /tmp/gitops-on --overwrite --defaults \
  --data '<domain>.<descriptor>_is_enabled=true' \
  --data '<domain>.<descriptor>.image.address=myregistry/myimage' \
  --data '<domain>.<descriptor>.image.tag=1.0.0'
ls /tmp/gitops-on/cluster/argocd/ | grep "<NNN>"       # expect the directory

# Render a real environment
copier copy . /tmp/gitops-env --overwrite --answers-file <env>/copier-answers.yaml

# No .jinja may leak into rendered output
find /tmp/gitops-env -name "*.jinja" | wc -l           # must be 0

# Every rendered YAML must parse
find /tmp/gitops-env -name "*.yaml" -exec python3 -c \
  "import yaml, sys; yaml.safe_load(open(sys.argv[1]))" {} \;

# Every Application's source.path must exist under the rendered tree
for app in $(find /tmp/gitops-env/cluster/argocd -name "application.yaml" -exec \
    python3 -c "import yaml,sys; print(yaml.safe_load(open(sys.argv[1]))['spec']['source']['path'])" {} \;); do
  [ -d "/tmp/gitops-env/$app" ] || echo "MISSING resources dir: $app"
done
```

Jinja2 syntax check without rendering (fast feedback while editing):

```bash
python3 -c "
from jinja2 import Environment
import pathlib
env = Environment()
for f in pathlib.Path('cluster').rglob('*.jinja'):
    env.parse(f.read_text()); print('OK:', f)
"
```

Helm-lint the rendered values against the chart when a chart path is available:

```bash
helm lint <chart-path> -f /tmp/gitops-on/cluster/resources/<NNN-name>/values.yaml
```

## 9. Consistency checklist

Before committing any resource addition or rename:

- [ ] The three-digit number is unique and in the correct band.
- [ ] The flag is declared in `copier.yaml` with `type: bool` and `default: false`.
- [ ] The ArgoCD `metadata.name` equals the directory name exactly.
- [ ] `spec.source.path` equals the `cluster/resources/` directory name exactly.
- [ ] The conditional expressions in `cluster/argocd/` and `cluster/resources/` are identical.
- [ ] `repoURL` and `targetRevision` use Jinja2 variables.
- [ ] `resources-finalizer.argocd.argoproj.io` is present in `metadata.finalizers`.
- [ ] `values.yaml.jinja` carries the `render()` macro and stripping footer.
- [ ] All optional variable references use `.get()` with a default.
- [ ] No secrets in any new or edited file.
- [ ] At least one environment's answer file enables and fully configures the resource.
- [ ] Rendered flag-off and flag-on, both validated.

## 10. Top mistakes to avoid

- Creating the ArgoCD app without the mirrored `cluster/resources/` directory — ArgoCD errors
  with path-not-found.
- Using different conditional expressions on the two sides — one renders, the other vanishes.
- Writing a raw `application.yaml` instead of `application.yaml.jinja`.
- Omitting the `resources-finalizer.argocd.argoproj.io` finalizer — deleting the app leaves
  orphaned cluster resources behind.
- Disabling `selfHeal` or `prune` without documenting why — it silently defeats the GitOps
  contract.
- Enabling a flag in an answer file without supplying the required image/config variables —
  the rendered values file has empty fields and Helm fails at sync.
- Editing `_src_path` or `_commit` by hand in an answer file — both are Copier metadata and
  wrong values break `copier update`.
- Skipping the render-and-diff step after editing an answer file — drift between answers and
  rendered output produces unexplained cluster state.

## 11. Connecting the repo to a cluster

Authoring stops at a rendered repository. Getting ArgoCD to *read* it is a separate,
cluster-side job, driven by the `.scripts/argo-cd/` shell module:

```bash
task argo-cd:0000-trust-repo-host      # once per git host — SSH known-hosts
task argo-cd:0100-connect-one          # preflight → repository Secret → bootstrap apply
task argo-cd:0200-grant-sso-readonly   # SSO users can see Applications
```

Full walkthrough, decision tables, and verification commands:
`references/connecting-a-repo.md`. Variables and flags: `references/connect-env-vars.md`.

Three things about this half are easy to get wrong:

- **It assumes ArgoCD is already installed.** Preflight only checks that the namespace
  exists; nothing here installs ArgoCD.
- **The repo must be pushed before connecting.** The bootstrap manifests are applied from
  local disk, but they point ArgoCD at a `repoURL` and `path` *in Git*. An unpushed render
  produces an app-of-apps that cannot find its own path.
- **Cluster folder names are a contract.** The connect scripts reference the rendered
  bootstrap directory by name. Renaming cluster folders in the template without updating the
  step scripts breaks onboarding — see `references/connecting-a-repo.md` §7, which also maps
  the numbered `cluster/0000-bootstrap` layout onto the `cluster/argocd/` layout used above.

Configuration for these operations lives in `.env` layers, which are **tooling inputs**.
Never let their values reach the deployed `cluster/**` manifests.