Merge pull request 'Add cluster-side connect operations to the ArgoCD skill' (#1) from 4.0.0.0/IIAA-XYZ-cluster-connect-operations-001 into test

Reviewed-on: home-v1-skills-code-agent/development-gitops-argo-cd#1
2026-08-12 13:27:37 -07:00
parent d029e3727d a2200988b5
commit 510b2bb590
4 changed files with 311 additions and 11 deletions
@@ -20,6 +20,8 @@ development-gitops-argo-cd/
 │   ├── helm-values.md                        # values.yaml.jinja, images, global block, probes
 │   ├── environments.md                       # copier-answers.yaml, flags, drift, copier update
 │   ├── adding-a-resource.md                  # End-to-end walkthrough across all three layers
+│   ├── connecting-a-repo.md                  # Cluster-side: repo Secret, bootstrap, host key, SSO RBAC
+│   ├── connect-env-vars.md                   # Variables and flags for the connect operations
 │   └── cogarchhubgitops-reference.md         # The CogArchHubGitOps instance
 └── assets/
    └── example-cluster-resource/             # Starting templates for a new resource
@@ -1,17 +1,17 @@
 ---
 name: development-gitops-argo-cd
 description: >
-  Create, review, refactor, debug, and maintain GitOps repositories where ArgoCD syncs
-  cluster state from a Copier-templated Git repo — app-of-apps ArgoCD Application manifests
-  under cluster/argocd/, Helm values.yaml.jinja templates under cluster/resources/, Jinja2
-  conditional directory names used as feature flags, copier.yaml variable schemas, and
-  per-environment copier-answers.yaml files. Use when adding a new operator, database, or
-  workload to a cluster; editing sync policies, prune/self-heal, finalizers, or destination
-  namespaces; wiring container image address/tag through Copier variables into Helm values;
-  enabling or disabling a feature flag for one environment; comparing environment drift or
-  running copier update to re-render a deployed environment; or debugging an app stuck
-  OutOfSync, a template that fails to render, or blank lines that break rendered YAML — even
-  when the user only names their repository (for example CogArchHubGitOps) instead of saying
+  Create, debug, and maintain GitOps repositories where ArgoCD syncs cluster state from a
+  Copier-templated Git repo — app-of-apps Application manifests under cluster/argocd/, Helm
+  values.yaml.jinja under cluster/resources/, Jinja2 conditional directory names used as
+  feature flags, copier.yaml schemas, per-environment copier-answers.yaml — and connect the
+  rendered repo to a live cluster. Use when adding an operator, database, or workload;
+  editing sync policies, finalizers, or destination namespaces; wiring image address/tag
+  through Copier variables into Helm values; toggling a feature flag for one environment;
+  running copier update; registering the repository Secret, applying the AppProject and
+  app-of-apps bootstrap, trusting the git host key, or granting SSO read access; or debugging
+  an app stuck OutOfSync, a failed render, or blank lines that break rendered YAML — even
+  when the user only names their repository (e.g. CogArchHubGitOps) instead of saying
  "GitOps" or "ArgoCD".
 license: Proprietary
 metadata:
@@ -32,6 +32,12 @@ The rules here are repo-agnostic. `references/cogarchhubgitops-reference.md` rec
 complete, concrete instance of every pattern (bands in use, flag names, global settings) —
 read it when working in that repository, or as a worked example when standing up a new one.

+Two distinct jobs live in this skill. **Authoring** — sections 1–10 — produces the repository:
+templates, flags, values, all of it files in git. **Operating** — section 11 — hands a
+rendered repository to a live cluster: repository credentials, bootstrap manifests, RBAC. The
+first never touches a cluster; the second never edits a template. If you are reaching for
+`kubectl`, you are in section 11.
+
 ## Scope routing

 Pick the reference file that matches the change; do not load all of them.
@@ -43,6 +49,8 @@ Pick the reference file that matches the change; do not load all of them.
 | A `values.yaml.jinja` — image wiring, `global` block, env vars, probes, replicas | `references/helm-values.md` |
 | A `<env>/copier-answers.yaml` — feature flags, image tags, drift, `copier update` | `references/environments.md` |
 | Adding a whole new resource across all three layers at once | `references/adding-a-resource.md` |
+| Connecting a rendered repo to a cluster — repo Secret, bootstrap apply, host key, SSO RBAC | `references/connecting-a-repo.md` |
+| Variables and flags for the connect operations | `references/connect-env-vars.md` |
 | CogArchHubGitOps specifically — bands in use, flag names, global settings | `references/cogarchhubgitops-reference.md` |

 ## 1. Canonical repository shape
@@ -250,3 +258,32 @@ Before committing any resource addition or rename:
  wrong values break `copier update`.
 - Skipping the render-and-diff step after editing an answer file — drift between answers and
  rendered output produces unexplained cluster state.
+
+## 11. Connecting the repo to a cluster
+
+Authoring stops at a rendered repository. Getting ArgoCD to *read* it is a separate,
+cluster-side job, driven by the `.scripts/argo-cd/` shell module:
+
+```bash
+task argo-cd:0000-trust-repo-host      # once per git host — SSH known-hosts
+task argo-cd:0100-connect-one          # preflight → repository Secret → bootstrap apply
+task argo-cd:0200-grant-sso-readonly   # SSO users can see Applications
+```
+
+Full walkthrough, decision tables, and verification commands:
+`references/connecting-a-repo.md`. Variables and flags: `references/connect-env-vars.md`.
+
+Three things about this half are easy to get wrong:
+
+- **It assumes ArgoCD is already installed.** Preflight only checks that the namespace
+  exists; nothing here installs ArgoCD.
+- **The repo must be pushed before connecting.** The bootstrap manifests are applied from
+  local disk, but they point ArgoCD at a `repoURL` and `path` *in Git*. An unpushed render
+  produces an app-of-apps that cannot find its own path.
+- **Cluster folder names are a contract.** The connect scripts reference the rendered
+  bootstrap directory by name. Renaming cluster folders in the template without updating the
+  step scripts breaks onboarding — see `references/connecting-a-repo.md` §7, which also maps
+  the numbered `cluster/0000-bootstrap` layout onto the `cluster/argocd/` layout used above.
+
+Configuration for these operations lives in `.env` layers, which are **tooling inputs**.
+Never let their values reach the deployed `cluster/**` manifests.
@@ -0,0 +1,70 @@
+# Connect — Environment Variables and Flags
+
+Configuration consumed by the `.scripts/argo-cd/` connect operations
+(`references/connecting-a-repo.md`). These are **tooling inputs**: they configure the machine
+running the connect, and must never reach the deployed `cluster/**` manifests.
+
+---
+
+## 1. Layered loading
+
+The base module loads three files in order, each overriding the previous:
+
+```text
+.env  ->  .env-<environment>  ->  .env-<environment>-credentials
+```
+
+- `.env` holds **only** the environment selector.
+- Connection configuration lives in the environment layer (`.env-test`).
+- With SSH auth there is no inline secret — the private key is a **file referenced by path**,
+  so the credentials layer stays empty unless a future component needs it.
+- Every layer has a tracked sample: `.sample.env`, `.sample.env-<env>`,
+  `.sample.env-<env>-credentials`. Adding a variable without adding it to the sample is how
+  the next operator gets a missing-variable failure on a fresh clone.
+
+Variable names narrow scope left to right: `<PREFIX> -> <domain> -> <sub> -> <leaf>`. The
+gondor family uses the `HL_V1_` prefix; the names below are shown with it.
+
+## 2. Variables
+
+| Variable | Layer | Required | Purpose |
+|---|---|---|---|
+| `HL_V1_ENVIRONMENT` | `.env` | yes | Selects which `.env-<env>` files load (e.g. `test`). |
+| `HL_V1_KUBE_CONFIG_PATH` | `.env-<env>` | yes | Path to the kubeconfig file. Supports `${HOME}`. |
+| `HL_V1_KUBE_CONFIG_CONTEXT` | `.env-<env>` | yes | Context inside that kubeconfig to target. |
+| `HL_V1_ARGOCD_NAMESPACE` | `.env-<env>` | yes | Namespace ArgoCD runs in (e.g. `argocd`). |
+| `HL_V1_ARGOCD_TARGET_DIR` | `.env-<env>` | yes | Local path to the rendered repo holding the bootstrap manifests. |
+| `HL_V1_ARGOCD_GIT_REPO_URL` | `.env-<env>` | yes | Repo ArgoCD pulls from — **SSH form** `git@host:owner/repo.git`. |
+| `HL_V1_ARGOCD_PROJECT` | `.env-<env>` | no | AppProject name → project-scoped repo. Empty = global. |
+| `HL_V1_ARGOCD_REPOSITORY_SECRET` | `.env-<env>` | when a Secret is created | Name of the ArgoCD repository Secret. |
+| `HL_V1_ARGOCD_GIT_REPO_SSH_KEY_PATH` | `.env-<env>` | private repo | Path to the **private** SSH key. Empty = public repo. |
+| `HL_V1_ARGOCD_RBAC_DEFAULT_POLICY` | `.env-<env>` | no | `policy.default` for `argocd-rbac-cm`. Defaults to `role:readonly`. |
+
+The five required variables are checked up front, so a missing one fails before any cluster
+call rather than halfway through onboarding.
+
+## 3. CLI flags
+
+Parsed in `lib/--env-vars-reader.sh`; each overrides the corresponding environment value for
+a single run:
+
+```text
+--kubeconfig      --kube-context      --namespace       --target-dir
+--repo-url        --repo-ssh-key-path --repo-secret-name
+```
+
+Flags reach the script after `--`:
+
+```bash
+task argo-cd:0100-connect-one -- --repo-url="git@host:owner/repo.git"
+```
+
+## 4. SSH auth
+
+The repo URL must be SSH form, and the **public** half of the key at
+`HL_V1_ARGOCD_GIT_REPO_SSH_KEY_PATH` must be registered as a read-only **deploy key** on the
+repository. The host key must also be trusted first — see `references/connecting-a-repo.md`
+§2.
+
+Never commit the private key. It stays a local file referenced by path; only the path travels
+in configuration.
@@ -0,0 +1,191 @@
+# Connecting a Repository to ArgoCD
+
+Cluster-side operations that make an **already-rendered** GitOps repo reachable by an
+**already-installed** ArgoCD: trusting the git host key, registering the repository Secret,
+applying the app-of-apps bootstrap, and granting SSO users read access.
+
+Read this when onboarding a repo to a cluster for the first time, rotating or scoping the
+repository credential, or debugging a connect run — kube context, namespaces, repo
+credentials, bootstrap paths. Every other reference in this skill concerns *authoring* the
+repo; this one concerns *operating* ArgoCD against it.
+
+Driven by the wrapper-first `.scripts/argo-cd/` shell module (`Taskfile -> api -> lib`; see
+the `development-scripts` skill for that architecture). Variables and CLI flags:
+`references/connect-env-vars.md`.
+
+---
+
+## 1. Operations
+
+| Operation | Task | Purpose |
+|---|---|---|
+| Trust the git repo SSH host key | `task argo-cd:0000-trust-repo-host` | run once per git host, before connect |
+| Onboard / connect a repo | `task argo-cd:0100-connect-one` | preflight → repo Secret → bootstrap apply |
+| Let SSO users view apps (RBAC) | `task argo-cd:0200-grant-sso-readonly` | ArgoCD install config, not repo state |
+
+The numbered task names encode run order. The module is expandable — sync, app management,
+UI, and uninstall operations belong here as they are built.
+
+## 2. Trust the repo host (run once, before connect)
+
+For SSH repos, ArgoCD's repo-server verifies the git server's host key against
+`argocd-ssh-known-hosts-cm`. A self-hosted host is not there by default, so `connect-one`
+fails with:
+
+```text
+ssh: handshake failed: knownhosts: key is unknown
+```
+
+```bash
+task argo-cd:0000-trust-repo-host      # parses host+port from the repo URL
+```
+
+`lib/-trust-repo-host.sh` parses host and port from the repo URL, `ssh-keyscan`s it,
+**appends** to `argocd-ssh-known-hosts-cm` — never overwriting the shipped defaults — and
+restarts `argocd-repo-server` so the mounted ConfigMap reloads. Idempotent: it skips when the
+host is already present.
+
+A ConfigMap patch alone can take ~60s to remount. The restart makes it deterministic.
+
+## 3. Onboard / connect
+
+```bash
+task argo-cd:0000-trust-repo-host      # once per host
+task argo-cd:0100-connect-one          # reads the .env layers; override with --repo-url=… etc.
+```
+
+Entry chain: `api/connect-one.sh` → `lib/--index.sh` (base loader → argo-cd reader → api
+index) → `_argocd_connect_one`:
+
+```text
+require KUBE_CONFIG_PATH KUBE_CONFIG_CONTEXT ARGOCD_NAMESPACE ARGOCD_TARGET_DIR ARGOCD_GIT_REPO_URL
+export KUBECONFIG=<kube config path>
+preflight → repo-register → bootstrap-apply
+```
+
+### Step 1 — preflight
+
+- `command -v kubectl`
+- `kubectl --context <ctx> get namespace <ns>` — ArgoCD reachable
+- the bootstrap directory exists in the target repo — the render must already have run
+
+### Step 2 — repo-register
+
+Builds a `kubectl create secret generic`, pipes it to `kubectl apply -f -`, then labels it
+`argocd.argoproj.io/secret-type=repository`:
+
+```text
+--from-literal=type=git
+--from-literal=url=<repo url>                    # SSH form: git@host:owner/repo.git
+[--from-literal=project=<appproject>]            # when set → project-scoped
+[--from-file=sshPrivateKey=<key path>]           # when set
+```
+
+| ssh key | project | Result |
+|---|---|---|
+| none | none | skip Secret (public, global repo) |
+| none | set | create Secret (type + url + project) — project-scoped public repo |
+| set | any | create Secret with `sshPrivateKey` |
+| set, file missing | any | fail fast |
+
+The matching **public** key must be a read-only deploy key on the repository.
+
+### Step 3 — bootstrap-apply
+
+Applies the bootstrap manifests in filename order — the AppProject (`0000-*`) before the
+app-of-apps (`0100-*`). ArgoCD then reconciles everything else **from Git**.
+
+The app-of-apps source must set `directory.recurse: true`, because feature Applications live
+in nested subdirectories. Without it only the top-level manifest is read and children never
+appear.
+
+## 4. Project-scoped repositories
+
+Setting an AppProject name makes the repo a **project-scoped repository**: the Secret carries
+a `project` field, so only Applications in that AppProject may use it.
+
+- The SSH key is optional (public repo); if a key path is set, the file must exist.
+- The Secret is skipped **only** when there is neither a key nor a project scope.
+- A public repo **with** a project scope still gets a Secret — type, url, project, no key.
+
+The AppProject's `sourceRepos` must permit the repo URL, and the app-of-apps `spec.project`
+must match.
+
+## 5. Let SSO users view apps (RBAC)
+
+With an empty `argocd-rbac-cm` `policy.default`, non-admin SSO/OIDC users see an **empty**
+Applications list. `admin` bypasses RBAC, so the apps exist but are invisible to SSO logins —
+which reads as "my apps disappeared" rather than as a permissions problem.
+
+```bash
+task argo-cd:0200-grant-sso-readonly   # sets policy.default (default role:readonly)
+```
+
+Idempotent; argocd-server reloads live, no restart needed.
+
+This is ArgoCD **install** configuration living on the cluster — not part of the GitOps repo,
+so it is neither rendered by Copier nor reconciled from Git.
+
+`role:readonly` grants **all** authenticated users read access to **all** applications. Scope
+a role to a specific SSO group when you need tighter control.
+
+## 6. Prerequisites the scripts do NOT handle
+
+- **ArgoCD must already be installed.** Preflight only *checks* that the namespace exists.
+- **The rendered repo must be pushed.** bootstrap-apply applies local manifests, but those
+  manifests point ArgoCD at a `repoURL` + `path` **in Git**. Unpushed → the app-of-apps
+  cannot find its path.
+- **The public deploy key must be on the repo**, read-only, for SSH auth to work.
+- **A successful render must exist first** — the target's bootstrap directory has to be on
+  disk. See `references/environments.md` for rendering an environment.
+- `kubectl` reachable at the configured kubeconfig and context.
+
+## 7. Bootstrap paths — two repository shapes
+
+The connect scripts reference the rendered repo's cluster folders by name, so those names are
+a contract between the template and the scripts. Two shapes are in use:
+
+| Purpose | `cluster/argocd/` shape (SKILL.md) | numbered shape |
+|---|---|---|
+| Bootstrap — AppProject + app-of-apps | root app applies it | `cluster/0000-bootstrap` |
+| Child Application manifests | `cluster/argocd/<NNN-name>/` | `cluster/0300-applications` |
+| Helm values / resources | `cluster/resources/<NNN-name>/` | `cluster/0500-resources` |
+
+The authoring rules in `SKILL.md` describe the first shape. Whichever a repo uses, preflight
+and bootstrap-apply must point at its **real** bootstrap directory — if the template renames
+cluster folders, update both step scripts in the same change.
+
+The deployed surface is only the manifests under paths an ArgoCD Application actually
+references.
+
+## 8. Verify without a cluster
+
+Render the repository Secret client-side to confirm project scoping before applying:
+
+```bash
+kubectl create secret generic <secret-name> \
+  --from-literal=type=git --from-literal=url=<repo-url> \
+  --from-literal=project=<appproject> \
+  --from-file=sshPrivateKey=<key-path> \
+  --dry-run=client -o yaml    # decode data.project → must equal the AppProject name
+```
+
+## 9. Post-onboard checks
+
+```bash
+kubectl -n <ns> get appproject <project>
+kubectl -n <ns> get applications
+kubectl -n <ns> get secret <repo-secret> -o jsonpath='{.data.project}' | base64 -d
+argocd app list    # if the argocd CLI is available
+```
+
+## 10. Anti-patterns
+
+- Committing the SSH private key instead of keeping it a local file referenced by path.
+- Letting `.env*` values reach the deployed `cluster/**` manifests — they are tooling inputs.
+- Adding a `:default` task wrapper; the env layers already supply values. Keep one env-driven
+  task and override with flags after `--`.
+- Hardcoding cluster folder names in step scripts without keeping them in sync with the
+  template (§7).
+- Keeping two connect modules. If a duplicate exists, prefer the wired one and delete the
+  other.