From 8f46c3faa5ba074b8a48971b87365e0d3e2a4c40 Mon Sep 17 00:00:00 2001 From: Oleg Lukasonok Date: Thu, 13 Aug 2026 00:10:03 +0300 Subject: [PATCH] =?UTF-8?q?feat:=20devsecops-ci-cd-gitea=20skill=20?= =?UTF-8?q?=E2=80=94=20Gitea=20Actions=20=E2=86=92=20Harbor=20pipelines?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generate and maintain Gitea Actions CI/CD pipelines that build & publish Docker images (applications-backend/*, applications-frontend/*) and OCI Helm charts (helm-charts/*) from a pnpm + Turborepo monorepo to a Harbor registry, authenticated by a project robot account via Gitea org secret + global vars. - SKILL.md: auth model, convention-driven discovery, publish gating, gotchas - references/harbor-auth.md: docker/helm login, robot accounts, secret/var taxonomy, bake-into-runner alternative, troubleshooting - references/pipeline-workflow.md: jobs, discovery loops, tagging, turbo prune - references/conventions.md: Dockerfile/chart locations, image & chart naming - assets/workflows/ci-cd.yaml: ready-to-drop .gitea/workflows pipeline Co-Authored-By: Claude Opus 4.8 --- README.md | 61 ++++++++++ SKILL.md | 160 ++++++++++++++++++++++++++ assets/workflows/ci-cd.yaml | 167 +++++++++++++++++++++++++++ references/conventions.md | 101 ++++++++++++++++ references/harbor-auth.md | 158 +++++++++++++++++++++++++ references/pipeline-workflow.md | 196 ++++++++++++++++++++++++++++++++ 6 files changed, 843 insertions(+) create mode 100644 README.md create mode 100644 SKILL.md create mode 100644 assets/workflows/ci-cd.yaml create mode 100644 references/conventions.md create mode 100644 references/harbor-auth.md create mode 100644 references/pipeline-workflow.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..397a44c --- /dev/null +++ b/README.md @@ -0,0 +1,61 @@ +# devsecops-ci-cd-gitea + +Agent Skill for **Gitea Actions** CI/CD pipelines that build and publish the artifacts of a +**pnpm + Turborepo** monorepo to a **Harbor** registry: + +```text +applications-backend//Dockerfile ─▶ image harbor//: +applications-frontend//Dockerfile ─▶ image harbor//: +helm-charts//Chart.yaml ─▶ OCI chart harbor//charts/: +``` + +Discovery is convention-driven — the pipeline finds every app with a `Dockerfile` and every +chart under `helm-charts/`, builds them, and pushes only on the default branch or a tag. +Authentication uses a Harbor **robot account** whose credential lives in Gitea Actions +secrets/variables — never a person's login, never a value committed to the repo. + +The skill owns the pipeline up to *"published to Harbor"*. Deploying the artifact is the +**development-gitops-argo-cd** skill's job. + +## Layout + +```text +devsecops-ci-cd-gitea/ +├── SKILL.md # Entry point — auth model, discovery, gating, gotchas +├── references/ +│ ├── harbor-auth.md # docker/helm login, robot accounts, Gitea vars/secrets, runner-bake +│ ├── pipeline-workflow.md # jobs, discovery loops, tagging, gating, turbo prune +│ └── conventions.md # where Dockerfiles/charts live, image & chart naming +└── assets/ + └── workflows/ + └── ci-cd.yaml # ready-to-drop /.gitea/workflows/ci-cd.yaml +``` + +## The one thing to get right up front + +**Build context is the repository root**, always: +`docker build -f applications-backend//Dockerfile … .` — the Dockerfile runs `turbo prune` +over the whole monorepo, so building from the app directory fails. And charts are **centralized +under `helm-charts/`**, not co-located under the app — discover images and charts independently, +with no assumed 1:1 mapping. + +## Registry auth in one table + +| Kind | Name | Scope | Example | +|---|---|---|---| +| Variable | `HL_V1_HARBOR_ADDRESS` | global | `harbor-v1.apps.lego-cloud.eu` | +| Variable | `HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME` | global | `robot$gondor-v1+gitea-actions-v1` | +| Secret | `HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET` | org | *(robot secret)* | + +Username and address are not secret → global variables (set once). Gitea has no global-secret +level, so the robot secret is a per-org secret. Details in `references/harbor-auth.md`. + +## Deploy + +```bash +cd ../skill-manager/scripts +task deploy -- --skill-dir="$(cd ../../devsecops-ci-cd-gitea && pwd)" +``` + +Built with the `skill-manager` skill, following the +[agentskills.io specification](https://agentskills.io/specification). diff --git a/SKILL.md b/SKILL.md new file mode 100644 index 0000000..0265c15 --- /dev/null +++ b/SKILL.md @@ -0,0 +1,160 @@ +--- +name: devsecops-ci-cd-gitea +description: > + Generate and maintain Gitea Actions CI/CD pipelines that build & publish Docker images and + Helm charts for every application in a pnpm + Turborepo monorepo, pushing both to a Harbor + registry. Use when adding or fixing a `.gitea/workflows/*` pipeline; building or pushing + container images from `applications-backend//Dockerfile` or + `applications-frontend//Dockerfile`; packaging and pushing OCI Helm charts from + `helm-charts//`; wiring `docker login` / `helm registry login` against Harbor with a + robot account; setting the Gitea Actions secrets and variables a pipeline needs; or debugging + a runner that can't authenticate, can't find Docker, or can't reach an action — even when the + user only says "the CI/CD thing", "the docker pipeline", or names their monorepo. +license: Proprietary +metadata: + author: home-v1-skills-code-agent + version: "1.0" + spec: agentskills.io/specification +compatibility: > + Targets Gitea Actions (act_runner, GitHub-Actions-compatible) pushing to a Harbor OCI + registry, for pnpm + Turborepo monorepos (see the development-monorepo-pnpm skill). Requires + a runner whose jobs can run docker and helm. Designed for Claude Code, Cline, GitHub Copilot, + OpenAI Codex, and other compatible agents. +--- + +# DevSecOps CI/CD — Gitea Actions → Harbor + +Generate the pipeline that turns a monorepo push into **published images and charts**: for +every app that has a `Dockerfile`, build and push its image; for every chart under +`helm-charts/`, package and push it as an OCI artifact. Both land in **Harbor**, authenticated +by a project **robot account** whose credential lives in Gitea Actions secrets/variables. + +This skill owns the pipeline up to *"the artifact is in Harbor"*. Deploying that artifact to +the cluster is the **development-gitops-argo-cd** skill's job — see [Handoff](#handoff). + +## When to use — and which reference to open + +| The task | Read | +|---|---| +| Wire or fix registry login (`docker login` / `helm registry login`), robot accounts, the Gitea secret/variable names, or the "bake into the runner" alternative | `references/harbor-auth.md` | +| Understand or edit the pipeline itself — jobs, discovery loops, tag derivation, PR-vs-publish gating, `turbo prune` | `references/pipeline-workflow.md` | +| Confirm where Dockerfiles and charts live, how image/chart names are derived, which Harbor project to push to | `references/conventions.md` | +| Drop a working pipeline into a repo | copy `assets/workflows/ci-cd.yaml` → `/.gitea/workflows/ci-cd.yaml` | + +## The mental model + +```text +git push ─▶ act_runner ─▶ [ discover apps ] + ├─ applications-backend//Dockerfile ─▶ docker build (repo root) ─▶ push harbor//: + ├─ applications-frontend//Dockerfile ─▶ docker build (repo root) ─▶ push harbor//: + └─ helm-charts//Chart.yaml ─▶ helm package ─▶ push oci://harbor//charts +``` + +Everything the runner needs to authenticate comes from **one org secret + two global +variables** (below). Nothing sensitive is committed to the repo. + +## 1. Registry authentication (the "docker thing") + +Harbor push/pull uses a **project-scoped robot account** — a non-human account with a long +username and a generated secret. The pipeline logs in with it; it never uses a person's +credentials. Full model and how the robot is created (palantir `harbor:robot:*` tasks) → +`references/harbor-auth.md`. + +The pipeline reads exactly three values: + +| Kind | Name | Scope in Gitea | Example value | +|---|---|---|---| +| Variable | `HL_V1_HARBOR_ADDRESS` | **global** (Site Admin → Actions → Variables) | `harbor-v1.apps.lego-cloud.eu` | +| Variable | `HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME` | **global** | `robot$gondor-v1+gitea-actions-v1` | +| Secret | `HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET` | **org** (per org, Settings → Actions → Secrets) | *(the robot's generated secret)* | + +Why this split: the username and address are **not secret**, so they go in *variables* — and +variables have a **global** level, so you set them once for all organisations. Gitea has **no +global-secret level**, so the one genuinely sensitive value is an **org secret** — the single +per-org chore when you onboard a new org. + +Login, in every job that touches the registry: + +```bash +echo "${HARBOR_SECRET}" | docker login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin +echo "${HARBOR_SECRET}" | helm registry login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin +``` + +> **Alternative — bake the login into the runner.** If you would rather not attach a secret per +> org at all, pre-seed the runner's Docker `config.json` so every job is pre-authenticated. It +> trades central rotation for zero per-org setup. Steps in `references/harbor-auth.md` §4. + +## 2. What gets built, and from where + +Discovery is **convention-driven** — the pipeline finds work, you don't list it. Details and +the naming rules in `references/conventions.md`; the essentials: + +- **Images**: one per `applications-backend/*/Dockerfile` and `applications-frontend/*/Dockerfile`. + The image repository is the **application directory name**. Build context is the + **repository root** (`docker build -f /Dockerfile … .`) because the Dockerfile runs + `turbo prune` over the whole monorepo — building from the app directory **will fail**. +- **Charts**: one per `helm-charts/*/Chart.yaml`. Charts are **centralized under `helm-charts/` + at the repo root**, *not* co-located inside the app directory. Do not look for + `applications-frontend//helm` — that is not where charts live in this monorepo family. +- **Tag**: the git tag on a tag build, otherwise the short commit SHA. On the default branch the + image also gets a moving `latest`. See `references/pipeline-workflow.md` §Tagging. + +## 3. Publish gating — build always, push selectively + +| Trigger | Images | Charts | +|---|---|---| +| pull request / feature branch | build only (verify it compiles) | `helm lint` + `helm package` (verify) | +| push to default branch (`test`) or `main` | build **and push** | package **and push** | +| tag `v*` | build **and push** (tagged with the git tag) | package **and push** | + +A PR must never publish — it proves the artifact builds, nothing more. The template computes a +`PUBLISH` flag from the event; do not remove it. + +## 4. Installing the pipeline into a monorepo + +1. Copy `assets/workflows/ci-cd.yaml` → `/.gitea/workflows/ci-cd.yaml`. +2. Set `runs-on:` to your runner's label (see Gotchas — it must be a label whose jobs provide + `docker`, `helm`, and `git`). Set `HL_V1_HARBOR_PROJECT` (default `gondor-v1`) to the Harbor + project the robot can **push** to. +3. Ensure the three Gitea values from §1 exist (2 global variables once; 1 org secret per org). +4. Commit, push a throwaway branch, and confirm the run **builds without pushing**. +5. Merge to the default branch and confirm images + charts appear in Harbor. + +## Handoff + +This skill stops at *published to Harbor*. To roll the new image/chart into the cluster, hand +the image reference and tag to **development-gitops-argo-cd**: the tag flows through the Copier +variables into the Helm values Argo CD syncs. Never wire a raw `kubectl apply` into this +pipeline — deployment is GitOps's responsibility, and a pipeline that also deploys bypasses the +one source of truth for cluster state. + +## Gotchas + +- **Build context is the repo root, always.** `docker build -f applications-backend//Dockerfile … .` + The Dockerfile's `COPY . .` + `turbo prune` need the whole monorepo. Building from the app + directory fails with missing lockfile / workspace errors. +- **Charts are centralized, not co-located.** They live in `helm-charts//`. An app can + ship an image with no chart, and a chart can exist with no matching app — discover the two + independently, do not assume a 1:1 pairing. +- **Gitea has no global secrets — only global variables.** The username/address are variables + (global, set once); the robot secret is an org secret (per org). Do not try to create a + "global secret"; the admin Actions page only offers global *variables*. +- **The robot secret is only shown once, at creation.** Harbor never reveals it again on GET. + It's cached under palantir `.harbor/-.yml` (git-ignored). To rotate, re-run + the palantir robot task, then update the Gitea org secret. See `references/harbor-auth.md`. +- **`runs-on` maps to a container image — it must have `docker`, `helm`, `git`.** `turbo prune` + runs *inside* the image build, so the job itself needs Node **only if** you add pre-build + steps; the default template needs just docker + helm + git. A bare `node:alpine` label has no + docker CLI and the build step will fail with `docker: not found`. +- **Marketplace actions may not resolve.** Gitea proxies `actions/*` to github.com by default; + if the runner has no egress, `actions/checkout` fails. The template keeps action use to + `actions/checkout@v4` only and documents a `git clone` fallback in + `references/pipeline-workflow.md`. +- **Frontend build args are public.** A frontend image bakes its build-time env into the + bundle. Never pass a secret as a `--build-arg`; anything per-environment on the frontend is + fetched at runtime, not built in. +- **`helm push` needs an OCI-enabled Helm ≥ 3.8 and a prior `helm registry login`.** Without + the login step, `helm push` fails with `unauthorized` even though `docker login` succeeded — + they are separate credential stores. +- **Never `docker login` with `-p` on the command line.** Use `--password-stdin`; a `-p` + argument lands in the process list and the runner logs. diff --git a/assets/workflows/ci-cd.yaml b/assets/workflows/ci-cd.yaml new file mode 100644 index 0000000..e28bbaf --- /dev/null +++ b/assets/workflows/ci-cd.yaml @@ -0,0 +1,167 @@ +# Drop-in Gitea Actions pipeline for a pnpm + Turborepo monorepo. +# Copy to /.gitea/workflows/ci-cd.yaml. +# +# It discovers every app with a Dockerfile and every chart under helm-charts/, builds them, +# and — only on the default branch or a tag — pushes images and OCI Helm charts to Harbor. +# +# Prerequisites (see the devsecops-ci-cd-gitea skill): +# Gitea GLOBAL variables (Site Admin -> Actions -> Variables), set once for all orgs: +# HL_V1_HARBOR_ADDRESS e.g. harbor-v1.apps.lego-cloud.eu +# HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME e.g. robot$gondor-v1+gitea-actions-v1 +# HL_V1_HARBOR_PROJECT (optional; defaults to gondor-v1 below) +# Gitea ORG secret (per org, Settings -> Actions -> Secrets): +# HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET +# +# TODO before first run: +# - set `runs-on` to your runner's label (its jobs must provide docker, helm, git) +name: ci-cd + +on: + push: + branches: [test, main] + tags: ["v*"] + pull_request: + branches: [test, main] + +env: + HARBOR_ADDRESS: ${{ vars.HL_V1_HARBOR_ADDRESS }} + HARBOR_USERNAME: ${{ vars.HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME }} + HARBOR_SECRET: ${{ secrets.HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET }} + HARBOR_PROJECT: ${{ vars.HL_V1_HARBOR_PROJECT || 'gondor-v1' }} + +jobs: + build-and-push-images: + runs-on: [self-hosted] # TODO: set to your runner label (needs docker + git) + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Compute tag and publish flag + id: meta + run: | + set -euo pipefail + if [ "${GITHUB_REF_TYPE}" = "tag" ]; then + TAG="${GITHUB_REF_NAME}" + else + TAG="$(git rev-parse --short HEAD)" + fi + if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then + PUBLISH=false + elif [ "${GITHUB_REF_TYPE}" = "tag" ] \ + || [ "${GITHUB_REF_NAME}" = "test" ] \ + || [ "${GITHUB_REF_NAME}" = "main" ]; then + PUBLISH=true + else + PUBLISH=false + fi + echo "tag=${TAG}" >> "$GITHUB_OUTPUT" + echo "publish=${PUBLISH}" >> "$GITHUB_OUTPUT" + echo "Tag=${TAG} Publish=${PUBLISH}" + + - name: Log in to Harbor (publish runs only) + if: steps.meta.outputs.publish == 'true' + run: | + set -euo pipefail + echo "${HARBOR_SECRET}" | docker login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin + + - name: Build and push images + env: + TAG: ${{ steps.meta.outputs.tag }} + PUBLISH: ${{ steps.meta.outputs.publish }} + run: | + set -euo pipefail + shopt -s nullglob + found=0 + for df in applications-backend/*/Dockerfile applications-frontend/*/Dockerfile; do + found=1 + app="$(basename "$(dirname "$df")")" + image="${HARBOR_ADDRESS}/${HARBOR_PROJECT}/${app}" + echo "::group::${app}" + # Build context is the REPOSITORY ROOT — the Dockerfile runs turbo prune / COPY . . + docker build -f "$df" -t "${image}:${TAG}" . + if [ "${PUBLISH}" = "true" ]; then + docker push "${image}:${TAG}" + # moving :latest on branch builds only (never on tags) + if [ "${GITHUB_REF_TYPE}" != "tag" ]; then + docker tag "${image}:${TAG}" "${image}:latest" + docker push "${image}:latest" + fi + else + echo "PUBLISH=false — built ${image}:${TAG}, not pushing" + fi + echo "::endgroup::" + done + [ "${found}" = "1" ] || echo "No Dockerfiles found under applications-backend/* or applications-frontend/*" + + - name: Log out + if: always() && steps.meta.outputs.publish == 'true' + run: docker logout "${HARBOR_ADDRESS}" || true + + package-and-push-charts: + runs-on: [self-hosted] # TODO: set to your runner label (needs helm + git) + needs: build-and-push-images + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Compute publish flag + id: meta + run: | + set -euo pipefail + if [ "${GITHUB_REF_TYPE}" = "tag" ]; then + TAG="${GITHUB_REF_NAME#v}" + else + TAG="" + fi + if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then + PUBLISH=false + elif [ "${GITHUB_REF_TYPE}" = "tag" ] \ + || [ "${GITHUB_REF_NAME}" = "test" ] \ + || [ "${GITHUB_REF_NAME}" = "main" ]; then + PUBLISH=true + else + PUBLISH=false + fi + echo "tag=${TAG}" >> "$GITHUB_OUTPUT" + echo "publish=${PUBLISH}" >> "$GITHUB_OUTPUT" + + - name: Log in to Harbor (publish runs only) + if: steps.meta.outputs.publish == 'true' + run: | + set -euo pipefail + echo "${HARBOR_SECRET}" | helm registry login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin + + - name: Lint, package and push charts + env: + TAG: ${{ steps.meta.outputs.tag }} + PUBLISH: ${{ steps.meta.outputs.publish }} + run: | + set -euo pipefail + shopt -s nullglob + mkdir -p .cicd-charts + found=0 + for chart in helm-charts/*/Chart.yaml; do + found=1 + dir="$(dirname "$chart")" + echo "::group::$(basename "$dir")" + helm lint "$dir" + if [ -n "${TAG}" ]; then + # On a tag build, align chart + app version with the release tag. + helm package "$dir" -d .cicd-charts --version "${TAG}" --app-version "${TAG}" + else + helm package "$dir" -d .cicd-charts + fi + echo "::endgroup::" + done + [ "${found}" = "1" ] || { echo "No charts under helm-charts/*"; exit 0; } + if [ "${PUBLISH}" = "true" ]; then + for tgz in .cicd-charts/*.tgz; do + helm push "$tgz" "oci://${HARBOR_ADDRESS}/${HARBOR_PROJECT}/charts" + done + else + echo "PUBLISH=false — packaged charts, not pushing" + fi + + - name: Log out + if: always() && steps.meta.outputs.publish == 'true' + run: helm registry logout "${HARBOR_ADDRESS}" || true diff --git a/references/conventions.md b/references/conventions.md new file mode 100644 index 0000000..8fd8340 --- /dev/null +++ b/references/conventions.md @@ -0,0 +1,101 @@ +# Discovery conventions + +Where the pipeline looks for work, and how names are derived. These conventions come from the +**development-monorepo-pnpm** skill (`references/containers-and-helm.md`) — this file restates +only what the pipeline depends on. If the monorepo skill and this file ever disagree, the +monorepo skill wins; update this one. + +--- + +## Repository shape the pipeline assumes + +```text +/ +├── applications-backend//Dockerfile # deployable services (NestJS by default) +├── applications-frontend//Dockerfile # deployable browser apps (React + Carbon) +├── helm-charts//Chart.yaml # ALL charts — centralized, not per-app +├── pnpm-workspace.yaml +├── turbo.json +└── pnpm-lock.yaml +``` + +Two independent discovery sets: + +| Artifact | Glob | Produced by | +|---|---|---| +| Image | `applications-backend/*/Dockerfile`, `applications-frontend/*/Dockerfile` | `docker build` from repo root | +| Chart | `helm-charts/*/Chart.yaml` | `helm package` | + +### Charts are centralized — do not look under the app directory + +A frequent wrong assumption is that a chart lives at `applications-frontend//helm/` or +`.../chart/`. It does **not** in this monorepo family. Charts are collected under +`helm-charts//` at the repo root. Practical consequences: + +- Discover images and charts **separately**; there is **no enforced 1:1** mapping. + - An app may have a `Dockerfile` and **no** chart (deployed by a shared/umbrella chart, or + not yet deployable). + - A chart may exist with **no** same-named app (an umbrella chart, or a chart that deploys a + third-party image). +- Do not infer a chart's existence from an app's, or vice versa. Loop each glob on its own. + +--- + +## Naming + +### Image repository = application directory name + +``` +${HARBOR_ADDRESS}/${HARBOR_PROJECT}/: +# e.g. harbor-v1.apps.lego-cloud.eu/gondor-v1/user-service:9f3a1c2 +``` + +- The **directory name** under `applications-backend/` or `applications-frontend/` is the image + repository name. It is not derived from the package.json `name` / scope — using the directory + name keeps the image path stable and needs no Node in the job. +- Backend and frontend images share the flat `/` namespace. If two apps across the + two roots ever share a directory name they would collide — keep directory names unique across + both roots (they already must be, since pnpm workspace names must be unique). + +### Chart repository = chart name, under a `charts` subpath + +``` +oci://${HARBOR_ADDRESS}/${HARBOR_PROJECT}/charts # push target (base) +# a chart named "user-service" becomes: +# ${HARBOR_ADDRESS}/${HARBOR_PROJECT}/charts/user-service: +``` + +Charts are pushed under a `charts` subpath so OCI chart artifacts and image artifacts do not +share the project's top-level namespace. The chart name is the `name:` field in `Chart.yaml` +(which should match its directory name). + +### Harbor project + +`HARBOR_PROJECT` (default `gondor-v1`) is the Harbor project everything is pushed into. It must +be a project the robot has **push** on. One project per monorepo is a clean default; if you host +several monorepos, either give each its own project (and a per-project push robot) or accept a +shared project with distinct `` names. + +--- + +## Tag → deploy handoff + +The pipeline publishes `:` (immutable) and, on the default branch, `:latest` (moving). The +**immutable** `:` is what a GitOps deployment should pin. This skill does not edit +deployment values — it hands the image reference and tag to **development-gitops-argo-cd**, where +the tag flows through Copier variables into the Helm values Argo CD syncs. + +--- + +## Quick audit of a repo before generating a pipeline + +```bash +echo "images:"; ls -d applications-backend/*/ applications-frontend/*/ 2>/dev/null \ + | while read d; do [ -f "$d/Dockerfile" ] && echo " $d"; done +echo "charts:"; ls -d helm-charts/*/ 2>/dev/null \ + | while read d; do [ -f "$d/Chart.yaml" ] && echo " $d"; done +``` + +If either list is empty, confirm you are at the repo root and that the monorepo actually follows +this layout before wiring the pipeline — an empty discovery set means the workflow will run +green while building nothing. diff --git a/references/harbor-auth.md b/references/harbor-auth.md new file mode 100644 index 0000000..a9afe2c --- /dev/null +++ b/references/harbor-auth.md @@ -0,0 +1,158 @@ +# Harbor authentication for Gitea Actions + +How a pipeline authenticates to the Harbor registry to **pull base images and push built +images/charts** — the robot account model, the exact Gitea secret/variable layout, the login +commands, and the "bake into the runner" alternative. + +--- + +## 1. Robot accounts — why, and what they look like + +CI must never push with a person's Harbor credentials. Harbor's answer is a **robot account**: +a non-human account scoped to a **project**, with a fixed long username and a **generated** +secret. + +- **Username** (public, not a secret): `robot$+` — e.g. + `robot$gondor-v1+gitea-actions-v1`. The `$` is part of the literal string, not shell syntax. +- **Secret** (sensitive): a generated token shown **once, at creation**. Harbor never returns + it again on a GET. If you lose it, you regenerate (which invalidates the old one). +- **Scope**: project-level. A robot with `push`/`pull` on project `gondor-v1` can only act + within `gondor-v1`. To publish images/charts to a project, the robot must have **push** there. + +In this platform the robot is created and cached by the **palantir** `harbor` module — do not +create robots by hand in the UI: + +```bash +# from the palantir-v1 repo, with KUBECONFIG set +task harbor:project:assure-one -- --name=gondor-v1 --public=false +task harbor:robot:gitea-actions-v1:assure-one # pull,push -> .harbor/gondor-v1-gitea-actions-v1.yml +task harbor:robot:gondor-v1:assure-one # pull -> used by the cluster's imagePullSecret +``` + +The secret is cached in git-ignored `.harbor/-.yml`. Read it with: + +```bash +yq '.secret' .harbor/gondor-v1-gitea-actions-v1.yml +``` + +There are two robots with different jobs — do not mix them up: + +| Robot | Permissions | Used by | Credential lands in | +|---|---|---|---| +| `gitea-actions-v1` | pull, **push** | Gitea Actions (this skill) | Gitea org secret (below) | +| `gondor-v1` | pull only | cluster `imagePullSecret` (Argo CD / workloads) | Bitwarden → ESO `ClusterExternalSecret` | + +--- + +## 2. The Gitea secret/variable layout + +Gitea Actions has two stores — **variables** (plaintext, visible) and **secrets** (masked) — +each available at repo, org, and (variables only) **global/admin** level: + +| | Repo | Org / User | Global (admin) | +|---|---|---|---| +| Variables | ✅ | ✅ | ✅ `…/-/admin/actions/variables` | +| Secrets | ✅ | ✅ | ❌ **no global level** | + +Because the username and address are **not secret**, put them in **global variables** — set +once, seen by every organisation. The robot secret is the only value that must be a secret, and +since there is no global secret, it is an **org secret** (repeat per org you onboard). + +**Global variables** — Site Administration → Actions → Variables, once: + +| Name | Value | +|---|---| +| `HL_V1_HARBOR_ADDRESS` | `harbor-v1.apps.lego-cloud.eu` | +| `HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME` | `robot$gondor-v1+gitea-actions-v1` | + +Optionally also global: `HL_V1_HARBOR_PROJECT` = `gondor-v1` (the push target; the template +defaults to `gondor-v1` if unset). + +**Org secret** — `https://gitea.lego-cloud.eu/org//settings/actions/secrets`, per org: + +| Name | Value | +|---|---| +| `HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET` | output of `yq '.secret' .harbor/gondor-v1-gitea-actions-v1.yml` | + +Referencing them in a workflow: + +```yaml +env: + HARBOR_ADDRESS: ${{ vars.HL_V1_HARBOR_ADDRESS }} + HARBOR_USERNAME: ${{ vars.HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME }} + HARBOR_SECRET: ${{ secrets.HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET }} + HARBOR_PROJECT: ${{ vars.HL_V1_HARBOR_PROJECT || 'gondor-v1' }} +``` + +--- + +## 3. Logging in — docker and helm are separate credential stores + +```bash +# container images +echo "${HARBOR_SECRET}" | docker login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin + +# OCI helm charts — a DIFFERENT store; docker login does not cover it +echo "${HARBOR_SECRET}" | helm registry login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin +``` + +- Always `--password-stdin`. A `-p ` argument leaks into the process list and runner + logs. +- `helm registry login` is required in addition to `docker login`; `helm push` uses Helm's own + registry credential store and will return `unauthorized` without it, even right after a + successful `docker login`. +- Log out at the end of the job if the runner is long-lived and shared: + `docker logout "${HARBOR_ADDRESS}"; helm registry logout "${HARBOR_ADDRESS}"`. + +**Rotation.** Re-run the palantir robot task to regenerate the secret (`.harbor/…` is +re-cached), then update the Gitea **org secret** with the new value. The username and address +do not change, so the global variables stay put. + +--- + +## 4. Alternative — bake the login into the runner + +If attaching an org secret to every organisation is more chore than you want, pre-authenticate +the **runner** itself so every job — in every org — can pull/push Harbor with **no** Gitea +secret referenced. + +Because act_runner runs each job in a container that talks to the host Docker daemon over the +mounted socket, `docker` inside a job reads *its own* config, not the host's. So seed a +`config.json` into every job container: + +1. Build the auth file locally (secret never printed to the terminal): + ```bash + U='robot$gondor-v1+gitea-actions-v1' + S=$(yq '.secret' .harbor/gondor-v1-gitea-actions-v1.yml) + AUTH=$(printf '%s:%s' "$U" "$S" | base64 | tr -d '\n') + umask 077 + printf '{ "auths": { "harbor-v1.apps.lego-cloud.eu": { "auth": "%s" } } }\n' "$AUTH" \ + > /tmp/harbor-docker-config.json + ``` +2. Copy it onto the NAS to a host path the runner can mount, e.g. + `/mnt//gitea-runner-harbor/config.json`, then `rm /tmp/harbor-docker-config.json`. +3. Wire it into the runner: + - **TrueNAS SCALE app**: Edit → add a read-only Host Path Volume + (`…/gitea-runner-harbor` → `/opt/harbor-docker`) and env `DOCKER_CONFIG=/opt/harbor-docker`. + - **Raw act_runner `config.yaml`**: under `container:` add + `options: "-v /mnt//gitea-runner-harbor:/opt/harbor-docker:ro -e DOCKER_CONFIG=/opt/harbor-docker"`, + then restart. +4. Jobs now run `docker pull/push harbor-v1.apps.lego-cloud.eu/...` with no secret reference. + +**Trade-off.** Zero per-org setup and it covers all orgs, but the Harbor secret is duplicated +onto the NAS and rotating it means rewriting that file. It also does **not** cover `helm push` +(Helm has its own store) — for charts you still `helm registry login` in the job, or add a +second baked Helm credential. Prefer the org-secret model (§2) when you want one source of +truth and central rotation. + +--- + +## 5. Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| `docker login` → `unauthorized: unauthorized to access repository` | wrong robot secret, or robot lacks push on the project | re-read `.harbor/…`, confirm the org secret matches; confirm the robot has push on `HARBOR_PROJECT` | +| `helm push` → `unauthorized` right after a good `docker login` | missing `helm registry login` | add the helm login step | +| `docker: not found` in the job | runner label maps to an image without the docker CLI | pick a label/image that has docker (and helm) | +| `denied: requested access to the resource is denied` on push | pushing to a project the robot can't write, or a typo in `HARBOR_PROJECT` | set `HARBOR_PROJECT` to the robot's project | +| login works locally, fails in CI | job can't reach Harbor (edge/DNS) or cert not trusted | confirm the runner resolves and trusts `harbor-v1.apps.lego-cloud.eu` | diff --git a/references/pipeline-workflow.md b/references/pipeline-workflow.md new file mode 100644 index 0000000..7ad0d52 --- /dev/null +++ b/references/pipeline-workflow.md @@ -0,0 +1,196 @@ +# The pipeline, explained + +Anatomy of the generated Gitea Actions workflow (`assets/workflows/ci-cd.yaml`) — the jobs, the +discovery loops, tag derivation, publish gating, and the `turbo prune` build model. Read this +when editing the workflow or debugging a run. + +--- + +## Jobs + +The workflow has two jobs; charts wait for images so a run doesn't publish a chart for an image +that failed to build: + +``` +build-and-push-images ─▶ package-and-push-charts (needs: build-and-push-images) +``` + +Both begin the same way: `actions/checkout`, then a **login** step (§Harbor login), then a +single **discover + act** shell step that loops over the convention paths. + +Why one shell step per job instead of an Actions `matrix`: discovery is dynamic (the set of +apps changes as the monorepo grows) and a shell loop needs no round-trip to compute a matrix, +no dependency on a marketplace action, and prints a clean per-item log. If you later want +parallelism across apps, split discovery into a `prepare` job that emits a matrix via +`GITHUB_OUTPUT` — but keep the loop as the default; it is the least fragile. + +--- + +## Triggers and publish gating + +```yaml +on: + push: + branches: [test, main] + tags: ["v*"] + pull_request: + branches: [test, main] +``` + +A single computed flag decides push vs. build-only. **Do not remove it** — a PR must prove the +artifact builds without publishing anything: + +```bash +# PUBLISH=true only on a push to a release branch or a tag; never on a pull_request +if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then + PUBLISH=false +elif [ "${GITHUB_REF_TYPE}" = "tag" ]; then + PUBLISH=true +elif [ "${GITHUB_REF_NAME}" = "test" ] || [ "${GITHUB_REF_NAME}" = "main" ]; then + PUBLISH=true +else + PUBLISH=false +fi +``` + +| Event | Images | Charts | +|---|---|---| +| `pull_request` | `docker build` (no push) | `helm lint` + `helm package` (no push) | +| push to `test` / `main` | build **and** push `:` + `:latest` | package **and** push | +| tag `v*` | build **and** push `:` | package **and** push (optionally `--version`/`--app-version` = tag) | + +--- + +## Tagging + +```bash +if [ "${GITHUB_REF_TYPE}" = "tag" ]; then + TAG="${GITHUB_REF_NAME}" # e.g. v1.4.2 +else + TAG="$(git rev-parse --short HEAD)" # e.g. 9f3a1c2 +fi +``` + +- On the default branch, images are pushed with **both** `:` and a moving `:latest`. The + immutable `:` is what GitOps should pin; `:latest` is a convenience, never a deploy + source. +- Chart version comes from `Chart.yaml`. On a tag build you may align it: + `helm package --version "${TAG#v}" --app-version "${TAG#v}"`. Keep chart `version` + (chart packaging) and `appVersion` (the image it deploys) conceptually distinct. +- Never hardcode a tag in a Helm template or values file committed to the repo — the tag is a + pipeline output that flows into GitOps values. See the monorepo skill's `containers-and-helm.md`. + +--- + +## The `turbo prune` build model — why context is the repo root + +Each app's Dockerfile is **multi-stage** and runs `turbo prune --scope=@/ --docker` +*inside* the build to isolate that app plus its workspace dependencies, then installs from the +pruned `out/json/` before copying `out/full/`. That is what keeps image builds cacheable and +independent across the monorepo. + +The consequence for the pipeline: the **build context must be the repository root**, and the +Dockerfile is selected with `-f`: + +```bash +docker build -f "applications-backend//Dockerfile" -t "${IMAGE}:${TAG}" . +# ^ context = repo root +``` + +Building with the app directory as context (`docker build applications-backend/`) fails — +`turbo prune` and `COPY . .` cannot see the workspace or the lockfile. This is the single most +common mistake; the template always builds from `.`. + +--- + +## Discovery loops + +Images: + +```bash +shopt -s nullglob +for df in applications-backend/*/Dockerfile applications-frontend/*/Dockerfile; do + app="$(basename "$(dirname "$df")")" + image="${HARBOR_ADDRESS}/${HARBOR_PROJECT}/${app}" + docker build -f "$df" -t "${image}:${TAG}" . + if [ "${PUBLISH}" = "true" ]; then + docker push "${image}:${TAG}" + if [ "${GITHUB_REF_TYPE}" != "tag" ]; then + docker tag "${image}:${TAG}" "${image}:latest" && docker push "${image}:latest" + fi + fi +done +``` + +Charts (OCI push to a `charts` subpath so charts and images don't collide in the project +namespace): + +```bash +shopt -s nullglob +mkdir -p .cicd-charts +for chart in helm-charts/*/Chart.yaml; do + dir="$(dirname "$chart")" + helm lint "$dir" + helm package "$dir" -d .cicd-charts +done +if [ "${PUBLISH}" = "true" ]; then + for tgz in .cicd-charts/*.tgz; do + helm push "$tgz" "oci://${HARBOR_ADDRESS}/${HARBOR_PROJECT}/charts" + done +fi +``` + +`nullglob` matters: without it, a repo with no frontend apps (or no charts) would iterate over +the literal unexpanded glob and try to build a file named `applications-frontend/*/Dockerfile`. + +--- + +## Harbor login step + +```yaml +- name: Log in to Harbor + run: | + echo "${HARBOR_SECRET}" | docker login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin + echo "${HARBOR_SECRET}" | helm registry login "${HARBOR_ADDRESS}" -u "${HARBOR_USERNAME}" --password-stdin + env: + HARBOR_ADDRESS: ${{ vars.HL_V1_HARBOR_ADDRESS }} + HARBOR_USERNAME: ${{ vars.HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_USERNAME }} + HARBOR_SECRET: ${{ secrets.HL_V1_HARBOR_ROBOT_GITEA_ACTIONS_V1_SECRET }} +``` + +On a `pull_request` the secret may be absent (org secrets are not exposed to fork PRs). The +template guards the login: it only logs in when `PUBLISH=true`, so PR builds still verify +without needing the credential. Base images pulled during the build must therefore be public, +or the build itself must run on a trusted (non-fork) branch. + +--- + +## `runs-on` and marketplace actions + +- `runs-on:` is a **runner label**, and each label maps to a container image. That image must + provide `docker`, `helm`, and `git`. The template ships `runs-on: [self-hosted]` — change it + to your registered label. +- The only marketplace action used is `actions/checkout@v4`. Gitea proxies `actions/*` to + github.com; if the runner has no egress, either mirror the action into your Gitea instance or + replace checkout with a manual clone: + ```yaml + - run: | + git init -q . + git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git" + git -c http.extraheader="AUTHORIZATION: bearer ${GITHUB_TOKEN}" fetch --depth=1 origin "${GITHUB_SHA}" + git checkout -q FETCH_HEAD + env: + GITHUB_TOKEN: ${{ secrets.GITEA_TOKEN }} + ``` + +--- + +## Verifying a change to the pipeline + +1. Push to a throwaway feature branch → confirm the run **builds** images and **lints/packages** + charts, and pushes **nothing** (`PUBLISH=false`). +2. Read the log for each app and chart — a silently skipped app means the glob didn't match + (wrong directory, missing `Dockerfile`/`Chart.yaml`). +3. Merge to `test` → confirm `:` and `:latest` images and the chart appear in Harbor under + the project. +4. Tag `v*` → confirm the tagged image and chart version.