49 lines
8.2 KiB
Markdown
49 lines
8.2 KiB
Markdown
# Channel Wiki: #jarvis-jr-v1-hermes-setup
|
||
_Channel ID: 1526844602303123466_
|
||
_Created: 2026-07-23 18:22 UTC_
|
||
_Last sync: 2026-08-11 04:15 UTC_
|
||
|
||
## Purpose
|
||
Hermes installation, persistent runtime configuration, secret management, and setup troubleshooting.
|
||
|
||
## Key Decisions
|
||
- **CONFIRMED — Gitea credential (2026-08-02):** authenticated operations against `gitea.lego-cloud.eu` must use the Bitwarden Secrets Manager key/environment variable `HL_V1_GITEA_ACCESS_TOKEN`. Do **not** use `GITHUB_TOKEN` for this Gitea instance and do not copy the Gitea token into `/opt/data/.env`.
|
||
- **CONFIRMED — secret naming:** Hermes exports Bitwarden secret keys exactly as named; it does not create aliases. The briefly reported `HL_V1_GITEA_TOKEN` alias was stale cache information and was explicitly corrected.
|
||
- **CONFIRMED — restart semantics:** `/reset` only resets a conversation and does not reload process environment. A gateway restart is required after Bitwarden keys change so the gateway imports them.
|
||
- **CONFIRMED — outage investigation constraint (2026-08-11):** Lego required a deep analysis of the multi-day Hermes outage, with no host changes, and requested a report by 11:00 UTC on 2026-08-11. Hermes completed and delivered the strictly read-only report at 00:36 UTC; no files, configuration, services, databases, or processes were changed by that investigation.
|
||
|
||
## Active Topics
|
||
- **Bitwarden setup:** enabled and operational. Live verification at sync time showed project access, `bws 2.0.0`, and one applied key: `HL_V1_GITEA_ACCESS_TOKEN` (value never exposed).
|
||
- **VERIFIED:** after gateway restart, `HL_V1_GITEA_ACCESS_TOKEN` is present in the Hermes runtime. Authenticated `GET /api/v1/user` against the internal Gitea upstream returned HTTP 200 for `jarvis-at-skic`; the account is active and not prohibited. Repository metadata for `skic-v1-playground/documentation` reported pull, push, and admin permissions.
|
||
- **ROOT CAUSE — public Nginx outage:** at `2026-08-02T18:07:35Z`, Argo CD application `0000-ingress-nginx` synced Helm chart `ingress-nginx` 4.11.3. Its DaemonSet declares host ports 80/443 on every Linux node, including Osgiliath (`192.168.148.249`), so Kubernetes CNI host-port NAT intercepts traffic before the host's system Nginx. System Nginx remains active, enabled, listening, and has valid vhost configuration, but public requests are answered by Kubernetes ingress instead. Since Kubernetes has only one Ingress (`argo-workflows.apps.lego-cloud.eu`), legacy hostnames such as Gitea and Keycloak get the ingress controller's fake `ingress.local` certificate and default HTTP 404. The internal Gitea upstream remains healthy. Durable repair must change the GitOps values file `cluster/0500-resources/0000-ingress-nginx/ingress-nginx/values.yml` in `home-v1/gondor-v1.git` to avoid hostPort 80/443 on Osgiliath (for example, schedule ingress only on worker nodes), while preserving a route for Kubernetes-hosted applications.
|
||
- **VERIFIED — Gondor node SSH:** Hermes can authenticate as `lego` to all three known Gondor v1 nodes: Osgiliath 000, Minas Tirith 010, and Minas Tirith 020. Verified node addresses are recorded in the `gondor-v1-nodes` skill. Lego authorized Hermes's existing ED25519 public key on both Minas Tirith workers.
|
||
- **INCIDENT ASSESSMENT — Hermes unavailable 2026-08-04 to 2026-08-11:** two distinct failures were found. **CONFIRMED / high confidence:** exhausted OpenAI Codex quota (`HTTP 429`, no available credentials; stated reset 2026-08-08 04:00:59 UTC) prevented normal model responses while Discord and cron still worked. **LIKELY / medium-high confidence:** a separate host or container-runtime availability failure then stopped gateway, scheduler, logs, and health activity after 2026-08-04 14:00:42 UTC until a physical restart. The restart booted the host at 2026-08-11 00:11:40 UTC, started Hermes around 00:13:54, reconnected Discord around 00:14, and restored responses without Hermes repair or configuration change.
|
||
- **UNKNOWN — exact low-level outage trigger:** preserved container evidence cannot distinguish TrueNAS Apps/Docker/containerd failure, kernel hang/panic, ZFS/storage I/O stall, hardware/watchdog/thermal event, power event, or complete power loss. Previous-boot host kernel, ZFS, Apps-runtime, middleware, and BMC/IPMI evidence was unavailable. A Discord-only failure and current `state.db` corruption were assessed as unlikely; immutable read-only SQLite `PRAGMA quick_check` returned `ok` after recovery.
|
||
|
||
## Key Context
|
||
- `HERMES_HOME=/opt/data`; this is persistent storage, so Bitwarden configuration survives gateway/container restarts and container recreation as long as `/opt/data` remains mounted and `HERMES_HOME` is unchanged.
|
||
- Persistent Bitwarden components: token bootstrap in `/opt/data/.env`, configuration in `/opt/data/config.yaml`, managed CLI at `/opt/data/bin/bws`, and cache at `/opt/data/cache/bws_cache.json`.
|
||
- Bitwarden configuration uses `access_token_env: BWS_ACCESS_TOKEN`, a 300-second cache TTL, automatic `bws` installation, and exact-key export.
|
||
- A permissions incident made `/opt/data/.env` unreadable and caused repeated gateway `PermissionError` responses. It was fixed; at 2026-08-02 18:06 UTC both `/opt/data/.env` and `/opt/data/config.yaml` were owned by `hermes:hermes` with mode `0600`.
|
||
- `gitea-repository-operations`, `gitea-docusaurus-projects`, and the Discord knowledge-wiki Gitea creation workflow were updated to require `HL_V1_GITEA_ACCESS_TOKEN` and reject `GITHUB_TOKEN` for `gitea.lego-cloud.eu`. GitHub-specific workflows may continue using `GITHUB_TOKEN` for `github.com`.
|
||
- Reusable skill: `/opt/data/skills/cloud-architecture/gondor-v1-nodes/SKILL.md`. It documents the persistent SSH identity (`/opt/data/home/.ssh/id_ed25519`, fingerprint `SHA256:7qkAFkIMxFHhCjd8FazFyQFLi75y+gMfUm08tHjYNjU`), node inventory, safe read-only checks, MicroK8s/Argo CD guidance, and the distinction between remote `authorized_keys` and client `known_hosts`.
|
||
- The blank human message `1533511724370624693` included `message.txt`; its CDN URL returned HTTP 403 when this sync attempted retrieval, so its contents remain **UNKNOWN**. Subsequent messages and live verification establish the setup outcome above.
|
||
|
||
## People & Roles
|
||
- **RootAtSkic / Lego:** configured Bitwarden and directed the Gitea credential migration.
|
||
- **Hermes:** validates secret integration without revealing values and maintains Gitea-related skills.
|
||
|
||
## Action Items
|
||
- [x] **Owner: infrastructure operator / Lego:** restart the Hermes gateway so `HL_V1_GITEA_ACCESS_TOKEN` is imported.
|
||
- [x] **Owner: Hermes:** verify gateway access to `HL_V1_GITEA_ACCESS_TOKEN` without displaying the value and authenticate to Gitea.
|
||
- [ ] **Owner: infrastructure operator / Lego — due date UNKNOWN:** choose and apply the ingress ownership model. Recommended: keep host system Nginx on Osgiliath ports 80/443, constrain the Argo CD-managed `ingress-nginx` DaemonSet to the two worker nodes via GitOps values, and proxy required `*.apps.lego-cloud.eu` routes from Osgiliath to worker ingress endpoints.
|
||
- [x] **Owner: Lego / Hermes:** authorize and verify Hermes SSH access to `gondor-v1-minas-tirith-010` and `gondor-v1-minas-tirith-020`; completed 2026-08-02.
|
||
- [x] **Owner: Hermes:** create and verify the reusable `gondor-v1-nodes` skill; completed 2026-08-02.
|
||
- [x] **Owner: Hermes — due 2026-08-11 11:00 UTC:** perform a deep, strictly read-only analysis of the multi-day outage and deliver the report; completed early at 2026-08-11 00:36 UTC.
|
||
- [ ] **Owner: infrastructure operator — due date UNKNOWN:** if a definitive host-level root cause is required, inspect retained TrueNAS previous-boot kernel/ZFS/Apps-runtime/middleware logs and BMC/IPMI power, thermal, watchdog, and hardware events; these sources are not exposed inside the Hermes container.
|
||
|
||
## Source Anchors
|
||
- Latest processed human message: `1536529995176878242` (2026-08-11 00:21 UTC), duplicating Lego's read-only outage-analysis directive and 11:00 UTC deadline from `1536529944094707763`.
|
||
- Delivered incident report: Discord messages `1536533693001236550` through `1536533698034409532` (2026-08-11 00:36 UTC).
|
||
- Current thread: human messages `1533543425167593484` through `1533557577361916045` (2026-08-02 18:34–19:30 UTC), with bot output used only to establish live Gitea authentication, outage diagnosis, node access, and skill implementation results.
|