knowledge-wiki/channels/1526844602303123466.md

Channel Wiki: #jarvis-jr-v1-hermes-setup

Channel ID: 1526844602303123466 Created: 2026-07-23 18:22 UTC Last sync: 2026-08-11 04:15 UTC

Purpose

Hermes installation, persistent runtime configuration, secret management, and setup troubleshooting.

Key Decisions

• CONFIRMED — Gitea credential (2026-08-02): authenticated operations against gitea.lego-cloud.eu must use the Bitwarden Secrets Manager key/environment variable HL_V1_GITEA_ACCESS_TOKEN. Do not use GITHUB_TOKEN for this Gitea instance and do not copy the Gitea token into /opt/data/.env.
• CONFIRMED — secret naming: Hermes exports Bitwarden secret keys exactly as named; it does not create aliases. The briefly reported HL_V1_GITEA_TOKEN alias was stale cache information and was explicitly corrected.
• CONFIRMED — restart semantics: /reset only resets a conversation and does not reload process environment. A gateway restart is required after Bitwarden keys change so the gateway imports them.
• CONFIRMED — outage investigation constraint (2026-08-11): Lego required a deep analysis of the multi-day Hermes outage, with no host changes, and requested a report by 11:00 UTC on 2026-08-11. Hermes completed and delivered the strictly read-only report at 00:36 UTC; no files, configuration, services, databases, or processes were changed by that investigation.

Active Topics

• Bitwarden setup: enabled and operational. Live verification at sync time showed project access, bws 2.0.0, and one applied key: HL_V1_GITEA_ACCESS_TOKEN (value never exposed).
• VERIFIED: after gateway restart, HL_V1_GITEA_ACCESS_TOKEN is present in the Hermes runtime. Authenticated GET /api/v1/user against the internal Gitea upstream returned HTTP 200 for jarvis-at-skic; the account is active and not prohibited. Repository metadata for skic-v1-playground/documentation reported pull, push, and admin permissions.
• ROOT CAUSE — public Nginx outage: at 2026-08-02T18:07:35Z, Argo CD application 0000-ingress-nginx synced Helm chart ingress-nginx 4.11.3. Its DaemonSet declares host ports 80/443 on every Linux node, including Osgiliath (192.168.148.249), so Kubernetes CNI host-port NAT intercepts traffic before the host's system Nginx. System Nginx remains active, enabled, listening, and has valid vhost configuration, but public requests are answered by Kubernetes ingress instead. Since Kubernetes has only one Ingress (argo-workflows.apps.lego-cloud.eu), legacy hostnames such as Gitea and Keycloak get the ingress controller's fake ingress.local certificate and default HTTP 404. The internal Gitea upstream remains healthy. Durable repair must change the GitOps values file cluster/0500-resources/0000-ingress-nginx/ingress-nginx/values.yml in home-v1/gondor-v1.git to avoid hostPort 80/443 on Osgiliath (for example, schedule ingress only on worker nodes), while preserving a route for Kubernetes-hosted applications.
• VERIFIED — Gondor node SSH: Hermes can authenticate as lego to all three known Gondor v1 nodes: Osgiliath 000, Minas Tirith 010, and Minas Tirith 020. Verified node addresses are recorded in the gondor-v1-nodes skill. Lego authorized Hermes's existing ED25519 public key on both Minas Tirith workers.
• INCIDENT ASSESSMENT — Hermes unavailable 2026-08-04 to 2026-08-11: two distinct failures were found. CONFIRMED / high confidence: exhausted OpenAI Codex quota (HTTP 429, no available credentials; stated reset 2026-08-08 04:00:59 UTC) prevented normal model responses while Discord and cron still worked. LIKELY / medium-high confidence: a separate host or container-runtime availability failure then stopped gateway, scheduler, logs, and health activity after 2026-08-04 14:00:42 UTC until a physical restart. The restart booted the host at 2026-08-11 00:11:40 UTC, started Hermes around 00:13:54, reconnected Discord around 00:14, and restored responses without Hermes repair or configuration change.
• UNKNOWN — exact low-level outage trigger: preserved container evidence cannot distinguish TrueNAS Apps/Docker/containerd failure, kernel hang/panic, ZFS/storage I/O stall, hardware/watchdog/thermal event, power event, or complete power loss. Previous-boot host kernel, ZFS, Apps-runtime, middleware, and BMC/IPMI evidence was unavailable. A Discord-only failure and current state.db corruption were assessed as unlikely; immutable read-only SQLite PRAGMA quick_check returned ok after recovery.

Key Context

• HERMES_HOME=/opt/data; this is persistent storage, so Bitwarden configuration survives gateway/container restarts and container recreation as long as /opt/data remains mounted and HERMES_HOME is unchanged.
• Persistent Bitwarden components: token bootstrap in /opt/data/.env, configuration in /opt/data/config.yaml, managed CLI at /opt/data/bin/bws, and cache at /opt/data/cache/bws_cache.json.
• Bitwarden configuration uses access_token_env: BWS_ACCESS_TOKEN, a 300-second cache TTL, automatic bws installation, and exact-key export.
• A permissions incident made /opt/data/.env unreadable and caused repeated gateway PermissionError responses. It was fixed; at 2026-08-02 18:06 UTC both /opt/data/.env and /opt/data/config.yaml were owned by hermes:hermes with mode 0600.
• gitea-repository-operations, gitea-docusaurus-projects, and the Discord knowledge-wiki Gitea creation workflow were updated to require HL_V1_GITEA_ACCESS_TOKEN and reject GITHUB_TOKEN for gitea.lego-cloud.eu. GitHub-specific workflows may continue using GITHUB_TOKEN for github.com.
• Reusable skill: /opt/data/skills/cloud-architecture/gondor-v1-nodes/SKILL.md. It documents the persistent SSH identity (/opt/data/home/.ssh/id_ed25519, fingerprint SHA256:7qkAFkIMxFHhCjd8FazFyQFLi75y+gMfUm08tHjYNjU), node inventory, safe read-only checks, MicroK8s/Argo CD guidance, and the distinction between remote authorized_keys and client known_hosts.
• The blank human message 1533511724370624693 included message.txt; its CDN URL returned HTTP 403 when this sync attempted retrieval, so its contents remain UNKNOWN. Subsequent messages and live verification establish the setup outcome above.

People & Roles

• RootAtSkic / Lego: configured Bitwarden and directed the Gitea credential migration.
• Hermes: validates secret integration without revealing values and maintains Gitea-related skills.

Action Items

• Owner: infrastructure operator / Lego: restart the Hermes gateway so HL_V1_GITEA_ACCESS_TOKEN is imported.
• Owner: Hermes: verify gateway access to HL_V1_GITEA_ACCESS_TOKEN without displaying the value and authenticate to Gitea.
• Owner: infrastructure operator / Lego — due date UNKNOWN: choose and apply the ingress ownership model. Recommended: keep host system Nginx on Osgiliath ports 80/443, constrain the Argo CD-managed ingress-nginx DaemonSet to the two worker nodes via GitOps values, and proxy required *.apps.lego-cloud.eu routes from Osgiliath to worker ingress endpoints.
• Owner: Lego / Hermes: authorize and verify Hermes SSH access to gondor-v1-minas-tirith-010 and gondor-v1-minas-tirith-020; completed 2026-08-02.
• Owner: Hermes: create and verify the reusable gondor-v1-nodes skill; completed 2026-08-02.
• Owner: Hermes — due 2026-08-11 11:00 UTC: perform a deep, strictly read-only analysis of the multi-day outage and deliver the report; completed early at 2026-08-11 00:36 UTC.
• Owner: infrastructure operator — due date UNKNOWN: if a definitive host-level root cause is required, inspect retained TrueNAS previous-boot kernel/ZFS/Apps-runtime/middleware logs and BMC/IPMI power, thermal, watchdog, and hardware events; these sources are not exposed inside the Hermes container.

Source Anchors

• Latest processed human message: 1536529995176878242 (2026-08-11 00:21 UTC), duplicating Lego's read-only outage-analysis directive and 11:00 UTC deadline from 1536529944094707763.
• Delivered incident report: Discord messages 1536533693001236550 through 1536533698034409532 (2026-08-11 00:36 UTC).
• Current thread: human messages 1533543425167593484 through 1533557577361916045 (2026-08-02 18:34–19:30 UTC), with bot output used only to establish live Gitea authentication, outage diagnosis, node access, and skill implementation results.