diff --git a/brain.md b/brain.md index 6ae5df8..e3da3c1 100644 --- a/brain.md +++ b/brain.md @@ -55,6 +55,7 @@ _Last updated: 2026-08-01 05:37 UTC_ - Gitea: gitea.lego-cloud.eu (org: home-v1) ✅ - **General Gitea repository skill:** ✅ `gitea-repository-operations` — reusable API + SSH workflow for any repo; full wiki runbook at `wiki/infrastructure/gitea-repository-operations.md` - **Gitea API credential (confirmed 2026-08-02):** use Bitwarden-provided `HL_V1_GITEA_ACCESS_TOKEN` for `gitea.lego-cloud.eu`; never use `GITHUB_TOKEN` for that instance or copy the Gitea token into `/opt/data/.env`. Bitwarden keys are exported without aliases and gateway restart is required after key changes. +- **Gondor v1 (2026-08-02):** Hermes SSH access as `lego` is verified to Osgiliath 000 and both Minas Tirith workers; use the `gondor-v1-nodes` skill. Open ingress incident: Argo CD's hostPort 80/443 `ingress-nginx` conflicts with Osgiliath's system Nginx; repair through `home-v1/gondor-v1.git`, not a transient live patch. - **world-v1/network-v1:** ✅ global Internet address book; IP-level mapping now separates registry holder, route origin, operator, and service. Initial records: `0.0.0.0` and `1.1.1.1`; wiki page at `wiki/infrastructure/network-v1.md` - Keycloak: keycloak.lego-cloud.eu ✅ - Hermes Dashboard: jarvis-jr-v1.lego-cloud.eu ✅ diff --git a/channels/1526844602303123466.md b/channels/1526844602303123466.md index 4d61d53..7b7457a 100644 --- a/channels/1526844602303123466.md +++ b/channels/1526844602303123466.md @@ -1,7 +1,7 @@ # Channel Wiki: #jarvis-jr-v1-hermes-setup _Channel ID: 1526844602303123466_ _Created: 2026-07-23 18:22 UTC_ -_Last sync: 2026-08-02 18:06 UTC_ +_Last sync: 2026-08-02 22:16 UTC_ ## Purpose Hermes installation, persistent runtime configuration, secret management, and setup troubleshooting. @@ -13,7 +13,9 @@ Hermes installation, persistent runtime configuration, secret management, and se ## Active Topics - **Bitwarden setup:** enabled and operational. Live verification at sync time showed project access, `bws 2.0.0`, and one applied key: `HL_V1_GITEA_ACCESS_TOKEN` (value never exposed). -- **PENDING:** the gateway process observed during the 2026-08-02 conversation predated the corrected Bitwarden cache and had not imported `HL_V1_GITEA_ACCESS_TOKEN`; run `/restart` once, then verify the variable is present without printing its value. +- **VERIFIED:** after gateway restart, `HL_V1_GITEA_ACCESS_TOKEN` is present in the Hermes runtime. Authenticated `GET /api/v1/user` against the internal Gitea upstream returned HTTP 200 for `jarvis-at-skic`; the account is active and not prohibited. Repository metadata for `skic-v1-playground/documentation` reported pull, push, and admin permissions. +- **ROOT CAUSE — public Nginx outage:** at `2026-08-02T18:07:35Z`, Argo CD application `0000-ingress-nginx` synced Helm chart `ingress-nginx` 4.11.3. Its DaemonSet declares host ports 80/443 on every Linux node, including Osgiliath (`192.168.148.249`), so Kubernetes CNI host-port NAT intercepts traffic before the host's system Nginx. System Nginx remains active, enabled, listening, and has valid vhost configuration, but public requests are answered by Kubernetes ingress instead. Since Kubernetes has only one Ingress (`argo-workflows.apps.lego-cloud.eu`), legacy hostnames such as Gitea and Keycloak get the ingress controller's fake `ingress.local` certificate and default HTTP 404. The internal Gitea upstream remains healthy. Durable repair must change the GitOps values file `cluster/0500-resources/0000-ingress-nginx/ingress-nginx/values.yml` in `home-v1/gondor-v1.git` to avoid hostPort 80/443 on Osgiliath (for example, schedule ingress only on worker nodes), while preserving a route for Kubernetes-hosted applications. +- **VERIFIED — Gondor node SSH:** Hermes can authenticate as `lego` to all three known Gondor v1 nodes: Osgiliath 000, Minas Tirith 010, and Minas Tirith 020. Verified node addresses are recorded in the `gondor-v1-nodes` skill. Lego authorized Hermes's existing ED25519 public key on both Minas Tirith workers. ## Key Context - `HERMES_HOME=/opt/data`; this is persistent storage, so Bitwarden configuration survives gateway/container restarts and container recreation as long as `/opt/data` remains mounted and `HERMES_HOME` is unchanged. @@ -21,6 +23,7 @@ Hermes installation, persistent runtime configuration, secret management, and se - Bitwarden configuration uses `access_token_env: BWS_ACCESS_TOKEN`, a 300-second cache TTL, automatic `bws` installation, and exact-key export. - A permissions incident made `/opt/data/.env` unreadable and caused repeated gateway `PermissionError` responses. It was fixed; at 2026-08-02 18:06 UTC both `/opt/data/.env` and `/opt/data/config.yaml` were owned by `hermes:hermes` with mode `0600`. - `gitea-repository-operations`, `gitea-docusaurus-projects`, and the Discord knowledge-wiki Gitea creation workflow were updated to require `HL_V1_GITEA_ACCESS_TOKEN` and reject `GITHUB_TOKEN` for `gitea.lego-cloud.eu`. GitHub-specific workflows may continue using `GITHUB_TOKEN` for `github.com`. +- Reusable skill: `/opt/data/skills/cloud-architecture/gondor-v1-nodes/SKILL.md`. It documents the persistent SSH identity (`/opt/data/home/.ssh/id_ed25519`, fingerprint `SHA256:7qkAFkIMxFHhCjd8FazFyQFLi75y+gMfUm08tHjYNjU`), node inventory, safe read-only checks, MicroK8s/Argo CD guidance, and the distinction between remote `authorized_keys` and client `known_hosts`. - The blank human message `1533511724370624693` included `message.txt`; its CDN URL returned HTTP 403 when this sync attempted retrieval, so its contents remain **UNKNOWN**. Subsequent messages and live verification establish the setup outcome above. ## People & Roles @@ -28,9 +31,12 @@ Hermes installation, persistent runtime configuration, secret management, and se - **Hermes:** validates secret integration without revealing values and maintains Gitea-related skills. ## Action Items -- [ ] **Owner: infrastructure operator / Lego — due date UNKNOWN:** restart the Hermes gateway once so `HL_V1_GITEA_ACCESS_TOKEN` is imported into the gateway environment. -- [ ] **Owner: Hermes — due date after restart:** verify gateway access to `HL_V1_GITEA_ACCESS_TOKEN` without displaying the value, then use it for future authenticated Gitea API operations. +- [x] **Owner: infrastructure operator / Lego:** restart the Hermes gateway so `HL_V1_GITEA_ACCESS_TOKEN` is imported. +- [x] **Owner: Hermes:** verify gateway access to `HL_V1_GITEA_ACCESS_TOKEN` without displaying the value and authenticate to Gitea. +- [ ] **Owner: infrastructure operator / Lego — due date UNKNOWN:** choose and apply the ingress ownership model. Recommended: keep host system Nginx on Osgiliath ports 80/443, constrain the Argo CD-managed `ingress-nginx` DaemonSet to the two worker nodes via GitOps values, and proxy required `*.apps.lego-cloud.eu` routes from Osgiliath to worker ingress endpoints. +- [x] **Owner: Lego / Hermes:** authorize and verify Hermes SSH access to `gondor-v1-minas-tirith-010` and `gondor-v1-minas-tirith-020`; completed 2026-08-02. +- [x] **Owner: Hermes:** create and verify the reusable `gondor-v1-nodes` skill; completed 2026-08-02. ## Source Anchors -- Latest processed human message: `1533516194106048725` (2026-08-02 16:45 UTC). -- Decision thread: human messages `1533507837391798413` through `1533516194106048725` (2026-08-02 16:12–16:45 UTC), with bot responses used only to establish implementation and verification outcomes. +- Latest processed human message: `1533557577361916045` (2026-08-02 19:30 UTC). +- Current thread: human messages `1533543425167593484` through `1533557577361916045` (2026-08-02 18:34–19:30 UTC), with bot output used only to establish live Gitea authentication, outage diagnosis, node access, and skill implementation results. diff --git a/meta/infrastructure.md b/meta/infrastructure.md index c9e572a..d09179e 100644 --- a/meta/infrastructure.md +++ b/meta/infrastructure.md @@ -9,8 +9,8 @@ | Service | Endpoint | Status | |---------|----------|--------| | TrueNAS SCALE | lego-cloud.eu | ✅ | -| Gitea | https://gitea.lego-cloud.eu (org: `home-v1`) | ✅ | -| Keycloak | https://keycloak.lego-cloud.eu (realm: `master`) | ✅ | +| Gitea | https://gitea.lego-cloud.eu (org: `home-v1`) | ⚠️ Internal upstream healthy; public ingress degraded 2026-08-02 | +| Keycloak | https://keycloak.lego-cloud.eu (realm: `master`) | ⚠️ Public ingress degraded 2026-08-02 | | Zot Registry | port 30264, OIDC via Keycloak (client: `zot`) | ✅ | | STT (Whisper) | http://192.168.1.218:39000/v1 (model: `Systran/faster-whisper-base`) | 🔲 | | Piper TTS | http://192.168.1.218:39001/v1 | 🔲 | @@ -24,3 +24,5 @@ - **Reusable Gitea operations skill:** ✅ `gitea-repository-operations` at `/opt/data/skills/software-development/gitea-repository-operations/SKILL.md`; covers safe discovery, create/clone/change/push, collaborators, issues/PRs, Actions, automation, and authenticated remote verification. Wiki runbook: [`gitea-repository-operations.md`](gitea-repository-operations.md). - **Global network repository:** ✅ [`world-v1/network-v1`](https://gitea.lego-cloud.eu/world-v1/network-v1) — source-backed Internet address book. IP mapping separates registry holder, route origin, operator, and service; details in [`network-v1.md`](network-v1.md). - **Gitea write access restored (verified 2026-07-20 15:08 UTC):** after Lego requested another access check at 11:08 UTC, authenticated SSH `git ls-remote` succeeded and local/remote `main` both resolved to `35eb834`. The disabled-account incident recorded at 11:06 UTC is resolved, and all queued commits reached the remote. +- **Gondor v1 node access (verified 2026-08-02):** reusable `gondor-v1-nodes` skill at `/opt/data/skills/cloud-architecture/gondor-v1-nodes/SKILL.md`. Hermes SSH access as `lego` is verified to Osgiliath 000 (`192.168.148.249`), Minas Tirith 010 (`192.168.153.55`), and Minas Tirith 020 (`192.168.6.208`). +- **Ingress incident (open, 2026-08-02):** Argo CD deployed `ingress-nginx` 4.11.3 as a DaemonSet with host ports 80/443 on Osgiliath, conflicting with its existing system Nginx. Kubernetes ingress intercepts legacy public hosts, returning the fake `ingress.local` certificate/default 404; `argo-workflows.apps.lego-cloud.eu` works and Gitea's internal upstream is healthy. Recommended durable repair is in GitOps (`home-v1/gondor-v1.git`, `cluster/0500-resources/0000-ingress-nginx/ingress-nginx/values.yml`): keep system Nginx as Osgiliath's public ingress, schedule Kubernetes ingress on workers, and proxy required app routes. Exact owner, due date, and final approved implementation remain **UNKNOWN**. diff --git a/meta/sync-log.md b/meta/sync-log.md index c8c9fc5..161e060 100644 --- a/meta/sync-log.md +++ b/meta/sync-log.md @@ -1,5 +1,40 @@ # Sync Log +### Run #96 — 2026-08-02 22:16 UTC ✅ +- **Channels scanned:** 26/26 (all accessible; up to the last 50 messages fetched from every current text channel; no channel-fetch failures). +- **New human activity since Run #95 (2026-08-02 18:06 UTC):** 12 messages in `#jarvis-jr-v1-hermes-setup` (18:34–19:30 UTC); no new human messages in the other 25 channels. +- **Durable knowledge captured:** gateway restart completed and `HL_V1_GITEA_ACCESS_TOKEN` authenticated successfully; public application outage traced to Argo CD-managed `ingress-nginx` claiming Osgiliath host ports 80/443 already used by system Nginx; no production repair was applied and the final implementation/owner/due date remain unknown. Hermes SSH access to both Minas Tirith workers was authorized and verified, and the reusable `gondor-v1-nodes` skill was created. The standing constraint remains unchanged: there is no ŽŪM AWS environment. +- **Processed anchors (latest human message in each fetched window):** + - `#atea-storm` → `1532359647435165829` (2026-07-30 12:10 UTC) + - `#general` → `1532690074498760745` (2026-07-31 10:03 UTC) + - `#zeta-functions` → `1531221244530397214` (2026-07-27 08:46 UTC) + - `#network-v1` → `1529573612510511214` (2026-07-22 19:39 UTC) + - `#vssa-storm-1st-contract` → `1532462731142103040` (2026-07-30 18:59 UTC) + - `#dpvp-phase-2-opportunity` → `1532001823597264957` (2026-07-29 12:28 UTC) + - `#vssa-storm` → `1533032343021813930` (2026-08-01 08:43 UTC) + - `#vssa-dpvp-storm` → `1531567826039734333` (2026-07-28 07:43 UTC) + - `#vssa-dpvp-cvpa-audit` → `1533039197777297428` (2026-08-01 09:10 UTC) + - `#cogarch-model-v1-storm` → `1528684615898759258` (2026-07-20 08:46 UTC) + - `#cogarch-storm` → `1532481752365273090` (2026-07-30 20:15 UTC) + - `#cogarch-storm-skills` → `1529471826697392228` (2026-07-22 12:54 UTC) + - `#cogarch-storm-drawio-custom-shapes` → `1531292165874126998` (2026-07-27 13:28 UTC) + - `#cogarch-storm-cogarch-hub-web` → no human-message anchor (empty history) + - `#playground-eduard-melman` → `1531913577462497350` (2026-07-29 06:37 UTC) + - `#lego-as-enterprise-architect-within-vssa` → `1523377235153780836` (2026-07-05 17:17 UTC) + - `#jarvis-jr-test-001` → `1532723701039108117` (2026-07-31 12:16 UTC) + - `#jarvis-jr-enablement-voice` → `1519063294973841571` (2026-06-23 19:35 UTC) + - `#jarvis-jr-general` → `1521626344599851211` (2026-06-30 21:19 UTC) + - `#jarvis-jr-knowledge-wiki` → `1532722144897794130` (2026-07-31 12:10 UTC) + - `#jarvis-jr-truenas` → `1519380863719047178` (2026-06-24 16:37 UTC) + - `#lego-planner-v1` → `1532474154954260683` (2026-07-30 19:45 UTC) + - `#vssa-account-strategy` → `1524753167483408526` (2026-07-09 12:24 UTC) + - `#jarvis-jr-v1-hermes-setup` → `1533557577361916045` (2026-08-02 19:30 UTC) **NEW** + - `#jarvis-jr-v1-hermes` → `1531933519285063771` (2026-07-29 07:56 UTC) + - `#skic-v1-playground` → `1527409474833350687` (2026-07-16 20:19 UTC) +- **Files updated:** `channels/1526844602303123466.md`, `infrastructure/README.md`, `MEMORY.md`, and `sync-log.md`. +- **Inventory:** unchanged; the authoritative 26-channel list remains synchronized with `MONITORED_CHANNELS` and `discord/README.md`; all 26 channel wiki files are present. +- **Context refresh and Gitea push:** executed after these edits; real results reported in the delivered run summary. + ### Run #95 — 2026-08-02 18:06 UTC ✅ - **Channels scanned:** 26/26 (all accessible; up to the last 50 messages fetched from every current text channel; no channel-fetch failures). - **New human activity since Run #94 (2026-08-02 14:02 UTC):** 16 messages in `#jarvis-jr-v1-hermes-setup` (16:12–16:45 UTC); no new human messages in the other 25 channels.