skic-v1-playground/knowledge-wiki
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

wiki: sync 2026-08-11 04:18 UTC — 5 file(s) updated

Browse Source
This commit is contained in:
jarvis-at-skic
2026-08-11 04:18:11 +00:00
parent 723e21d7e4
commit bb13c49299
5 changed files with 53 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
channels/1526844602303123466.md
+8 -2
View File
@@ -1,7 +1,7 @@
# Channel Wiki: #jarvis-jr-v1-hermes-setup
_Channel ID: 1526844602303123466_
_Created: 2026-07-23 18:22 UTC_
_Last sync: 2026-08-02 22:16 UTC_
_Last sync: 2026-08-11 04:15 UTC_
## Purpose
Hermes installation, persistent runtime configuration, secret management, and setup troubleshooting.
@@ -10,12 +10,15 @@ Hermes installation, persistent runtime configuration, secret management, and se
- **CONFIRMED — Gitea credential (2026-08-02):** authenticated operations against `gitea.lego-cloud.eu` must use the Bitwarden Secrets Manager key/environment variable `HL_V1_GITEA_ACCESS_TOKEN`. Do **not** use `GITHUB_TOKEN` for this Gitea instance and do not copy the Gitea token into `/opt/data/.env`.
- **CONFIRMED — secret naming:** Hermes exports Bitwarden secret keys exactly as named; it does not create aliases. The briefly reported `HL_V1_GITEA_TOKEN` alias was stale cache information and was explicitly corrected.
- **CONFIRMED — restart semantics:** `/reset` only resets a conversation and does not reload process environment. A gateway restart is required after Bitwarden keys change so the gateway imports them.
- **CONFIRMED — outage investigation constraint (2026-08-11):** Lego required a deep analysis of the multi-day Hermes outage, with no host changes, and requested a report by 11:00 UTC on 2026-08-11. Hermes completed and delivered the strictly read-only report at 00:36 UTC; no files, configuration, services, databases, or processes were changed by that investigation.
## Active Topics
- **Bitwarden setup:** enabled and operational. Live verification at sync time showed project access, `bws 2.0.0`, and one applied key: `HL_V1_GITEA_ACCESS_TOKEN` (value never exposed).
- **VERIFIED:** after gateway restart, `HL_V1_GITEA_ACCESS_TOKEN` is present in the Hermes runtime. Authenticated `GET /api/v1/user` against the internal Gitea upstream returned HTTP 200 for `jarvis-at-skic`; the account is active and not prohibited. Repository metadata for `skic-v1-playground/documentation` reported pull, push, and admin permissions.
- **ROOT CAUSE — public Nginx outage:** at `2026-08-02T18:07:35Z`, Argo CD application `0000-ingress-nginx` synced Helm chart `ingress-nginx` 4.11.3. Its DaemonSet declares host ports 80/443 on every Linux node, including Osgiliath (`192.168.148.249`), so Kubernetes CNI host-port NAT intercepts traffic before the host's system Nginx. System Nginx remains active, enabled, listening, and has valid vhost configuration, but public requests are answered by Kubernetes ingress instead. Since Kubernetes has only one Ingress (`argo-workflows.apps.lego-cloud.eu`), legacy hostnames such as Gitea and Keycloak get the ingress controller's fake `ingress.local` certificate and default HTTP 404. The internal Gitea upstream remains healthy. Durable repair must change the GitOps values file `cluster/0500-resources/0000-ingress-nginx/ingress-nginx/values.yml` in `home-v1/gondor-v1.git` to avoid hostPort 80/443 on Osgiliath (for example, schedule ingress only on worker nodes), while preserving a route for Kubernetes-hosted applications.
- **VERIFIED — Gondor node SSH:** Hermes can authenticate as `lego` to all three known Gondor v1 nodes: Osgiliath 000, Minas Tirith 010, and Minas Tirith 020. Verified node addresses are recorded in the `gondor-v1-nodes` skill. Lego authorized Hermes's existing ED25519 public key on both Minas Tirith workers.
- **INCIDENT ASSESSMENT — Hermes unavailable 2026-08-04 to 2026-08-11:** two distinct failures were found. **CONFIRMED / high confidence:** exhausted OpenAI Codex quota (`HTTP 429`, no available credentials; stated reset 2026-08-08 04:00:59 UTC) prevented normal model responses while Discord and cron still worked. **LIKELY / medium-high confidence:** a separate host or container-runtime availability failure then stopped gateway, scheduler, logs, and health activity after 2026-08-04 14:00:42 UTC until a physical restart. The restart booted the host at 2026-08-11 00:11:40 UTC, started Hermes around 00:13:54, reconnected Discord around 00:14, and restored responses without Hermes repair or configuration change.
- **UNKNOWN — exact low-level outage trigger:** preserved container evidence cannot distinguish TrueNAS Apps/Docker/containerd failure, kernel hang/panic, ZFS/storage I/O stall, hardware/watchdog/thermal event, power event, or complete power loss. Previous-boot host kernel, ZFS, Apps-runtime, middleware, and BMC/IPMI evidence was unavailable. A Discord-only failure and current `state.db` corruption were assessed as unlikely; immutable read-only SQLite `PRAGMA quick_check` returned `ok` after recovery.
## Key Context
- `HERMES_HOME=/opt/data`; this is persistent storage, so Bitwarden configuration survives gateway/container restarts and container recreation as long as `/opt/data` remains mounted and `HERMES_HOME` is unchanged.
@@ -36,7 +39,10 @@ Hermes installation, persistent runtime configuration, secret management, and se
- [ ] **Owner: infrastructure operator / Lego — due date UNKNOWN:** choose and apply the ingress ownership model. Recommended: keep host system Nginx on Osgiliath ports 80/443, constrain the Argo CD-managed `ingress-nginx` DaemonSet to the two worker nodes via GitOps values, and proxy required `*.apps.lego-cloud.eu` routes from Osgiliath to worker ingress endpoints.
- [x] **Owner: Lego / Hermes:** authorize and verify Hermes SSH access to `gondor-v1-minas-tirith-010` and `gondor-v1-minas-tirith-020`; completed 2026-08-02.
- [x] **Owner: Hermes:** create and verify the reusable `gondor-v1-nodes` skill; completed 2026-08-02.
- [x] **Owner: Hermes — due 2026-08-11 11:00 UTC:** perform a deep, strictly read-only analysis of the multi-day outage and deliver the report; completed early at 2026-08-11 00:36 UTC.
- [ ] **Owner: infrastructure operator — due date UNKNOWN:** if a definitive host-level root cause is required, inspect retained TrueNAS previous-boot kernel/ZFS/Apps-runtime/middleware logs and BMC/IPMI power, thermal, watchdog, and hardware events; these sources are not exposed inside the Hermes container.
## Source Anchors
- Latest processed human message: `1533557577361916045` (2026-08-02 19:30 UTC).
- Latest processed human message: `1536529995176878242` (2026-08-11 00:21 UTC), duplicating Lego's read-only outage-analysis directive and 11:00 UTC deadline from `1536529944094707763`.
- Delivered incident report: Discord messages `1536533693001236550` through `1536533698034409532` (2026-08-11 00:36 UTC).
- Current thread: human messages `1533543425167593484` through `1533557577361916045` (2026-08-02 18:34–19:30 UTC), with bot output used only to establish live Gitea authentication, outage diagnosis, node access, and skill implementation results.
channels/1532415849762127902.md
+7 -2
View File
@@ -1,6 +1,6 @@
# vssa-storm
_Last sync: 2026-08-01 09:40 UTC_
_Last sync: 2026-08-11 04:15 UTC_
## Purpose
@@ -21,11 +21,16 @@ VSSA client investigation and evidence gathering.
- Each run audits all 290 records, deeply researches up to 10 missing/weak/stale dossiers, updates the index, commits and pushes evidence-backed changes, and verifies remote state.
- Research must prefer primary sources and must not infer providers/systems without corroborating evidence.
## Availability continuity
- On 2026-08-04 at 20:05–20:42 UTC, Lego (`root_at_skic`) repeatedly asked whether Hermes was alive. These messages are evidence of the broader Hermes availability incident, not a change to the VSSA investigation scope. The incident was subsequently investigated in `#jarvis-jr-v1-hermes-setup`.
## Terminology
- In repository and Gitea workflow guidance, **mutation** means an operation that changes data or state (for example editing, committing, pushing, changing settings, or deleting). “Discovery before mutation” means inspect and confirm the target repository and its current state before making changes.
## Source anchor
- Latest processed human message: `1533032343021813930` (2026-08-01 08:43 UTC). This message requested clarification of the term “mutation”; no project scope or status changed.
- Latest processed human message: `1534300468530905170` (2026-08-04 20:42 UTC). Lego asked whether Hermes was alive; no VSSA project scope or status changed.
- Previous terminology message: `1533032343021813930` (2026-08-01 08:43 UTC), requesting clarification of “mutation”.
- Previous durable directive: `1532810727847624884` (2026-07-31 18:02 UTC), confirming the concise one-file-per-client format, plain-language system meaning, contractor/company research, deep public investigation, and iterative improvement of `client-investigator`.