wiki: sync 2026-08-11 04:18 UTC — 5 file(s) updated

2026-08-11 04:18:11 +00:00
parent 723e21d7e4
commit bb13c49299
5 changed files with 53 additions and 5 deletions
@@ -26,3 +26,4 @@
 - **Gitea write access restored (verified 2026-07-20 15:08 UTC):** after Lego requested another access check at 11:08 UTC, authenticated SSH `git ls-remote` succeeded and local/remote `main` both resolved to `35eb834`. The disabled-account incident recorded at 11:06 UTC is resolved, and all queued commits reached the remote.
 - **Gondor v1 node access (verified 2026-08-02):** reusable `gondor-v1-nodes` skill at `/opt/data/skills/cloud-architecture/gondor-v1-nodes/SKILL.md`. Hermes SSH access as `lego` is verified to Osgiliath 000 (`192.168.148.249`), Minas Tirith 010 (`192.168.153.55`), and Minas Tirith 020 (`192.168.6.208`).
 - **Ingress incident (open, 2026-08-02):** Argo CD deployed `ingress-nginx` 4.11.3 as a DaemonSet with host ports 80/443 on Osgiliath, conflicting with its existing system Nginx. Kubernetes ingress intercepts legacy public hosts, returning the fake `ingress.local` certificate/default 404; `argo-workflows.apps.lego-cloud.eu` works and Gitea's internal upstream is healthy. Recommended durable repair is in GitOps (`home-v1/gondor-v1.git`, `cluster/0500-resources/0000-ingress-nginx/ingress-nginx/values.yml`): keep system Nginx as Osgiliath's public ingress, schedule Kubernetes ingress on workers, and proxy required app routes. Exact owner, due date, and final approved implementation remain **UNKNOWN**.
+- **Hermes multi-day outage (2026-08-04–11):** read-only analysis separated two failures. Codex quota exhaustion is the confirmed cause of failed model responses while Discord/cron still ran. After the final preserved event at 2026-08-04 14:00:42 UTC, gateway, cron, logs, and health activity all ceased until the physical host restart at 2026-08-11 00:11:40 UTC; restart restored service without a Hermes repair. A host/container-runtime availability failure is most likely, but the exact TrueNAS Apps, kernel, ZFS/storage, hardware, watchdog, thermal, or power trigger is **UNKNOWN** because previous-boot host telemetry is unavailable inside the container. Current SQLite integrity and capacity checks are healthy.