knowledge-wiki/channels/1519072621130547220.md

Channel Wiki: #jarvis-jr-truenas

Channel ID: 1519072621130547220 Last sync: 2026-08-18 03:05 UTC

Purpose

TrueNAS, self-hosted infrastructure, registry, proxy, and platform setup discussions.

Key Context

• Keycloak hostname: keycloak.lego-cloud.eu; historical discussion referenced the master realm.
• Gitea organization URL: https://gitea.lego-cloud.eu/home-v1
• Repository referenced for shared access: https://gitea.lego-cloud.eu/home-v1/arda-v1-code-agent
• Historical work investigated Harbor installation on TrueNAS SCALE using a custom-app YAML.
• The external Nginx reverse proxy already existed in front of TrueNAS; Harbor planning should not add an unnecessary duplicate Nginx layer.
• Services exposed through the existing HTTPS path still require explicit internal/service ports and routing.

Behavioral Decisions

• Teach and train without negativity or pointing out what others lack.
• Operate autonomously enough that RootAtSkic does not need to micromanage routine investigation.

Active Topics

• TrueNAS application deployment patterns.
• Harbor/container-registry architecture and certificate/routing requirements.
• CONFIRMED capacity objective (RootAtSkic, 2026-08-17): maximize this TrueNAS host's memory. For the documented W680D4U-2L2T/G5 / i9-14900KS platform, the target is 192 GB ECC (4 × 48 GB DDR5 ECC UDIMM), not a smaller diagnostic kit. Procurement and validation constraints are recorded below.

TrueNAS Stability Incident — 2026-08-17

• TrueNAS 26.0.0-BETA.2 on nas.lego-cloud.eu underwent full-host boots at 11:02:52, 12:16:18, 12:32:13, and 13:52:25 Europe/Vilnius time. Docker/containerd restarts were consequences, not the initial cause.
• Recursive recovery of /var/lib/systemd/pstore proved that all observed Aug 17 outages followed explicit kernel panics. Fifteen panic records are persisted from Aug 4–17, all using kernel 6.18.23-production+truenas and OpenZFS 2.4.1-1.
• A repeated signature across earlier boots faults on the same impossible address 0x04000034 in __mutex_lock, called through dbuf_find / dbuf_hold_impl in ZFS. The latest panic occurred at 16:01:52 Europe/Vilnius and rebooted at about 16:03:50: process rg under UID 10000 (verified as the Hermes user) triggered an OverlayFS xattr read that faulted on non-canonical pointer 0x7fff8a3979c65828 in nvt_lookup_name_type -> nvlist_lookup_common -> zpl_xattr_get_sa [zfs]. A prior panic also involved Hermes rg and the same ZFS xattr/nvlist area. Hermes should avoid search_files/ripgrep on this TrueNAS-hosted container until the kernel/OpenZFS issue is fixed, using bounded read_file and Python traversal instead. Other panics involve ZFS ARC/dbuf/list corruption and kernel page accounting. This makes a TrueNAS 26/OpenZFS/kernel regression or use-after-free a strong primary candidate; broad userspace segfaults leave CPU/RAM corruption as a material secondary possibility.
• Current pool and Apps recovered healthy, with no storage or memory-capacity pressure. BMC reports no current power, fan, or overload fault; historical last_power_event is ac failed without a timestamp. EDAC counters are 0 corrected / 0 uncorrected on both controllers, and no MCE/APEI event appears in panic captures.
• CPU is an Intel i9-14900KS running microcode 0x12F (current Intel mitigation level). Stable boot environment 25.10.3 remains available; kdump_enabled is false.
• Best discriminator is booting 25.10.3: if panics stop under the same workload, the beta kernel/OpenZFS stack is implicated; if they continue, proceed to offline RAM/CPU testing and conservative BIOS defaults. Preserve pstore files for a TrueNAS bug report.
• TrueNAS support feedback requested hardware elimination before SA-xattr investigation: several full MemTest86+ passes, verify latest board BIOS and Intel-default CPU power/voltage settings, test known-good ECC UDIMMs if possible, and attach all 15 pstore records. A 15-record archive was prepared at /opt/data/tmp/truenas-pstore-15-panics-2026-08-17.tar.gz (SHA-256 ff82f38c265931f6132ea377d4f14aa25741d32fcb3e2a268148b1614b817b81). The files contain the private hostname/pool name, so attach them to the support ticket rather than publishing indiscriminately.
• ECC-memory investigation: target is the board maximum, 192 GB ECC = 4 × 48 GB DDR5 ECC UDIMM. ASRock Rack officially supports 288-pin DDR5 1.1 V ECC/non-ECC UDIMMs only on W680D4U-2L2T/G5, maximum 48 GB/DIMM with 14th-gen Core. Four 48 GB 2Rx8 DIMMs are 2DPC dual-rank and should operate at the board-supported 3600 MT/s; 5600 is the DIMM rating, not expected installed speed. Intel officially lists ECC support for i9-14900KS. The exact 48 GB QVL module is SMART SR6G7UD5385MB01; specification-compatible but not individually QVL-listed alternatives include Kingston KSM56E46BD8KM-48HA, Micron MTC20C2085S1EC56BR, and Samsung M323R6GA3BB0-CWM. Buy four identical modules from one revision/lot; never RDIMM/LRDIMM. No Lithuanian retailer or EU distributor page checked on 2026-08-17 verified four exact 48 GB modules in stock, so the QVL SMART part likely needs special ordering. For lower capacities, exact QVL SKUs include Kingston KSM48E40BD8KM-32HM/KSM48E40BS8KM-16HM, Micron MTC20C2085S1EC48BA1, and Samsung M324R4GA3BB0-CQK0L; do not confuse Samsung M323R4GA3BB0-CQK0L (QVL non-ECC) with ECC.