After a monitor no-signal incident and hard power-off during development, John reported a changed CPU fan sound and requested additional logging, potentially to kelp or gravlax. This setup records diagnostic data without running a load/stress test. The cause of the incident remains unproven.
Local source: /home/john/py/rainbow/tools/host_health.py, Python standard library only. User service definition: tools/rainbow-health.service. Remote receiver deployed on gravlax at /home/john/.local/lib/rainbow-health/host_health.py. SSH uses the existing john@gravlax alias and strict host-key verification; actual remote account reports admin with HOME=/home/john. No listening ports, credentials, firewall changes or reverse tunnels were added.
Local rotating samples: /home/john/py/rainbow/output/health/samples.jsonl. Remote rotating samples: gravlax:/home/john/.local/state/rainbow-health/pomelo.jsonl. Each retains an active file and four 16-MiB backups, approximately 80 MiB maximum per host. Local monitor.log retains an active 2-MiB file and two backups; remote-status.json records the last durable acknowledgement.
systemctl --user link /home/john/py/rainbow/tools/rainbow-health.service
systemctl --user enable --now rainbow-health.service
systemctl --user status rainbow-health.service
cat /home/john/py/rainbow/output/health/remote-status.json
ssh john@gravlax 'tail -n 2 /home/john/.local/state/rainbow-health/pomelo.jsonl'Service is enabled and active. John explicitly instructed SSH localhost to run the service commands, which succeeded. The service manager initially supplied an empty GnuPG SSH agent; the sender now reuses an existing unlocked user-owned agent found in the inherited environment or a private user-owned /tmp/ssh-/agent. socket. It queries identities without exposing, copying, loading or changing keys, and repeats discovery on reconnect. Fresh remote fsync acknowledgements were verified. Check remote-status.json and advancing remote sample timestamps before assuming current delivery is healthy.
Samples every two seconds include UTC and monotonic time, boot and collector identity, sequence, per-core CPU usage deltas, load, available memory and swap, dirty/writeback memory, CPU/memory/I/O pressure, CPU frequency range, readable hwmon temperatures/power/frequency, GPU busy percentage and relevant process names with CPU ticks/RSS. Kernel journal following captures GPU/DRM, thermal, watchdog, OOM and related messages. Journal availability and sensor-read errors are visible in samples. No fan RPM sensor is exposed on pomelo; fan_rpm is therefore empty, not evidence that the fan stopped. Process arguments, environment variables and credentials are not collected.
Local samples are fsynced. The sender uses a bounded asynchronous queue and a persistent SSH command running the receiver. Gravlax appends and fsyncs each record before acknowledging its sequence. Queue overflow and connection failures are logged; local sampling continues. Failed acknowledgements may cause duplicate remote records: deduplicate by collector_id and sequence. Local-only records are not automatically replayed after an extended outage. A hard freeze can lose events since the last sample/ACK; remote receipt does not establish the failure's cause.
Service has Nice=10, CPUQuota=5% of one CPU, MemoryMax=128M, bounded restart behavior and rotating files. Six logger tests passed, local CPU-delta/kernel following worked, and real continuous two-second samples with remote receipt and no queue drops were verified during resumed builds/renders. The source at fd26be7 adds automatic unlocked-agent selection. After the service was healthy, modest two-thread workloads produced images; no captured kernel fault explained the original blackout.
systemctl --user disable --now rainbow-health.serviceStopping collection preserves all logs. The initial standalone test receiver was explicitly terminated. Current remote receivers are SSH child processes owned by the managed local service; they are not independently installed daemons. Remote software remains available for reconnection.