Parent: README
Read this when something that is supposed to be running is not doing its job: a tunnel or remote-access path is down, a service reports healthy but nothing comes through, a connection is refused or closed unexpectedly, or you are about to restart something to see if that fixes it. The subject of this note is the gap between a system reporting health and a system actually working.
Everything here was paid for once, on 2026-09-25, diagnosing why pomelo's reverse tunnels to kelp had been dead for eleven hours while every service involved reported itself healthy. Worked example: infrastructure/kelp-dante-socks5.
A restart destroys the evidence that would have told you why it broke. Something that works again after a restart has not been diagnosed, it has been disturbed — and it will break again, at a time you do not choose, with the same cause and no more information than you have now.
John, 2026-09-25: "Investigate and gather information. Don't just restart and say 'fixed' — then get to the root of the problem." If you do restart for service restoration before the cause is known, say so plainly and say that the cause is still open. Do not let restoration be mistaken for a fix.
Every check below separates what a component reports about itself from what it is actually doing. Always prefer the second. The single highest-value question is: what observable effect should this thing be producing, and is that effect present?
systemctl lies in both directions, and each direction means something specific.
active (running) with a hung process. systemd supervises liveness, not function. A process blocked forever in a read is alive and will be reported as running indefinitely. Uptime in the status output is the least informative field on a broken service — a long uptime with no recent log lines is a warning sign, not reassurance.inactive (dead) with the daemon nonetheless running. This means somebody bypassed systemd — usually a previous operator or agent who hand-launched the binary to work around a problem. The consequences are silent: no restart-on-failure, no start at boot, and monitoring that reads unit state sees a service that is "off" while traffic flows. Check ps -o pid,ppid,lstart,cmd against the unit's Main PID, and treat a mismatch as a finding in its own right.1. A blocking read with a long timeout is indistinguishable from working. A socket read with a 24-hour timeout (--timeout 86400) and no TCP keepalive cannot detect a peer that went away without sending FIN or RST. The process sits in recv(), emits nothing, and is reported active (running) for the entire timeout. On 2026-09-25 this produced eleven hours of a completely dead SSH tunnel with a healthy-looking service and an empty log.
Diagnostic: a service that has logged nothing since its last successful action is a suspect, not a bystander. Compare the timestamp of the last log line against how often the thing should be doing something. Confirm with ls -la /proc/PID/fd and ss -tnp — an ESTABLISHED socket with zero queued bytes for hours, on a link that should carry traffic, is very likely a half-open connection that only this end still believes in.
Design corollary: when writing anything that waits on a socket, set SO_KEEPALIVE or use a timeout short enough that a dead peer is noticed, and reconnect. A long timeout looks conservative and is actually a way to hide failure for exactly that long.
2. Restart=always converts a resource conflict into a silent crash-loop. A service that cannot acquire what it needs — most often a port already bound by something else — will restart every RestartSec seconds forever, doing nothing, raising nothing. It had been doing so for over a day before anyone noticed, because the only visible symptom was at the far end of the chain. Restart=always guarantees the process exists; it guarantees nothing about the process working.
3. Silence in a log is not evidence of no traffic — read the log level first. stunnel at debug = warning logs errors but not successful connections, so a working connection and a connection that never arrived look identical: nothing. Before concluding "it never reached us", establish what a success would have written. Reason about the absence of a log line only once you know that line would have existed.
4. Orphan processes from earlier sessions squat resources. On 2026-09-25 two stray socat forwarders were occupying the ports the tunnel needed. One was traced, via /proc/PID/environ, to an unrelated Claude Code session working in a different project two days earlier, which had nohup'd it and exited. It had no systemd unit, no documentation, and no owner — and it broke production infrastructure belonging to a different project entirely.
Diagnostic: when a port is in use, ask who owns it, not merely whether it is bound. ss -ltnp gives the PID; then ps -o pid,ppid,lstart,cmd -p PID for age and command line, cat /proc/PID/cgroup to see whether it sits under a systemd unit or under somebody's interactive shell, and /proc/PID/environ to identify the session that started it. A process whose cgroup is a terminal scope is nobody's managed service.
Agent hygiene, the other side of the same lesson: never leave a background process behind on a host. If something must outlive the session, give it a systemd unit and document it in the relevant note. A nohup that survives your session is an undocumented, unsupervised, unattributable dependency that someone will debug for hours months later. See AI Operational Safety Rules.
5. Two listeners on the same port number need not conflict. 127.0.0.1:5900 and 172.31.41.106:5900 are different sockets and coexist happily. Do not assume a port collision from the number alone, and do not assume safety either — read the bind address in the ss output, every time.
A cloud firewall is a separate layer that is invisible from inside the instance. iptables and ufw will not show an AWS security group, so "the host firewall allows it" proves nothing about reachability. Check it directly:
# The instance's SG rules -- note the region; the CLI default (us-east-1) is
# almost never the right one for these hosts.
aws ec2 describe-instances --region eu-central-1 \
--filters Name=instance-state-name,Values=running \
--query "Reservations[].Instances[].[InstanceId,PrivateIpAddress,Tags[?Key=='Name'].Value|[0],SecurityGroups]"
aws ec2 describe-security-groups --region eu-central-1 \
--group-ids sg-XXXX --query "SecurityGroups[0].IpPermissions"An empty result from an aws command is ambiguous — it means "no match", which is equally consistent with the wrong region, the wrong tag, or a genuinely absent resource. Confirm with aws sts get-caller-identity and aws configure list before believing a negative.
Write down the full path before testing anything, listing every hop and what each is supposed to produce. Then walk it, checking the effect at each hop rather than the status of the process implementing it. The point of naming the hops first is that it exposes hops you would otherwise skip — in the 2026-09-25 case, an entire relay service and a security-group rule that no note mentioned.
Test from the point closest to the failure that you can actually reach, and remember that a connection test only tests the path you actually used. Connecting to 127.0.0.1:22222 from the wrong host proves nothing about kelp:22222; the design's real entry point may be a different port entirely.
Useful inversion: an error that gets further is progress. Permission denied (publickey) from an SSH relay means the whole chain carried a real protocol negotiation to a real sshd — that is a working tunnel and an auth problem, not a broken tunnel. Distinguish Connection refused (nothing listening), Connection reset/closed-remotely (something accepted then gave up, usually a backend failure behind a proxy), and a hang (silently dropped, usually a firewall).
Where a listener is created on demand rather than held open — a SOCKS5 BIND, a socket-activated service, anything with a re-acquire loop — there is a window during which a connection legitimately gets refused. This means two things. A single failed attempt is not proof of a broken system; and "it works now" is not proof that you fixed it, since it may be proof that you retried outside the window. Where the distinction matters, confirm the fix by its mechanism rather than by a successful retry.
Handover notes, incident notes and previous sessions' conclusions record what someone believed at the time, often written while the problem was still unfolding. The 2026-09-25 handover said the tunnel services were down and listed restarts to perform. Following it would have restored service and permanently hidden the actual cause, which was a port-squatting orphan process and a systemd bypass, neither of which it mentioned.
Read prior notes for the topology and the commands — that part is durable and saves real time. Treat their conclusions about current state as claims to re-verify. When your findings contradict the note, fix the note: that contradiction is the most valuable thing you will produce, and the next session will otherwise repeat your work. See README/agent on re-verifying rather than restating.
README/lessons — specific operational lessons, including the davfs2 write-caching incident, which is the same species: an operation reporting success while nothing arrived.
README/agent — working practices: find before investigate, verify before asserting, re-verify when doubted.
infrastructure/kelp-dante-socks5 — the tunnel this note was learned on: topology, per-hop diagnosis table, and the full incident history.