Exposes services on pomelo (home machine, behind NAT) to the internet via kelp (AWS EC2), using SOCKS5 BIND over a stunnel mTLS tunnel. No SSH reverse tunnel; all tunnelling is done by stunnel and Dante.
No SSH reverse tunnel. Tunnelling is always stunnel (mTLS). SSH is for shell access only.
SOCKS5 BIND, not CONNECT. The proxy opens an inbound listener on kelp; pomelo's client connects outward, avoiding the NAT problem.
Dante external: 127.0.0.1. BIND listeners land on loopback, not ens5. SSH reaches its listener only through the public socat relay; remote VNC reaches its listener only through the mTLS stunnel endpoint on port 15900.
socat relay uses port 22222 internally. 0.0.0.0:2222 (public) → 127.0.0.1:22222 (Dante BIND). The two ports differ so socat's 0.0.0.0:2222 doesn't block Dante's bind(127.0.0.1:22222).
One BIND, one accepted connection. Each socks5_bind_client.py process owns one SOCKS5 BIND control connection, accepts one incoming connection, relays it to the local service, and exits. Restart=always then starts a completely fresh process and BIND. There is no fork, overlapping BIND, or in-process server loop. This restores the original single-accept design and avoids two generations competing for the fixed Dante port.
TIME_WAIT retry loop. After a session ends, Dante may give a random port (TIME_WAIT on 22222). The retry loop detects this, closes the wrong BIND, and retries every 5 s until 22222 is available again (~60 s worst case).
1. User's SSH client connects to kelp_public_ip:2222.
2. socat-ssh-relay (kelp) accepts on 0.0.0.0:2222, connects to 127.0.0.1:22222.
3. 127.0.0.1:22222 is Dante's BIND listener — Dante sends the second BIND reply to the control connection.
4. socks5_bind_client.py (pomelo) receives the reply, connects to 127.0.0.1:22 (pomelo's own sshd).
5. Dante relays data between socat (kelp side) and socks5_bind_client.py (pomelo side).
6. When that relay closes, the client process exits. systemd waits RestartSec=5, starts a new process, and that process establishes the next BIND listener. Connections are sequential; there is deliberately no simultaneous second BIND.
VNC has two access paths. Both terminate at the same ephemeral Dante BIND listener on kelp's 127.0.0.1:5900; they differ only in how you reach it.
accept = localhost:5900, connect = kelp.critchley.biz:15900) → mTLS over the internet → stunnel-vnc.service on kelp (0.0.0.0:15900) → 127.0.0.1:5900 (Dante VNC BIND) → Dante relays over the existing SOCKS5 control connection → socks5_bind_client.py on pomelo → 127.0.0.1:5900 (x11vnc, display :0).nc → 127.0.0.1:5900 directly, skipping the stunnel hop since it is already on loopback. Everything downstream is identical.| Port | Host | Listens on | Service | Notes |
|---|---|---|---|---|
| 12081 | pomelo | 127.0.0.1 | stunnel [kelp-dante] client | mTLS to kelp:12080 |
| 12080 | kelp | 0.0.0.0 | stunnel [dante-socks5h] server | → Dante :2080 |
| 2080 | kelp | 127.0.0.1 | Dante sockd | SOCKS5 with BIND; loopback only |
| 22222 | kelp | 127.0.0.1 | Dante BIND (SSH) | ephemeral; created on demand |
| 5900 | kelp | 127.0.0.1 | Dante BIND (VNC) | ephemeral; created on demand |
| 2222 | kelp | 0.0.0.0 | socat-ssh-relay.service | public; → 127.0.0.1:22222 |
| 15900 | kelp | 0.0.0.0 | stunnel-vnc.service [vnc-5900] | public, mTLS; → 127.0.0.1:5900. THE remote VNC entry point. SG opens 15900-15909 |
| 15901 | kelp | 0.0.0.0 | stunnel-vnc.service [vnc-5901] | public, mTLS; forwards to kelp's own Xtigervnc desktop on localhost:5901 (verified 2026-10-17), so unavailable for Radagast |
| 22 | pomelo | 0.0.0.0 | sshd | target for SSH relay |
| 5900 | pomelo | 127.0.0.1 | x11vnc (display :0) | target for VNC relay |
[kelp-dante]
client = yes
accept = 127.0.0.1:12081
connect = kelp.critchley.biz:12080
checkHost = John Critchley
socket = r:SO_KEEPALIVE=1
socket = r:TCP_KEEPIDLE=60
socket = r:TCP_KEEPINTVL=20
socket = r:TCP_KEEPCNT=3
[vnc-kelp-watchdog]
client = yes
accept = 127.0.0.1:15990
connect = kelp.critchley.biz:15900
checkHost = John Critchley[Unit]
Description=SOCKS5 BIND – expose pomelo SSH on kelp:2222
After=network-online.target stunnel4.service
Wants=network-online.target
[Service]
User=john
ExecStart=/usr/bin/python3 /home/john/bin/socks5_bind_client.py \
--proxy-host 127.0.0.1 \
--proxy-port 12081 \
--bind-port 22222 \
--target-port 22 \
--timeout 86400
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target[Unit]
Description=SOCKS5 BIND – expose pomelo VNC (:0) on kelp:5900
After=network-online.target stunnel4.service
Wants=network-online.target
[Service]
User=john
ExecStart=/usr/bin/python3 /home/john/bin/socks5_bind_client.py \
--proxy-host 127.0.0.1 \
--proxy-port 12081 \
--bind-port 5900 \
--target-port 5900 \
--timeout 86400
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.targetlogoutput: stderr
internal: 127.0.0.1 port = 2080
external: 127.0.0.1
socksmethod: none
clientmethod: none
user.privileged: root
user.unprivileged: nobody
client pass {
from: 127.0.0.0/8 to: 0.0.0.0/0
log: error
}
socks pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
command: connect bind udpassociate
socksmethod: none
log: error
}
socks pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
command: bindreply udpreply
log: error
}[Unit]
Description=socat relay – expose kelp 0.0.0.0:2222 → 127.0.0.1:22222
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/bin/socat TCP-LISTEN:2222,bind=0.0.0.0,reuseaddr,fork TCP:127.0.0.1:22222
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target[dante-socks5h]
accept = 12080
connect = 127.0.0.1:2080A separate stunnel instance from the one in /etc/stunnel/stunnel.conf, run by its own unit stunnel-vnc.service from a config in John's home directory. It is easy to miss: it does not appear in /etc/stunnel/, and the process shows as stunnel rather than stunnel4. Config: /home/john/py/ssl/stunnel-server.conf. Client counterpart on each remote machine: accept = localhost:5900, connect = kelp.critchley.biz:15900, with a CA-signed client cert (verify = 2 is enforced server-side).
; /home/john/py/ssl/stunnel-server.conf (run by stunnel-vnc.service)
foreground = yes
cert = /etc/stunnel/server.pem
key = /etc/stunnel/server.key
verify = 2
CAfile = /etc/stunnel/server.pem
[vnc-5900]
accept = 15900
connect = localhost:5900
[vnc-5901]
accept = 15901
connect = localhost:5901Stunnel mTLS (verify=2): both pomelo and kelp present certificates signed by the shared CA. A compromised connection cannot reach Dante without the client cert.
Dante: loopback only. internal: 127.0.0.1 and external: 127.0.0.1 mean the proxy and all BIND ports are on loopback. The proxy is not reachable from the internet even if the stunnel cert were somehow compromised.
SSH port 2222: public on kelp (EC2 security group open), protected by SSH public-key auth only — no password. Equivalent security to port 22 on kelp or gravlax.
VNC: port 5900 is not open in the security group, but port 15900 is (0.0.0.0/0, and the rule spans 15900-15909). So VNC is reachable from the internet, via stunnel-vnc.service. x11vnc has no password, therefore the stunnel mTLS client certificate is the only thing protecting the desktop — not a loopback restriction, as an earlier version of this note claimed. Treat the client key in /home/john/py/ssl/ and its counterpart on each client as desktop-equivalent credentials. Anyone holding a CA-signed client cert has unauthenticated access to pomelo's session.
Idle-connection resilience and application check (updated 2026-09-25): Pomelo's [kelp-dante] stunnel client enables TCP keepalive on its remote socket after 60 idle seconds, probing every 20 seconds and failing after three unanswered probes. SOCKS5 BIND has no legal heartbeat frame while awaiting its second reply, so kelp-bind-watchdog.timer performs an end-to-end check every five minutes. For an idle service, /home/john/bin/check-kelp-bind-tunnels connects and reads only the public SSH banner through kelp:2222 or the VNC RFB banner through local stunnel endpoint 127.0.0.1:15990 to Kelp's mTLS port 15900. This consumes the one-shot BIND; the client exits and systemd creates a fresh BIND. Before probing, the watchdog checks whether the service process already has an established relay to local port 22 or 5900. It skips an active relay, so a legitimate SSH or VNC session is never mistaken for a missing listener and restarted. A failed idle probe restarts only the corresponding service. No SSH client, login, agent, or SSH private key is used.
# From any host with access to kelp:2222
ssh -p 2222 john@kelp.critchley.biz
# Should reach pomelo's sshdssh kelp
nc -w3 127.0.0.1 5900 | od -An -tx1 | head -1
# Should print: 52 46 42 20 30 30 33 2e 30 30 38 0a (RFB 003.008)sudo systemctl status socks5-bind-ssh socks5-bind-vnc
sudo journalctl -u socks5-bind-ssh -n 20 --no-pagerssh kelp 'sudo netstat -tlnp | grep -E "2222|5900|2080|12080"'Work from pomelo outwards. Every hop below can fail while every systemctl status in the chain still says active (running) — so check the effect in the right-hand column, never the unit state. General method and the reasoning behind it: README/diagnosis.
| # | Hop | Check (run where stated) | Healthy looks like |
|---|---|---|---|
| 1 | pomelo BIND client | journalctl -u socks5-bind-ssh -n 20 (or -vnc) | Idle: a current BIND_LISTEN line. Active session: BIND_INCOMING and RELAY_TARGET, with no listener until that one relay ends. The process exits after the relay and systemd starts a fresh one. |
| 2 | pomelo stunnel client | sudo systemctl status stunnel4; ss -tnp | grep 12081 | An ESTABLISHED pair on 127.0.0.1:12081. A long-lived one with no traffic may be half-open; correlate with hop 1. |
| 3 | kelp Dante | ssh kelp 'systemctl status dante-sockd; ps -o lstart,cmd -p $(pgrep sockd|head -1)' | Unit active AND the running sockd is the unit's Main PID. Unit dead + process alive = someone bypassed systemd; fix that first. |
| 4 | kelp BIND listener | ssh kelp 'sudo ss -ltnp | grep -E "22222|5900"' | sockd owns 127.0.0.1:22222 (SSH) and/or 127.0.0.1:5900 (VNC). If ANY other process owns them, that is your fault — see the 2026-09-25 incident below. |
| 5 | kelp public relay | ssh kelp 'sudo ss -ltnp | grep -E "2222|15900"' | socat-ssh-relay owns 0.0.0.0:2222; stunnel owns 0.0.0.0:15900. A missing one is usually crash-looping on a squatted port, silently. |
| 6 | kelp stunnel errors | ssh kelp 'sudo tail -40 /var/log/stunnel.log' | Errors only. debug=warning does NOT log successful connects, so silence here is ambiguous — it does not prove traffic never arrived. |
| 7 | AWS security group | aws ec2 describe-security-groups --region eu-central-1 --group-ids sg-09b115cb2f93edb4f --query 'SecurityGroups[0].IpPermissions' | 22, 2222, 11080, 12080, 15900-15909 open. Not visible to iptables/ufw on the host — must be checked via the API. |
| 8 | DNS vs real IP | dig +short kelp.critchley.biz; aws ec2 describe-instances --region eu-central-1 --instance-ids i-00fbb7584cd890340 --query 'Reservations[0].Instances[0].PublicIpAddress' | The two agree. kelp's public IP changes across stop/start, so a stale A record or a hardcoded IP in a note is a real failure mode. |
| 9 | End to end | ssh -p 2222 john@kelp.critchley.biz (SSH); VNC viewer to localhost:5900 through the client stunnel (VNC) | Reaches pomelo. 'Permission denied (publickey)' means the tunnel WORKS and only auth failed — that is a pass for hops 1-8. |
Symptom shortcuts. Connection refused at hop 9 can be the expected five-second systemd restart window after the preceding one-shot connection; retry once before investigating. Connection closed remotely after a successful TLS handshake means stunnel accepted the connection but the Dante BIND target was absent at that instant. During a legitimate active session the absence of another listener is expected. A hang with no response at all points toward security-group or client-side network filtering.
One active relay per service, by design. SOCKS5 BIND accepts one incoming connection. While SSH or VNC is in use, there is no second listener and concurrent attempts are refused. After the session closes, systemd starts a fresh one-shot client after five seconds. The watchdog detects and preserves an active relay.
TIME_WAIT on 22222. After each SSH session, Dante cannot rebind to 22222 for ~60 s. The retry loop handles this automatically; there is no operator action needed.
WebDAV password rotation needed. The password was briefly exposed in shell output during initial setup (2026-09-23).
`vnc-localhost-relay.service` stub — leftover from an earlier design iteration, stopped/disabled since before this note existed. Deleted 2026-09-25 (unit file removed from kelp, daemon-reloaded, failed-state cleared).
2026-09-25 outage, root cause: unmanaged/orphan processes squatting the relay ports. Found and partly fixed during a pomelo reconnect investigation:
Radagast remote access (added 2026-10-17) — same Dante/TLS BIND design, separate public SSH port 2225 and VNC TLS port 15905; fixed kelp BIND ports 22225 and 5905. Existing pomelo and kelp desktop paths were preserved.