Client stuck on "Not ready" with self-hosted server (public rs-ny works, same network)

[pata2004]

Environment

Client

• OS: AlmaLinux 9.x (GNOME)
• RustDesk client: 1.4.4 (AppImage)
• Network: home connection behind mobile 4G/5G (CGNAT), no IPv6

Server

• OS: AlmaLinux 9.x (VM on Proxmox, public static IP)
• RustDesk server: hbbs / hbbr 1.1.14
• Hostname: rustdesk.myserver.com
• Ports:
  • hbbs: UDP/TCP 21116, TCP 21115
  • hbbr: TCP 21117
• No HTTP/API server

---

Summary

If I use the free public server rs-ny.rustdesk.com, the client quickly becomes “Ready” and everything works, even from home behind 5G CGNAT.

If I change only the configuration to use my self-hosted server (rustdesk.myserver.com), the same client, same machine, same network stays “Not ready”. Other peers trying to connect to this client see: “ID does not exist”.

The self-hosted server appears healthy: hbbs and hbbr are running, ports are open, and logs show the key correctly loaded.

---

Server configuration

hbbs and hbbr are running as systemd services:

$ ps aux | egrep 'hbbs|hbbr'
hbbs 1.1.14
hbbr 1.1.14

$ ss -ltnup | egrep '21115|21116|21117'
udp   UNCONN 0      0      *:21116    *:*    users:(("hbbs",pid=...,fd=...))
tcp   LISTEN 0      128    *:21116    *:*    users:(("hbbs",pid=...,fd=...))
tcp   LISTEN 0      128    *:21115    *:*    users:(("hbbs",pid=...,fd=...))
tcp   LISTEN 0      128    *:21117    *:*    users:(("hbbr",pid=...,fd=...))


/var/log/rustdesk/signalserver.log (hbbs):
Private key comes from id_ed25519
Key: XXXXXXXXXX=
DB_URL=./db_v2.sqlite3
serial=0
rendezvous-servers=[]
Listening on tcp/udp :21116
Listening on tcp :21115, extra port for NAT test
Listening on websocket :21118


/var/log/rustdesk/relayserver.log (hbbr):
#blacklist(blacklist.txt): 0
#blocklist(blocklist.txt): 0
Listening on tcp :21117
Listening on websocket :21119
Start
...
(relay requests from other clients appear here)



Client configuration (self-hosted server)

After wiping config:
pkill rustdesk 2>/dev/null || true
mv ~/.config/rustdesk ~/.config/rustdesk.backup.$(date +%s)
mv ~/.local/share/RustDesk ~/.local/share/RustDesk.backup.$(date +%s) 2>/dev/null || true
mv ~/.local/share/logs/RustDesk ~/.local/share/logs/RustDesk.backup.$(date +%s) 2>/dev/null || true


~/.config/rustdesk/RustDesk2.toml:
rendezvous_server = 'rustdesk.myserver.com:21116'
nat_type = 1
serial = 0
unlock_pin = ''
trusted_devices = ''

[options]
local-ip-addr             = '192.168.x.x'
custom-rendezvous-server  = 'rustdesk.myserver.com'
relay-server              = 'rustdesk.myserver.com'
key                       = 'XXXXXXXXXX='
av1-test                  = 'Y'
disable-udp               = 'N'


~/.config/rustdesk/RustDesk.toml (after confirming key manually):
enc_id = '...'
password = ''
salt = '...'
key_pair = [ ... ]
key_confirmed = true

[keys_confirmed]
rustdesk   = true
rs-ny      = true
myserver   = true
"rustdesk.myserver.com" = true
"X.X.X.X"               = true   # server public IP


enc_id = '...'
password = ''
salt = '...'
key_pair = [ ... ]
key_confirmed = true

[keys_confirmed]
rustdesk   = true
rs-ny      = true
myserver   = true
"rustdesk.myserver.com" = true
"X.X.X.X"               = true   # server public IP


Client log with self-hosted server (problem case)

From ~/.local/share/logs/RustDesk/rustdesk_rCURRENT.log:
INFO  server not started: Connection refused (os error 111), no_server: false
WARN  Failed to bind IPv6 socket: Network is unreachable (os error 101)
ERROR Failed to get public IPv6 address: UDP socket error

INFO  start rendezvous mediator of rustdesk.myserver.com:21116
INFO  start udp: rustdesk.myserver.com:21116

DEBUG Got nat response from rustdesk.myserver.com:21116: port=10499
DEBUG Got nat response from rustdesk.myserver.com:21115: port=10499

INFO  register_pk of rustdesk.myserver.com due to key not confirmed
INFO  register_pk of rustdesk.myserver.com due to key not confirmed
INFO  register_pk of rustdesk.myserver.com due to key not confirmed
... (repeats every few seconds, GUI stays "Not ready")



Client log with public server (works)

With a clean config and default settings (using rs-ny.rustdesk.com), I see for example:
INFO  start rendezvous mediator of rs-ny.rustdesk.com
INFO  start udp: rs-ny.rustdesk.com:21116
INFO  register_pk of rs-ny due to key not confirmed
INFO  Generated new keypair for id:
DEBUG Update rendezvous_server in config to rs-ny.rustdesk.com:21116
INFO  UUID_MISMATCH received from rs-ny.rustdesk.com:21116

The GUI then shows “Ready” and other peers can connect to this client without problems, even behind 5G CGNAT.


Network checks

From the home client to the self-hosted server:
ping rustdesk.myserver.com              # OK
nc -zv rustdesk.myserver.com 21115      # Connected
nc -zv rustdesk.myserver.com 21116      # Connected
nc -zv rustdesk.myserver.com 21117      # Connected
Only IPv4 is available here, so the IPv6 warnings seem expected and also appear with the public server.



Steps to reproduce
1-Run hbbs and hbbr 1.1.14 on a public server as shown above (default ports 21115/21116/21117, with key generated in id_ed25519).
2- Configure a Linux client (RustDesk 1.4.4 AppImage on AlmaLinux) behind 4G/5G mobile CGNAT to use the self-hosted server:
2.1-rendezvous_server = 'your-domain:21116'
2.2-custom-rendezvous-server = 'your-domain'
2.3-relay-server = 'your-domain'
2.4-key = '<server key from id_ed25519.pub>'
3-Start RustDesk and check the status: it stays on “Not ready”.
4-On another machine, try to connect using that client ID → “ID does not exist”.
5-Reset client config and use the default public server (rs-ny.rustdesk.com) → status becomes “Ready” and connections work.


Expected behavior
With a valid key and a reachable hbbs/hbbr server, the client should become “Ready” when using the self-hosted server, just like it does with rs-ny.rustdesk.com.


Actual behavior
- Client stays “Not ready” when using the self-hosted server.
- Server logs look normal (key is loaded, ports listening, relay in use by other clients).
- Client log keeps printing:
        register_pk ... due to key not confirmed
even though key_confirmed = true and the [keys_confirmed] section contains the rendezvous hostname and IP.


Question

Is there anything wrong in my self-hosted configuration (especially around server key / rendezvous vs relay configuration) that would explain why key confirmation never succeeds and the client never reaches the “Ready” state behind mobile 4G/5G?

Or could this be a compatibility/logic issue between the 1.4.x client and 1.1.14 self-hosted servers with a custom key?

Any hints on what to check or how to debug the register_pk ... due to key not confirmed loop would be very welcome.