Self-host a RustDesk Server: Two Ways — Linux Docker and Proxmox VE LXC
A private remote-control setup that uses fast P2P direct connections at home and your own server as a relay gateway when you're away.
- Why bother self-hosting RustDesk
- What a self-hosted RustDesk server does
- Option A: Install on a Linux server with Docker Compose
- Option B: Install on Proxmox VE as an LXC
- Common port forwarding and the security key
- Client setup for Windows, macOS, and Linux
- Connection test and shortcut tips
1. Why bother self-hosting RustDesk
My main reason for running RustDesk as a self-hosted server is simple. In an environment that mixes Mac, Windows, and Linux, every existing remote-control option felt a little awkward in its own way.
Apple VNC is quite fast between Apple devices thanks to its own codec. But the moment you go from a Mac to Windows or Linux, you need a separate VNC client and the perceived speed drops considerably.
Connecting Windows and Linux means setting up an RDP-style config separately. RDP under Wayland in particular is tricky to deal with, and Remmina has a clunky UI with settings that feel complicated.
A self-hosted RustDesk server is a choice to reduce that friction. At home you connect as directly as possible at LAN speed, and when you're away you use the server you built as the relay point. In other words, it unifies cross-OS remote control into a single client experience while bringing the relay point onto your side.
2. What a self-hosted RustDesk server does
If you just use RustDesk as-is, it locates device IDs and attempts connections through the default public server. Setting up your own server brings that role into your home or your own server. Instead of relying on an external company's relay, your server takes on the roles of ID server and relay server.
A RustDesk server is split into two main services. hbbs is the ID server that finds where a device is, and hbbr is the relay server that forwards screen and input data on your behalf when a direct P2P connection fails.
3. Option A: Install with Linux Docker Compose
For a regular Ubuntu, Debian, mini PC, or NAS-style Linux server, Docker Compose is the easiest to manage. Server data stays in a separate folder, and you can bring just the containers up and down easily.
2-1. Create a working folder
mkdir -p ~/rustdesk-server/data cd ~/rustdesk-server
2-2. Write compose.yml
In the example below, change 192.168.0.15 to your actual Linux server's internal IP or DDNS address. As in the original docs, adding the -k _ option requires clients to enter the public key before they can use the server.
services:
hbbs:
container_name: rustdesk-hbbs
image: rustdesk/rustdesk-server:latest
command: hbbs -r 192.168.0.15:21117 -k _
volumes:
- ./data:/root
ports:
- "21115:21115"
- "21116:21116"
- "21116:21116/udp"
- "21118:21118"
depends_on:
- hbbr
restart: unless-stopped
hbbr:
container_name: rustdesk-hbbr
image: rustdesk/rustdesk-server:latest
command: hbbr -k _
volumes:
- ./data:/root
ports:
- "21117:21117"
- "21119:21119"
restart: unless-stopped
network_mode: "host" approach on Linux Docker. If port mapping gets confusing, host networking is simpler; if you want to clearly separate ports as on a shared server or NAS, run it with the port-mapping approach above.2-3. Run the containers
docker compose up -d docker compose ps
2-4. Check the public key
This value goes into every client's settings. Copy the entire long string that's printed.
cat data/id_ed25519.pub
4. Option B: Install on Proxmox VE as an LXC
If you already run Proxmox VE, it's cleaner to isolate the RustDesk server in an LXC container. The original docs' example sets the LXC internal IP to 192.168.0.15 and accesses the web admin dashboard at http://192.168.0.15:21114.
3-1. Create the LXC from the Proxmox VE Shell
If you use a community Helper Script, you can run the RustDesk Server LXC script from the Proxmox VE Shell to create the container. It's a good idea to review the script's contents once before running it.
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/rustdeskserver.sh)"
3-2. Check the LXC IP and dashboard
192.168.0.15
http://192.168.0.15:21114
3-3. Check the public key from the LXC console
In the Proxmox web GUI, select the RustDesk LXC, go to the Console tab, and run the command below.
cat /var/lib/rustdesk-server/id_ed25519.pub
The long string that's printed is the public key for the client's Key field. Enter the same value in every client — macOS, Windows, and Linux.
3-4. Reset the admin password
If you can't log into the web dashboard or don't know the initial password, set a new admin password from the LXC console.
cd /var/lib/rustdesk-api rustdesk-api reset-admin-pwd your-desired-password
Then open http://192.168.0.15:21114 in a browser and log in with the username admin and the password you just set.
5. Port forwarding
To connect from outside your home, your router must forward ports to the RustDesk server's IP. If you'll only use it internally, you can skip router port forwarding — but if you want external access, check the ports below.
| Port | Protocol | Role | Notes |
|---|---|---|---|
| 21114 | TCP | Web admin dashboard / API | Open carefully, only if you view the dashboard externally |
| 21115 | TCP | NAT type test and auth handling | Required |
| 21116 | TCP / UDP | ID registration, heartbeat, hole punching | Most important. UDP is needed too |
| 21117 | TCP | Relay server hbbr | Relays when P2P fails |
| 21118 | TCP | WebSocket for the web client | Optional if you don't use the web client |
| 21119 | TCP | Relay WebSocket | Optional if you don't use the web client |
myhome.duckdns.org as the ID and relay server address, you can keep using the same settings from outside your home.6. Client setup: common to Windows, macOS, and Linux
Whether you built the server with Docker or a PVE LXC, the client setup is almost the same. Launch RustDesk on every device, then go to Network or ID/Relay Server settings from the menu next to your ID.
- ID Server: Enter the RustDesk server's IP or DDNS address. e.g.
192.168.0.15ormyhome.duckdns.org - Relay Server: Enter the same address. It's sometimes inferred if left blank, but in a homelab it's less confusing to specify it.
- API Server: For an OSS server, usually leave it blank. Enter it only when using Pro or web-console account features.
- Key: Paste the entire
id_ed25519.pubstring you got from the server. - Press Apply and confirm the bottom of the main screen shows Ready.
Example client input
If you test on the LAN first, put the RustDesk server's internal IP into both the ID server and Relay server as shown below. The example uses 192.168.0.15 and leaves the API server blank. In the last Key field, paste the entire id_ed25519.pub public key copied from the server.
Required permissions when macOS is the remote target
When using a Mac as the controlled host, you must grant macOS security permissions. In System Settings → Privacy & Security, find RustDesk and allow the following.
- Screen Recording
- Accessibility
- Input Monitoring: grant this too when you need complex shortcuts or input control
After changing permissions, you must fully quit and relaunch RustDesk for them to take effect.
Installing the Linux viewer
When Linux acts only as the viewer that watches the remote screen, it runs into fewer screen-capture permission issues. On Ubuntu/Debian, install the official release .deb package, or use Flatpak/AppImage regardless of distribution.
sudo apt update sudo apt install ./rustdesk-*.deb
flatpak install flathub com.rustdesk.RustDesk
7. Connection test and shortcut tips
Check the 9-digit ID shown on the RustDesk screen of the computer to be controlled, then enter that ID in the remote-desktop field on the controlling computer and connect. Enter the password or approve it on the other side's screen, and the session opens.
For a device you connect to all the time, set a Permanent Password on the remote target so you don't have to check the ever-changing temporary password each time.
Win + C in the Linux viewer is delivered as Cmd + C (copy) on the Mac host.If it doesn't feel right, adjust the Ctrl-to-Command mapping in the keyboard options on the top toolbar during a RustDesk session.
Final checklist
- Confirm both
hbbsandhbbrare running on the server - Confirm the client's ID Server, Relay Server, and Key all point to the same server
- If external access fails, first check whether 21116 UDP port forwarding is missing
- If the Mac screen isn't visible, check Screen Recording permission; if mouse/keyboard don't work, check Accessibility permission
- If you regenerate the public key or change the data folder, update the Key on every client again
댓글