Self-Hosted RustDesk Server Setup: Docker & Proxmox VE LXC (+ Client Config Guide)

RustDesk Self-hosted Guide

Self-host a RustDesk Server: Two Ways — Linux Docker and Proxmox VE LXC

A private remote-control setup that uses fast P2P direct connections at home and your own server as a relay gateway when you're away.

Linux Docker Proxmox VE LXC hbbs / hbbr macOS / Windows / Linux Client

What this post covers

1. Why bother self-hosting RustDesk
2. What a self-hosted RustDesk server does
3. Option A: Install on a Linux server with Docker Compose
4. Option B: Install on Proxmox VE as an LXC
5. Common port forwarding and the security key
6. Client setup for Windows, macOS, and Linux
7. Connection test and shortcut tips

1. Why bother self-hosting RustDesk

My main reason for running RustDesk as a self-hosted server is simple. In an environment that mixes Mac, Windows, and Linux, every existing remote-control option felt a little awkward in its own way.

The limits of Mac VNC

Apple VNC is quite fast between Apple devices thanks to its own codec. But the moment you go from a Mac to Windows or Linux, you need a separate VNC client and the perceived speed drops considerably.

The hassle of Windows ↔ Linux

Connecting Windows and Linux means setting up an RDP-style config separately. RDP under Wayland in particular is tricky to deal with, and Remmina has a clunky UI with settings that feel complicated.

Services like TeamViewer or Chrome Remote Desktop are convenient, but their relay servers live outside your network. So speed depends on the state of an external server and the network path, and there's a lingering security concern about entrusting your personal machines to them indefinitely.

A self-hosted RustDesk server is a choice to reduce that friction. At home you connect as directly as possible at LAN speed, and when you're away you use the server you built as the relay point. In other words, it unifies cross-OS remote control into a single client experience while bringing the relay point onto your side.

2. What a self-hosted RustDesk server does

If you just use RustDesk as-is, it locates device IDs and attempts connections through the default public server. Setting up your own server brings that role into your home or your own server. Instead of relying on an external company's relay, your server takes on the roles of ID server and relay server.

A RustDesk server is split into two main services. hbbs is the ID server that finds where a device is, and hbbr is the relay server that forwards screen and input data on your behalf when a direct P2P connection fails.

When two devices at home are on the same LAN, they connect directly via P2P whenever possible; when a direct connection is hard because of an external network or NAT, they are relayed through your RustDesk server.

3. Option A: Install with Linux Docker Compose

For a regular Ubuntu, Debian, mini PC, or NAS-style Linux server, Docker Compose is the easiest to manage. Server data stays in a separate folder, and you can bring just the containers up and down easily.

2-1. Create a working folder

mkdir -p ~/rustdesk-server/data
cd ~/rustdesk-server

2-2. Write compose.yml

In the example below, change 192.168.0.15 to your actual Linux server's internal IP or DDNS address. As in the original docs, adding the -k _ option requires clients to enter the public key before they can use the server.

services:
  hbbs:
    container_name: rustdesk-hbbs
    image: rustdesk/rustdesk-server:latest
    command: hbbs -r 192.168.0.15:21117 -k _
    volumes:
      - ./data:/root
    ports:
      - "21115:21115"
      - "21116:21116"
      - "21116:21116/udp"
      - "21118:21118"
    depends_on:
      - hbbr
    restart: unless-stopped

  hbbr:
    container_name: rustdesk-hbbr
    image: rustdesk/rustdesk-server:latest
    command: hbbr -k _
    volumes:
      - ./data:/root
    ports:
      - "21117:21117"
      - "21119:21119"
    restart: unless-stopped

The official RustDesk docs also recommend the network_mode: "host" approach on Linux Docker. If port mapping gets confusing, host networking is simpler; if you want to clearly separate ports as on a shared server or NAS, run it with the port-mapping approach above.

2-3. Run the containers

docker compose up -d
docker compose ps

2-4. Check the public key

This value goes into every client's settings. Copy the entire long string that's printed.

cat data/id_ed25519.pub

4. Option B: Install on Proxmox VE as an LXC

If you already run Proxmox VE, it's cleaner to isolate the RustDesk server in an LXC container. The original docs' example sets the LXC internal IP to 192.168.0.15 and accesses the web admin dashboard at http://192.168.0.15:21114.

3-1. Create the LXC from the Proxmox VE Shell

If you use a community Helper Script, you can run the RustDesk Server LXC script from the Proxmox VE Shell to create the container. It's a good idea to review the script's contents once before running it.

bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/rustdeskserver.sh)"

3-2. Check the LXC IP and dashboard

Example LXC internal IP

192.168.0.15

Web admin dashboard

http://192.168.0.15:21114

3-3. Check the public key from the LXC console

In the Proxmox web GUI, select the RustDesk LXC, go to the Console tab, and run the command below.

cat /var/lib/rustdesk-server/id_ed25519.pub

The long string that's printed is the public key for the client's Key field. Enter the same value in every client — macOS, Windows, and Linux.

3-4. Reset the admin password

If you can't log into the web dashboard or don't know the initial password, set a new admin password from the LXC console.

cd /var/lib/rustdesk-api
rustdesk-api reset-admin-pwd your-desired-password

Then open http://192.168.0.15:21114 in a browser and log in with the username admin and the password you just set.

5. Port forwarding

To connect from outside your home, your router must forward ports to the RustDesk server's IP. If you'll only use it internally, you can skip router port forwarding — but if you want external access, check the ports below.

Port	Protocol	Role	Notes
21114	TCP	Web admin dashboard / API	Open carefully, only if you view the dashboard externally
21115	TCP	NAT type test and auth handling	Required
21116	TCP / UDP	ID registration, heartbeat, hole punching	Most important. UDP is needed too
21117	TCP	Relay server hbbr	Relays when P2P fails
21118	TCP	WebSocket for the web client	Optional if you don't use the web client
21119	TCP	Relay WebSocket	Optional if you don't use the web client

For an external address, a DDNS hostname is more convenient than a public IP. For example, if you set myhome.duckdns.org as the ID and relay server address, you can keep using the same settings from outside your home.

6. Client setup: common to Windows, macOS, and Linux

Whether you built the server with Docker or a PVE LXC, the client setup is almost the same. Launch RustDesk on every device, then go to Network or ID/Relay Server settings from the menu next to your ID.

1. ID Server: Enter the RustDesk server's IP or DDNS address. e.g. 192.168.0.15 or myhome.duckdns.org
2. Relay Server: Enter the same address. It's sometimes inferred if left blank, but in a homelab it's less confusing to specify it.
3. API Server: For an OSS server, usually leave it blank. Enter it only when using Pro or web-console account features.
4. Key: Paste the entire id_ed25519.pub string you got from the server.
5. Press Apply and confirm the bottom of the main screen shows Ready.

Example client input

If you test on the LAN first, put the RustDesk server's internal IP into both the ID server and Relay server as shown below. The example uses 192.168.0.15 and leaves the API server blank. In the last Key field, paste the entire id_ed25519.pub public key copied from the server.

RustDesk Settings → Network → ID/Relay server input example. For internal testing, enter the server's internal IP; for external access, switch to a DDNS address.

Required permissions when macOS is the remote target

When using a Mac as the controlled host, you must grant macOS security permissions. In System Settings → Privacy & Security, find RustDesk and allow the following.

• Screen Recording
• Accessibility
• Input Monitoring: grant this too when you need complex shortcuts or input control

After changing permissions, you must fully quit and relaunch RustDesk for them to take effect.

Installing the Linux viewer

When Linux acts only as the viewer that watches the remote screen, it runs into fewer screen-capture permission issues. On Ubuntu/Debian, install the official release .deb package, or use Flatpak/AppImage regardless of distribution.

sudo apt update
sudo apt install ./rustdesk-*.deb

flatpak install flathub com.rustdesk.RustDesk

7. Connection test and shortcut tips

Check the 9-digit ID shown on the RustDesk screen of the computer to be controlled, then enter that ID in the remote-desktop field on the controlling computer and connect. Enter the password or approve it on the other side's screen, and the session opens.

For a device you connect to all the time, set a Permanent Password on the remote target so you don't have to check the ever-changing temporary password each time.

When controlling a Mac from a Linux or Windows keyboard, the Win/Super key usually acts like macOS's Command key. For example, Win + C in the Linux viewer is delivered as Cmd + C (copy) on the Mac host.

If it doesn't feel right, adjust the Ctrl-to-Command mapping in the keyboard options on the top toolbar during a RustDesk session.

Final checklist

• Confirm both hbbs and hbbr are running on the server
• Confirm the client's ID Server, Relay Server, and Key all point to the same server
• If external access fails, first check whether 21116 UDP port forwarding is missing
• If the Mac screen isn't visible, check Screen Recording permission; if mouse/keyboard don't work, check Accessibility permission
• If you regenerate the public key or change the data folder, update the Key on every client again