netbird server setup behind nat

netbird | pfsync | haproxy | nginx | client

tested on debian12

                               ┌──────────────────┐       CARP      ┌──────────────────┐       
                               │                  │    PUBLIC VIP   │                  │       
                               │   pfsync1        │                 │   pfsync2        │       
                               │   10.1.0.28/16   │       CARP      │   10.1.0.29/16   │       
                               │                  │   INTERNAL VIP  │                  │       
                               └──────────────────┘ 10.1.255.254/16 └──────────────────┘       

                                                    │ │         │                              
                                                    │ │         │                              
                                                    │ │         └───────────────────┐          
                    ┌───────────────────────────────┘ │                             │          
                    │                                 │ 10000/tcp (http)            │          
         80,443/tcp │                                 │                             │          
                    ▼                                 │ 3478,5349/tcp               │          
┌────────────────────────┐                            │ 3478,5349/udp               │          
│                        │                            │ 49152:49999/udp             │          
│   dnc-haproxy          │                            ▼                             │          
│   10.1.0.27/16         │                  ┌────────────────────────┐              │          
│                        ├────────────────► │                        │              │          
└────────────────────────┘                  │   dnc-netbird          │  ┌───────────┤          
                                            │   10.1.0.20/16         │  │           │          
                                            │                        │  │ 51853/udp │ 51854/udp
                                            └────────────────────────┘  │           │          
                                                                        ▼           ▼          
                                                              ┌──────────────┐ ┌──────────────┐
                                                              │              │ │              │
                                                              │ bastion53    │ │ bastion54    │
                                                              │ 10.1.0.53/16 │ │ 10.1.0.54/16 │
                                                              │              │ │              │
                                                              └──────────────┘ └──────────────┘

warning // lessons learned

public poc

that poc needs to be on the public network as NETBIRD_DOMAIN resolves while SSL is required – not sure it’s possible to PoC within with static name resolution

behind a reverse-proxy

in the compose file, api mgmt endpoints need to be fixed manually

NETBIRD_MGMT_API_ENDPOINT
NETBIRD_MGMT_GRPC_API_ENDPOINT

mgmt grpc needs to be reverse-proxied in a special way

as for signal grpc, we did not manage: haproxy was constantly reporting a 503, therefore we switched to plain text http for that haproxy backend.

network requirements

you need an auth server and it needs to resolve from within the docker instances e.g.

    ping -c1 keycloak.demo.nethence.com
    nmap -p 443 keycloak.demo.nethence.com

see netbird-pfsync if you want DNAT

and eventually a reverse-proxy e.g. haproxy or nginx

system requirements

apt install docker.io docker-compose # unless you want docker-ce
docker ps

apt install jq curl gettext-base

keycloak setup

see keycloak and setup as such

realm       netbird
user        netbird & credentials (not temporary)
clients     netbird-client (OIDC & oauth2 device auth grant) -- NETBIRD_AUTH_CLIENT_ID
  root url  https://netbird.demo.nethence.com/
  redirect  https://netbird.demo.nethence.com/* & http://localhost:53000
  post logout   https://netbird.demo.nethence.com/*
  web origins   +
client scope    api (default & OIDC)
  mapper audience netbird-audience (client audience netbird-client)
clients     netbird-client / client scopes --> api (default)
clients     netbird-backend (OIDC & oauth2 device auth grant) -- NETBIRD_IDP_MGMT_CLIENT_ID
            + client auth
            + service access grants?
        credentials -- NETBIRD_IDP_MGMT_CLIENT_SECRET
        roles + view-users

install

grab latest tag

ver=0.27.6

wget https://github.com/netbirdio/netbird/archive/refs/tags/v$ver.tar.gz
tar xzf v$ver.tar.gz
rm -f v$ver.tar.gz

setup

cd netbird-$ver/infrastructure_files/
ls -lF setup.env.example
vi setup.env # new file

NETBIRD_DOMAIN="netbird.demo.nethence.com"
NETBIRD_TURN_EXTERNAL_IP="@@@@@ front-facing public ip here @@@@@"

# keycloak setup
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://keycloak.demo.nethence.com/realms/netbird/.well-known/openid-configuration"
NETBIRD_USE_AUTH0=false
NETBIRD_AUTH_CLIENT_ID="netbird-client"
NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"
NETBIRD_AUTH_AUDIENCE="netbird-client"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID="netbird-client"
NETBIRD_MGMT_IDP="keycloak"
NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend"
NETBIRD_IDP_MGMT_CLIENT_SECRET="@@@@@ netbird-backend client secret here @@@@@"
NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://keycloak.demo.nethence.com/admin/realms/netbird"

# we already got an ssl certificate
NETBIRD_DISABLE_LETSENCRYPT=true
NETBIRD_DISABLE_ANONYMOUS_METRICS=true
NETBIRD_MGMT_DNS_DOMAIN=netbird.selfhosted

NETBIRD_MGMT_API_PORT="443"
NETBIRD_SIGNAL_PORT="443"
TURN_MIN_PORT="49152"
TURN_MAX_PORT="49999"

proceed

./configure.sh

cd artifacts/

check that you obtained the right kc setup (mgmt needs that to get a token endpoint)

cat openid-configuration.json
#curl https://keycloak.demo.nethence.com/realms/netbird/.well-known/openid-configuration \
#   -o openid-configuration.json
vi docker-compose.yml

(make the dashboard listen 80 only)
(remove nginx ssl)
(remove letsencrypt)

tuning 1/3

in our case we also had to add static name resolution (anyhow it might help, makes the path shorter)

  dashboard:
    extra_hosts:
      - "keycloak.demo.nethence.com=10.1.0.27"
      - "netbird.demo.nethence.com=10.1.0.27"

  management:
    extra_hosts:
      - "keycloak.demo.nethence.com=10.1.0.27"
      - "netbird.demo.nethence.com=10.1.0.27"

tuning 2/3

vi docker-compose.yml

  dashboard:
    environment:
      - NETBIRD_MGMT_API_ENDPOINT=https://netbird.demo.nethence.com
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.demo.nethence.com

tuning 3/3

vi management.json

    "Signal": {
        "Proto": "http",

ready to go

#docker compose down
docker compose up -d
# --force-recreate

docker ps -a
netstat -lntup

https://netbird.demo.nethence.com/ ==> login as netbird (owner)

troubleshooting

docker compose logs signal
docker compose logs management
docker compose logs coturn
docker compose logs dashboard

troubleshooting

    Fatal glibc error: CPU does not support x86-64-v2

==> either rebuild keycloak (w/o RHEL9 as base) or change the VPS – works fine on xen/pvh

Error: failed retrieving a new idp manager with err: keycloak IdP configuration is incomplete, TokenEndpoint is missing

==> fix openid-configuration.json

resources

https://docs.netbird.io/selfhosted/selfhosted-guide

https://docs.netbird.io/selfhosted/identity-providers#keycloak

kc setup

https://docs.netbird.io/selfhosted/identity-providers#keycloak

troubles

https://github.com/netbirdio/netbird/issues/1247 ==> using advanced install and kc instead


HOME | GUIDES | LECTURES | LAB | SMTP HEALTH | HTML5 | CONTACT
Copyright © 2024 Pierre-Philipp Braun