Nginx Proxy Manager Load Balancing

I 100% struggled with this for a bit now. It's been keeping me from deploying apps with HA or scale in mind. Today, I finally figured it out!

Here's what I'm working with.

From the generic image above, I'm using Cloudflare to Proxy my DNS and terminate SSL, then forwarding traffic to my homelab firewall (OPNsense), then forwarding that traffic to a Nginx Proxy Manager VM that has a wildcard certificate for my domain running on my Proxmox cluster. Previously I've just used single host in NPM for the proxy setup simply because I couldn't figure out the proper way to do this. Now I have Docker Swarm setup and am slowly migrating services over to it. The first one up, is 13ft, a simple yet effective paywall bypass app from https://github.com/wasi-master/13ft.

NPM Configuration

Log into your NPM instance and go to Proxy Hosts. Click Add Proxy Host and fill it out like below:

Change up your domain name and port number as you see fit, but use "backend" as the Forward Hostname/IP. Click over to the SSL tab and select your SSL Certificate of choice, then finally to the advanced tab and fill out your "Custom Nginx Configuration" to look like what I have below:

Click Save and SSH to your NPM instance and do the following:

mkdir /data/nginx/custom
touch /data/nginx/http_top.conf

Then, you'll need to modify the newly created file with the following:

upstream backend {
    server 192.168.254.6:5000;
    server 192.168.254.7:5000;
    server 192.168.254.8:5000;
}

Save the file and start testing. I was able to use tcpdump to verify it was load balancing as expected. This specific configuration without additional definition uses "round robin" as the load balancing method.

If you want to see the traffic with tcpudmp, use the following from your ssh session:

tcpdump port 5000

You'll get output similar to this:

10:28:22.430847 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 3914647141 ecr 202375768], length 0
10:28:22.430888 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [F.], seq 577, ack 176, win 501, options [nop,nop,TS val 3914647141 ecr 202375768], length 0
10:28:22.431403 IP 192.168.254.8.5000 > npm.homelabdomain.tld.54454: Flags [F.], seq 176, ack 578, win 498, options [nop,nop,TS val 202375769 ecr 3914647141], length 0
10:28:22.431407 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 3914647141 ecr 202375769], length 0
10:33:22.330079 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [S], seq 180177399, win 64240, options [mss 1460,sackOK,TS val 33715079 ecr 0,nop,wscale 7], length 0
10:33:22.331293 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [S.], seq 799044855, ack 180177400, win 64308, options [mss 1410,sackOK,TS val 523646323 ecr 33715079,nop,wscale 7], length 0
10:33:22.331302 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 33715080 ecr 523646323], length 0
10:33:22.331347 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [P.], seq 1:578, ack 1, win 502, options [nop,nop,TS val 33715080 ecr 523646323], length 577
10:33:22.332055 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [.], ack 578, win 498, options [nop,nop,TS val 523646324 ecr 33715080], length 0
10:33:22.333427 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [P.], seq 1:176, ack 578, win 498, options [nop,nop,TS val 523646325 ecr 33715080], length 175
10:33:22.333432 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 33715082 ecr 523646325], length 0
10:33:22.333529 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [F.], seq 578, ack 176, win 501, options [nop,nop,TS val 33715083 ecr 523646325], length 0
10:33:22.334460 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [F.], seq 176, ack 579, win 498, options [nop,nop,TS val 523646326 ecr 33715083], length 0
10:33:22.334467 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 33715083 ecr 523646326], length 0
10:38:22.506149 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [S], seq 3866721961, win 64240, options [mss 1460,sackOK,TS val 992168023 ecr 0,nop,wscale 7], length 0
10:38:22.507256 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [S.], seq 1024843093, ack 3866721962, win 64308, options [mss 1410,sackOK,TS val 2515934452 ecr 992168023,nop,wscale 7], length 0
10:38:22.507265 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 992168024 ecr 2515934452], length 0
10:38:22.507311 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [P.], seq 1:578, ack 1, win 502, options [nop,nop,TS val 992168024 ecr 2515934452], length 577
10:38:22.508198 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [.], ack 578, win 498, options [nop,nop,TS val 2515934453 ecr 992168024], length 0
10:38:22.509597 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [P.], seq 1:176, ack 578, win 498, options [nop,nop,TS val 2515934455 ecr 992168024], length 175
10:38:22.509603 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 992168027 ecr 2515934455], length 0
10:38:22.509650 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [F.], seq 578, ack 176, win 501, options [nop,nop,TS val 992168027 ecr 2515934455], length 0
10:38:22.510594 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [F.], seq 176, ack 579, win 498, options [nop,nop,TS val 2515934456 ecr 992168027], length 0
10:38:22.510600 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 992168028 ecr 2515934456], length 0

You'll notice all 3 IP's listed in the "backend" configuration are getting traffic.

WOOT WOOT!!