Nginx Proxy Manager Load Balancing

I 100% struggled with this for a bit now. It’s been keeping me from deploying apps with HA or scale in mind. Today, I finally figured it out!

Here’s what I’m working with.

From the generic image above, I’m using Cloudflare to Proxy my DNS and terminate SSL, then forwarding traffic to my homelab firewall (OPNsense), then forwarding that traffic to a Nginx Proxy Manager VM that has a wildcard certificate for my domain running on my Proxmox cluster. Previously I’ve just used single host in NPM for the proxy setup simply because I couldn’t figure out the proper way to do this. Now I have Docker Swarm setup and am slowly migrating services over to it. The first one up, is 13ft, a simple yet effective paywall bypass app from https://github.com/wasi-master/13ft.

NPM Configuration

Log into your NPM instance and go to Proxy Hosts. Click Add Proxy Host and fill it out like below:

Change up your domain name and port number as you see fit, but use “backend” as the Forward Hostname/IP. Click over to the SSL tab and select your SSL Certificate of choice, then finally to the advanced tab and fill out your “Custom Nginx Configuration” to look like what I have below:

Click Save and SSH to your NPM instance and do the following:

mkdir /data/nginx/customtouch /data/nginx/http_top.conf

Then, you’ll need to modify the newly created file with the following:

upstream backend {    server 192.168.254.6:5000;    server 192.168.254.7:5000;    server 192.168.254.8:5000;}

Save the file and start testing. I was able to use tcpdump to verify it was load balancing as expected. This specific configuration without additional definition uses “round robin” as the load balancing method.

If you want to see the traffic with tcpudmp, use the following from your ssh session:

tcpdump port 5000

You’ll get output similar to this:

10:28:22.430847 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 3914647141 ecr 202375768], length 010:28:22.430888 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [F.], seq 577, ack 176, win 501, options [nop,nop,TS val 3914647141 ecr 202375768], length 010:28:22.431403 IP 192.168.254.8.5000 > npm.homelabdomain.tld.54454: Flags [F.], seq 176, ack 578, win 498, options [nop,nop,TS val 202375769 ecr 3914647141], length 010:28:22.431407 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 3914647141 ecr 202375769], length 010:33:22.330079 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [S], seq 180177399, win 64240, options [mss 1460,sackOK,TS val 33715079 ecr 0,nop,wscale 7], length 010:33:22.331293 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [S.], seq 799044855, ack 180177400, win 64308, options [mss 1410,sackOK,TS val 523646323 ecr 33715079,nop,wscale 7], length 010:33:22.331302 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 33715080 ecr 523646323], length 010:33:22.331347 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [P.], seq 1:578, ack 1, win 502, options [nop,nop,TS val 33715080 ecr 523646323], length 57710:33:22.332055 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [.], ack 578, win 498, options [nop,nop,TS val 523646324 ecr 33715080], length 010:33:22.333427 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [P.], seq 1:176, ack 578, win 498, options [nop,nop,TS val 523646325 ecr 33715080], length 17510:33:22.333432 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 33715082 ecr 523646325], length 010:33:22.333529 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [F.], seq 578, ack 176, win 501, options [nop,nop,TS val 33715083 ecr 523646325], length 010:33:22.334460 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [F.], seq 176, ack 579, win 498, options [nop,nop,TS val 523646326 ecr 33715083], length 010:33:22.334467 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 33715083 ecr 523646326], length 010:38:22.506149 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [S], seq 3866721961, win 64240, options [mss 1460,sackOK,TS val 992168023 ecr 0,nop,wscale 7], length 010:38:22.507256 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [S.], seq 1024843093, ack 3866721962, win 64308, options [mss 1410,sackOK,TS val 2515934452 ecr 992168023,nop,wscale 7], length 010:38:22.507265 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 992168024 ecr 2515934452], length 010:38:22.507311 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [P.], seq 1:578, ack 1, win 502, options [nop,nop,TS val 992168024 ecr 2515934452], length 57710:38:22.508198 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [.], ack 578, win 498, options [nop,nop,TS val 2515934453 ecr 992168024], length 010:38:22.509597 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [P.], seq 1:176, ack 578, win 498, options [nop,nop,TS val 2515934455 ecr 992168024], length 17510:38:22.509603 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 992168027 ecr 2515934455], length 010:38:22.509650 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [F.], seq 578, ack 176, win 501, options [nop,nop,TS val 992168027 ecr 2515934455], length 010:38:22.510594 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [F.], seq 176, ack 579, win 498, options [nop,nop,TS val 2515934456 ecr 992168027], length 010:38:22.510600 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 992168028 ecr 2515934456], length 0

You’ll notice all 3 IP’s listed in the “backend” configuration are getting traffic.

WOOT WOOT!!