Tag: Docker

  • Nginx Proxy Manager Load Balancing

    I 100% struggled with this for a bit now. It’s been keeping me from deploying apps with HA or scale in mind. Today, I finally figured it out!

    Here’s what I’m working with.

    Homelab Traffic Diagram Simplified

    From the generic image above, I’m using Cloudflare to Proxy my DNS and terminate SSL, then forwarding traffic to my homelab firewall (OPNsense), then forwarding that traffic to a Nginx Proxy Manager VM that has a wildcard certificate for my domain running on my Proxmox cluster. Previously I’ve just used single host in NPM for the proxy setup simply because I couldn’t figure out the proper way to do this. Now I have Docker Swarm setup and am slowly migrating services over to it. The first one up, is 13ft, a simple yet effective paywall bypass app from https://github.com/wasi-master/13ft.

    NPM Configuration

    Log into your NPM instance and go to Proxy Hosts. Click Add Proxy Host and fill it out like below:

    NPM Proxy Host Configuration

    Change up your domain name and port number as you see fit, but use “backend” as the Forward Hostname/IP. Click over to the SSL tab and select your SSL Certificate of choice, then finally to the advanced tab and fill out your “Custom Nginx Configuration” to look like what I have below:

    NPM Proxy Host Advanced

    Click Save and SSH to your NPM instance and do the following:

    mkdir /data/nginx/customtouch /data/nginx/http_top.conf

    Then, you’ll need to modify the newly created file with the following:

    upstream backend {    server 192.168.254.6:5000;    server 192.168.254.7:5000;    server 192.168.254.8:5000;}

    Save the file and start testing. I was able to use tcpdump to verify it was load balancing as expected. This specific configuration without additional definition uses “round robin” as the load balancing method.

    If you want to see the traffic with tcpudmp, use the following from your ssh session:

    tcpdump port 5000

    You’ll get output similar to this:

    10:28:22.430847 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 3914647141 ecr 202375768], length 010:28:22.430888 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [F.], seq 577, ack 176, win 501, options [nop,nop,TS val 3914647141 ecr 202375768], length 010:28:22.431403 IP 192.168.254.8.5000 > npm.homelabdomain.tld.54454: Flags [F.], seq 176, ack 578, win 498, options [nop,nop,TS val 202375769 ecr 3914647141], length 010:28:22.431407 IP npm.homelabdomain.tld.54454 > 192.168.254.8.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 3914647141 ecr 202375769], length 010:33:22.330079 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [S], seq 180177399, win 64240, options [mss 1460,sackOK,TS val 33715079 ecr 0,nop,wscale 7], length 010:33:22.331293 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [S.], seq 799044855, ack 180177400, win 64308, options [mss 1410,sackOK,TS val 523646323 ecr 33715079,nop,wscale 7], length 010:33:22.331302 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 33715080 ecr 523646323], length 010:33:22.331347 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [P.], seq 1:578, ack 1, win 502, options [nop,nop,TS val 33715080 ecr 523646323], length 57710:33:22.332055 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [.], ack 578, win 498, options [nop,nop,TS val 523646324 ecr 33715080], length 010:33:22.333427 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [P.], seq 1:176, ack 578, win 498, options [nop,nop,TS val 523646325 ecr 33715080], length 17510:33:22.333432 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 33715082 ecr 523646325], length 010:33:22.333529 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [F.], seq 578, ack 176, win 501, options [nop,nop,TS val 33715083 ecr 523646325], length 010:33:22.334460 IP 192.168.254.7.5000 > npm.homelabdomain.tld.35660: Flags [F.], seq 176, ack 579, win 498, options [nop,nop,TS val 523646326 ecr 33715083], length 010:33:22.334467 IP npm.homelabdomain.tld.35660 > 192.168.254.7.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 33715083 ecr 523646326], length 010:38:22.506149 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [S], seq 3866721961, win 64240, options [mss 1460,sackOK,TS val 992168023 ecr 0,nop,wscale 7], length 010:38:22.507256 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [S.], seq 1024843093, ack 3866721962, win 64308, options [mss 1410,sackOK,TS val 2515934452 ecr 992168023,nop,wscale 7], length 010:38:22.507265 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 992168024 ecr 2515934452], length 010:38:22.507311 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [P.], seq 1:578, ack 1, win 502, options [nop,nop,TS val 992168024 ecr 2515934452], length 57710:38:22.508198 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [.], ack 578, win 498, options [nop,nop,TS val 2515934453 ecr 992168024], length 010:38:22.509597 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [P.], seq 1:176, ack 578, win 498, options [nop,nop,TS val 2515934455 ecr 992168024], length 17510:38:22.509603 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 176, win 501, options [nop,nop,TS val 992168027 ecr 2515934455], length 010:38:22.509650 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [F.], seq 578, ack 176, win 501, options [nop,nop,TS val 992168027 ecr 2515934455], length 010:38:22.510594 IP 192.168.254.6.5000 > npm.homelabdomain.tld.41766: Flags [F.], seq 176, ack 579, win 498, options [nop,nop,TS val 2515934456 ecr 992168027], length 010:38:22.510600 IP npm.homelabdomain.tld.41766 > 192.168.254.6.5000: Flags [.], ack 177, win 501, options [nop,nop,TS val 992168028 ecr 2515934456], length 0

    You’ll notice all 3 IP’s listed in the “backend” configuration are getting traffic.

    WOOT WOOT!!

  • a simple homelab

    a simple homelab

    I’ve tried writing this post a dozen times now and I think I’ll go with the most simplistic route of just giving a high level overview. You can ask questions or Google your heart out to find the missing pieces. The purpose of this document is to inspire you to build a homelab, not a full step-by-step so here we go….

    Proxmox is a free, enterprise grade, open-source virtualization platform that allows you to run multiple virtual machines on a single host machine. This is a great way to experiment with different operating systems and software without having to dedicate a physical machine to each one. Docker is a containerization platform that allows you to package an application with all of its dependencies into a self-contained unit. This makes it easy to deploy and run applications consistently across different environments. OpenMediaVault is a network-attached storage (NAS) solution that allows you to create a centralized storage pool for your data.

    By combining these three technologies, you can create a powerful and versatile homelab that can be used for a variety of purposes. For example, you could use Proxmox to run a virtual machine for your Ghost blog, a virtual machine for a web server, and a virtual machine for a media server like Plex. You could then use Docker to deploy containers for additional services, such as a database or a development environment. Finally, you could use OpenMediaVault to create a centralized storage pool for your data with NFS or SMB, such as your blog posts, media files, and backups.

    Links

    Proxmox Server Solutions
    Proxmox develops powerful and efficient open-source server solutions like the Proxmox VE platform, Proxmox Backup Server, and Proxmox Mail Gateway.

    Proxmox – Virtualization

    Docker: Accelerated Container Application Development
    Docker is a platform designed to help developers build, share, and run container applications. We handle the tedious setup, so you can focus on the code.

    Docker – Containerization

    Kubernetes and Docker Container Management Software
    Portainer is your container management software to deploy, troubleshoot, and secure applications across cloud, datacenter, and Industrial IoT use cases.

    Portainer – Kubernetes and Docker Management

    Proxmox VE Helper-Scripts
    A Front-end for the Proxmox VE Helper-Scripts (Community) Repository. Featuring over 200+ scripts to help you manage your Proxmox VE environment.

    Proxmox Helper Scripts to Launch Servers Quickly

    openmediavault – The open network attached storage solution
    openmediavault is the next generation network attached storage (NAS) solution based on Debian Linux. It contains services like SSH, (S)FTP, SMB/CIFS, AFS, UPnP media server, DAAP media server, RSync, BitTorrent client and many more.

    OpenMediaVault – Network Attached Storage

    Ghost: The best open source blog & newsletter platform
    Beautiful, modern publishing with email newsletters and paid subscriptions built-in. Used by Platformer, 404Media, Lever News, Tangle, The Browser, and thousands more.

    Ghost – Content Management

    Get Started

    I am building this on a 12 year old Dell Optiplex 7010 with an i7 CPU, 12 Gb RAM, and a 240Gb SSD. It’s not much, but it’ll more than get the job done and get you on your way to building a new homelab or expanding your current one.

    We will start off easy, but also be fully functional by the time this is done. On top of Docker we will run 3 service VM’s. Nginx Proxy Manager, Ghost CMS and Watchtower. Ghost is the server we will present to the world. Nginx Proxy Manager will handle SSL offload and some security fundementals. Watchtower will keep the Docker containers updated. Docker will be managed via Portainer giving a very well polished and extensable Web UI for building your homelab. Essentially, this is what we are building:

    Bottom Up – Hardware –> Proxmox –> Hypervisor –> OS’ –> Docker

    Proxmox

    Here’s a tl;dr without the actual long read:

    Here’s a starting point for IP’s and resource allocation.

    hostname ip/url cpu ram disk1 disk2
    pm1 https://192.168.254.5:8006 8 Cores 12Gb 240Gb
    ovm1 https://192.168.254.6 2vCPU 2048Gb 32Gb 50Gb
    docker1 https://192.168.254.7:9443 2vCPU 4096Gb 16Gb

    Download and Install Proxmox on your hardware. Use the information presented in this post to answer as many questions as you can. This has been done a thousand times over and well documented on the internet. Proxmox’s wiki is a great reference.

    1. Download ISO – https://enterprise.proxmox.com/iso/proxmox-ve_8.3-1.iso
    2. Use Etcher to copy that ISO to a USB drive – https://pve.proxmox.com/wiki/Prepare_Installation_Media & https://etcher.io/
    3. Use the USB drive to install Proxmox on your hardware – https://pve.proxmox.com/wiki/Installation#chapter_installation

    Once you have Proxmox installed, login via the browser to the IP you set the server to. In the example here https://192.168.254.5:8006.

    OpenMediaVault

    Download the ISO for OpenMediaVault to your computer and upload that ISO to “local” storage on Proxmox. Use the information above as a reference. From there we install OVM via Proxmox’s UI. In the example here we create a VM with the OVM ISO using mostly defaults with 2vCPU Cores, 2Gb RAM and a single 32Gb Disk. You can create the 2nd 50Gb Disk for NFS here as well. Then we login to OVM’s UI, do updates, change any settings we need to (timezone, password, IP, etc.). Then we provision the 2nd disk as an ext4 filesystem, create a /docker shared folder and finally expose that shared folder via NFS.

    1. Download OVM ISO – https://sourceforge.net/projects/openmediavault/files/iso/7.0-32/openmediavault_7.0-32-amd64.iso
    2. Upload the OVM ISO to Proxmox via the Proxmox UI
    3. Create a new VM via the Proxmox UI with the following: 2vCPU Cores, 2Gb RAM, 2 Disks (1st a 32Gb for the OS and a 2nd 50Gb for NFS)
    4. Login to the OVM UI and make your settings changes, run updates and do a final reboot before starting the rest.
    5. Create an ext4 filesystem on the 50Gb disk.
    6. Create a /docker shared folder
    7. Expose the /docker shared folder via NFS with the following: client – 192.168.254.0/24, permission – read/write, extra options – subtree_check,insecure,no_root_squash. Click Save, then Apply.

    Ubuntu 24.04 LXC w/ Docker

    For this step, we’ll be using Proxmox VE Helper-Scripts to install an Ubuntu 24.04 LTS LXC (linux container) that installs Docker and Portainer for us. There’s a few modifications we need make in addition to deploying the LXC like modifying the LXC conf file on the Proxmox Host, permissions related settings on the LXC guest, and then the NFS client setup. After that we’ll be able to deploy some Docker containers and start having fun.

    1. From your computer go to https://community-scripts.github.io/ProxmoxVE/scripts?id=docker and copy the .sh script URL to your clipboard and paste it into the Proxmox Console (I recommend opening a new console or using SSH on the Proxmox host) and paste the .sh script URL. This will kick off an installer script prompt that you’ll need to answer.

    I rarely go with defaults for this for some reason so here’s what I do:

    Select Advanced Then…

    • Container: Priviliged
    • Hostname: docker1
    • Disk size: 8Gb
    • CPU: 2
    • RAM: 2
    • Network: vmbr0
    • IP Address: 192.168.254.6/24 #make this your IP
    • Gateway: 192.168.254.254 #make this your default GW
    • Disable IPv6: yes
    • DNS Search Domain: blank
    • DNS Server IP: 1.1.1.1 #or whatever you use
    • VLAN: blank
    • Root SSH: yes
    • Verbose Mode: no
    1. After your LXC Container is deployed, use the Proxmox UI to shut it down. Take note of the ID Number of the LXC Container you crated. Then from the Proxmox shell you’ll need to add a line to that container’s config file with this command:
      echo -e "lxc.apparmor.profile = unconfined" >> /etc/pve/lxc/103.conf #the 103 needs to be replaced with your ID Number of the LXC you created.
    2. Now under that LXC in the Proxmox UI, you’ll want to go to Options –> Features and check NFS, Fuse and ensure Nesting is also checked. Save and start the CT.
    3. Login to the Docker server with SSH
    4. Create a new user with useradd dockeradmin
    5. Set the password with passwd dockeradmin
    6. Add that user to the /etc/sudoers file with the following syntax:
    visudo /etc/sudoersdockeradmin ALL=(ALL) ALLcontrol XYEnter
    1. Run the following commands:
    apt update && apt upgrade -yapt install curl gpg nfs-common -ymkdir -p /opt/docker
    1. Modify the /etc/fstab to permanently mount the newly minted NFS share from OMV with the following:
    echo -e "192.168.254.5:/docker /opt/docker nfs vers=4.2,rw,hard,intr 0 0" >> /etc/fstab
    1. Reboot the Docker server, log back in via ssh and run the command:
    mount | grep nfs

    If you see the mount, you are good to go onto the Portainer step.

    Portainer

    Portainer was installed during the LXC script process (or should have been). It’s pretty easy to install if you missed that step. In Portainer, I like to use “Stacks” as it helps me keep track of the docker compose elements I run as well as modify later without the feeling of starting from scratch…or doing everything from VI/CLI.

    Let’s deploy our 3 stacks. Below will be 3 already modified compose.yml files that you’ll copy into Portainer’s Web Config under Stacks. Modify the particulars if you know what your doing. You’ll notice the volumes use the previously created /opt/docker directory that is NFS mounted to OMV.

    services:  app:    image: 'jc21/nginx-proxy-manager:latest'    restart: unless-stopped    ports:      - '80:80' # Public HTTP Port      - '443:443' # Public HTTPS Port      - '81:81' # Admin Web Port    environment:      DB_SQLITE_FILE: "/data/database.sqlite"      DISABLE_IPV6: 'true'      INITIAL_ADMIN_EMAIL: [email protected]      INITIAL_ADMIN_PASSWORD: reallySTRONGpassword    volumes:      - /opt/docker/npm/data:/data      - /opt/docker/npm/letsencrypt:/etc/letsencrypt

    Nginx Proxy Manager Docker Compose

    services:  ghost:    image: ghost:5-alpine    restart: always    ports:      - 2368:2368    environment:      database__client: mysql      database__connection__host: db      database__connection__user: root      database__connection__password: REALLYstrongPASSWORD      database__connection__database: ghost      url: https://fqdn.publicsite.tld #UPDATE THIS WITH YOUR HOST.DOMAIN.TLD    volumes:      - /opt/docker/ghost/content:/var/lib/ghost/content  db:    image: mysql:8.0    restart: always    environment:      MYSQL_ROOT_PASSWORD: REALLYstrongPASSWORD    volumes:      - /opt/docker/ghost/db:/var/lib/mysqlvolumes:  ghost:  db:

    Ghost w/ MariaDB Docker Compose

    services:  watchtower:    image: containrrr/watchtower    volumes:      - /var/run/docker.sock:/var/run/docker.sock    command: --interval 300 # Check for updates every 5 minutes    restart: always

    Watchtower Docker Compose