Setup

Some notes on setting up a new Proxmox server:

• Install on ZFS root (set advanced options ashift to either 9 for 512-byte sector ssds, or 12 for 4k sectors)
  • (Check what sector sizes an SSD supports with nvme id-ns -H /dev/nvme0n1 | grep “Relative Performance”
• Run community post-pve-install to enable open-source repos, update and disable nag screen
• Join node into cluster (datacenter→clusters grab join info from existing cluster, join cluster on new node)

• NUC with e1000e interface drops offline after a few hours/days: nic-offloading-fix

https://pve.proxmox.com/wiki/HTTPSCertificateConfiguration

• Copy the tsig key to /usr/local/share/nsupdate.key
• Navigate to ACME plugins and create a new one:
  • ID: FreeIPA
  • Type: RFC2136
  • Key: /usr/local/share/nsupdate.key
  • Server: ares.jellybean.sihnon.net
• Create the ACME account via webui
• Create the certificate request using DNS type and FreeIPA plugin
• Order the new certificate

https://forum.proxmox.com/threads/proxmox-3-0-web-ui-on-port-443-instead-of-8006.13964/

• Edit vmbr0, remove physical nic and IP address
• Create OVSBridge ovsbr0
• Create OVSIntPort mgmt attached to ovsbr0 and add the management IP back onto this
• Create OVSBond bond0 attached to ovsbr0 and add all physical nics to this, with mode balance-tcp
• Apply changes
• Update switch configuration to enable LACP
• Confirm connectivity
• Datacenter→SDN→Apply to setup vlans

apt-get install openvswitch-switch

• Install dependencies:

  apt install frr-pythontools
  systemctl enable --now frr.service

• Datacenter→SDN→Zones
  • Create lan zone of type vlan bound to ovsbr0
• Datacenter→SDN-VNets
  • Create one vnet for each vlan, setting the zone to lan and entering the vlan id in tag field.
  • When creating VMs, select this vnet to assign to the vlan without needing to re-enter the vlan id.

TrueNAS SCALE demands unique serial numbers on each VM disk for stable imports. Proxmox doesn't have UI for this, but it can be done by editing the VM config files.

• Generate a unique serial number with:

  uuidgen | sed -E "s|-||g" | cut -c 1-20

• Edit the VM definition at /etc/pve/qemu-server/${VMID}.conf to add serial=XXX to the end of the scsi line(s)