convert 'proxmox' node from proxmox to truenas

[billimek]

Currently the host proxmox acts as both a NSF as well as a proxmox node (with an associated k3s VM).

It is not ideal to have the NAS act as double-duty as another purpose, especially in this environment with about 8 other physical nodes.

Therefore, the purpose of this issue to is facilitate the conversion of this host from a proxmox node to a standalone NAS, using the TrueNAS solution.

[billimek]

Workloads that will need to be addressed:

• NFS clients (see /etc/exports)
• borgbackup 'server' and associated configuration (see /etc/borgmatic/config.yaml, /etc/systemd/system/borgmatic.service, /etc/systemd/system/borgmatic.timer)
• rcone backup and associated configuration (see /etc/systemd/system/rclone-sync.service & /etc/systemd/system/rclone-sync.timer)
• UPS monitor and associated configuration (see /etc/nut directory)
• manual backup scripts (see /root/backup directory)
• prometheus-node-exporter
• prometheus-upsc-exporter (see /etc/systemd/system/prometheus-pve-exporter.service
• syslog exporting to promtail

[billimek]

Rough migration plan:

1. 'handle' workloads above (probably backup configs for later reference in TrueNAS solution)
2. Stop nvr VM (uses NFS)
3. Scale-down the following k8s workloads that use NFS: flood, jackett, lazylibrarian, nzbget, plex, radarr, sonarr, tesla-dashcam, thanos, loki, minio
4. export tank zpool: zpool export tank
5. shutdown proxmox node
6. boot TrueNAS USB/ISO
7. Install TrueNAS to SSD Samsung_SSD_850_EVO_250GB_S21NNXBG304131Y (use same host IP of 10.0.7.7)
8. Import tank zpool and properly configure
9. Configure TrueNAS for NFS clients as before
10. Verify proxmox NFS client connectivity
11. Verify NFS client connectivity
12. Scale-up and verify k8s workloads NFS client connectivity
13. Go/No-Go decision on moving forward with TrueNAS
14. Remove proxmox node from proxmox cluster pvecm delnode proxmox from one of the other proxmox nodes

[billimek]

• During step 4, proxmox would not cleanly export the tank zpool and I had to just forcefully shut it down. The subsequent step deprecate dedicated 10.0.10.0 subnet #8 still worked properly however
• Step 9 had a snag in that TrueNAS mounts the zpool inside /mnt (e.g. /mnt/tank) while proxmox had them mounted at root. I did not want to mess with forcing trueNAS to mount it to a non-standard mountpoint and instead modified the proxmox and other NFS client configurations to mount them at the new mount-point.
• Step 10 required fixing of permissions in /mnt/tank/proxmox/images to work properly
• Step 12 required deleting and re-creating the nfs PVs and PVCs because the NFS 'path' is immutable

[billimek]

UPS configuration required manual setup outside of what the GUI can provide. If the GUI configuration is attempted again, it will reset the configuration. This is not ideal.

For future ref:

truenas# cat ups.conf
maxretry = 3

[ups]
        driver = usbhid-ups
        port = auto
        desc = "Cyber Power System, Inc. CP1500 AVR UPS"
        pollinterval = 20

truenas# cat upsd.conf
LISTEN 127.0.0.1 3493
LISTEN ::1 3493
MAXAGE 25

truenas# cat upsmon.conf
MONITOR ups@localhost:3493 1 upsmon <REDACTED>MASTER
#NOTIFYCMD "/usr/local/sbin/upssched"
NOTIFYCMD "/etc/local/nut/upscommand.sh"
NOTIFYFLAG ONBATT SYSLOG+EXEC
NOTIFYFLAG LOWBATT SYSLOG+EXEC
NOTIFYFLAG ONLINE SYSLOG+EXEC
NOTIFYFLAG COMMBAD SYSLOG+EXEC
NOTIFYFLAG COMMOK SYSLOG+EXEC
NOTIFYFLAG REPLBATT SYSLOG+EXEC
NOTIFYFLAG NOCOMM SYSLOG+EXEC
NOTIFYFLAG FSD SYSLOG+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG+EXEC
SHUTDOWNCMD "/sbin/shutdown -p now"
POWERDOWNFLAG /etc/nokillpower
HOSTSYNC 15
NOCOMMWARNTIME 300
MINSUPPLIES 1
POLLFREQ 5
POLLFREQALERT 5
DEADTIME 25
RBWARNTIME 43200
FINALDELAY 5

[billimek]

It is not possible to install the starship prompt because there are no FreeBSD binaries readily available that work in TueNAS

[billimek]

rclone backup solution implemented using built-in 'cloud sync' features to back-up /mnt/tank/backups/borg/ and /mnt/tank/data/minio/velero/restic/ directories.

[billimek]

borgbackup solution implemented by creating a 'jail' with configuration and settings persisted in /mnt/tank/iocage/jails/borg. This jail is mounting /mnt/tank/backups/borg/ to /mnt/borg inside it's 'container'.

Added the necessary ssh keys and installed borg to the jail, and updated the client configurations (home, proxmox-b, proxmox-c).