Upgrade Proxmox ZFS boot drive with mirroring

From https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#chapter_zfs # zpool status pool: rpool state: ONLINE scan: resilvered 9.32M in 00:00:00 with 0 errors on Thu Apr 3 23:20:51 2025 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 nvme-eui.0025388581b66796-part3 ONLINE 0 0 0 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 1.8T 0 disk --sda1 8:1 0 1.8T 0 part zd16 230:16 0 32G 0 disk --zd16p1 230:17 0 100M 0 part --zd16p2 230:18 0 892M 0 part --zd16p3 230:19 0 31G 0 part zd32 230:32 0 10G 0 disk --zd32p1 230:33 0 9.5G 0 part --zd32p2 230:34 0 1K 0 part --zd32p5 230:37 0 510M 0 part nvme0n1 259:0 0 476.9G 0 disk --nvme0n1p1 259:1 0 1007K 0 part --nvme0n1p2 259:2 0 1G 0 part --nvme0n1p3 259:3 0 475.9G 0 part Duplicate the partition tables on the new drive (/dev/sda) # sgdisk /dev/nvme0n1 -R /dev/sda change the GUID so they are not the same # sgdisk -G /dev/sda Use parted to fdisk to expand partition 3 into the full capacity of the new disk # fdisk /dev/sda Welcome to fdisk (util-linux 2.38.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors Disk model: SSD Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 126F0F8E-624E-4F4D-8CD4-89F8B2EDE74A Device Start End Sectors Size Type /dev/sda1 34 2047 2014 1007K BIOS boot /dev/sda2 2048 2099199 2097152 1G EFI System /dev/sda3 2099200 1000215182 998115983 475.9G Solaris /usr & Apple ZFS Command (m for help): d Partition number (1-3, default 3): 3 Partition 3 has been deleted. Command (m for help): n Partition number (3-128, default 3): 3 First sector (2099200-3907029134, default 2099200): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2099200-3907029134, default 3907028991): Created a new partition 3 of type 'Linux filesystem' and of size 1.8 TiB. Command (m for help): p Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors Disk model: SSD Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 126F0F8E-624E-4F4D-8CD4-89F8B2EDE74A Device Start End Sectors Size Type /dev/sda1 34 2047 2014 1007K BIOS boot /dev/sda2 2048 2099199 2097152 1G EFI System /dev/sda3 2099200 3907028991 3904929792 1.8T Linux filesystem Command (m for help): Change the label back to “Solaris /usr & Apple ZFS” Command (m for help): t Partition number (1-3, default 3): 3 Partition type or alias (type L to list all): 157 Changed type of partition 'Linux filesystem' to 'Solaris /usr & Apple ZFS'. Command (m for help): p Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors Disk model: SSD Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 126F0F8E-624E-4F4D-8CD4-89F8B2EDE74A Device Start End Sectors Size Type /dev/sda1 34 2047 2014 1007K BIOS boot /dev/sda2 2048 2099199 2097152 1G EFI System /dev/sda3 2099200 3907028991 3904929792 1.8T Solaris /usr & Apple ZFS Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. Check what type of boot partition you have (Grub / EUFI) # proxmox-boot-tool status Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. System currently booted with uefi F0A5-6343 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) Copy boot partition detail to the new disk # proxmox-boot-tool format /dev/sda2 UUID="" SIZE="1073741824" FSTYPE="" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sda" MOUNTPOINT="" Formatting '/dev/sda2' as vfat.. mkfs.fat 4.2 (2021-01-31) Done. # proxmox-boot-tool init /dev/sda2 Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. UUID="F84D-06C6" SIZE="1073741824" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sda" MOUNTPOINT="" Mounting '/dev/sda2' on '/var/tmp/espmounts/F84D-06C6'. Installing systemd-boot.. Created "/var/tmp/espmounts/F84D-06C6/EFI/systemd". Created "/var/tmp/espmounts/F84D-06C6/EFI/BOOT". Created "/var/tmp/espmounts/F84D-06C6/loader". Created "/var/tmp/espmounts/F84D-06C6/loader/entries". Created "/var/tmp/espmounts/F84D-06C6/EFI/Linux". Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/F84D-06C6/EFI/systemd/systemd-bootx64.efi". Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/F84D-06C6/EFI/BOOT/BOOTX64.EFI". Random seed file /var/tmp/espmounts/F84D-06C6/loader/random-seed successfully written (32 bytes). Created EFI boot entry "Linux Boot Manager". Configuring systemd-boot.. Unmounting '/dev/sda2'. Adding '/dev/sda2' to list of synced ESPs.. Refreshing kernels and initrds.. Running hook script 'proxmox-auto-removal'.. Running hook script 'zz-proxmox-boot'.. Copying and configuring kernels on /dev/disk/by-uuid/F0A5-6343 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve Copying and configuring kernels on /dev/disk/by-uuid/F84D-06C6 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve Add the new disk to rpool as a mirror device. Important - you have to use partition 3, not the just the disk designation. # zpool status pool: rpool state: ONLINE scan: resilvered 9.32M in 00:00:00 with 0 errors on Thu Apr 3 23:20:51 2025 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 nvme-eui.0025388581b66796-part3 ONLINE 0 0 0 errors: No known data errors # zpool attach rpool nvme-eui.0025388581b66796-part3 /dev/sda3 # zpool status pool: rpool state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Apr 4 03:35:18 2025 378G / 378G scanned, 853M / 378G issued at 35.5M/s 841M resilvered, 0.22% done, 03:01:01 to go config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvme-eui.0025388581b66796-part3 ONLINE 0 0 0 sda3 ONLINE 0 0 0 (resilvering) errors: No known data errors # zpool status pool: rpool state: ONLINE scan: resilvered 371G in 03:10:09 with 0 errors on Sat Apr 5 11:24:50 2025 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvme-eui.0025388581b65b82-part3 ONLINE 0 0 0 sda3 ONLINE 0 0 0 Once synced up, Power off and replace the old drive with the new drive. System should still boot if UEFI. Once booted, you’ll have a degraded mirror, you can safely remove the old drive # zpool status pool: rpool state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J scan: resilvered 371G in 03:10:09 with 0 errors on Sat Apr 5 11:24:50 2025 config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 14929228184534084580 UNAVAIL 0 0 0 was /dev/disk/by-id/nvme-eui.0025388581b65b82-part3 nvme0n1p3 ONLINE 0 0 0 errors: No known data errors # zpool detach rpool 14929228184534084580 # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 472G 367G 105G - 1.35T 21% 77% 1.00x ONLINE - If you expanded partition 3 to be larger, tell zfs that you want to expand into the available space. # zpool set autoexpand=on rpool # zpool online -e rpool nvme0n1p3 # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 1.82T 367G 1.46T - - 5% 19% 1.00x ONLINE -

April 6, 2025 · 6 min

Proxmox GPU Passthrough for LXC for Docker, for apps WebODM, immich

Proxmox GPU Passthrough for Docker using LXC to host WebODM with ClusterODM References: NVIDIA Drivers, NVIDIA Driver Search NVIDIA Container Toolkit Docs Ansible Playbook that runs the recommendations below GPU Passthrough on Proxmox ClusterODM Setup Guide Using Specific GPU with NodeODM ClusterODM Project Remove Old NVIDIA Drivers List existing NVIDIA or CUDA packages: apt list --installed | egrep -i "nvidia|cuda" | cut -d/ -f1 If drivers are listed, uninstall the current NVIDIA runfile driver: sudo ./NVIDIA-Linux-*.run --uninstall Re-check installed packages: apt list --installed | egrep -i "nvidia|cuda" | cut -d/ -f1 If any packages remain, remove them: apt list --installed | egrep -i "nvidia|cuda" | cut -d/ -f1 | xargs apt remove -y Setting Up GPU Passthrough on Proxmox Server Install required packages: apt install pve-headers dkms pciutils ...

March 30, 2025 · 3 min

Using Fit-statUSB to provide visual server health in my homelab using proxmox

https://www.amazon.com/Compulab-FIT-STATUSB-fit-statUSB/dp/B07CKFLQ5V https://www.amazon.com/Compulab-FIT-STATUSB-fit-statUSB/dp/B07CKFLQ5V #!/bin/bash # Define color variables BLUE="000011" RED="010000" YELLOW="050500" GREEN="000100" WHITE="111111" OFF="000000" DECAY="#FF0000" # Initialize the serial port usbreset fit_StatUSB if [ $? -ne 0 ]; then echo "Device not found. Aborting." exit 1 fi sleep 5 stty -F /dev/ttyACM0 9600 raw -echo -echoe -echok -echoctl -echoke sleep 5 # Function to send a color command to Fit-statUSB send_color() { echo Sending: "B#${1}-250#000000-1000${DECAY}-9999" echo -e "B#${1}-250#000000-1000${DECAY}-9999" > /dev/ttyACM0 sleep 1 } echo -e "F0001" > /dev/ttyACM0; sleep 1 # Minimal Transition echo -e "#${RED}" > /dev/ttyACM0; sleep 1 # Red echo -e "#${GREEN}" > /dev/ttyACM0; sleep 1 # Green echo -e "#${BLUE}" > /dev/ttyACM0; sleep 1 # Blue echo -e "#${WHITE}" > /dev/ttyACM0; sleep 1 # White echo -e "#${OFF}" > /dev/ttyACM0; sleep 1 # Off while true; do # Get processor idle time using vmstat idle=$(vmstat 1 2 | tail -1 | awk '{print $15}') # Get Proxmox health state expected_votes=$(pvecm status | grep 'Expected votes:' | awk '{print $2}') total_votes=$(pvecm status | grep 'Total votes:' | awk '{print $2}') flags=$(pvecm status | grep 'Flags:' | awk '{print $2}') if [ "$flags" != "Quorate" ]; then proxmox_status="$RED" elif [ "$expected_votes" != "$total_votes" ]; then proxmox_status="$YELLOW" else proxmox_status="$GREEN" fi # Check network connectivity if ping -c 1 8.8.8.8 &> /dev/null; then network_status="$GREEN" else network_status="$RED" fi # Determine processor state color if [ "$idle" -lt 10 ]; then processor_status="$RED" elif [ "$idle" -lt 20 ]; then processor_status="$YELLOW" else processor_status="$GREEN" fi # Repeat the sequences 6 times before re-evaluating the system state for i in {1..6}; do # Create and send blink sequence with breaks send_color "$BLUE" # Initial Blue indicating start of the dataset send_color "$processor_status" send_color "$proxmox_status" send_color "$network_status" done done

June 17, 2024 · 2 min

Armbian custom builds for different SOCs, using docker

I’m impressed with how well Armbian works with SOCs. Since I couldn’t find a Raspberry Pi recently, I tried out a “LePotato” board, which has performed well overall. The main issue is the lack of a headless install option for Armbian. Without a FAT partition for /boot, configuring the OS on a PC or Mac before installation is challenging. I attempted adding /boot to an existing image but struggled. Eventually, I found I could compile Armbian with a revised lepotato.conf file by adding BOOTFS_TYPE="fat". Typically, compiling OS builds requires specific hardware, compilers, libraries, etc., making it a hassle. However, Armbian’s DIY-focused approach made the process surprisingly easy. I even used Docker, so my main OS stayed clean—very cool indeed! ...

October 8, 2022 · 2 min

LePotato and NanoPi Neo Plus2 Goodness

Using DietPi to install current versions of Debian is fantastic! The little NanoPi, while missing 2 USB ports, boasts WiFi, Bluetooth, and 8Gb emmc! Move OS to the NanoPi Neo Plus2 8GB emmc: Get an Armbian Image for the NanoPi and boot it from the microSD. After initially configuring, run /sbin/nand-sata-install. Follow the prompts to copy the root filesystem to the emmc. Remove the SD card. Installing DietPi to the 8GB eMMC flash on the NanoPi: Booted into Armbian on the internal SD card and put The dietPi image in a USB to microSD dongle. The DietPi image was small, so I chose to create an fsarchiver image of it on the Armbian filesystem temporarily. This may or may not work for you if you don’t have enough space on the booted OS in the internal microSD card. ...

October 1, 2022 · 4 min

Home Lab KVM with MeshCommander

No Homelab Remote KVM? Intel Chipset? No Problem with Intel’s Management Engine and MeshCommander! Mesh Commander is an application that can communicate to the Intel Management Engine (IME) available on most systems using an intel chipset. Once IME is configured, Mesh Commander will provide an entry point into the system and can provide a whole range of options from power cycling the system, remote controlling the system, and even accessing the BIOS. So how does it work? Here are the steps I go through to enable it: ...

August 26, 2022 · 2 min

Where's the truth...

By not being on social media sites, I have a choice in what I consume from a news perspective. I want sources that give me the data to form my opinion rather than get opinions framed as news. So this is what works for me: Use a web browser “incognito” mode so tracking cookies aren’t used. This will help prevent news shaping. I use a web browser called Duck Duck Go that helps prevent tracking data from being used. I highly recommend using it for a browser on your phone and the search engine on your desktop. Once google has some history on you, the shaping algorithms will take hold quickly. I use multiple sources. If I watch CNN, I also try to watch Fox News and so on. It helps to see the different perspectives and the spin applied on all sides. For any of the “news” sites, I first look at the ad’s that I’m bombarded with. I want to determine right off the bat how the site makes money, and what generates the most revenue (typically its the product you see most of). They will usually not tell you directly, but you’ll get a sense of why they want you to buy while you stay on the site. I use Axios; they are trying to be a news source with little spin. You can see their mission statement here: https://www.axios.com/about/ They tell you how they gather information, how it’s gathered and qualified, and how they make money. Most of the content is short and concise with little fluff. I appreciate the attempt. I use allsides.com frequently. allsides is an aggregate site that tried to rate news stories as right, left, and center. It’s a good site to get an honest perspective on things. Their take is no news is unbiased, so they show you how. There was an exciting science project from Middle Schooler where they looked at bias in google search engine using data from allsides. More detail on the science project is located at: https://www.allsides.com/blog/teen-proves-media-bias-google-search-results-can-influence-political-opinions Is the content focused on the subject, and is the opinion kind? If the content attacks aperson rather than a position, it’s typically because the position they are trying to make is weak and doesn’t hold up well on its own. I’m sure I can keep rambling, but the above list encompasses most of what I do. Let me know if you have better methods!

January 19, 2021 · 2 min

zfs glory and snaphot hell

This page is to document my trials with zfs snapshots for backup purposes. There a problem I found that entails when incremental snapshot sends are performed when the receive side has changed in some way. I’ll provide complete details soon. Good news my zfs retention script looks to be running well. I’ll document that as well soon. Here’s a teaser…

October 11, 2020 · 1 min

Rock64 System on a Chip (SoC) Review

I’ll review my experience with this SoC shortly…

June 4, 2020 · 1 min

Living on a Raspberry Pi!

This feels a little weird! Playing with the overclocking and it really makes a difference! The settings below look stable but make the proc very hot (over 85 degrees c.) From /boot/config.txt: Overclock settings - disabled until heat sink is added. 170327 SeanK #arm_freq=1350 #core_freq=500 #over_voltage=4 #disable_splash=1 ##force_turbo=1 #boot_delay=1 #sdram_freq=500 Also created a script to put the governor in ondemand mode and put it in the init.d directory: root@webpi:/sys/devices/system/cpu/cpu0/cpufreq# cat /etc/init.d/sk-perf-set-cpu-governor.sh #!/bin/sh # # 20170327, Sean Kennedy # # From /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors: # conservative ondemand userspace powersave performance governor="ondemand" echo $governor > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor root@webpi:/sys/devices/system/cpu/cpu0/cpufreq# Also got ganglia to report on CPU Frequency and Temp using this init script…. ...

March 5, 2017 · 2 min