Native ZFS encryption support (Cryptrebooting fails with "No such file or directory @ dir_chdir")

[martinhoefling]

Trying to cryptreboot fails with the following error:

Extracting initramfs... To speed things up, future versions will employ cache.
Running (echo lz4 \"-t\"; strace -f --trace=execve -z -qq --signal=\!all unmkinitramfs /boot/initrd.img-6.8.12-10-pve /tmp/d20250503-1661553-u8qwpo) 2>&1 | grep --line-buffered lz4
        lz4 "-t"
Finished in 12.978 seconds with exit status 0 (successful)
Running mount -t ramfs -o mode\=700 none /tmp/d20250503-1661553-xgpdqm
Finished in 0.005 seconds with exit status 0 (successful)
Running umount /tmp/d20250503-1661553-xgpdqm
Finished in 0.004 seconds with exit status 0 (successful)
/var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/archiver.rb:10:in `chdir': No such file or directory @ dir_chdir - /tmp/d20250503-1661553-xgpdqm/files (Errno::ENOENT)
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/archiver.rb:10:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/patcher.rb:14:in `block in call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:13:in `block (2 levels) in call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/mounter.rb:17:in `run'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/mounter.rb:11:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:12:in `block in call'
        from /usr/lib/ruby/3.1.0/tmpdir.rb:96:in `mktmpdir'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:11:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/safe_temp/directory.rb:11:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/initramfs/patcher.rb:11:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/patched_initramfs_generator.rb:8:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/kexec_patching_loader.rb:12:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/cli/params_parsing_executor.rb:25:in `configure_and_exec'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/lib/crypt_reboot/cli/params_parsing_executor.rb:9:in `call'
        from /var/lib/gems/3.1.0/gems/crypt_reboot-0.3.1/exe/cryptreboot:8:in `<top (required)>'
        from /usr/local/bin/cryptreboot:25:in `load'
        from /usr/local/bin/cryptreboot:25:in `<main>'

tmp is mounted

rpool/var/tmp on /var/tmp type zfs (rw,relatime,xattr,posixacl,casesensitive)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=65625572k,nr_inodes=409600,inode64)

various kernel (pve / proxmox) are present:

config-5.13.19-6-pve  efi                       initrd.img-6.8.12-10-pve  System.map-5.13.19-6-pve  vmlinuz-5.13.19-6-pve
config-5.15.30-1-pve  efi2                      initrd.img-6.8.12-4-pve   System.map-5.15.30-1-pve  vmlinuz-5.15.30-1-pve
config-6.8.12-10-pve  grub                      initrd.img-6.8.12-9-pve   System.map-6.8.12-10-pve  vmlinuz-6.8.12-10-pve
config-6.8.12-4-pve   initrd.img-5.13.19-6-pve  lost+found                System.map-6.8.12-4-pve   vmlinuz-6.8.12-4-pve
config-6.8.12-9-pve   initrd.img-5.15.30-1-pve  pve                       System.map-6.8.12-9-pve   vmlinuz-6.8.12-9-pve

System is a fully updated bookworm / proxmox.

Let me know if should provide further information to debug the issue.

[pepawel]

You're not asked for a passphrase. You should be at this point. Probably cryptreboot couldn't understand how your drive is encrypted.
Please provide the output of the following commands:

cat /etc/crypttab
ls -l /dev/mapper
cat /etc/fstab
mount

[martinhoefling]

OK I, realized that I'm probably missing an important bit of information. I could imagine that detection of the actual volume to unlock fails due to my non-standard setup.

The system has two ZFS pools. One backed by SSDs which use ZFS native encryption. This pool is used for booting and the system. There's a second pool backed by spinning disks (luks encrypted for historic reasosn) used for data. The second pool backing disks are unlocked later in the boot process using keyfiles.

# ls /dev/zvol/
datapool  rpool

Crypttab to unlock HDDs using a key stored on the encrypted zfs volume on SSDs.

# cat /etc/crypttab
hdd3zfscrypt /dev/disk/by-id/ata-WDC_WD80EFAX-68KNBN0_VAHEBWWL /etc/hddkeyfile luks
hdd4zfscrypt /dev/disk/by-id/ata-WDC_WD80EFAX-68KNBN0_VAHEESAL /etc/hddkeyfile luks

Mapping of these volumes

# ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 May  3 14:24 control
lrwxrwxrwx 1 root root       7 May  3 14:24 hdd3zfscrypt -> ../dm-0
lrwxrwxrwx 1 root root       7 May  3 14:24 hdd4zfscrypt -> ../dm-1

fstab has efi and boot and swap partitions only...

total 0
crw------- 1 root root 10, 236 May  3 14:24 control
lrwxrwxrwx 1 root root       7 May  3 14:24 hdd3zfscrypt -> ../dm-0
lrwxrwxrwx 1 root root       7 May  3 14:24 hdd4zfscrypt -> ../dm-1`

mount output

# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=65591468k,nr_inodes=16397867,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=13125720k,mode=755,inode64)
rpool/ROOT/debian on / type zfs (rw,relatime,xattr,posixacl,casesensitive)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5403)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
rpool/home on /home type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/home/root on /root type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/proxmox on /rpool/proxmox type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/cache on /var/cache type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/log on /var/log type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/nfs on /var/nfs type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/spool on /var/spool type zfs (rw,relatime,xattr,posixacl,casesensitive)
rpool/var/tmp on /var/tmp type zfs (rw,relatime,xattr,posixacl,casesensitive)
/dev/sda3 on /boot type ext4 (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=65628592k,nr_inodes=409600,inode64)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdb2 on /boot/efi2 type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
datapool/proxmox on /datapool/proxmox type zfs (rw,noatime,xattr,noacl,casesensitive)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=13125716k,nr_inodes=3281429,mode=700,inode64)

[pepawel]

Thanks. I still don't see /etc/fstab contents, it seems you duplicated ls -l /dev/mapper output there.

Using ZFS is an important piece of information. Could you provide me with the output of:

zfs list

On Ubuntu 24.04 it looks like this:

NAME                                               USED  AVAIL  REFER  MOUNTPOINT
bpool                                             97.3M  1.40G    96K  /boot
bpool/BOOT                                        96.8M  1.40G    96K  none
bpool/BOOT/ubuntu_awvp0x                          96.7M  1.40G  96.7M  /boot
rpool                                             3.97G  6.68G   192K  /
rpool/ROOT                                        3.94G  6.68G   192K  none
rpool/ROOT/ubuntu_awvp0x                          3.94G  6.68G  2.86G  /
rpool/ROOT/ubuntu_awvp0x/srv                       192K  6.68G   192K  /srv
rpool/ROOT/ubuntu_awvp0x/usr                       580K  6.68G   192K  /usr
rpool/ROOT/ubuntu_awvp0x/usr/local                 388K  6.68G   388K  /usr/local
rpool/ROOT/ubuntu_awvp0x/var                      1.08G  6.68G   192K  /var
rpool/ROOT/ubuntu_awvp0x/var/games                 192K  6.68G   192K  /var/games
rpool/ROOT/ubuntu_awvp0x/var/lib                  1.07G  6.68G   976M  /var/lib
rpool/ROOT/ubuntu_awvp0x/var/lib/AccountsService   212K  6.68G   212K  /var/lib/AccountsService
rpool/ROOT/ubuntu_awvp0x/var/lib/NetworkManager    224K  6.68G   224K  /var/lib/NetworkManager
rpool/ROOT/ubuntu_awvp0x/var/lib/apt              75.4M  6.68G  75.4M  /var/lib/apt
rpool/ROOT/ubuntu_awvp0x/var/lib/dpkg             47.8M  6.68G  47.8M  /var/lib/dpkg
rpool/ROOT/ubuntu_awvp0x/var/log                  4.16M  6.68G  4.16M  /var/log
rpool/ROOT/ubuntu_awvp0x/var/mail                  192K  6.68G   192K  /var/mail
rpool/ROOT/ubuntu_awvp0x/var/snap                 2.16M  6.68G  2.16M  /var/snap
rpool/ROOT/ubuntu_awvp0x/var/spool                 244K  6.68G   244K  /var/spool
rpool/ROOT/ubuntu_awvp0x/var/www                   192K  6.68G   192K  /var/www
rpool/USERDATA                                    6.10M  6.68G   192K  none
rpool/USERDATA/home_r9xgh7                        5.46M  6.68G  5.46M  /home
rpool/USERDATA/root_r9xgh7                         464K  6.68G   464K  /root
rpool/keystore                                    22.5M  6.69G  16.5M  -

There is a keystore volume at the bottom. It contains an LUKS-encrypted ext4 filesystem.
Do you have an equivalent of that?

I don't see it in your /dev/mapper directory and mount output, but maybe it's just not unlocked and not currently mounted.

[martinhoefling]

Sure thing. I fixed the output in my previous comment to show the command line. /dev/mapper and mount was already included.

fstab

# cat /etc/fstab
/dev/disk/by-uuid/21e4d148-156f-4130-9da2-d3fc29891f27 /boot ext4 defaults 0 0
/dev/disk/by-uuid/51D4-EF56 /boot/efi vfat defaults 0 0
/dev/disk/by-uuid/8E8E-DAC5 /boot/efi2 vfat defaults 0 0
/dev/zvol/rpool/swap none swap defaults 0 0

ZFS list

# zfs list
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
datapool                             4.30T  2.84T    96K  /data
datapool/home                        4.07T  2.84T    96K  /data/home
datapool/proxmox                      232G  2.84T    96K  /datapool/proxmox
datapool/proxmox/vm-102-disk-1        232G  2.94T  82.0G  -
rpool                                 311G   139G   192K  none
rpool/ROOT                            143G   139G   160K  none
rpool/ROOT/debian                     143G   139G  31.1G  /
rpool/home                           57.3G   139G   360K  /home
rpool/proxmox                        90.5G   139G   200K  /rpool/proxmox
rpool/proxmox/vm-101-disk-0          90.5G   171G  7.53G  -
rpool/swap                           16.3G   139G  16.3G  -
rpool/var                            3.20G   139G   296K  none
rpool/var/cache                       125M   139G  76.1M  /var/cache
rpool/var/log                        3.06G   139G   849M  /var/log
rpool/var/nfs                         176K   139G   176K  /var/nfs
rpool/var/spool                      16.3M   139G  3.05M  /var/spool
rpool/var/tmp                         536K   139G   292K  /var/tmp

I don't have a keystore volume or equivalent.

[pepawel]

Thanks. Cryptreboot currently supports LUKS encryption only, the ZFS support uses LUKS underneath.
On Ubuntu, when ZFS is chosen during installation, creates a special keystore volume. This volume contains key material for the native ZFS encryption and is encrypted with LUKS. On boot, this keystore volume is decrypted. This allows retrieval of a key to unlock the ZFS root filesystem.

As your system doesn't contain the keystore or equivalent, the current version of cryptreboot won't be able to unlock it.
Unfortunately, I don't think I will be able to implement this feature in the near future due to a lack of time :/

I don't have any experience with Proxmox, but I did some preliminary research. I installed Proxmox VE 8.4, however, it seems it doesn't support activating disk encryption during the installation. On this page I found following sentence:

There is currently no support for booting from pools with encrypted datasets using GRUB, and only limited support for automatically unlocking encrypted datasets on boot.

Did you use a guide or tutorial to set up the ZFS system encryption? It would be helpful if you shared it. This way, I could try to implement this feature when I have some free time.

[martinhoefling]

Thanks Pawel, I really appreciate you taking time digging into this and for the explanation. Don't hesitate to deprioritize this feature for now.

I essentially have a set up with limited support for automatically unlocking encrypted datasets on boot.. My boot partition is unencrypted . I had an encrypted boot partition on a separate encrypted ZFS pool before but that was a pain to maintain (grub is incompatible with certain ZFS pool features).

Regardingn installation, I recall following the ZFS docs for debian in the past but the system has been upgraded multiple times.

[pepawel]

That's no problem at all, and thanks for the understanding and the context! I appreciate you sharing the details about your setup and the link to the ZFS docs for Debian—that's helpful background.

[martinhoefling]

Did dig a bit deeper into the rabbithole and it seems as there is no standard mechanism yet to provide the passphrase for key decryption in initramfs. openzfs/zfs#13757 has some context.
I checked my /usr/share/initramfs-tools in particular the contained zfsunlock script but didn't see a way to inject the passphrase.

My installed zfs-initramfs version is

zfs-initramfs/stable,now 2.2.7-pve2 all [installed]
  OpenZFS root filesystem capabilities for Linux - initramfs

[pepawel]

Thanks for the interesting info. The issue you linked explains the Ubuntu approach.

My current idea is to rewrite cryptreboot to replace zfs binary with a script calling the renamed original file. The script would inject the key material.

Of course, the zfs would be replaced only in the in-memory patched initramfs.

I believe this approach would free cryptreboot from having to know the (possibly changing) internals of the ZFS unlocking implementation in each distro.

[martinhoefling]

What might even be easier than replacing the binary is to prepare an initramfs hook script that unlocks the pools. One option could be to to read out pools with loaded keys as a first step. Then ask for the passphrases which are then used to forge the hook script. The zfs unlock process as far as I can see is only used if the pool is not unlocked.

Do you have an existing testbed for distros / test cases?

[martinhoefling]

Actually, not hook scripts but boot scripts would help. Hook scripts interact with initramfs building as far as I understood. So you might be right in that it's easier to patch the zfs script and wrap into a loader for keys.