Ubuntu 20.04 → 24.04 Dist-Upgrade: LUKS2, GRUB, Secure Boot and systemd-boot

This page documents debugging a failed Ubuntu release upgrade that left the system unbootable. The root causes were an interplay between LUKS2 encryption, GRUB 2.12, Secure Boot and missing dotfiles — each amplifying the others into a long chain of cascading problems.

Starting Point

  • Xubuntu 20.04, LUKS2-encrypted root partition
  • Dual-boot with Windows on the same NVMe SSD
  • A second encrypted partition (Veracrypt/exFAT) as shared storage, mounted at /home/user/data
  • Personal dotfiles (xmonad, zsh, vim, etc.) stored as symlinks on this Veracrypt partition
  • Neo2 keyboard layout, xmonad as window manager, Alacritty as terminal

Phase 1: Failed Dist-Upgrade

do-release-upgrade from 20.04 → 22.04 was started. The upgrade stalled on the package ConTeXt MarkIV (part of TeXLive) — a very large package whose post-install script runs for a long time. It was manually interrupted.

The system no longer booted afterwards. GRUB dropped directly into its interactive command line (grub>) without showing a boot menu.

Phase 2: Chroot Repair via Live-USB

The system was booted from a Live-USB. The encrypted partition was opened manually and a chroot environment was set up:

cryptsetup luksOpen /dev/sdXY cryptroot
mount /dev/mapper/cryptroot /mnt
mount /dev/sdXZ /mnt/boot/efi          # EFI partition
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /run /mnt/run
mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
chroot /mnt

Inside the chroot:

  • dpkg --configure -a and apt --fix-broken install repaired broken packages
  • The kernel was missing entirely (only initrd present, no vmlinuz): apt install --reinstall linux-image-generic
  • The upgrade was completed with do-release-upgrade to 22.04 and then to 24.04

Phase 3: GRUB + LUKS2 + Secure Boot — The Core Problem

Symptom

After the upgrade GRUB still showed no menu entries. Debugging in the GRUB command line:

grub> ls (hd1,gpt7)
# → "No known filesystem detected"

grub> insmod luks2
# → "file '/boot/grub/x86_64-efi/luks2.mod' not found"

Root Cause

The real problem is a fundamental incompatibility:

Canonical’s pre-signed grubx64.efi deliberately does not include the luks2 module.

GRUB 2.12 is technically capable of decrypting LUKS2 (provided PBKDF2 is used as the key derivation function rather than Argon2), but the Secure-Boot-signed EFI binary shipped by Ubuntu intentionally omits this module — it is still considered not fully stable.

grub-install --modules="... luks2 ..." therefore has no effect: with Secure Boot active, grub-install always uses the pre-signed image and ignores the --modules flag.

A custom image built with grub-mkimage that includes luks2 support is immediately rejected by Secure Boot:

Verification failed: (0x1A) Security Violation

Additionally, grub-install failed silently inside the chroot because EFI variables cannot be written from within a chroot environment (grub-efi/install_devices_failed: true in debconf-show grub-efi).

Options

Option Effort Downside
Disable Secure Boot low Secure Boot inactive
Convert LUKS2 → LUKS1 low No backup header
Own MOK key + self-signed GRUB high Must be repeated on every GRUB update
Switch to systemd-boot medium One-time setup effort

Converting to LUKS1 was ruled out because LUKS2’s backup header had already proven invaluable once before. systemd-boot was chosen as the solution.

Phase 4: Switching to systemd-boot

Why systemd-boot?

  • Secure Boot is natively supported (the binary is signed by the distribution)
  • LUKS is not opened by systemd-boot itself — that is handled by the initramfs during kernel startup, which eliminates the LUKS2 problem entirely
  • Dual boot with Windows works out of the box: systemd-boot automatically detects the Windows EFI entry
  • No shim intermediary required (unlike GRUB)

Installation

bootctl install --esp-path=/boot/efi

The kernel and initrd are automatically copied to the EFI partition by kernel-install. A new boot entry is created on every kernel update.

LUKS Parameters

The automatically generated boot entry contained no LUKS parameters. These are stored permanently in /etc/kernel/cmdline:

rd.luks.uuid=<UUID-of-LUKS-partition> rd.luks.name=<UUID>=cryptroot root=/dev/mapper/cryptroot rw quiet splash

kernel-install add <version> /boot/vmlinuz-<version> regenerates the entry with these parameters. Future kernel updates will automatically use the correct parameters.

Secure Boot Problem

systemd-bootx64.efi was not signed on this system — Ubuntu does not ship a .efi.signed package for systemd-boot. Temporary solution: Secure Boot disabled in BIOS.

For a clean solution with Secure Boot re-enabled:

# Generate own signing key
openssl req -newkey rsa:4096 -nodes -keyout /root/mok.key \
  -new -x509 -sha256 -days 3650 \
  -subj "/CN=My Boot Signing Key/" -out /root/mok.crt

# Sign systemd-boot
sbsign --key /root/mok.key --cert /root/mok.crt \
  --output /boot/efi/EFI/systemd/systemd-bootx64.efi \
  /boot/efi/EFI/systemd/systemd-bootx64.efi

# Register key with Secure Boot
mokutil --import /root/mok.crt
# → Confirm once in the MOK manager on next boot

EFI Boot Order

The old GRUB entry was removed and systemd-boot set as the first entry:

efibootmgr --delete-bootnum --bootnum XXXX   # remove old Ubuntu/GRUB entry
efibootmgr --bootorder 0001                  # Linux Boot Manager first

Phase 5: Veracrypt Partition — The Actual Mount Error

Symptom

The Veracrypt partition was not mounted at boot. The journal showed:

mount: wrong fs type, bad option, bad superblock on /dev/mapper/data,
missing codepage or program, or other error

This looked like a serious problem — but it wasn’t.

Diagnosis with -v

cryptsetup open /dev/sdXY data \
  --type tcrypt --veracrypt \
  --key-file /home/user/.data.key -v
# → Error: device already exists

device already exists means: cryptsetup had already successfully opened the partition. The only problem was the subsequent mount step.

Cause: Invalid fstab Options for exFAT

The /etc/fstab contained for this partition:

/dev/mapper/data  /home/user/data  auto  permissions,users,nosuid,nodev,nofail,...

permissions is not a valid mount option for exFAT filesystems and causes an immediate mount failure. The corrected line:

/dev/mapper/data  /home/user/data  exfat  nofail,fmask=133,dmask=022,uid=1000,gid=1000  0  0

Phase 6: Keyboard Layout Chaos Due to Missing Dotfiles

What Happened

Because the Veracrypt partition was not mounted, all symlink targets were missing:

  • .xmonad/ → xmonad did not start; XFCE was loaded as a fallback
  • .zshrc → attempted to source files from the unmounted partition
  • .vimrc → dangling symlink
  • xkb/xmodmap configuration missing

XFCE initialized the keyboard layout without the user-specific xkb configuration. The result was a BadKey (0xcf) registered as the mod2 modifier in X11 — visible with:

xmodmap -pm
# mod2  BadKey (0xcf)

This invalid modifier entry activated Layer 4 special characters instead of normal input. In Neo2, Layer 4 maps the number keys to symbols like º, £, ¤ — which explains why the keyboard appeared completely broken.

Solution

After correctly mounting the Veracrypt partition, xmonad started normally with the existing dotfiles. The keyboard layout worked immediately without any further intervention, as the xmodmap configuration was loaded correctly.

Summary: Lessons Learned

LUKS2 + GRUB + Secure Boot is currently not possible out of the box on Ubuntu. The combination requires either a custom MOK key with a self-built GRUB image, converting to LUKS1, or switching to a different bootloader. systemd-boot is the most elegant solution here since it delegates LUKS to the initramfs and supports Secure Boot natively.

Dotfiles on encrypted partitions are a double-edged sword: they are secure, but if the partition is unavailable at boot, the entire session initialisation breaks silently and with cryptic downstream errors. A minimal fallback configuration directly in the home directory is advisable.

cryptsetup -v is essential when debugging — without verbose output, “device already exists” looks identical to a real failure.

Open Items

  • Re-enable Secure Boot via own MOK key for systemd-boot
  • Clean up stale /boot fstab entry
  • Python symlink: sudo ln -s /usr/bin/python3 /usr/bin/python
  • Investigate audio errors (alsactl) in the journal