Ubuntu 20.04 → 24.04 Dist-Upgrade: LUKS2, GRUB, Secure Boot and systemd-boot
This page documents debugging a failed Ubuntu release upgrade that left the system unbootable. The root causes were an interplay between LUKS2 encryption, GRUB 2.12, Secure Boot and missing dotfiles — each amplifying the others into a long chain of cascading problems.
Starting Point
- Xubuntu 20.04, LUKS2-encrypted root partition
- Dual-boot with Windows on the same NVMe SSD
- A second encrypted partition (Veracrypt/exFAT) as shared storage,
mounted at
/home/user/data - Personal dotfiles (xmonad, zsh, vim, etc.) stored as symlinks on this Veracrypt partition
- Neo2 keyboard layout, xmonad as window manager, Alacritty as terminal
Phase 1: Failed Dist-Upgrade
do-release-upgrade from 20.04 → 22.04 was started. The upgrade stalled on the
package ConTeXt MarkIV (part of TeXLive) — a very large package whose
post-install script runs for a long time. It was manually interrupted.
The system no longer booted afterwards. GRUB dropped directly into its interactive
command line (grub>) without showing a boot menu.
Phase 2: Chroot Repair via Live-USB
The system was booted from a Live-USB. The encrypted partition was opened manually and a chroot environment was set up:
cryptsetup luksOpen /dev/sdXY cryptroot
mount /dev/mapper/cryptroot /mnt
mount /dev/sdXZ /mnt/boot/efi # EFI partition
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /run /mnt/run
mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
chroot /mnt
Inside the chroot:
dpkg --configure -aandapt --fix-broken installrepaired broken packages- The kernel was missing entirely (only initrd present, no vmlinuz):
apt install --reinstall linux-image-generic - The upgrade was completed with
do-release-upgradeto 22.04 and then to 24.04
Phase 3: GRUB + LUKS2 + Secure Boot — The Core Problem
Symptom
After the upgrade GRUB still showed no menu entries. Debugging in the GRUB command line:
grub> ls (hd1,gpt7)
# → "No known filesystem detected"
grub> insmod luks2
# → "file '/boot/grub/x86_64-efi/luks2.mod' not found"
Root Cause
The real problem is a fundamental incompatibility:
Canonical’s pre-signed
grubx64.efideliberately does not include theluks2module.
GRUB 2.12 is technically capable of decrypting LUKS2 (provided PBKDF2 is used as the key derivation function rather than Argon2), but the Secure-Boot-signed EFI binary shipped by Ubuntu intentionally omits this module — it is still considered not fully stable.
grub-install --modules="... luks2 ..." therefore has no effect: with Secure Boot
active, grub-install always uses the pre-signed image and ignores the
--modules flag.
A custom image built with grub-mkimage that includes luks2 support is
immediately rejected by Secure Boot:
Verification failed: (0x1A) Security Violation
Additionally, grub-install failed silently inside the chroot because EFI
variables cannot be written from within a chroot environment
(grub-efi/install_devices_failed: true in debconf-show grub-efi).
Options
| Option | Effort | Downside |
|---|---|---|
| Disable Secure Boot | low | Secure Boot inactive |
| Convert LUKS2 → LUKS1 | low | No backup header |
| Own MOK key + self-signed GRUB | high | Must be repeated on every GRUB update |
| Switch to systemd-boot | medium | One-time setup effort |
Converting to LUKS1 was ruled out because LUKS2’s backup header had already proven invaluable once before. systemd-boot was chosen as the solution.
Phase 4: Switching to systemd-boot
Why systemd-boot?
- Secure Boot is natively supported (the binary is signed by the distribution)
- LUKS is not opened by systemd-boot itself — that is handled by the initramfs during kernel startup, which eliminates the LUKS2 problem entirely
- Dual boot with Windows works out of the box: systemd-boot automatically detects the Windows EFI entry
- No shim intermediary required (unlike GRUB)
Installation
bootctl install --esp-path=/boot/efi
The kernel and initrd are automatically copied to the EFI partition by
kernel-install. A new boot entry is created on every kernel update.
LUKS Parameters
The automatically generated boot entry contained no LUKS parameters. These are
stored permanently in /etc/kernel/cmdline:
rd.luks.uuid=<UUID-of-LUKS-partition> rd.luks.name=<UUID>=cryptroot root=/dev/mapper/cryptroot rw quiet splash
kernel-install add <version> /boot/vmlinuz-<version> regenerates the entry with
these parameters. Future kernel updates will automatically use the correct
parameters.
Secure Boot Problem
systemd-bootx64.efi was not signed on this system — Ubuntu does not ship a
.efi.signed package for systemd-boot. Temporary solution: Secure Boot
disabled in BIOS.
For a clean solution with Secure Boot re-enabled:
# Generate own signing key
openssl req -newkey rsa:4096 -nodes -keyout /root/mok.key \
-new -x509 -sha256 -days 3650 \
-subj "/CN=My Boot Signing Key/" -out /root/mok.crt
# Sign systemd-boot
sbsign --key /root/mok.key --cert /root/mok.crt \
--output /boot/efi/EFI/systemd/systemd-bootx64.efi \
/boot/efi/EFI/systemd/systemd-bootx64.efi
# Register key with Secure Boot
mokutil --import /root/mok.crt
# → Confirm once in the MOK manager on next boot
EFI Boot Order
The old GRUB entry was removed and systemd-boot set as the first entry:
efibootmgr --delete-bootnum --bootnum XXXX # remove old Ubuntu/GRUB entry
efibootmgr --bootorder 0001 # Linux Boot Manager first
Phase 5: Veracrypt Partition — The Actual Mount Error
Symptom
The Veracrypt partition was not mounted at boot. The journal showed:
mount: wrong fs type, bad option, bad superblock on /dev/mapper/data,
missing codepage or program, or other error
This looked like a serious problem — but it wasn’t.
Diagnosis with -v
cryptsetup open /dev/sdXY data \
--type tcrypt --veracrypt \
--key-file /home/user/.data.key -v
# → Error: device already exists
device already exists means: cryptsetup had already successfully opened the
partition. The only problem was the subsequent mount step.
Cause: Invalid fstab Options for exFAT
The /etc/fstab contained for this partition:
/dev/mapper/data /home/user/data auto permissions,users,nosuid,nodev,nofail,...
permissions is not a valid mount option for exFAT filesystems and causes an
immediate mount failure. The corrected line:
/dev/mapper/data /home/user/data exfat nofail,fmask=133,dmask=022,uid=1000,gid=1000 0 0
Phase 6: Keyboard Layout Chaos Due to Missing Dotfiles
What Happened
Because the Veracrypt partition was not mounted, all symlink targets were missing:
.xmonad/→ xmonad did not start; XFCE was loaded as a fallback.zshrc→ attempted to source files from the unmounted partition.vimrc→ dangling symlink- xkb/xmodmap configuration missing
XFCE initialized the keyboard layout without the user-specific xkb configuration.
The result was a BadKey (0xcf) registered as the mod2 modifier in X11 —
visible with:
xmodmap -pm
# mod2 BadKey (0xcf)
This invalid modifier entry activated Layer 4 special characters instead of
normal input. In Neo2, Layer 4 maps the number keys to symbols like º, £,
¤ — which explains why the keyboard appeared completely broken.
Solution
After correctly mounting the Veracrypt partition, xmonad started normally with the existing dotfiles. The keyboard layout worked immediately without any further intervention, as the xmodmap configuration was loaded correctly.
Summary: Lessons Learned
LUKS2 + GRUB + Secure Boot is currently not possible out of the box on Ubuntu. The combination requires either a custom MOK key with a self-built GRUB image, converting to LUKS1, or switching to a different bootloader. systemd-boot is the most elegant solution here since it delegates LUKS to the initramfs and supports Secure Boot natively.
Dotfiles on encrypted partitions are a double-edged sword: they are secure, but if the partition is unavailable at boot, the entire session initialisation breaks silently and with cryptic downstream errors. A minimal fallback configuration directly in the home directory is advisable.
cryptsetup -v is essential when debugging — without verbose output,
“device already exists” looks identical to a real failure.
Open Items
- Re-enable Secure Boot via own MOK key for systemd-boot
- Clean up stale
/bootfstab entry - Python symlink:
sudo ln -s /usr/bin/python3 /usr/bin/python - Investigate audio errors (alsactl) in the journal