📄 ir/blog.md
pipeline › ir › blog.md
pipeline › ir › blog.md
/* I Trusted Ubuntu With a Kernel Update. We Have Trust Issues Now. · 10 min read */
--tag=linux --tag=ubuntu --tag=kernel --tag=boot-recovery --tag=debugging --level=intermediate

Let me tell you about this laptop before I tell you about the panic. It’s almost half my age. It started its life with my oldest sister, got passed down to another sister, and eventually found its way to me. Three owners. Years of memories baked into its keyboard. It’s the kind of machine that has a personality. I know which hinge is a little stiff, which key needs a firmer press. It’s not just hardware. It’s family history.

So when it greeted me one morning with a black screen and a blinking cursor instead of a desktop, my stomach dropped. And then I saw this:

Kernel panic: not syncing: VFS: Unable to mount root filesystem
 VFS: Unable to mount root fs on unknown-block(0,0)

My first thought wasn’t technical. It was: please don’t be hardware. Please don’t make me replace it.

The words “kernel panic” sounded catastrophic, like something had physically broken inside. A dying drive. A failed chip. Something irreversible. I was already mentally preparing to say goodbye to a machine I was genuinely attached to.

But I decided not to give up without a fight. Before assuming the worst, I would investigate. And that decision is what this post is really about. Because it wasn’t hardware at all. It was a single missing file. And this is the story of how I found it.

First, What Even Is a Kernel Panic?

The kernel is the core of your operating system. It’s the layer of software that sits between your hardware and everything else, managing your CPU, memory, disk, and all your devices. Think of it as the manager of a very complex restaurant. It coordinates everything: the chefs (CPU), the waiters (memory), the storage room (your disk).

A kernel panic is Linux’s version of a crash so severe that the system can’t recover on its own. It halts everything and displays an error rather than risk making things worse. The manager shows up to work and the entire building is gone.

In my case, the manager showed up, looked around for the storage room, and found absolutely nothing. No disk. No partitions. No data. Just vibes.

The technical reason? Ubuntu had updated me to a shiny new kernel (6.17.0-19-generic) but forgot to pack something critical: the initramfs.

The initramfs (short for initial RAM filesystem) is a tiny, temporary filesystem that loads into memory before your actual system does. Its only job is to load the bare minimum drivers needed to find your real disk, then hand over control to your actual Ubuntu installation. Think of it as a small rescue crew that arrives first to unlock the building before the rest of the staff can get in.

No initramfs → no disk drivers → no disk → no boot → kernel panic. Human panic shortly after.

The Crime Scene

Here’s what was actually happening under the hood:

  1. The Trigger: Ubuntu auto-updated to kernel 6.17.0-19-generic but failed to generate a valid initrd.img (the initramfs file) for it.
  2. The Conspiracy: The vmlinuz symlink updated itself to point at the broken new kernel. A symlink is just a shortcut file that points to another file. vmlinuz is a shortcut that always points to whichever kernel Ubuntu considers current. It quietly updated itself to point at the broken new one.
  3. The Betrayal: GRUB (the boot menu program that runs before Ubuntu and lets you choose which kernel to start) tried to boot the new kernel. The kernel loaded, looked for disk drivers inside the missing initramfs, found none, and panicked.
  4. The Resolution: Rebuilt the initramfs from a Live USB using chroot. The laptop booted. Trust partially restored.

The Journey Through Boot Hell

What made this particularly special was that the fix wasn’t straightforward. Every layer of recovery revealed a new problem.

Layer 1: The GRUB Shell

The normal GRUB boot menu never appeared. Instead I got a blinking grub> prompt, meaning GRUB had started but couldn’t find its own config files to display the menu. I had to manually type commands to find my partition (a dedicated section of the hard drive where Ubuntu lives) and load the menu:

set root=(hd0,gpt2)
 set prefix=(hd0,gpt2)/boot/grub
 insmod normal
 normal

Layer 2: The Missing initrd

After getting into the GRUB editor and trying to boot, I discovered the initrd line was absent. Gone. initrd (short for initial RAM disk) is the line in GRUB that tells the kernel where to find its initramfs file. Without it, the kernel boots completely blind. Every boot entry in GRUB has two key lines: one pointing to the kernel file, and one pointing to the initramfs file. The second line was missing entirely. I added it manually, only to be told the file didn’t exist. Because it didn’t.

Layer 3: The initramfs Emergency Shell

I eventually got the system far enough to drop into an (initramfs) emergency shell. This is a bare-bones command prompt that appears when the initramfs loads but fails to find the real system. Progress! Except:

ls /dev/sd*
 # No such file or directory

 ls /dev/nvme*
 # No such file or directory

In Linux, every piece of hardware appears as a file under /dev/. Hard drives show up as /dev/sda or /dev/nvme0n1. If they aren’t there, the kernel simply cannot see the hardware. My disk was completely invisible because no storage drivers had been loaded. I tried modprobe (the command used to manually load a hardware driver into the kernel) for both common disk types:

modprobe nvme   # driver for NVMe SSDs
 modprobe ahci   # driver for SATA drives

Nothing. The disk simply did not exist as far as this kernel was concerned.

The hard truth: When the kernel can’t see any storage devices from the initramfs shell, there is nothing I can do from within that shell. I needed a Live USB.

Layer 4: The Live USB Rescue

A Live USB is a USB drive with a full Ubuntu system on it that you can boot and run entirely without touching your hard drive. It’s the standard recovery tool for situations like this.

Booting from it, I could now see my disk normally. The fix involved using chroot to operate on the broken installation from the outside. chroot (short for change root) is a command that makes your terminal think a different folder is the root of the system. It’s how you work on a broken installation while booted into a different one, like performing surgery on a patient from another hospital.

sudo mount /dev/sda2 /mnt
 sudo mount /dev/sda1 /mnt/boot/efi
 sudo mount --bind /dev /mnt/dev
 sudo mount --bind /proc /mnt/proc
 sudo mount --bind /sys /mnt/sys
 sudo mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
 sudo chroot /mnt

 update-initramfs -u -k all
 update-grub
 grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu

A quick note on efi: modern computers use a system called UEFI to handle the very first step of booting. It requires a small dedicated partition on your drive called the EFI partition where bootloader files live. The grub-install command reinstalls GRUB into that partition so the computer knows how to start Ubuntu again.

And just like that, after two hours, three error messages, one emergency shell, and a brief existential crisis, the laptop booted.

What Actually Caused It

Ubuntu installed kernel 6.17.0-19-generic but the initrd.img for it was either not generated or generated incorrectly. The system had two kernels:

  • 6.14.0-37 — with a working initrd ✅
  • 6.17.0-19 — without one ❌

GRUB defaulted to the newer broken one. Everything collapsed.

One missing file. Two hours of my life. Zero warning from Ubuntu.

How to Never Let This Happen to You

1. Make GRUB always visible

By default Ubuntu hides the GRUB menu and boots immediately. Turning it back on means you can always pick an older working kernel if a new one breaks. Edit /etc/default/grub:

GRUB_TIMEOUT_STYLE=menu
 GRUB_TIMEOUT=10

Then apply the change:

sudo update-grub

This alone would have saved me an hour.

2. Pin a working kernel

apt-mark hold tells Ubuntu’s package manager to never automatically update or remove a specific package. Use it to protect a kernel you know works:

sudo apt-mark hold linux-image-$(uname -r)
 sudo apt-mark hold linux-headers-$(uname -r)

3. Rebuild initramfs after major updates

Get in the habit of running this after kernel updates. The -u flag means update, -k all means do it for every installed kernel:

sudo update-initramfs -u -k all

4. Keep a Live USB handy

Flash a Ubuntu ISO to a spare USB and keep it in your bag. It takes 10 minutes to make and has saved me twice now.

If It Already Happened to You

If you’re reading this from another device while your machine stares back at you with a kernel panic, here’s the express version:

# 1. Boot from Live USB, open terminal

 # 2. lsblk lists all drives and partitions so you can identify yours
 lsblk
 sudo mount /dev/sda2 /mnt            # mount your Ubuntu partition
 sudo mount /dev/sda1 /mnt/boot/efi   # mount your EFI partition

 # 3. Mount system directories so chroot works properly
 sudo mount --bind /dev /mnt/dev
 sudo mount --bind /proc /mnt/proc
 sudo mount --bind /sys /mnt/sys
 sudo mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars

 # 4. Chroot into your broken system and fix it
 sudo chroot /mnt
 update-initramfs -u -k all
 update-grub
 grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu

 # 5. Exit and reboot
 exit
 sudo reboot

Your data is completely safe throughout this process. We’re only touching boot files. Your documents, photos, and everything in /home are untouched.

⎇ main