blog

Debian 12's latest kernel update is freezing some servers. Check before you reboot

If you run a server on Debian 12, don't reboot it this week until you've checked which kernel it will come back up on. The latest kernel update, 6.1.0-55, is making some machines freeze a few minutes after boot.

There's no fix yet, but there is a simple way round it. Here's what's going on and what I'd do.

What happened

Debian 12 ("bookworm") got a new kernel package through its security updates, linux-image-6.1.0-55-amd64, version 6.1.192-1. On Sunday evening someone filed Debian bug 1154101. It says their server stopped answering "after few minutes (imap, smtp, http, ssh...)". They rated it critical, because it "breaks the whole system".

By Monday morning four more bug reports had been merged into it. They all point at the same place: a crash in the kernel's ext4 code, the part that handles the filesystem. One of the people affected says they "had to use a backup of the system disk" to get going again.

It isn't tied to one type of machine. The first report is from a physical server. Others are on Xen virtual machines, and one person says "Hyper-V VMs are affected too".

The Debian kernel team has picked it up and tagged it as an upstream kernel problem. As I write this, no fixed package has been released.

Does it affect you?

You're in scope if all three of these are true:

It's Debian 12. Debian 13 runs a newer kernel line and isn't in these reports. Ubuntu builds its own kernels, so this bug isn't about Ubuntu either.

It uses ext4. That's the filesystem Debian's installer picks by default, so most Debian servers do.

It has 6.1.0-55 installed. This is the bit that catches people out. Unattended upgrades on Debian will happily install a new kernel, but by default they don't reboot. So your server can look perfectly healthy today and fall over the next time it restarts, which might be months from now or might be tonight after a power cut.

Hosting company VPSs count too. If it's Debian 12 and you chose the image, it's probably yours to look after.

How to check

Log in and run:

uname -r

That's the kernel running now. Then see what's installed:

dpkg -l 'linux-image-6.1.0-5*'

If you see 6.1.0-55 in the list but uname -r still says 6.1.0-54, the new one is waiting for your next reboot. If uname -r already says 6.1.0-55 and the server's been fine for days, you may be one of the lucky ones, but I'd still keep an eye on it.

What to do this week

1. Don't reboot casually. Hold off on planned restarts until a fixed kernel is out, unless you've done step 3 first.

2. Don't clean up old kernels. Skip apt autoremove for now. The 6.1.0-54 kernel is your way back, so keep it installed.

3. Make 6.1.0-54 the default. The workaround in the bug report is to "select the previous kernel version through Grub (6.1.0-54)". That's fine if you can see the boot menu. On a cloud server you often can't, so this is my advice rather than Debian's: set it as the default instead. Find the exact menu titles with:

sudo grep -E "menuentry |submenu " /boot/grub/grub.cfg

Then in /etc/default/grub set something like:

GRUB_DEFAULT="Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 6.1.0-54-amd64"

Run sudo update-grub, and copy the titles exactly as your server shows them. Write yourself a note to put it back to 0 once the fix lands.

4. If it's not installed yet, stop it arriving. sudo apt-mark hold linux-image-amd64 stops apt pulling in the new kernel. Again, that's my suggestion, and it needs undoing later with apt-mark unhold.

5. Take a snapshot. If your host offers snapshots, take one before any reboot. Make sure you know how to reach the server's console too, because SSH is no use to you if the machine has frozen.

6. Watch the bug. When a fixed kernel appears, unhold, update, put GRUB back to normal and reboot in a quiet hour.

The trade-off

Staying on 6.1.0-54 means going without whatever security fixes came in 6.1.0-55 for a few days. For most small business servers, a few days on the previous kernel is a far smaller risk than a server that dies every time it boots. Just don't let "a few days" quietly turn into "until Easter".

The bigger point

Updates are still the right thing to do. Most weeks I'm writing here to tell you to install them faster. But every update needs a way back, and a reboot nobody's watching is when you find out whether you have one.

That's why I'd always take a snapshot first and test that backups actually restore. If the server runs a bespoke app that your business depends on, the team at Dev Partners look after existing software, including the updates underneath it, so a bad kernel doesn't take the app down with it.

And if you'd rather not be the one reading Debian bug reports on a Monday morning, that's what my support retainers are for.