Linux Hardened No More

I changed from Arch Linux linux-hardened to plain linux kernels on my Dell R530 server.

I did my habitual weekly pacman -Syu today, and rebooted the R530. It did not come back. According to the monitor in my machine room (a.k.a. furnace room in the basement) there was a kernel problem, something to do with interrupts.

I recovered by power cycling, and getting to the GRUB menu. As it happens, I have the linux-lts kernel installed and ready to use for exactly this circumstance. The GRUB menu has linux-lts as a selectable option. I highly recommend having something like linux-lts installed side-by-side with regular linux or linux-hardened kernels so that recovery is easier.

To get the R530 to boot linux, I followed the same steps as when I replaced linux-hardened on my Qotom fanless server.

  1. sudo pacman -R linux-hardened linux-hardened-headers
  2. sudo pacman -Sy linux linux-headers
  3. sudo mv grub.cfg old.grub.cfg

At this point, I decided to finally figure out how to choose the default kernel to boot. GRUB somehow decides which one of the multiple kernels installed is the default, and should be booted.

At least for Arch Linux, the choice is made in /etc/default/grub, the line GRUB_DEFAULT=... Apparently kernels are numbered based on a 0-based index of menuentry lines in /boot/grub/grub.cfg.

The menuentry lines look something like this:

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux, with Linux linux-lts' --class ...
    ...
menuentry 'Arch Linux, with Linux linux' --class ...
    ...

Because I had linux-hardened installed previously, I recreated grub.cfg: sudo grub-mkconfig -o /boot/grub/grub.cfg

I double-checked that linux-hardened no longer appeared in the file. I counted where menuentry 'Arch Linux, with Linux linux' appeared in the GRUB config file was - second menuentry. I edited /etc/default/grub so that the line read GRUB_DEFAULT=1. The menuentry with linux-lts is first, index 0, and the menuentry with regular linux is second, index 1. I recreated grub.cfg again, to get linux as the default kernel.

  1. sudo grub-mkconfig -o /boot/grub/grub.cfg
  2. sudo vim grub.cfg
    • remove “quiet” from 2 invocations of /vmlinuz-linux so I can see what goes on at boot.
  3. sync;sync;sync;sudo systemctl reboot

This time, the Dell R530 completed booting the regular linux kernel:

1061 % uname -a
Linux monarch 7.2.4-arch1-2 #1 SMP PREEMPT_DYNAMIC Tue, 08 Sep 2026 10:22:31 +0000 x86_64 GNU/Linux

For completeness sake, the GRUB_DEFAULT value appears in /boot/grub/grub.cfg in a line that looks like set default="1" (or whatever value GRUB_DEFAULT has). You probably want to set the default in /etc/default/grub so that the next time the grub package gets updated you get the kernel you anticipate.

This is the second machine that’s had a problem with linux-hardened in two months. Additionally, the linux-hardened maintainer has had problems putting out the package on the same schedule as the linux package. Updating kernels is always “exciting”, I’m going to try plain linux package kernels for a while to see if they give me as many problems.