Forcing a direct kernel reboot is sometimes needed and recommended

Normally when we need to reboot one of our servers, I type 'reboot' and don't think twice about it. This is sometimes a mistake (for reasons other than rebooting the wrong server), as I was reminded yesterday when I ran 'reboot' on a server and had it hang during the reboot process. To put the summary up front, if the server I'm rebooting is in a dodgy state, such as having a piece of hardware drop off the PCIe bus, probably I want to proceed straight to an abrupt, direct kernel reboot. I can always selectively shut down certain important and safe services beforehand.

(The problem with not going straight to an abrupt reboot is that you usually don't get a second chance. If you trigger a regular reboot and it hangs, you probably need to power cycle the system, which often requires physical access.)

When you use 'reboot' to reboot a modern Linux server, there are several things that can go wrong. The normal reboot process goes through your init system to perform an orderly shutdown (these days that's usually systemd), and so during that orderly shutdown process programs and services can hang. There are a series of classical ordering mistakes that can create service hangs (often involving the kernel); for example, shutting down networking with NFS mounts still present (also), or trying to shut down swap space with a lot of stuff swapped out (this doesn't necessarily permanently hang the system but it certainly can take a long time).

More seriously, you can be left with unkillable processes that your system's shutdown process is waiting for. We've seen this with processes that have hit a kernel OOPS, which turns the process into an unkillable zombie (these don't normally trigger an immediate reboot). Based on symptoms, I think this can also happen if a process on the system is holding a reference to some hardware resource (for example by having an open file descriptor for a GPU), terminating the process releases the hold (for example, by closing the file descriptor) and this triggers cleanups in the hardware driver, and either the driver or the hardware hangs. Of course ideally neither the driver nor the hardware has problems, but sometimes we're rebooting because they do.

I suspect that the Linux kernel goes through some degree of driver shutdown and release on a regular reboot, although I haven't tried to navigate through the kernel code to see for sure (but it does seem like you'd want to do things like cancel outstanding DMA activities). If it does, that's another place where a regular shutdown can hang, if the driver either has bugs itself or runs into issues where it gets stuck because the hardware isn't behaving as expected. However, I don't know if forced, abrupt reboots bypass that kernel level driver shutdown.

Systemd offers a nuanced set of options for halting and rebooting, as covered in 'systemctl halt'. If you use '--force' once, it will skip service shutdowns (which I believe will skip shutting down the network) but will still kill all processes and unmount all filesystems, which might take a while or possibly hang. If you use '--force' twice, the systemctl process asks the kernel for an immediate halt or reboot, which bypasses a lot of this except perhaps kernel level driver shutdown.

My alternate, default approach to abrupt reboots is to write a 'b' into /proc/sysrq-trigger, for example 'echo b >/proc/sysrq-trigger'. One advantage of this is that it's easy for me to remember, and another is that it only needs shell builtins ('echo' is built in to most shells, especially root shells). If the system is sufficiently badly off that I want to reboot it by force, this is about as low level as I can get.

If I feel like it, I can be slightly less abrupt by at least trying to force some things to disk:

echo s >/proc/sysrq-trigger
echo u >/proc/sysrq-trigger
# maybe: sleep 5
echo b >/proc/sysrq-trigger

This will try to sync and then unmount everything before doing the abrupt reboot. I don't think you can get this through systemd, because the important bit about this sequence is that you're not waiting forever for either the sync or the unmount; after at most a few seconds, you're telling the kernel to force the reboot.

If you've loaded a new kernel with kexec and want to force an immediate reboot into it, this is done with 'kexec -e' or perhaps 'systemctl kexec --force --force'. As far as I know there's no way to do this through /proc/sysrq-trigger. You might want and need to do this if you hit OOPSes in an installer environment, for one example that's happened to us.