Incorrect rebooting of marvin 19.09.2017

Introduction

Following a much delayed maintenance period for marvin, the latest published update from RedHat on the 10 Sept 2017 (i.e. kernel 2.6.32-696.10.2.el6.x86_64) was implemented on the nodes and marvin itself.

Progress

There are usually some small problems, such as irregularities with RedHat's subscription manager software or with the repositories. Marvin had one such referring to an opensuse URL having something to do with Ocaml. This was disabled and the update went fine.

Problem

The nodes were fine, but marvin itself did not boot properly

Forgotten

The precaution of disabling automatic mounting of the INTERNAL, SHELF and STORAGE was not followed. This was because an attempt was made to manually unmount the systems before the closedown, which is not a necessary operation. The manual umount's were not successful. There has been no problem in the past just rebooting while these were mounted. So they were uncommented in the fstab file.

Truth is, it was thought that these filesystems were set to noauto, but this was not true, they are set up with normal defaults. So, after the manual unmounting attempt, these lines were not commented out again, so their automatic mounting was set again.

The problem here is that - and the reason is not understood - automatic mounting (especially of STORAGE) causes a hiccup in the RedHat boot process. All indications point to a hanging at the fsck stage. SO this is why marvin did not boot up properly, and also they reason why this was unexpected, because uncommenting was not meant to allow automatic mounting becaus eit was though the noauto would then come into force. That was a mistake.

Needless, to say, much heartbreak can be avoid if these things are not forgotten.

To remedy this, a live Linux distribution on a USB stick is required. The CIBOX branded stick has the "SystemRescueCD" distribution on it. It recognises LVM, so you can simply go into the /dev/mapper foler and mount