Frontend Restart

Introduction

Contains notes on how to restart marvin.

Measures

Bring all nodes down before restart

This is possibly the most useful measure. Primarily, it is due to the nodes using marvin to keep various filesystems mounted, and the havoc they experience when marvin stops doing this. NFS4 stale filehandles then appear and are hard to get rid of. This measure is not immediately obvious, becuase all the nodes are updated on a rolling basis and often do not need to be turned off. And then, when marvin is back up, and once its filesystems are verified, the nodes maybe brought back up. Of course this seems like quite alot of extra work, but it's worth it in terms of saving later debugging time.

Provisos

Restarting marvin is a major operation, as all running jobs are lost.

It is therefore necessary to advise all users well in advance, as to when it might happen.