Why a Queue Manager?

Introduction

With so many personal devices around nowadays: laptops, smartphones, etc., it's normal to ask the question, "why must I submit to a queue manager?".

Most of this can be explained by the personal nature of such devices, and how this can lure users into thinking that queuing is really a bad dream from the past which is happily becoming rarer now.

However, a look around modern life will show that perhaps the activity of queuing takes less time now (especially if one compares to periods of extreme shortages of staples such as bread and gasoline) but that there are in fact more queues now.

What has occurred to decrease the time spent queuing is that queue management is taken much more seriously now than in the past. Many techniques have now been put in place to make queuing both shorter and less stressful. We interact with a higher number of queues in modern times because there has been a growth in centralisation and sharing of services, and modern technologies have made queuing less of burden, and, at time, undetectable.

This is the case with marvin computer cluster, it is a shared resource which has a very efficient queue manager called Grid Engine, which will be explained here. Yes, it does mean some small inconveniences, but if one undertands that it is shared by a significant number of people working on different research projects

Why?

computational intensity

Scientific research projects tend to attract some very compute-intensive workloads. These are not, in fact, so common in the biological sciences (especially when compared to physics and chemistry), but nevertheless, research projects are often trying to solve some thorny problems, and some programs may resort to trying out all possible solutions and seeing which gives the right answer. Because the machines are on 24 hours a day, and users also do not have to supervise the processing (they can go away and come back later), some workloads are capable of taking up all possible resources from other programs.

This is not a case of bad manners, it is natural consequence of finding out things that have not been discovered yet ... it will logically tend to occur. So this is why using a queue manager is a good thing to consider as routine.

Large Resource

The compute cluster can be classed as a large resource, as opposed to laptops or personal computers, which are small. The difference is not particularly big, and it's possible to find cases where a laptop may be faster at processing something than a compute cluster*. It's the same with many tools, cured by finding out how big a problem is before choosing the tool.

There is a much greater obligation to efficiently use a large costly resource than a small cheap one. A laptop maybe left switched off doing nothing for long periods of time, because it is too small to be considered a waste. A large resourcfe however needs to justify itself, and this it does by being available all the time, and keep a record of how often it is used.

A queue manager also takes care of this issue, and includes an accounting module, showing on whihc jobs its time was spent.

*Interestingly, a similar phenomenon has occurred between smartphones and laptops: many smartphones can compete with laptops in tems of processing power. However it is often their small size and dedicated nature which limits their usefulness in terms of general processing.

Is using the queue always necessary?

On the marvin frontend, half of its 32 processors are reserved for queue usage. The other 16 are outside the queue and are for small tasks which are generally part of interactive sessions. We can define a small task by saying that it finishes in under 20 minutes and only ever uses one processor. This is a rough rule of thumb, as sometimes one doesn't know how long a task will take, so there is leniency, and it is in general hoped that user will remember to look out for apparently small jobs that end up taking alot of time.