Difference between revisions of "Singularity with grid engine"

From wiki
Jump to: navigation, search
 
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
JW may 2019
+
We have been working hard in the background to allow users to be able to run Docker images.
 +
Good news you can now do this via Singularity (Don’t worry this is all installed and should just work).
 +
Try Conda to install your package of interest first. If it is not on Conda, you can try the following:
  
 +
Please remember, you must run this through qsub and not directly on the head node.
  
−
Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/
+
Full singularity documentation is here;
 +
https://www.sylabs.io/guides/3.2/user-guide/
  
−
Pete installed singularity on phylo
+
where to get the images from? search here:
 +
https://hub.docker.com/
 +
if you search funannotate you will see the phrase: nextgenusfs/funannotate
 +
this is what you want
  
 +
to download the image, simply type:
 +
singularity pull docker:name_of_image
 +
e.g.  singularity pull docker:nextgenusfs/funannotate
  
−
now a requestable resource with "-l singularity" to make sure you get a node with singularity on
+
Be slightly careful about versions. Teh command may not actually pull down the laters see this below, also how to do this on Rocky Linux 8:
 +
My download command (which pulled down an old version):
 +
singularity pull docker:docker broadinstitute/gatk  (did not pull the latest - check versions)
  
 +
what did pull down the correct version (Redhat 6.9, running on centos 7):
  
−
Example:
+
  singularity pull docker:broadinstitute/gatk:latest
−
  qsub -l singularity -b y singularity run /full_path_to/ubuntu.sif ./test_script.sh
 
−
replace:ubuntu.sif with whatever image you are trying to run
 
  
 +
Now trying on Rocky Linux 8. This worked:
  
−
example2:
+
  singularity pull docker://broadinstitute/gatk
−
  qsub -pe multi 8 -l singularity -b y singularity run /full_path/funannotate_latest.sif /full_path/fun_singularity.sh
 
−
The shell must have the curent working directory full path in it as cd /ful_path/  
 
−
the #!cwd command will not work!
 
  
  
−
To add another node with singularity on:
+
Please, do not run this on the head node, this must be run through the qsub system.
  
 +
Example of how to run via qsub:
 +
qsub -l singularity -b y singularity run /full_path_to/ubuntu.sif /full_path_to/test_script.sh
 +
replace: ubuntu.sif with whatever image you are trying to run
  
−
qconf -me <nodename>
+
Lets go through that command in more depth:
 +
qsub -l singularity -b y singularity run
  
−
On the complex_values line remove NONE if present, and add "singularity=TRUE"
+
this is a special command so singularity will run on a specific server, you don't need to alter this, just copy it.
 +
  /full_path_to/ubuntu.sif
 +
this is the image you download for the software you are interested in
  
 +
  /full_path_to/test_script.sh
 +
this needs to contain the commands you want to run
  
  
 +
example 2, running the image with qsub:
 +
qsub -pe multi 8 -l singularity -b y singularity run /full_path/funannotate_latest.sif /full_path/fun_singularity.sh
 +
The shell must have the current working directory full path in it as cd /ful_path/ 
 +
putting #!cwd command in your shell scripts will not work!
 +
cd /ful_path/
 +
-pe multi 8    this asks for 8 cores, just as normal.
  
−
-b is a normal qsub command
+
please note that in the shell script:
 +
full paths to everything are required!
  
−
      -b y[es]|n[o]
 
−
              Available for qsub, qrsh only. Qalter does not allow changing this option. This  option  cannot
 
−
              be embedded in the script file itself.
 
  
−
              Gives  the  user  the  possibility  to indicate explicitly whether command should be treated as
+
Notes for Admin:
−
              binary or script. If the argument of -b is ’y’, then command  may be a binary or  script.  The
+
To add another node with singularity on:
−
              command  might not be accessible from the submission host.  Nothing except the path of the com-
 
−
              mand will be transferred from the submission host to the execution host. Path aliasing will  be
 
−
              applied to the path of command before it is executed.
 
  
−
              If the argument of -b is ’n’ then command needs to be a script, and it will be handled as such.
+
qconf -me <nodename>
−
              The script file has to be accessible by the submission host. It will be transferred to the exe-
 
−
              cution host. qsub/qrsh will search directive prefixes within scripts.
 
  
−
              qsub will implicitly use -b n, whereas qrsh will apply the -b y option if nothing else is spec-
+
On the complex_values line remove NONE if present, and add "singularity=TRUE"
−
              ified.
+
Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/
−
 
+
now a request-able resource with "-l singularity" to make sure you get a node with singularity on
−
              The value specified with this option or the corresponding value specified in qmon will only  be
 
−
              passed  to defined JSV instances if the value is yes. The name of the parameter will be b. The
 
−
              value will be y also when the long form yes was specified during submission.  (See -jsv  option
 
−
              below or find more information concerning JSV in jsv(1).)
 
−
 
 
−
              Please  note  that  submission of command as a script (-b n) can have a significant performance
 
−
              impact, especially for short running jobs and big job scripts.  Script submission adds a number
 
−
              of operations to the submission process: The job script needs to be
 
−
              - parsed at client side (for special comments)
 
−
              - transferred from submit client to qmaster
 
−
              - spooled in qmaster
 
−
              - transferred to execd at job execution
 
−
              - spooled in execd
 
−
              - removed from spooling both in execd and qmaster once the job is done
 
−
              If job scripts are available on the execution nodes, e.g. via NFS, binary submission can be the
 
−
              better choice.
 

Latest revision as of 09:21, 21 January 2022

We have been working hard in the background to allow users to be able to run Docker images. Good news you can now do this via Singularity (Don’t worry this is all installed and should just work). Try Conda to install your package of interest first. If it is not on Conda, you can try the following:

Please remember, you must run this through qsub and not directly on the head node.

Full singularity documentation is here; 

https://www.sylabs.io/guides/3.2/user-guide/

where to get the images from? search here: 

https://hub.docker.com/

if you search funannotate you will see the phrase: nextgenusfs/funannotate this is what you want

to download the image, simply type: 

singularity pull docker:name_of_image
e.g.  singularity pull docker:nextgenusfs/funannotate

Be slightly careful about versions. Teh command may not actually pull down the laters see this below, also how to do this on Rocky Linux 8: My download command (which pulled down an old version): 

singularity pull docker:docker broadinstitute/gatk  (did not pull the latest - check versions)

what did pull down the correct version (Redhat 6.9, running on centos 7): 

singularity pull docker:broadinstitute/gatk:latest

Now trying on Rocky Linux 8. This worked: 

singularity pull docker://broadinstitute/gatk


Please, do not run this on the head node, this must be run through the qsub system.

Example of how to run via qsub: 

qsub -l singularity -b y singularity run /full_path_to/ubuntu.sif /full_path_to/test_script.sh
replace: ubuntu.sif with whatever image you are trying to run

Lets go through that command in more depth: 

qsub -l singularity -b y singularity run

this is a special command so singularity will run on a specific server, you don't need to alter this, just copy it. 

 /full_path_to/ubuntu.sif

this is the image you download for the software you are interested in 

 /full_path_to/test_script.sh
this needs to contain the commands you want to run


example 2, running the image with qsub: 

qsub -pe multi 8 -l singularity -b y singularity run /full_path/funannotate_latest.sif /full_path/fun_singularity.sh

The shell must have the current working directory full path in it as cd /ful_path/ 

putting #!cwd command in your shell scripts will not work!
cd /ful_path/
-pe multi 8     this asks for 8 cores, just as normal.

please note that in the shell script: 

full paths to everything are required!


Notes for Admin: To add another node with singularity on: 

qconf -me <nodename>

On the complex_values line remove NONE if present, and add "singularity=TRUE" Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/ now a request-able resource with "-l singularity" to make sure you get a node with singularity on

Retrieved from "http://stab.st-andrews.ac.uk/wiki/index.php?title=Singularity_with_grid_engine&oldid=3510"