Difference between revisions of "Singularity with grid engine"

From wiki
Jump to: navigation, search
Line 1: Line 1:
JW may 2019
+
We have been working hard in the background to allow users to be able to run Docker images.
 +
Good news you can now do this via Singularity (Don’t worry this is all installed and should just work).
 +
Try Conda to install your package of interest first. If it is not on Conda, you can try the following:
  
 +
Please remember, you must run this through qsub and not directly on the head node.
  
Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/
 
  
Pete installed singularity on phylo
 
  
 +
Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/
  
 
now a request-able resource with "-l singularity" to make sure you get a node with singularity on
 
now a request-able resource with "-l singularity" to make sure you get a node with singularity on
  
Full singularity documentation is here; https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/  
+
Full singularity documentation is here; https://www.sylabs.io/guides/3.2/user-guide/
  
 
Example:
 
Example:
Line 32: Line 34:
 
  cd /ful_path/
 
  cd /ful_path/
 
  -pe multi 8    this asks for 8 cores, just as normal.  
 
  -pe multi 8    this asks for 8 cores, just as normal.  
 +
 +
 +
where to get the images from?
 +
to get an image: (search here: https://hub.docker.com/)
 +
if you search funannotate you will see the phrase: nextgenusfs/funannotate
 +
this is what you want
 +
 +
to get the image
 +
singularity pull docker:nextgenusfs/funannotate
 +
 +
Please, do not run this on the head node, this must be run through the qsub system.
 +
  
  

Revision as of 11:00, 24 May 2019

We have been working hard in the background to allow users to be able to run Docker images. Good news you can now do this via Singularity (Don’t worry this is all installed and should just work). Try Conda to install your package of interest first. If it is not on Conda, you can try the following:

Please remember, you must run this through qsub and not directly on the head node.


Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/

now a request-able resource with "-l singularity" to make sure you get a node with singularity on

Full singularity documentation is here; https://www.sylabs.io/guides/3.2/user-guide/

Example:

qsub -l singularity -b y singularity run /full_path_to/ubuntu.sif ./test_script.sh
replace: ubuntu.sif with whatever image you are trying to run

Lets go through that command in more depth:

qsub -l singularity -b y singularity run

this is a special command so singularity will run on a specific server

 /full_path_to/ubuntu.sif 

this is the image you download for the software you are interested in

 ./test_script.sh
this needs to contain the commands you want to run


example 2:

qsub -pe multi 8 -l singularity -b y singularity run /full_path/funannotate_latest.sif /full_path/fun_singularity.sh

The shell must have the current working directory full path in it as cd /ful_path/

putting #!cwd command in your shell scripts will not work!
cd /ful_path/
-pe multi 8     this asks for 8 cores, just as normal. 


where to get the images from?

to get an image: (search here: https://hub.docker.com/)

if you search funannotate you will see the phrase: nextgenusfs/funannotate this is what you want

to get the image

singularity pull docker:nextgenusfs/funannotate

Please, do not run this on the head node, this must be run through the qsub system.


To add another node with singularity on:


qconf -me <nodename>

On the complex_values line remove NONE if present, and add "singularity=TRUE"