http://stab.st-andrews.ac.uk/wiki/api.php?action=feedcontributions&feedformat=atom&user=PeterThorpewiki - User contributions [en]2026-04-16T08:33:05ZUser contributionsMediaWiki 1.30.0http://stab.st-andrews.ac.uk/wiki/index.php?title=Singularity_with_grid_engine&diff=3510Singularity with grid engine2022-01-21T09:21:45Z<p>PeterThorpe: </p>
<hr />
<div>We have been working hard in the background to allow users to be able to run Docker images. <br />
Good news you can now do this via Singularity (Don’t worry this is all installed and should just work). <br />
Try Conda to install your package of interest first. If it is not on Conda, you can try the following:<br />
<br />
Please remember, you must run this through qsub and not directly on the head node. <br />
<br />
Full singularity documentation is here; <br />
https://www.sylabs.io/guides/3.2/user-guide/<br />
<br />
where to get the images from? search here: <br />
https://hub.docker.com/<br />
if you search funannotate you will see the phrase: nextgenusfs/funannotate<br />
this is what you want<br />
<br />
to download the image, simply type:<br />
singularity pull docker:name_of_image<br />
e.g. singularity pull docker:nextgenusfs/funannotate<br />
<br />
Be slightly careful about versions. Teh command may not actually pull down the laters see this below, also how to do this on Rocky Linux 8:<br />
My download command (which pulled down an old version): <br />
singularity pull docker:docker broadinstitute/gatk (did not pull the latest - check versions)<br />
<br />
what did pull down the correct version (Redhat 6.9, running on centos 7):<br />
<br />
singularity pull docker:broadinstitute/gatk:latest<br />
<br />
Now trying on Rocky Linux 8. This worked:<br />
<br />
singularity pull docker://broadinstitute/gatk<br />
<br />
<br />
Please, do not run this on the head node, this must be run through the qsub system. <br />
<br />
Example of how to run via qsub:<br />
qsub -l singularity -b y singularity run /full_path_to/ubuntu.sif /full_path_to/test_script.sh<br />
replace: ubuntu.sif with whatever image you are trying to run<br />
<br />
Lets go through that command in more depth:<br />
qsub -l singularity -b y singularity run<br />
<br />
this is a special command so singularity will run on a specific server, you don't need to alter this, just copy it. <br />
/full_path_to/ubuntu.sif <br />
this is the image you download for the software you are interested in<br />
<br />
/full_path_to/test_script.sh<br />
this needs to contain the commands you want to run<br />
<br />
<br />
example 2, running the image with qsub:<br />
qsub -pe multi 8 -l singularity -b y singularity run /full_path/funannotate_latest.sif /full_path/fun_singularity.sh<br />
The shell must have the current working directory full path in it as cd /ful_path/ <br />
putting #!cwd command in your shell scripts will not work!<br />
cd /ful_path/<br />
-pe multi 8 this asks for 8 cores, just as normal. <br />
<br />
please note that in the shell script:<br />
full paths to everything are required!<br />
<br />
<br />
Notes for Admin:<br />
To add another node with singularity on:<br />
<br />
qconf -me <nodename><br />
<br />
On the complex_values line remove NONE if present, and add "singularity=TRUE"<br />
Followed guide here: https://blogs.univa.com/2019/01/using-univa-grid-engine-with-singularity/<br />
now a request-able resource with "-l singularity" to make sure you get a node with singularity on</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Conda&diff=3509Conda2020-07-28T08:56:26Z<p>PeterThorpe: </p>
<hr />
<div>= conda =<br />
<br />
'''This is the future'''<br />
<br />
<br />
Conda on Marvin on a per user basis, so you have total control over it. <br />
Log in and from the command line type Only ever do this once or you will get error pessages):<br />
<br />
install-bioconda<br />
<br />
Either log out and back in, or type:<br />
source ~/.bashrc<br />
<br />
This will make a user specific version of conda avaible to you.<br />
<br />
'''To find the package you want, search here:'''<br />
'''<br />
https://bioconda.github.io/conda-recipe_index.html'''<br />
<br />
'''or google search:''' - as there are multiple channels!!<br />
conda name_of_tool_you_want<br />
<br />
We strongly advise using environments: You can do this in many ways. Please see the link for more details (here you can specifiy exact version etc ..)<br />
https://conda.io/docs/user-guide/tasks/manage-environments.html <br />
<br />
The easiest usage would be: e.g. <br />
<br />
conda create -n NAME_OF_ENV PACKAGE_TO_INSTALL<br />
<br />
This is an example of a toold called roary<br />
<br />
conda create -n roaryENV roary<br />
<br />
once it has installed, you can activate the environment by typing:<br />
<br />
conda activate roaryENV<br />
<br />
As a lot of tools have dependencies, all the dependencies should be installed during this process. It is a good idea to keep them looked up in their own ENV, so they dont interfere with dependencies and spcific version required for other things you have installed. <br />
<br />
To get the latest version<br />
<br />
conda update roary<br />
<br />
<br />
You are now ready to use this package.<br />
<br />
conda deactivate to leave this environment.<br />
<br />
If you want to install a specific version of a tool (use the equals sign and the version you want):<br />
<br />
conda create -n samtools1.3 samtools=1.3<br />
<br />
or <br />
<br />
conda create -n python27 python=27<br />
<br />
once you have this version of python installed, you can easily use (after you have activated the python env:<br />
pip install biopython (or whatever you require)<br />
<br />
'''conda and perl''' <br />
Firstly, perl is a pain to look after. The way the cluster has been historically set up, all perl versions and modules are different on all the nodes. The new cluster will not be like this. Therefore, we cannot install modules for every user on all the nodes. What we recommend is you having you own perl version using conda:<br />
conda create -n perlEnv perl<br />
<br />
conda activate perlEnv<br />
<br />
then install the modules you want<br />
<br />
http://www.cpan.org/modules/INSTALL.html<br />
<br />
cpan App::cpanminus<br />
<br />
cpanm Term::ReadKey<br />
<br />
Bioperl is difficult to install. Fact. But someone has put this in coda. So lets use that:<br />
<br />
conda create -n bioperlEnv perl-bioperl<br />
<br />
<br />
To list all the environments you have created (you will forget the envs after a while, so name them well!!):<br />
<br />
conda info –envs<br />
<br />
To list all the dependencies you have within an env<br />
<br />
conda list -n ''envname''<br />
<br />
For example, trinity comes with jellyfish, which can be difficult to install. Therefore some of these tools are a treasure chest of cools tools, which can save many hours and kg of pain in installation process. <br />
<br />
To install multiple tools in one env:<br />
conda create -n python36_bioperl python=3.6 perl-bioperl mummer scipy numpy biopython matplotlib<br />
<br />
You can activate multiple conda envs, they have preference as the latest is used first in you path: <br />
conda activate trinityenv<br />
conda activate python27<br />
conda activate python36<br />
in this example, a simple python command will use the python3 version as this was the last to be activated, if you want python 2. You have to specifiy python2. The perl will be the one in the monster python36 env i created. But jelly fish will be from the trinity env. Make sense? <br />
<br />
ENJOY!!!<br />
<br />
<br />
installing packages: https://bioconda.github.io/conda-recipe_index.html<br />
<br />
<br />
'''conda and samtools problems'''<br />
<br />
Bascially sometimes this get installed from the wrong channel. This may happen with other tools. We havent come across this yet. <br />
<br />
The problem will look like this if you try to use samtools:<br />
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory).<br />
<br />
The solution:<br />
install from the bioconda channel: -c bioconda <br />
conda install -c bioconda samtools<br />
<br />
or specifically the problem was found with unicycler:<br />
remove the old unicycler failed install:<br />
conda remove --name unicyclerENV --all <br />
<br />
remake this env from the bioconda channel:<br />
conda create -n unicyclerENV -c bioconda unicycler<br />
<br />
<br />
to update the base conda, if required<br />
<br />
conda update -n base conda<br />
<br />
<br />
'''conda hanging on solving environment'''<br />
<br />
Can you try this please, then try re installing the beastie package.<br />
<br />
conda config --remove channels conda-forge<br />
conda config --add channels conda-forge<br />
<br />
if this still doesnt work try:<br />
conda config --set channel_priority strict<br />
<br />
<br />
More samtools problems, try:<br />
https://github.com/bioconda/bioconda-recipes/issues/12100<br />
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory<br />
conda install -c bioconda samtools openssl=1.0</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Firewall_and_iptables&diff=3508Firewall and iptables2020-07-22T18:21:05Z<p>PeterThorpe: Created page with "firstly the iptables was accidentally wiped when trying to mount marvin to kennedy. SGE failed and so did ldap authentication from the nodes on new users. solution for ldap..."</p>
<hr />
<div>firstly the iptables was accidentally wiped when trying to mount marvin to kennedy. SGE failed and so did ldap authentication from the nodes on new users. <br />
<br />
solution for ldap<br />
Fixed. Ldap is not ldap. It is smbladp which listens on port 1544. OPen up port 1544 (see below on how to open up ports)<br />
<br />
restart ldap<br />
service slapd restart<br />
<br />
How to turn on and off firwll and save to iptables. <br />
https://www.cyberciti.biz/faq/turn-on-turn-off-firewall-in-linux/<br />
<br />
/etc/init.d/iptables save<br />
/etc/init.d/iptables stop<br />
<br />
<br />
SGE stopped working: <br />
Got it working was on port 6444<br />
<br />
To open firewall ports. Log in as root, but ssh as root with -x<br />
system-config-firewall <br />
<br />
"other ports" on the left - open the bad boys you want open.</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Main_Page&diff=3507Main Page2020-07-22T18:14:33Z<p>PeterThorpe: /* Cluster Administration */</p>
<hr />
<div><br />
'''KENNEDY HPC for Bioinf community '''<br />
* [[Kennedy manual]]<br />
<br />
<br />
= Usage of Cluster=<br />
* [[Cluster Manual]]<br />
* [[Kennedy manual]]<br />
* [[Why a Queue Manager?]]<br />
* [[Available Software]]<br />
* [[how to use the cluster training course]]<br />
* [[windows network connect]]<br />
<br />
= Documented Programs =<br />
<br />
The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.<br />
<br />
{|style="width:85%"<br />
|* [[abacas]]<br />
|* [[albacore]]<br />
|* [[ariba]]<br />
|* [[aspera]]<br />
|* [[assembly-stats]]<br />
|* [[augustus]]<br />
|-<br />
|* [[BamQC]]<br />
|* [[bamtools]]<br />
|* [[banjo]]<br />
|* [[bcftools]]<br />
|* [[bedtools]]<br />
|* [[bgenie]]<br />
|-<br />
|* [[BLAST]]<br />
|* [[Blat]]<br />
|* [[blast2go: b2g4pipe]]<br />
|* [[bowtie]]<br />
|* [[bowtie2]]<br />
|* [[bwa]]<br />
|-<br />
|* [[BUSCO]]<br />
|* [[CAFE]]<br />
|* [[canu]]<br />
|* [[cd-hit]]<br />
|* [[cegma]]<br />
|* [[clustal]]<br />
|-<br />
|* [[cramtools]]<br />
|* [[conda]]<br />
|* [[deeptools]]<br />
|* [[detonate]]<br />
|* [[diamond]]<br />
|* [[ea-utils]]<br />
|* [[ensembl]]<br />
|-<br />
|* [[ETE]]<br />
|* [[FASTQC and MultiQC]]<br />
|* [[Archaeopteryx and Forester]]<br />
|* [[GapFiller]]<br />
|* [[GenomeTools]]<br />
|* [[gubbins]]<br />
|-<br />
|* [[JBrowse]]<br />
|* [[kallisto]]<br />
|* [[kentUtils]]<br />
|* [[last]]<br />
|* [[lastz]]<br />
|* [[macs2]]<br />
|-<br />
|* [[Mash]]<br />
|* [[mega]]<br />
|* [[meryl]]<br />
|* [[MUMmer]]<br />
|* [[NanoSim]]<br />
|* [[nseq]]<br />
|-<br />
|* [[OrthoFinder]]<br />
|* [[PASA]]<br />
|* [[perl]]<br />
|* [[PGAP]]<br />
|* [[picard-tools]]<br />
|* [[poRe]]<br />
|-<br />
|* [[poretools]]<br />
|* [[prokka]]<br />
|* [[pyrad]]<br />
|* [[python]]<br />
|* [[qualimap]]<br />
|* [[quast]]<br />
|-<br />
|* [[qiime2]]<br />
|* [[R]]<br />
|* [[RAxML]]<br />
|* [[Repeatmasker]]<br />
|* [[Repeatmodeler]]<br />
|* [[rnammer]]<br />
|-<br />
|* [[roary]]<br />
|* [[RSeQC]]<br />
|* [[samtools]]<br />
|* [[Satsuma]]<br />
|* [[sickle]]<br />
|* [[SPAdes]]<br />
|-<br />
|* [[squid]]<br />
|* [[sra-tools]]<br />
|* [[srst2]]<br />
|* [[SSPACE]]<br />
|* [[stacks]]<br />
|* [[Thor]]<br />
|-<br />
|* [[Tophat]]<br />
|* [[trimmomatic]]<br />
|* [[Trinity]]<br />
|* [[t-coffee]]<br />
|* [[Unicycler]]<br />
|* [[velvet]]<br />
|-<br />
|* [[ViennaRNA]]<br />
|}<br />
<br />
= Queue Manager Tips =<br />
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:<br />
* [[Queue Manager Tips]]<br />
* [[Queue Manager : shell script command]]<br />
* [[Queue Manager emailing when jobs run]]<br />
* [[General Command-line Tips]]<br />
* [[DRMAA for further Gridengine automation]]<br />
<br />
= Data Examples =<br />
* [[Two Eel Scaffolds]]<br />
<br />
= Procedures =<br />
(short sequence of tasks with a certain short-term goal, often, a simple script)<br />
* [[Calculating coverage]]<br />
* [[MinION Coverage sensitivity analysis]]<br />
<br />
= Navigating genomic data websites=<br />
* [[Patric]]<br />
* [[NCBI]]<br />
* [[IGSR/1000 Genomes]]<br />
<br />
= Explanations=<br />
* [[ITUcourse]]<br />
* [[VCF]]<br />
* [[Maximum Likelihood]]<br />
* [[SNP Analysis and phylogenetics]]<br />
* [[Normalization]]<br />
<br />
= Pipelines =<br />
(Workflow with specific end-goals)<br />
* [[Trinity_Protocol]]<br />
* [[STAR BEAST]]<br />
* [[callSNPs.py]]<br />
* [[pairwiseCallSNPs]]<br />
* [[mapping.py]]<br />
* [[Edgen RNAseq]]<br />
* [[Miseq Prokaryote FASTQ analysis]]<br />
* [[snpcallphylo]]<br />
* [[Bottlenose dolphin population genomic analysis]]<br />
* [[ChIP-Seq Top2 in Yeast]]<br />
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]<br />
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]<br />
* [[Bisulfite Sequencing]]<br />
* [[microRNA and Salmo Salar]]<br />
<br />
=Protocols=<br />
(Extensive workflows with different with several possible end goals)<br />
* [[Synthetic Long reads]]<br />
* [[MinION (Oxford Nanopore)]]<br />
* [[MinKNOW folders and log files]]<br />
* [[Research Data Management]]<br />
* [[MicroRNAs]]<br />
<br />
= Tech Reviews =<br />
* [[SWATH-MS Data Analysis]]<br />
<br />
= Cluster Administration =<br />
* [[StABDMIN]]<br />
* [[Hardware Issues]]<br />
* [[marvin and IPMI (remote hardware control)]]<br />
* [[restart a node]]<br />
* [[mounting drives]]<br />
* [[Admin Tips]]<br />
* [[RedHat]]<br />
* [[Globus_gridftp]]<br />
* [[Galaxy Setup]]<br />
* [[Son of Gridengine]]<br />
* [[Blas Libraries]]<br />
* [[CMake]]<br />
* [[conda bioconda]]<br />
* [[Users and Groups]]<br />
* [[Installing software on marvin]]<br />
* [[emailing]]<br />
* [[biotime machine]]<br />
* [[SCAN-pc laptop]]<br />
* [[node1 issues]]<br />
* [[6TB storage expansion]]<br />
* [[PIs storage sacrifice]]<br />
* [[SAN relocation task]]<br />
* [[Home directories max-out incident 28.11.2016]]<br />
* [[Frontend Restart]]<br />
* [[environment-modules]]<br />
* [[H: drive on cluster]]<br />
* [[Incident: Can't connect to BerkeleyDB]]<br />
* [[Bioinformatics Wordpress Site]]<br />
* [[Backups]]<br />
* [[users disk usage]]<br />
* [[Updating BLAST databases]]<br />
* [[Python DRMAA]]<br />
* [[message of the day]]<br />
* [[SAN disconnect incident 10.01.2017]]<br />
* [[Memory repair glitch 16.02.2017]]<br />
* [[node9 network failure incident 16-20.03.2017]]<br />
* [[Incorrect rebooting of marvin 19.09.2017]]<br />
* [[ansible]]<br />
* [[webstie and word press]]<br />
* [[allow user access to other peoples data]]<br />
* [[RAM and RAM slots]]<br />
* [[ldap is not ldap]]<br />
* [[reset a password]]<br />
* [[sending emails from command line examples]]<br />
* [[disk management after shelf disk failure]]<br />
* [[firewall and iptables]]<br />
<br />
= Courses =<br />
<br />
==I2U4BGA==<br />
* [[Original schedule]]<br />
* [[New schedule]]<br />
* [[Actual schedule]]<br />
* [[Course itself]]<br />
* [[Biolinux Source course]]<br />
* [[Directory Organization Exercise]]<br />
* [[Glossary]]<br />
* [[Key Bindings]]<br />
* [[one-liners]]<br />
* [[Cheatsheets]]<br />
* [[Links]]<br />
* [[pandoc modified manual]]<br />
* [[Command Line Exercises]]<br />
<br />
= hdi2u =<br />
<br />
The half-day linux course held on 20th April. Modified version of I2U4BGA.<br />
<br />
* [[hdi2u_intro]]<br />
* [[hdi2u_commandbased_exercises]]<br />
* [[hdi2u_dirorg_exercise]]<br />
* [[hdi2u_rendertotsv_exercise]]<br />
<br />
= RNAseq for DGE =<br />
* [[Theoretical background]]<br />
* [[Quality Control and Preprocessing]]<br />
* [[Mapping to Reference]]<br />
* [[Mapping Quality Exercise]]<br />
* [[Key Aspects of using R]]<br />
* [[Estimating Gene Count Exercise]]<br />
* [[Differential Expression Exercise]]<br />
* [[Functional Analysis Exercise]]<br />
<br />
= Introduction to Unix 2017 =<br />
* [[Introduction_to_Unix_2017]]<br />
<br />
<br />
<br />
==Templates==<br />
* [[edgenl2g]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Users_and_Groups&diff=3506Users and Groups2020-07-22T09:32:22Z<p>PeterThorpe: /* Usage: How to add a new user */</p>
<hr />
<div>= Introduction=<br />
<br />
Some, though not all, of the tips here are for setting up users and groups.<br />
<br />
The tool of choice is smbldap.<br />
<br />
= Usage: How to add a new user =<br />
==Users==<br />
<br />
it may be required now to discable the firewall when creating new accouts. Make sure you turn it back on:<br />
https://www.cyberciti.biz/faq/turn-on-turn-off-firewall-in-linux/<br />
<br />
/etc/init.d/iptables save<br />
/etc/init.d/iptables stop<br />
<br />
* To create a new user(s)<br />
Root has a script in bin/creasu.sh, so as root:<br />
sh bin/creasu.sh <user> <user1> <user2><br />
<br />
if this line fails go to the admin page which talk about ldap.<br />
Manually doing the command from the script worked for me when this failed<br />
<br />
# (only if needed - perl errors) service restart slapd<br />
<br />
NU=test06<br />
smbldap-groupadd -a $NU<br />
smbldap-useradd -g $NU -a $NU<br />
smbldap-passwd $NU<br />
bash_files=/etc/skel<br />
basepath=/storage/home/users<br />
path=$basepath/$NU<br />
echo $path<br />
cd $basepath<br />
cp -r $bash_files/.{m,n,b,g}* $NU<br />
chown -R $NU:$NU $path<br />
smbldap-groupadd -a $NU<br />
chown -R $NU:$NU $path<br />
chmod 0701 $NU<br />
chcon 'unconfined_u:object_r:user_home_dir_t:s0' $path<br />
<br />
will create groups, accounts, home folder and all relevant files into the new home folder. <br />
Then you need to setup passwords with (password promt will appear):<br />
smbldap-passwd <user><br />
for each of the users.<br />
<br />
Then setup an ssh key for logging into the nodes by doing the following: <br />
<br />
as root user, login a user via<br />
su - <newuserid><br />
<br />
and execute<br />
ssh-keygen<br />
<br />
and just accept all the suggestions, keep accepting then as they are ...<br />
.ssh/id_rsa and .ssh/id_rsa.pub, then get created.<br />
<br />
then<br />
cp .ssh/id_rsa.pub .ssh/authorized_keys<br />
<br />
and<br />
chmod 600 .ssh/authorized_keys<br />
<br />
then ssh node1 should log in to node1 without password (no need to test other nodes).<br />
<br />
Then tell the user to change their password by doing:<br />
passwd<br />
<br />
==Groups==<br />
* To create a new group (we dont have groups YET!)<br />
smbldap-groupadd -a <newgrpname><br />
<br />
* To add users to a certain group (note that this seems to take some time to propagate, as well as only working on fresh logins)<br />
smbldap-groupmod -m <list,of,users> <targetgroup><br />
<br />
as root turn firewall back on<br />
<br />
/etc/init.d/iptables start<br />
<br />
= change a password =<br />
<br />
when a user forgets their password and asks for a new one:<br />
as root<br />
smbldap-passwd <user></div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Disk_management_after_shelf_disk_failure&diff=3505Disk management after shelf disk failure2020-06-19T11:15:11Z<p>PeterThorpe: Created page with " The software for your RAID controller was LSI (now Broadcom) MegaCli, I've not been able to find any evidence that there's a Redhat package but I have found a download. The..."</p>
<hr />
<div><br />
<br />
The software for your RAID controller was LSI (now Broadcom) MegaCli, I've not been able to find any evidence that there's a Redhat package but I have found a download. The URL below should get you a zip file that contains a rpm and other installers for other operating systems.<br />
<br />
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.broadcom.com%2Fdocs%2F12351587&amp;data=02%7C01%7Cjrc9%40st-andrews.ac.uk%7Cbe3c1405b577499c6ef808d813989466%7Cf85626cb0da849d3aa5864ef678ef01a%7C0%7C0%7C637280893338362007&amp;sdata=p93NQ8JyhxGgqCHMiAZEHq8xSdODq1urak0LKSNo3%2F0%3D&amp;reserved=0<br />
<br />
If you can install the RPM that you should be able to display the current status of the array to confirm it's safe to remove the failed or failing disk and if necessary configure the new disks. The following commands should show the status of any RAID volumes and physical disks, if you can pipe the output into a couple of files and send it over I can advise on what needs to be done.<br />
<br />
/opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aall<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aall<br />
<br />
The first command should produce something like this, we'll be able to tell if the array is degraded or not.<br />
<br />
> Adapter 0 -- Virtual Drive Information:<br />
> Virtual Drive: 0 (Target Id: 0)<br />
> Name :<br />
> RAID Level : Primary-1, Secondary-3, RAID Level Qualifier-0<br />
> Size : 203.25 GB<br />
> Sector Size : 512<br />
> Mirror Data : 203.25 GB<br />
> State : Optimal<br />
> Strip Size : 64 KB<br />
> Number Of Drives per span:2<br />
> Span Depth : 3<br />
> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache <br />
> if Bad BBU Current Cache Policy: WriteBack, ReadAdaptive, Direct, No <br />
> Write Cache if Bad BBU Default Access Policy: Read/Write Current <br />
> Access Policy: Read/Write<br />
> Disk Cache Policy : Disk's Default<br />
> Encryption Type : None<br />
> Is VD Cached: No<br />
<br />
The second command will produce a lot of output for each disk. The 'Firmware State' line should show how each disk is configured, most should be 'Online' or 'Hotspare' . Once the new disks are added we'll need to re-run this, they'll probably be listed as 'Unconfigured (Good)'. With some information about the disk position the command to configure the disks will be something like the following, I'll confirm when we know what the variable are.<br />
<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -PhysDrv\[$ENCLOSURE:$SLOT\] -a$ARRAY<br />
<br />
Then if the array is degraded it should automatically start to rebuild using one or more of the new disks.<br />
<br />
If it's still rebuilding this should show the progress:<br />
<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv\[8:13\] -a0<br />
<br />
I added one additional disk to each tray, they should be in Enclosure 8 Slot 14 and Enclosure 9 Slot 13. Confirm that with the output from PDList, if that's correct this should mark them as hotspares.<br />
<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Set -PhysDrv\[8:14\] -a0<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP - Set -PhysDrv\[9:13\] -a0<br />
<br />
Confirm by running PDList again, their Firmware State should have updated from Unconfigured Good to Configured Hotspare or something similar.<br />
<br />
<br />
to set as hot spares<br />
<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Set -PhysDrv\[8:14\] -a0<br />
/opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Set -PhysDrv\[9:13\] -a0<br />
<br />
<br />
From the verbose MegaCli -h … all the options in some kind of order<br />
> MegaCli -PDHSP {-Set [-Dedicated [-ArrayN|-Array0,1,2...]] [-EnclAffinity] [-nonRevertible]}<br />
> |-Rmv -PhysDrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Main_Page&diff=3504Main Page2020-06-19T11:12:28Z<p>PeterThorpe: /* Cluster Administration */</p>
<hr />
<div><br />
'''KENNEDY HPC for Bioinf community '''<br />
* [[Kennedy manual]]<br />
<br />
<br />
= Usage of Cluster=<br />
* [[Cluster Manual]]<br />
* [[Kennedy manual]]<br />
* [[Why a Queue Manager?]]<br />
* [[Available Software]]<br />
* [[how to use the cluster training course]]<br />
* [[windows network connect]]<br />
<br />
= Documented Programs =<br />
<br />
The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.<br />
<br />
{|style="width:85%"<br />
|* [[abacas]]<br />
|* [[albacore]]<br />
|* [[ariba]]<br />
|* [[aspera]]<br />
|* [[assembly-stats]]<br />
|* [[augustus]]<br />
|-<br />
|* [[BamQC]]<br />
|* [[bamtools]]<br />
|* [[banjo]]<br />
|* [[bcftools]]<br />
|* [[bedtools]]<br />
|* [[bgenie]]<br />
|-<br />
|* [[BLAST]]<br />
|* [[Blat]]<br />
|* [[blast2go: b2g4pipe]]<br />
|* [[bowtie]]<br />
|* [[bowtie2]]<br />
|* [[bwa]]<br />
|-<br />
|* [[BUSCO]]<br />
|* [[CAFE]]<br />
|* [[canu]]<br />
|* [[cd-hit]]<br />
|* [[cegma]]<br />
|* [[clustal]]<br />
|-<br />
|* [[cramtools]]<br />
|* [[conda]]<br />
|* [[deeptools]]<br />
|* [[detonate]]<br />
|* [[diamond]]<br />
|* [[ea-utils]]<br />
|* [[ensembl]]<br />
|-<br />
|* [[ETE]]<br />
|* [[FASTQC and MultiQC]]<br />
|* [[Archaeopteryx and Forester]]<br />
|* [[GapFiller]]<br />
|* [[GenomeTools]]<br />
|* [[gubbins]]<br />
|-<br />
|* [[JBrowse]]<br />
|* [[kallisto]]<br />
|* [[kentUtils]]<br />
|* [[last]]<br />
|* [[lastz]]<br />
|* [[macs2]]<br />
|-<br />
|* [[Mash]]<br />
|* [[mega]]<br />
|* [[meryl]]<br />
|* [[MUMmer]]<br />
|* [[NanoSim]]<br />
|* [[nseq]]<br />
|-<br />
|* [[OrthoFinder]]<br />
|* [[PASA]]<br />
|* [[perl]]<br />
|* [[PGAP]]<br />
|* [[picard-tools]]<br />
|* [[poRe]]<br />
|-<br />
|* [[poretools]]<br />
|* [[prokka]]<br />
|* [[pyrad]]<br />
|* [[python]]<br />
|* [[qualimap]]<br />
|* [[quast]]<br />
|-<br />
|* [[qiime2]]<br />
|* [[R]]<br />
|* [[RAxML]]<br />
|* [[Repeatmasker]]<br />
|* [[Repeatmodeler]]<br />
|* [[rnammer]]<br />
|-<br />
|* [[roary]]<br />
|* [[RSeQC]]<br />
|* [[samtools]]<br />
|* [[Satsuma]]<br />
|* [[sickle]]<br />
|* [[SPAdes]]<br />
|-<br />
|* [[squid]]<br />
|* [[sra-tools]]<br />
|* [[srst2]]<br />
|* [[SSPACE]]<br />
|* [[stacks]]<br />
|* [[Thor]]<br />
|-<br />
|* [[Tophat]]<br />
|* [[trimmomatic]]<br />
|* [[Trinity]]<br />
|* [[t-coffee]]<br />
|* [[Unicycler]]<br />
|* [[velvet]]<br />
|-<br />
|* [[ViennaRNA]]<br />
|}<br />
<br />
= Queue Manager Tips =<br />
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:<br />
* [[Queue Manager Tips]]<br />
* [[Queue Manager : shell script command]]<br />
* [[Queue Manager emailing when jobs run]]<br />
* [[General Command-line Tips]]<br />
* [[DRMAA for further Gridengine automation]]<br />
<br />
= Data Examples =<br />
* [[Two Eel Scaffolds]]<br />
<br />
= Procedures =<br />
(short sequence of tasks with a certain short-term goal, often, a simple script)<br />
* [[Calculating coverage]]<br />
* [[MinION Coverage sensitivity analysis]]<br />
<br />
= Navigating genomic data websites=<br />
* [[Patric]]<br />
* [[NCBI]]<br />
* [[IGSR/1000 Genomes]]<br />
<br />
= Explanations=<br />
* [[ITUcourse]]<br />
* [[VCF]]<br />
* [[Maximum Likelihood]]<br />
* [[SNP Analysis and phylogenetics]]<br />
* [[Normalization]]<br />
<br />
= Pipelines =<br />
(Workflow with specific end-goals)<br />
* [[Trinity_Protocol]]<br />
* [[STAR BEAST]]<br />
* [[callSNPs.py]]<br />
* [[pairwiseCallSNPs]]<br />
* [[mapping.py]]<br />
* [[Edgen RNAseq]]<br />
* [[Miseq Prokaryote FASTQ analysis]]<br />
* [[snpcallphylo]]<br />
* [[Bottlenose dolphin population genomic analysis]]<br />
* [[ChIP-Seq Top2 in Yeast]]<br />
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]<br />
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]<br />
* [[Bisulfite Sequencing]]<br />
* [[microRNA and Salmo Salar]]<br />
<br />
=Protocols=<br />
(Extensive workflows with different with several possible end goals)<br />
* [[Synthetic Long reads]]<br />
* [[MinION (Oxford Nanopore)]]<br />
* [[MinKNOW folders and log files]]<br />
* [[Research Data Management]]<br />
* [[MicroRNAs]]<br />
<br />
= Tech Reviews =<br />
* [[SWATH-MS Data Analysis]]<br />
<br />
= Cluster Administration =<br />
* [[StABDMIN]]<br />
* [[Hardware Issues]]<br />
* [[marvin and IPMI (remote hardware control)]]<br />
* [[restart a node]]<br />
* [[mounting drives]]<br />
* [[Admin Tips]]<br />
* [[RedHat]]<br />
* [[Globus_gridftp]]<br />
* [[Galaxy Setup]]<br />
* [[Son of Gridengine]]<br />
* [[Blas Libraries]]<br />
* [[CMake]]<br />
* [[conda bioconda]]<br />
* [[Users and Groups]]<br />
* [[Installing software on marvin]]<br />
* [[emailing]]<br />
* [[biotime machine]]<br />
* [[SCAN-pc laptop]]<br />
* [[node1 issues]]<br />
* [[6TB storage expansion]]<br />
* [[PIs storage sacrifice]]<br />
* [[SAN relocation task]]<br />
* [[Home directories max-out incident 28.11.2016]]<br />
* [[Frontend Restart]]<br />
* [[environment-modules]]<br />
* [[H: drive on cluster]]<br />
* [[Incident: Can't connect to BerkeleyDB]]<br />
* [[Bioinformatics Wordpress Site]]<br />
* [[Backups]]<br />
* [[users disk usage]]<br />
* [[Updating BLAST databases]]<br />
* [[Python DRMAA]]<br />
* [[message of the day]]<br />
* [[SAN disconnect incident 10.01.2017]]<br />
* [[Memory repair glitch 16.02.2017]]<br />
* [[node9 network failure incident 16-20.03.2017]]<br />
* [[Incorrect rebooting of marvin 19.09.2017]]<br />
* [[ansible]]<br />
* [[webstie and word press]]<br />
* [[allow user access to other peoples data]]<br />
* [[RAM and RAM slots]]<br />
* [[ldap is not ldap]]<br />
* [[reset a password]]<br />
* [[sending emails from command line examples]]<br />
* [[disk management after shelf disk failure]]<br />
<br />
= Courses =<br />
<br />
==I2U4BGA==<br />
* [[Original schedule]]<br />
* [[New schedule]]<br />
* [[Actual schedule]]<br />
* [[Course itself]]<br />
* [[Biolinux Source course]]<br />
* [[Directory Organization Exercise]]<br />
* [[Glossary]]<br />
* [[Key Bindings]]<br />
* [[one-liners]]<br />
* [[Cheatsheets]]<br />
* [[Links]]<br />
* [[pandoc modified manual]]<br />
* [[Command Line Exercises]]<br />
<br />
= hdi2u =<br />
<br />
The half-day linux course held on 20th April. Modified version of I2U4BGA.<br />
<br />
* [[hdi2u_intro]]<br />
* [[hdi2u_commandbased_exercises]]<br />
* [[hdi2u_dirorg_exercise]]<br />
* [[hdi2u_rendertotsv_exercise]]<br />
<br />
= RNAseq for DGE =<br />
* [[Theoretical background]]<br />
* [[Quality Control and Preprocessing]]<br />
* [[Mapping to Reference]]<br />
* [[Mapping Quality Exercise]]<br />
* [[Key Aspects of using R]]<br />
* [[Estimating Gene Count Exercise]]<br />
* [[Differential Expression Exercise]]<br />
* [[Functional Analysis Exercise]]<br />
<br />
= Introduction to Unix 2017 =<br />
* [[Introduction_to_Unix_2017]]<br />
<br />
<br />
<br />
==Templates==<br />
* [[edgenl2g]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Users_and_Groups&diff=3503Users and Groups2020-06-18T12:03:29Z<p>PeterThorpe: </p>
<hr />
<div>= Introduction=<br />
<br />
Some, though not all, of the tips here are for setting up users and groups.<br />
<br />
The tool of choice is smbldap.<br />
<br />
= Usage: How to add a new user =<br />
==Users==<br />
* To create a new user(s)<br />
Root has a script in bin/creasu.sh, so as root:<br />
sh bin/creasu.sh <user> <user1> <user2><br />
<br />
if this line fails go to the admin page which talk about ldap.<br />
Manually doing the command from the script worked for me when this failed<br />
<br />
# (only if needed - perl errors) service restart slapd<br />
<br />
NU=test06<br />
smbldap-groupadd -a $NU<br />
smbldap-useradd -g $NU -a $NU<br />
smbldap-passwd $NU<br />
bash_files=/etc/skel<br />
basepath=/storage/home/users<br />
path=$basepath/$NU<br />
echo $path<br />
cd $basepath<br />
cp -r $bash_files/.{m,n,b,g}* $NU<br />
chown -R $NU:$NU $path<br />
smbldap-groupadd -a $NU<br />
chown -R $NU:$NU $path<br />
chmod 0701 $NU<br />
chcon 'unconfined_u:object_r:user_home_dir_t:s0' $path<br />
<br />
will create groups, accounts, home folder and all relevant files into the new home folder. <br />
Then you need to setup passwords with (password promt will appear):<br />
smbldap-passwd <user><br />
for each of the users.<br />
<br />
Then setup an ssh key for logging into the nodes by doing the following: <br />
<br />
as root user, login a user via<br />
su - <newuserid><br />
<br />
and execute<br />
ssh-keygen<br />
<br />
and just accept all the suggestions, keep accepting then as they are ...<br />
.ssh/id_rsa and .ssh/id_rsa.pub, then get created.<br />
<br />
then<br />
cp .ssh/id_rsa.pub .ssh/authorized_keys<br />
<br />
and<br />
chmod 600 .ssh/authorized_keys<br />
<br />
then ssh node1 should log in to node1 without password (no need to test other nodes).<br />
<br />
Then tell the user to change their password by doing:<br />
passwd<br />
<br />
==Groups==<br />
* To create a new group (we dont have groups YET!)<br />
smbldap-groupadd -a <newgrpname><br />
<br />
* To add users to a certain group (note that this seems to take some time to propagate, as well as only working on fresh logins)<br />
smbldap-groupmod -m <list,of,users> <targetgroup><br />
<br />
<br />
= change a password =<br />
<br />
when a user forgets their password and asks for a new one:<br />
as root<br />
smbldap-passwd <user></div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Main_Page&diff=3502Main Page2020-06-15T08:40:27Z<p>PeterThorpe: /* Cluster Administration */</p>
<hr />
<div><br />
'''KENNEDY HPC for Bioinf community '''<br />
* [[Kennedy manual]]<br />
<br />
<br />
= Usage of Cluster=<br />
* [[Cluster Manual]]<br />
* [[Kennedy manual]]<br />
* [[Why a Queue Manager?]]<br />
* [[Available Software]]<br />
* [[how to use the cluster training course]]<br />
* [[windows network connect]]<br />
<br />
= Documented Programs =<br />
<br />
The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.<br />
<br />
{|style="width:85%"<br />
|* [[abacas]]<br />
|* [[albacore]]<br />
|* [[ariba]]<br />
|* [[aspera]]<br />
|* [[assembly-stats]]<br />
|* [[augustus]]<br />
|-<br />
|* [[BamQC]]<br />
|* [[bamtools]]<br />
|* [[banjo]]<br />
|* [[bcftools]]<br />
|* [[bedtools]]<br />
|* [[bgenie]]<br />
|-<br />
|* [[BLAST]]<br />
|* [[Blat]]<br />
|* [[blast2go: b2g4pipe]]<br />
|* [[bowtie]]<br />
|* [[bowtie2]]<br />
|* [[bwa]]<br />
|-<br />
|* [[BUSCO]]<br />
|* [[CAFE]]<br />
|* [[canu]]<br />
|* [[cd-hit]]<br />
|* [[cegma]]<br />
|* [[clustal]]<br />
|-<br />
|* [[cramtools]]<br />
|* [[conda]]<br />
|* [[deeptools]]<br />
|* [[detonate]]<br />
|* [[diamond]]<br />
|* [[ea-utils]]<br />
|* [[ensembl]]<br />
|-<br />
|* [[ETE]]<br />
|* [[FASTQC and MultiQC]]<br />
|* [[Archaeopteryx and Forester]]<br />
|* [[GapFiller]]<br />
|* [[GenomeTools]]<br />
|* [[gubbins]]<br />
|-<br />
|* [[JBrowse]]<br />
|* [[kallisto]]<br />
|* [[kentUtils]]<br />
|* [[last]]<br />
|* [[lastz]]<br />
|* [[macs2]]<br />
|-<br />
|* [[Mash]]<br />
|* [[mega]]<br />
|* [[meryl]]<br />
|* [[MUMmer]]<br />
|* [[NanoSim]]<br />
|* [[nseq]]<br />
|-<br />
|* [[OrthoFinder]]<br />
|* [[PASA]]<br />
|* [[perl]]<br />
|* [[PGAP]]<br />
|* [[picard-tools]]<br />
|* [[poRe]]<br />
|-<br />
|* [[poretools]]<br />
|* [[prokka]]<br />
|* [[pyrad]]<br />
|* [[python]]<br />
|* [[qualimap]]<br />
|* [[quast]]<br />
|-<br />
|* [[qiime2]]<br />
|* [[R]]<br />
|* [[RAxML]]<br />
|* [[Repeatmasker]]<br />
|* [[Repeatmodeler]]<br />
|* [[rnammer]]<br />
|-<br />
|* [[roary]]<br />
|* [[RSeQC]]<br />
|* [[samtools]]<br />
|* [[Satsuma]]<br />
|* [[sickle]]<br />
|* [[SPAdes]]<br />
|-<br />
|* [[squid]]<br />
|* [[sra-tools]]<br />
|* [[srst2]]<br />
|* [[SSPACE]]<br />
|* [[stacks]]<br />
|* [[Thor]]<br />
|-<br />
|* [[Tophat]]<br />
|* [[trimmomatic]]<br />
|* [[Trinity]]<br />
|* [[t-coffee]]<br />
|* [[Unicycler]]<br />
|* [[velvet]]<br />
|-<br />
|* [[ViennaRNA]]<br />
|}<br />
<br />
= Queue Manager Tips =<br />
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:<br />
* [[Queue Manager Tips]]<br />
* [[Queue Manager : shell script command]]<br />
* [[Queue Manager emailing when jobs run]]<br />
* [[General Command-line Tips]]<br />
* [[DRMAA for further Gridengine automation]]<br />
<br />
= Data Examples =<br />
* [[Two Eel Scaffolds]]<br />
<br />
= Procedures =<br />
(short sequence of tasks with a certain short-term goal, often, a simple script)<br />
* [[Calculating coverage]]<br />
* [[MinION Coverage sensitivity analysis]]<br />
<br />
= Navigating genomic data websites=<br />
* [[Patric]]<br />
* [[NCBI]]<br />
* [[IGSR/1000 Genomes]]<br />
<br />
= Explanations=<br />
* [[ITUcourse]]<br />
* [[VCF]]<br />
* [[Maximum Likelihood]]<br />
* [[SNP Analysis and phylogenetics]]<br />
* [[Normalization]]<br />
<br />
= Pipelines =<br />
(Workflow with specific end-goals)<br />
* [[Trinity_Protocol]]<br />
* [[STAR BEAST]]<br />
* [[callSNPs.py]]<br />
* [[pairwiseCallSNPs]]<br />
* [[mapping.py]]<br />
* [[Edgen RNAseq]]<br />
* [[Miseq Prokaryote FASTQ analysis]]<br />
* [[snpcallphylo]]<br />
* [[Bottlenose dolphin population genomic analysis]]<br />
* [[ChIP-Seq Top2 in Yeast]]<br />
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]<br />
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]<br />
* [[Bisulfite Sequencing]]<br />
* [[microRNA and Salmo Salar]]<br />
<br />
=Protocols=<br />
(Extensive workflows with different with several possible end goals)<br />
* [[Synthetic Long reads]]<br />
* [[MinION (Oxford Nanopore)]]<br />
* [[MinKNOW folders and log files]]<br />
* [[Research Data Management]]<br />
* [[MicroRNAs]]<br />
<br />
= Tech Reviews =<br />
* [[SWATH-MS Data Analysis]]<br />
<br />
= Cluster Administration =<br />
* [[StABDMIN]]<br />
* [[Hardware Issues]]<br />
* [[marvin and IPMI (remote hardware control)]]<br />
* [[restart a node]]<br />
* [[mounting drives]]<br />
* [[Admin Tips]]<br />
* [[RedHat]]<br />
* [[Globus_gridftp]]<br />
* [[Galaxy Setup]]<br />
* [[Son of Gridengine]]<br />
* [[Blas Libraries]]<br />
* [[CMake]]<br />
* [[conda bioconda]]<br />
* [[Users and Groups]]<br />
* [[Installing software on marvin]]<br />
* [[emailing]]<br />
* [[biotime machine]]<br />
* [[SCAN-pc laptop]]<br />
* [[node1 issues]]<br />
* [[6TB storage expansion]]<br />
* [[PIs storage sacrifice]]<br />
* [[SAN relocation task]]<br />
* [[Home directories max-out incident 28.11.2016]]<br />
* [[Frontend Restart]]<br />
* [[environment-modules]]<br />
* [[H: drive on cluster]]<br />
* [[Incident: Can't connect to BerkeleyDB]]<br />
* [[Bioinformatics Wordpress Site]]<br />
* [[Backups]]<br />
* [[users disk usage]]<br />
* [[Updating BLAST databases]]<br />
* [[Python DRMAA]]<br />
* [[message of the day]]<br />
* [[SAN disconnect incident 10.01.2017]]<br />
* [[Memory repair glitch 16.02.2017]]<br />
* [[node9 network failure incident 16-20.03.2017]]<br />
* [[Incorrect rebooting of marvin 19.09.2017]]<br />
* [[ansible]]<br />
* [[webstie and word press]]<br />
* [[allow user access to other peoples data]]<br />
* [[RAM and RAM slots]]<br />
* [[ldap is not ldap]]<br />
* [[reset a password]]<br />
* [[sending emails from command line examples]]<br />
<br />
= Courses =<br />
<br />
==I2U4BGA==<br />
* [[Original schedule]]<br />
* [[New schedule]]<br />
* [[Actual schedule]]<br />
* [[Course itself]]<br />
* [[Biolinux Source course]]<br />
* [[Directory Organization Exercise]]<br />
* [[Glossary]]<br />
* [[Key Bindings]]<br />
* [[one-liners]]<br />
* [[Cheatsheets]]<br />
* [[Links]]<br />
* [[pandoc modified manual]]<br />
* [[Command Line Exercises]]<br />
<br />
= hdi2u =<br />
<br />
The half-day linux course held on 20th April. Modified version of I2U4BGA.<br />
<br />
* [[hdi2u_intro]]<br />
* [[hdi2u_commandbased_exercises]]<br />
* [[hdi2u_dirorg_exercise]]<br />
* [[hdi2u_rendertotsv_exercise]]<br />
<br />
= RNAseq for DGE =<br />
* [[Theoretical background]]<br />
* [[Quality Control and Preprocessing]]<br />
* [[Mapping to Reference]]<br />
* [[Mapping Quality Exercise]]<br />
* [[Key Aspects of using R]]<br />
* [[Estimating Gene Count Exercise]]<br />
* [[Differential Expression Exercise]]<br />
* [[Functional Analysis Exercise]]<br />
<br />
= Introduction to Unix 2017 =<br />
* [[Introduction_to_Unix_2017]]<br />
<br />
<br />
<br />
==Templates==<br />
* [[edgenl2g]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Main_Page&diff=3501Main Page2020-06-15T08:40:01Z<p>PeterThorpe: /* Cluster Administration */</p>
<hr />
<div><br />
'''KENNEDY HPC for Bioinf community '''<br />
* [[Kennedy manual]]<br />
<br />
<br />
= Usage of Cluster=<br />
* [[Cluster Manual]]<br />
* [[Kennedy manual]]<br />
* [[Why a Queue Manager?]]<br />
* [[Available Software]]<br />
* [[how to use the cluster training course]]<br />
* [[windows network connect]]<br />
<br />
= Documented Programs =<br />
<br />
The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.<br />
<br />
{|style="width:85%"<br />
|* [[abacas]]<br />
|* [[albacore]]<br />
|* [[ariba]]<br />
|* [[aspera]]<br />
|* [[assembly-stats]]<br />
|* [[augustus]]<br />
|-<br />
|* [[BamQC]]<br />
|* [[bamtools]]<br />
|* [[banjo]]<br />
|* [[bcftools]]<br />
|* [[bedtools]]<br />
|* [[bgenie]]<br />
|-<br />
|* [[BLAST]]<br />
|* [[Blat]]<br />
|* [[blast2go: b2g4pipe]]<br />
|* [[bowtie]]<br />
|* [[bowtie2]]<br />
|* [[bwa]]<br />
|-<br />
|* [[BUSCO]]<br />
|* [[CAFE]]<br />
|* [[canu]]<br />
|* [[cd-hit]]<br />
|* [[cegma]]<br />
|* [[clustal]]<br />
|-<br />
|* [[cramtools]]<br />
|* [[conda]]<br />
|* [[deeptools]]<br />
|* [[detonate]]<br />
|* [[diamond]]<br />
|* [[ea-utils]]<br />
|* [[ensembl]]<br />
|-<br />
|* [[ETE]]<br />
|* [[FASTQC and MultiQC]]<br />
|* [[Archaeopteryx and Forester]]<br />
|* [[GapFiller]]<br />
|* [[GenomeTools]]<br />
|* [[gubbins]]<br />
|-<br />
|* [[JBrowse]]<br />
|* [[kallisto]]<br />
|* [[kentUtils]]<br />
|* [[last]]<br />
|* [[lastz]]<br />
|* [[macs2]]<br />
|-<br />
|* [[Mash]]<br />
|* [[mega]]<br />
|* [[meryl]]<br />
|* [[MUMmer]]<br />
|* [[NanoSim]]<br />
|* [[nseq]]<br />
|-<br />
|* [[OrthoFinder]]<br />
|* [[PASA]]<br />
|* [[perl]]<br />
|* [[PGAP]]<br />
|* [[picard-tools]]<br />
|* [[poRe]]<br />
|-<br />
|* [[poretools]]<br />
|* [[prokka]]<br />
|* [[pyrad]]<br />
|* [[python]]<br />
|* [[qualimap]]<br />
|* [[quast]]<br />
|-<br />
|* [[qiime2]]<br />
|* [[R]]<br />
|* [[RAxML]]<br />
|* [[Repeatmasker]]<br />
|* [[Repeatmodeler]]<br />
|* [[rnammer]]<br />
|-<br />
|* [[roary]]<br />
|* [[RSeQC]]<br />
|* [[samtools]]<br />
|* [[Satsuma]]<br />
|* [[sickle]]<br />
|* [[SPAdes]]<br />
|-<br />
|* [[squid]]<br />
|* [[sra-tools]]<br />
|* [[srst2]]<br />
|* [[SSPACE]]<br />
|* [[stacks]]<br />
|* [[Thor]]<br />
|-<br />
|* [[Tophat]]<br />
|* [[trimmomatic]]<br />
|* [[Trinity]]<br />
|* [[t-coffee]]<br />
|* [[Unicycler]]<br />
|* [[velvet]]<br />
|-<br />
|* [[ViennaRNA]]<br />
|}<br />
<br />
= Queue Manager Tips =<br />
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:<br />
* [[Queue Manager Tips]]<br />
* [[Queue Manager : shell script command]]<br />
* [[Queue Manager emailing when jobs run]]<br />
* [[General Command-line Tips]]<br />
* [[DRMAA for further Gridengine automation]]<br />
<br />
= Data Examples =<br />
* [[Two Eel Scaffolds]]<br />
<br />
= Procedures =<br />
(short sequence of tasks with a certain short-term goal, often, a simple script)<br />
* [[Calculating coverage]]<br />
* [[MinION Coverage sensitivity analysis]]<br />
<br />
= Navigating genomic data websites=<br />
* [[Patric]]<br />
* [[NCBI]]<br />
* [[IGSR/1000 Genomes]]<br />
<br />
= Explanations=<br />
* [[ITUcourse]]<br />
* [[VCF]]<br />
* [[Maximum Likelihood]]<br />
* [[SNP Analysis and phylogenetics]]<br />
* [[Normalization]]<br />
<br />
= Pipelines =<br />
(Workflow with specific end-goals)<br />
* [[Trinity_Protocol]]<br />
* [[STAR BEAST]]<br />
* [[callSNPs.py]]<br />
* [[pairwiseCallSNPs]]<br />
* [[mapping.py]]<br />
* [[Edgen RNAseq]]<br />
* [[Miseq Prokaryote FASTQ analysis]]<br />
* [[snpcallphylo]]<br />
* [[Bottlenose dolphin population genomic analysis]]<br />
* [[ChIP-Seq Top2 in Yeast]]<br />
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]<br />
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]<br />
* [[Bisulfite Sequencing]]<br />
* [[microRNA and Salmo Salar]]<br />
<br />
=Protocols=<br />
(Extensive workflows with different with several possible end goals)<br />
* [[Synthetic Long reads]]<br />
* [[MinION (Oxford Nanopore)]]<br />
* [[MinKNOW folders and log files]]<br />
* [[Research Data Management]]<br />
* [[MicroRNAs]]<br />
<br />
= Tech Reviews =<br />
* [[SWATH-MS Data Analysis]]<br />
<br />
= Cluster Administration =<br />
* [[StABDMIN]]<br />
* [[Hardware Issues]]<br />
* [[marvin and IPMI (remote hardware control)]]<br />
* [[restart a node]]<br />
* [[mounting drives]]<br />
* [[Admin Tips]]<br />
* [[RedHat]]<br />
* [[Globus_gridftp]]<br />
* [[Galaxy Setup]]<br />
* [[Son of Gridengine]]<br />
* [[Blas Libraries]]<br />
* [[CMake]]<br />
* [[conda bioconda]]<br />
* [[Users and Groups add a new user]]<br />
* [[Installing software on marvin]]<br />
* [[emailing]]<br />
* [[biotime machine]]<br />
* [[SCAN-pc laptop]]<br />
* [[node1 issues]]<br />
* [[6TB storage expansion]]<br />
* [[PIs storage sacrifice]]<br />
* [[SAN relocation task]]<br />
* [[Home directories max-out incident 28.11.2016]]<br />
* [[Frontend Restart]]<br />
* [[environment-modules]]<br />
* [[H: drive on cluster]]<br />
* [[Incident: Can't connect to BerkeleyDB]]<br />
* [[Bioinformatics Wordpress Site]]<br />
* [[Backups]]<br />
* [[users disk usage]]<br />
* [[Updating BLAST databases]]<br />
* [[Python DRMAA]]<br />
* [[message of the day]]<br />
* [[SAN disconnect incident 10.01.2017]]<br />
* [[Memory repair glitch 16.02.2017]]<br />
* [[node9 network failure incident 16-20.03.2017]]<br />
* [[Incorrect rebooting of marvin 19.09.2017]]<br />
* [[ansible]]<br />
* [[webstie and word press]]<br />
* [[allow user access to other peoples data]]<br />
* [[RAM and RAM slots]]<br />
* [[ldap is not ldap]]<br />
* [[reset a password]]<br />
* [[sending emails from command line examples]]<br />
<br />
= Courses =<br />
<br />
==I2U4BGA==<br />
* [[Original schedule]]<br />
* [[New schedule]]<br />
* [[Actual schedule]]<br />
* [[Course itself]]<br />
* [[Biolinux Source course]]<br />
* [[Directory Organization Exercise]]<br />
* [[Glossary]]<br />
* [[Key Bindings]]<br />
* [[one-liners]]<br />
* [[Cheatsheets]]<br />
* [[Links]]<br />
* [[pandoc modified manual]]<br />
* [[Command Line Exercises]]<br />
<br />
= hdi2u =<br />
<br />
The half-day linux course held on 20th April. Modified version of I2U4BGA.<br />
<br />
* [[hdi2u_intro]]<br />
* [[hdi2u_commandbased_exercises]]<br />
* [[hdi2u_dirorg_exercise]]<br />
* [[hdi2u_rendertotsv_exercise]]<br />
<br />
= RNAseq for DGE =<br />
* [[Theoretical background]]<br />
* [[Quality Control and Preprocessing]]<br />
* [[Mapping to Reference]]<br />
* [[Mapping Quality Exercise]]<br />
* [[Key Aspects of using R]]<br />
* [[Estimating Gene Count Exercise]]<br />
* [[Differential Expression Exercise]]<br />
* [[Functional Analysis Exercise]]<br />
<br />
= Introduction to Unix 2017 =<br />
* [[Introduction_to_Unix_2017]]<br />
<br />
<br />
<br />
==Templates==<br />
* [[edgenl2g]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Frontend_Restart&diff=3500Frontend Restart2020-05-20T19:28:16Z<p>PeterThorpe: /* sungrid engine not running on the centos 7 nodes */</p>
<hr />
<div>= Introduction =<br />
<br />
This is a highly critical operation, and though most of the possible pitfalls are documented here, there is plenty scope for new ones which could render the entire cluster inaccessible. While this does not mean lasting damage, but rather delays in the order of days, may be even over a week.<br />
<br />
Because of various precautions and checks the process usually requires 2-3 hours.<br />
<br />
Given these aspects, the question may be asked: why do it? The answer is to update the software, specifically, the kernel software.<br />
<br />
Short process:<br />
* Give plenty of warning<br />
* Make sure the STORAGE line in fstab is commented out<br />
* shutdown all the nodes<br />
* Double check the STORAGE line in fstab is commented out<br />
* reboot Marvin<br />
* re-mount the STORAGE partition<br />
* comment out the STORAGE line in fstab<br />
* bring the nodes back up with ipmiconfig<br />
<br />
= Measures =<br />
<br />
== Bring all nodes down before restart ==<br />
first comment out in ''/etc/fstab'' , The lines after - "The Three vital LVs" must be commented out. <br />
<br />
To bring a node down, simply (as super user, then log into the node via ssh)<br />
shutdown now<br />
<br />
<br />
This is possibly the most useful measure. Primarily, it is due to the nodes using marvin to keep various filesystems mounted, and the havoc they experience when marvin stops doing this. NFS4 stale filehandles then appear and are hard to get rid of. This measure is not immediately obvious, because all the nodes are updated on a rolling basis and often do not need to be switched off.<br />
<br />
And then, when marvin is back up, and once its filesystems are verified, the nodes maybe brought back up. It is best not to bring them up at the same time, so that they all don't immediately grab the central filesystem at the same time. They may be brought up say 5 minutes from each other. Simply another precaution<br />
<br />
Of course this seems like quite alot of extra work, but it's worth it in terms of saving later debugging time. It's clearly related to necessary manual mounting of the STORAGE volume documented below.<br />
<br />
Nodes can be restarted with<br />
pwonck <node number><br />
or <br />
pwcycck <node number> <br />
as root.<br />
<br />
== Try to get console access to the frontend ==<br />
<br />
This can be solved with IPMI. IPMI IP is 138.251.13.220, username is ADMIN and the password is in the red folder. <br />
<br />
There are various options:<br />
<br />
* via the '''ipmiconfig''' tool, this is command line only.<br />
* via the IPMIView tool, GUI.<br />
* via the IPMI device's webserver<br />
* via the SOL (part of ipmiconfig)<br />
<br />
SOL is closest to being at the terminal, with the added advantage of being able to use linux screen's history capability to record a session. Unfortunately, it seldom works. The webserver and the IPMIView tool have an alternative console program using java, termed "KVM". This uses the Iced Tea jnlp environment, but it can recetnly it has been demanding keys (only for marvin) and will probably not work.<br />
<br />
Also many of these tools are rather old, which is often not a problem if the tool's function is simple. For example "power on" and "power off" are simple functions. There terinal program is not a simple function, and sometimes it may required an old version of java to run (i.e. version 6).<br />
<br />
== startup services==<br />
<br />
All important startup services are launched automatically, however it is a good idea to verify this using the "chkconfig" command. i.e. to check the runlevels (runlevel 5 being the most important) on the network time daemon (ntpd)<br />
<br />
chkconfig --list ntpd<br />
<br />
To enable it for automatic startup: <br />
<br />
chkconfig ntpd on --level 35<br />
<br />
which also ensures it launches even when on the minimal level 3.<br />
<br />
= The main critical issue =<br />
<br />
All in all, restarting marvin simply means typing "reboot", and it will power down and then power up. This will happen smoothly on the nodes for example, and is a very fortunate series of events, because it means that no BIOS interaction (key presses) are required (however the baseboard event logger sometimes fills up and may disrupt this, by asking for a key press, and so bring the boot-up process to a total stop).<br />
<br />
Something else however causes an interruption in the boot-up of the frontend and it is the automatic mounting of the STORAGE filesystem which holds all the users home directories. This is a networked storage system, but the marvin system configures it under LVM (Logical Volume Management system) and when one is ready, the other one isn't which stalls the automatic procedure. Manual intervention is therefore required. One can detect this happening by running '''vgdisplay''' and noticing that the STORAGE volume is unavailable.<br />
<br />
Because this discrepancy halts the boot-up process it must be done manually and the mount directive available via '''/etc/fstab''' should always be commented out. In any case, the manual command is very simple, so one just performs "reboot" on marvin, and once it is back on line (could take as long as 10 minutes), the following command should be invoked:<br />
<br />
vgchange -a y STORAGE<br />
<br />
one should check via "vgdisplay" that STORAGE is now available, and one can decomment the appropriate line in '''/etc/fstab''' and run<br />
<br />
mount /storage<br />
<br />
Of course this should then be followed by the commenting out (once again) of the storage line in '''/etc/fstab'''.<br />
<br />
<br />
<br />
NOTE: if you get many unfixable satle file handle problems ... which I have .. feel my pain (Im tired now - LOOOONg day):<br />
<br />
# on head node<br />
showmount -e marvin<br />
exportfs -f<br />
vgchange -a y STORAGE<br />
mount /storage<br />
<br />
# on the slaves<br />
showmount -e marvin<br />
service nfs restart<br />
service nfs restart<br />
mount -a<br />
df -h<br />
<br />
= Provisos =<br />
<br />
Restarting marvin is a major operation, as all running jobs are lost.<br />
<br />
It is therefore necessary to advise all users well in advance, as to when it might happen.<br />
<br />
= sungrid engine not running on the centos 7 nodes=<br />
restart the services. <br />
<br />
cd into <br />
cd/etc/init.d<br />
<br />
http://www.softpanorama.org/HPC/Grid_engine/Troubleshooting/starting_and_killing_sge_daemons.shtml<br />
<br />
cd /etc/init.d<br />
<br />
./sgeexecd.p6444 stop<br />
Shutting down Grid Engine execution daemon<br />
./sgemaster.p6444 stop<br />
shutting down Grid Engine qmaster<br />
<br />
service sgeexec.p6444 stop && sgeexec.p6444 start<br />
<br />
if it fails due to "shepherd of job"<br />
<br />
then<br />
<br />
./sgeexecd.p6444 softstop # does not kill sheperd jobs<br />
<br />
doesnt work!!!!<br />
<br />
also useful:<br />
<br />
https://www.linuxquestions.org/questions/linux-newbie-8/installing-gridengine-in-centos-7-a-4175596488-print/<br />
<br />
This does work:<br />
just run the script:<br />
/etc/init.d/sgeexecd.p6444<br />
<br />
chkconfig sgeexecd.p6444 on</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Frontend_Restart&diff=3499Frontend Restart2020-05-20T19:07:37Z<p>PeterThorpe: /* sungrid engine not running on the centos 7 nodes */</p>
<hr />
<div>= Introduction =<br />
<br />
This is a highly critical operation, and though most of the possible pitfalls are documented here, there is plenty scope for new ones which could render the entire cluster inaccessible. While this does not mean lasting damage, but rather delays in the order of days, may be even over a week.<br />
<br />
Because of various precautions and checks the process usually requires 2-3 hours.<br />
<br />
Given these aspects, the question may be asked: why do it? The answer is to update the software, specifically, the kernel software.<br />
<br />
Short process:<br />
* Give plenty of warning<br />
* Make sure the STORAGE line in fstab is commented out<br />
* shutdown all the nodes<br />
* Double check the STORAGE line in fstab is commented out<br />
* reboot Marvin<br />
* re-mount the STORAGE partition<br />
* comment out the STORAGE line in fstab<br />
* bring the nodes back up with ipmiconfig<br />
<br />
= Measures =<br />
<br />
== Bring all nodes down before restart ==<br />
first comment out in ''/etc/fstab'' , The lines after - "The Three vital LVs" must be commented out. <br />
<br />
To bring a node down, simply (as super user, then log into the node via ssh)<br />
shutdown now<br />
<br />
<br />
This is possibly the most useful measure. Primarily, it is due to the nodes using marvin to keep various filesystems mounted, and the havoc they experience when marvin stops doing this. NFS4 stale filehandles then appear and are hard to get rid of. This measure is not immediately obvious, because all the nodes are updated on a rolling basis and often do not need to be switched off.<br />
<br />
And then, when marvin is back up, and once its filesystems are verified, the nodes maybe brought back up. It is best not to bring them up at the same time, so that they all don't immediately grab the central filesystem at the same time. They may be brought up say 5 minutes from each other. Simply another precaution<br />
<br />
Of course this seems like quite alot of extra work, but it's worth it in terms of saving later debugging time. It's clearly related to necessary manual mounting of the STORAGE volume documented below.<br />
<br />
Nodes can be restarted with<br />
pwonck <node number><br />
or <br />
pwcycck <node number> <br />
as root.<br />
<br />
== Try to get console access to the frontend ==<br />
<br />
This can be solved with IPMI. IPMI IP is 138.251.13.220, username is ADMIN and the password is in the red folder. <br />
<br />
There are various options:<br />
<br />
* via the '''ipmiconfig''' tool, this is command line only.<br />
* via the IPMIView tool, GUI.<br />
* via the IPMI device's webserver<br />
* via the SOL (part of ipmiconfig)<br />
<br />
SOL is closest to being at the terminal, with the added advantage of being able to use linux screen's history capability to record a session. Unfortunately, it seldom works. The webserver and the IPMIView tool have an alternative console program using java, termed "KVM". This uses the Iced Tea jnlp environment, but it can recetnly it has been demanding keys (only for marvin) and will probably not work.<br />
<br />
Also many of these tools are rather old, which is often not a problem if the tool's function is simple. For example "power on" and "power off" are simple functions. There terinal program is not a simple function, and sometimes it may required an old version of java to run (i.e. version 6).<br />
<br />
== startup services==<br />
<br />
All important startup services are launched automatically, however it is a good idea to verify this using the "chkconfig" command. i.e. to check the runlevels (runlevel 5 being the most important) on the network time daemon (ntpd)<br />
<br />
chkconfig --list ntpd<br />
<br />
To enable it for automatic startup: <br />
<br />
chkconfig ntpd on --level 35<br />
<br />
which also ensures it launches even when on the minimal level 3.<br />
<br />
= The main critical issue =<br />
<br />
All in all, restarting marvin simply means typing "reboot", and it will power down and then power up. This will happen smoothly on the nodes for example, and is a very fortunate series of events, because it means that no BIOS interaction (key presses) are required (however the baseboard event logger sometimes fills up and may disrupt this, by asking for a key press, and so bring the boot-up process to a total stop).<br />
<br />
Something else however causes an interruption in the boot-up of the frontend and it is the automatic mounting of the STORAGE filesystem which holds all the users home directories. This is a networked storage system, but the marvin system configures it under LVM (Logical Volume Management system) and when one is ready, the other one isn't which stalls the automatic procedure. Manual intervention is therefore required. One can detect this happening by running '''vgdisplay''' and noticing that the STORAGE volume is unavailable.<br />
<br />
Because this discrepancy halts the boot-up process it must be done manually and the mount directive available via '''/etc/fstab''' should always be commented out. In any case, the manual command is very simple, so one just performs "reboot" on marvin, and once it is back on line (could take as long as 10 minutes), the following command should be invoked:<br />
<br />
vgchange -a y STORAGE<br />
<br />
one should check via "vgdisplay" that STORAGE is now available, and one can decomment the appropriate line in '''/etc/fstab''' and run<br />
<br />
mount /storage<br />
<br />
Of course this should then be followed by the commenting out (once again) of the storage line in '''/etc/fstab'''.<br />
<br />
<br />
<br />
NOTE: if you get many unfixable satle file handle problems ... which I have .. feel my pain (Im tired now - LOOOONg day):<br />
<br />
# on head node<br />
showmount -e marvin<br />
exportfs -f<br />
vgchange -a y STORAGE<br />
mount /storage<br />
<br />
# on the slaves<br />
showmount -e marvin<br />
service nfs restart<br />
service nfs restart<br />
mount -a<br />
df -h<br />
<br />
= Provisos =<br />
<br />
Restarting marvin is a major operation, as all running jobs are lost.<br />
<br />
It is therefore necessary to advise all users well in advance, as to when it might happen.<br />
<br />
= sungrid engine not running on the centos 7 nodes=<br />
restart the services. <br />
<br />
cd into <br />
cd/etc/init.d<br />
<br />
http://www.softpanorama.org/HPC/Grid_engine/Troubleshooting/starting_and_killing_sge_daemons.shtml<br />
<br />
cd /etc/init.d<br />
<br />
./sgeexecd.p6444 stop<br />
Shutting down Grid Engine execution daemon<br />
./sgemaster.p6444 stop<br />
shutting down Grid Engine qmaster<br />
<br />
service sgeexec.p6444 stop && sgeexec.p6444 start<br />
<br />
if it fails due to "shepherd of job"<br />
<br />
then<br />
<br />
./sgeexecd.p6444 softstop # does not kill sheperd jobs<br />
<br />
doesnt work!!!!<br />
<br />
also useful:<br />
<br />
https://www.linuxquestions.org/questions/linux-newbie-8/installing-gridengine-in-centos-7-a-4175596488-print/<br />
<br />
This does work:<br />
just run the script:<br />
/etc/init.d/sgeexecd.p6444</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Frontend_Restart&diff=3498Frontend Restart2020-05-19T18:36:20Z<p>PeterThorpe: /* sungrid engine not running on the centos 7 nodes */</p>
<hr />
<div>= Introduction =<br />
<br />
This is a highly critical operation, and though most of the possible pitfalls are documented here, there is plenty scope for new ones which could render the entire cluster inaccessible. While this does not mean lasting damage, but rather delays in the order of days, may be even over a week.<br />
<br />
Because of various precautions and checks the process usually requires 2-3 hours.<br />
<br />
Given these aspects, the question may be asked: why do it? The answer is to update the software, specifically, the kernel software.<br />
<br />
Short process:<br />
* Give plenty of warning<br />
* Make sure the STORAGE line in fstab is commented out<br />
* shutdown all the nodes<br />
* Double check the STORAGE line in fstab is commented out<br />
* reboot Marvin<br />
* re-mount the STORAGE partition<br />
* comment out the STORAGE line in fstab<br />
* bring the nodes back up with ipmiconfig<br />
<br />
= Measures =<br />
<br />
== Bring all nodes down before restart ==<br />
first comment out in ''/etc/fstab'' , The lines after - "The Three vital LVs" must be commented out. <br />
<br />
To bring a node down, simply (as super user, then log into the node via ssh)<br />
shutdown now<br />
<br />
<br />
This is possibly the most useful measure. Primarily, it is due to the nodes using marvin to keep various filesystems mounted, and the havoc they experience when marvin stops doing this. NFS4 stale filehandles then appear and are hard to get rid of. This measure is not immediately obvious, because all the nodes are updated on a rolling basis and often do not need to be switched off.<br />
<br />
And then, when marvin is back up, and once its filesystems are verified, the nodes maybe brought back up. It is best not to bring them up at the same time, so that they all don't immediately grab the central filesystem at the same time. They may be brought up say 5 minutes from each other. Simply another precaution<br />
<br />
Of course this seems like quite alot of extra work, but it's worth it in terms of saving later debugging time. It's clearly related to necessary manual mounting of the STORAGE volume documented below.<br />
<br />
Nodes can be restarted with<br />
pwonck <node number><br />
or <br />
pwcycck <node number> <br />
as root.<br />
<br />
== Try to get console access to the frontend ==<br />
<br />
This can be solved with IPMI. IPMI IP is 138.251.13.220, username is ADMIN and the password is in the red folder. <br />
<br />
There are various options:<br />
<br />
* via the '''ipmiconfig''' tool, this is command line only.<br />
* via the IPMIView tool, GUI.<br />
* via the IPMI device's webserver<br />
* via the SOL (part of ipmiconfig)<br />
<br />
SOL is closest to being at the terminal, with the added advantage of being able to use linux screen's history capability to record a session. Unfortunately, it seldom works. The webserver and the IPMIView tool have an alternative console program using java, termed "KVM". This uses the Iced Tea jnlp environment, but it can recetnly it has been demanding keys (only for marvin) and will probably not work.<br />
<br />
Also many of these tools are rather old, which is often not a problem if the tool's function is simple. For example "power on" and "power off" are simple functions. There terinal program is not a simple function, and sometimes it may required an old version of java to run (i.e. version 6).<br />
<br />
== startup services==<br />
<br />
All important startup services are launched automatically, however it is a good idea to verify this using the "chkconfig" command. i.e. to check the runlevels (runlevel 5 being the most important) on the network time daemon (ntpd)<br />
<br />
chkconfig --list ntpd<br />
<br />
To enable it for automatic startup: <br />
<br />
chkconfig ntpd on --level 35<br />
<br />
which also ensures it launches even when on the minimal level 3.<br />
<br />
= The main critical issue =<br />
<br />
All in all, restarting marvin simply means typing "reboot", and it will power down and then power up. This will happen smoothly on the nodes for example, and is a very fortunate series of events, because it means that no BIOS interaction (key presses) are required (however the baseboard event logger sometimes fills up and may disrupt this, by asking for a key press, and so bring the boot-up process to a total stop).<br />
<br />
Something else however causes an interruption in the boot-up of the frontend and it is the automatic mounting of the STORAGE filesystem which holds all the users home directories. This is a networked storage system, but the marvin system configures it under LVM (Logical Volume Management system) and when one is ready, the other one isn't which stalls the automatic procedure. Manual intervention is therefore required. One can detect this happening by running '''vgdisplay''' and noticing that the STORAGE volume is unavailable.<br />
<br />
Because this discrepancy halts the boot-up process it must be done manually and the mount directive available via '''/etc/fstab''' should always be commented out. In any case, the manual command is very simple, so one just performs "reboot" on marvin, and once it is back on line (could take as long as 10 minutes), the following command should be invoked:<br />
<br />
vgchange -a y STORAGE<br />
<br />
one should check via "vgdisplay" that STORAGE is now available, and one can decomment the appropriate line in '''/etc/fstab''' and run<br />
<br />
mount /storage<br />
<br />
Of course this should then be followed by the commenting out (once again) of the storage line in '''/etc/fstab'''.<br />
<br />
<br />
<br />
NOTE: if you get many unfixable satle file handle problems ... which I have .. feel my pain (Im tired now - LOOOONg day):<br />
<br />
# on head node<br />
showmount -e marvin<br />
exportfs -f<br />
vgchange -a y STORAGE<br />
mount /storage<br />
<br />
# on the slaves<br />
showmount -e marvin<br />
service nfs restart<br />
service nfs restart<br />
mount -a<br />
df -h<br />
<br />
= Provisos =<br />
<br />
Restarting marvin is a major operation, as all running jobs are lost.<br />
<br />
It is therefore necessary to advise all users well in advance, as to when it might happen.<br />
<br />
= sungrid engine not running on the centos 7 nodes=<br />
restart the services. <br />
<br />
cd into <br />
cd/etc/init.d<br />
<br />
http://www.softpanorama.org/HPC/Grid_engine/Troubleshooting/starting_and_killing_sge_daemons.shtml<br />
<br />
cd /etc/init.d<br />
<br />
./sgeexecd.p6444 stop<br />
Shutting down Grid Engine execution daemon<br />
./sgemaster.p6444 stop<br />
shutting down Grid Engine qmaster<br />
<br />
service sgeexec.p6444 stop && sgeexec.p6444 start<br />
<br />
if it fails due to "shepherd of job"<br />
<br />
then<br />
<br />
./sgeexecd.p6444 softstop # does not kill sheperd jobs<br />
<br />
doesnt work!!!!<br />
<br />
also useful:<br />
<br />
https://www.linuxquestions.org/questions/linux-newbie-8/installing-gridengine-in-centos-7-a-4175596488-print/</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Frontend_Restart&diff=3497Frontend Restart2020-05-19T18:05:53Z<p>PeterThorpe: /* sungrid engine not running on the centos 7 nodes */</p>
<hr />
<div>= Introduction =<br />
<br />
This is a highly critical operation, and though most of the possible pitfalls are documented here, there is plenty scope for new ones which could render the entire cluster inaccessible. While this does not mean lasting damage, but rather delays in the order of days, may be even over a week.<br />
<br />
Because of various precautions and checks the process usually requires 2-3 hours.<br />
<br />
Given these aspects, the question may be asked: why do it? The answer is to update the software, specifically, the kernel software.<br />
<br />
Short process:<br />
* Give plenty of warning<br />
* Make sure the STORAGE line in fstab is commented out<br />
* shutdown all the nodes<br />
* Double check the STORAGE line in fstab is commented out<br />
* reboot Marvin<br />
* re-mount the STORAGE partition<br />
* comment out the STORAGE line in fstab<br />
* bring the nodes back up with ipmiconfig<br />
<br />
= Measures =<br />
<br />
== Bring all nodes down before restart ==<br />
first comment out in ''/etc/fstab'' , The lines after - "The Three vital LVs" must be commented out. <br />
<br />
To bring a node down, simply (as super user, then log into the node via ssh)<br />
shutdown now<br />
<br />
<br />
This is possibly the most useful measure. Primarily, it is due to the nodes using marvin to keep various filesystems mounted, and the havoc they experience when marvin stops doing this. NFS4 stale filehandles then appear and are hard to get rid of. This measure is not immediately obvious, because all the nodes are updated on a rolling basis and often do not need to be switched off.<br />
<br />
And then, when marvin is back up, and once its filesystems are verified, the nodes maybe brought back up. It is best not to bring them up at the same time, so that they all don't immediately grab the central filesystem at the same time. They may be brought up say 5 minutes from each other. Simply another precaution<br />
<br />
Of course this seems like quite alot of extra work, but it's worth it in terms of saving later debugging time. It's clearly related to necessary manual mounting of the STORAGE volume documented below.<br />
<br />
Nodes can be restarted with<br />
pwonck <node number><br />
or <br />
pwcycck <node number> <br />
as root.<br />
<br />
== Try to get console access to the frontend ==<br />
<br />
This can be solved with IPMI. IPMI IP is 138.251.13.220, username is ADMIN and the password is in the red folder. <br />
<br />
There are various options:<br />
<br />
* via the '''ipmiconfig''' tool, this is command line only.<br />
* via the IPMIView tool, GUI.<br />
* via the IPMI device's webserver<br />
* via the SOL (part of ipmiconfig)<br />
<br />
SOL is closest to being at the terminal, with the added advantage of being able to use linux screen's history capability to record a session. Unfortunately, it seldom works. The webserver and the IPMIView tool have an alternative console program using java, termed "KVM". This uses the Iced Tea jnlp environment, but it can recetnly it has been demanding keys (only for marvin) and will probably not work.<br />
<br />
Also many of these tools are rather old, which is often not a problem if the tool's function is simple. For example "power on" and "power off" are simple functions. There terinal program is not a simple function, and sometimes it may required an old version of java to run (i.e. version 6).<br />
<br />
== startup services==<br />
<br />
All important startup services are launched automatically, however it is a good idea to verify this using the "chkconfig" command. i.e. to check the runlevels (runlevel 5 being the most important) on the network time daemon (ntpd)<br />
<br />
chkconfig --list ntpd<br />
<br />
To enable it for automatic startup: <br />
<br />
chkconfig ntpd on --level 35<br />
<br />
which also ensures it launches even when on the minimal level 3.<br />
<br />
= The main critical issue =<br />
<br />
All in all, restarting marvin simply means typing "reboot", and it will power down and then power up. This will happen smoothly on the nodes for example, and is a very fortunate series of events, because it means that no BIOS interaction (key presses) are required (however the baseboard event logger sometimes fills up and may disrupt this, by asking for a key press, and so bring the boot-up process to a total stop).<br />
<br />
Something else however causes an interruption in the boot-up of the frontend and it is the automatic mounting of the STORAGE filesystem which holds all the users home directories. This is a networked storage system, but the marvin system configures it under LVM (Logical Volume Management system) and when one is ready, the other one isn't which stalls the automatic procedure. Manual intervention is therefore required. One can detect this happening by running '''vgdisplay''' and noticing that the STORAGE volume is unavailable.<br />
<br />
Because this discrepancy halts the boot-up process it must be done manually and the mount directive available via '''/etc/fstab''' should always be commented out. In any case, the manual command is very simple, so one just performs "reboot" on marvin, and once it is back on line (could take as long as 10 minutes), the following command should be invoked:<br />
<br />
vgchange -a y STORAGE<br />
<br />
one should check via "vgdisplay" that STORAGE is now available, and one can decomment the appropriate line in '''/etc/fstab''' and run<br />
<br />
mount /storage<br />
<br />
Of course this should then be followed by the commenting out (once again) of the storage line in '''/etc/fstab'''.<br />
<br />
<br />
<br />
NOTE: if you get many unfixable satle file handle problems ... which I have .. feel my pain (Im tired now - LOOOONg day):<br />
<br />
# on head node<br />
showmount -e marvin<br />
exportfs -f<br />
vgchange -a y STORAGE<br />
mount /storage<br />
<br />
# on the slaves<br />
showmount -e marvin<br />
service nfs restart<br />
service nfs restart<br />
mount -a<br />
df -h<br />
<br />
= Provisos =<br />
<br />
Restarting marvin is a major operation, as all running jobs are lost.<br />
<br />
It is therefore necessary to advise all users well in advance, as to when it might happen.<br />
<br />
= sungrid engine not running on the centos 7 nodes=<br />
restart the services. <br />
<br />
cd into <br />
cd/etc/init.d<br />
<br />
http://www.softpanorama.org/HPC/Grid_engine/Troubleshooting/starting_and_killing_sge_daemons.shtml<br />
<br />
cd /etc/init.d<br />
<br />
./sgeexecd.p6444 stop<br />
Shutting down Grid Engine execution daemon<br />
./sgemaster.p6444 stop<br />
shutting down Grid Engine qmaster<br />
<br />
service sgeexec.p6444 stop && sgeexec.p6444 start<br />
<br />
if it fails due to "shepherd of job"<br />
<br />
then<br />
<br />
./sgeexecd.p6444 softstop # does not kill sheperd jobs</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Frontend_Restart&diff=3496Frontend Restart2020-05-19T17:39:05Z<p>PeterThorpe: </p>
<hr />
<div>= Introduction =<br />
<br />
This is a highly critical operation, and though most of the possible pitfalls are documented here, there is plenty scope for new ones which could render the entire cluster inaccessible. While this does not mean lasting damage, but rather delays in the order of days, may be even over a week.<br />
<br />
Because of various precautions and checks the process usually requires 2-3 hours.<br />
<br />
Given these aspects, the question may be asked: why do it? The answer is to update the software, specifically, the kernel software.<br />
<br />
Short process:<br />
* Give plenty of warning<br />
* Make sure the STORAGE line in fstab is commented out<br />
* shutdown all the nodes<br />
* Double check the STORAGE line in fstab is commented out<br />
* reboot Marvin<br />
* re-mount the STORAGE partition<br />
* comment out the STORAGE line in fstab<br />
* bring the nodes back up with ipmiconfig<br />
<br />
= Measures =<br />
<br />
== Bring all nodes down before restart ==<br />
first comment out in ''/etc/fstab'' , The lines after - "The Three vital LVs" must be commented out. <br />
<br />
To bring a node down, simply (as super user, then log into the node via ssh)<br />
shutdown now<br />
<br />
<br />
This is possibly the most useful measure. Primarily, it is due to the nodes using marvin to keep various filesystems mounted, and the havoc they experience when marvin stops doing this. NFS4 stale filehandles then appear and are hard to get rid of. This measure is not immediately obvious, because all the nodes are updated on a rolling basis and often do not need to be switched off.<br />
<br />
And then, when marvin is back up, and once its filesystems are verified, the nodes maybe brought back up. It is best not to bring them up at the same time, so that they all don't immediately grab the central filesystem at the same time. They may be brought up say 5 minutes from each other. Simply another precaution<br />
<br />
Of course this seems like quite alot of extra work, but it's worth it in terms of saving later debugging time. It's clearly related to necessary manual mounting of the STORAGE volume documented below.<br />
<br />
Nodes can be restarted with<br />
pwonck <node number><br />
or <br />
pwcycck <node number> <br />
as root.<br />
<br />
== Try to get console access to the frontend ==<br />
<br />
This can be solved with IPMI. IPMI IP is 138.251.13.220, username is ADMIN and the password is in the red folder. <br />
<br />
There are various options:<br />
<br />
* via the '''ipmiconfig''' tool, this is command line only.<br />
* via the IPMIView tool, GUI.<br />
* via the IPMI device's webserver<br />
* via the SOL (part of ipmiconfig)<br />
<br />
SOL is closest to being at the terminal, with the added advantage of being able to use linux screen's history capability to record a session. Unfortunately, it seldom works. The webserver and the IPMIView tool have an alternative console program using java, termed "KVM". This uses the Iced Tea jnlp environment, but it can recetnly it has been demanding keys (only for marvin) and will probably not work.<br />
<br />
Also many of these tools are rather old, which is often not a problem if the tool's function is simple. For example "power on" and "power off" are simple functions. There terinal program is not a simple function, and sometimes it may required an old version of java to run (i.e. version 6).<br />
<br />
== startup services==<br />
<br />
All important startup services are launched automatically, however it is a good idea to verify this using the "chkconfig" command. i.e. to check the runlevels (runlevel 5 being the most important) on the network time daemon (ntpd)<br />
<br />
chkconfig --list ntpd<br />
<br />
To enable it for automatic startup: <br />
<br />
chkconfig ntpd on --level 35<br />
<br />
which also ensures it launches even when on the minimal level 3.<br />
<br />
= The main critical issue =<br />
<br />
All in all, restarting marvin simply means typing "reboot", and it will power down and then power up. This will happen smoothly on the nodes for example, and is a very fortunate series of events, because it means that no BIOS interaction (key presses) are required (however the baseboard event logger sometimes fills up and may disrupt this, by asking for a key press, and so bring the boot-up process to a total stop).<br />
<br />
Something else however causes an interruption in the boot-up of the frontend and it is the automatic mounting of the STORAGE filesystem which holds all the users home directories. This is a networked storage system, but the marvin system configures it under LVM (Logical Volume Management system) and when one is ready, the other one isn't which stalls the automatic procedure. Manual intervention is therefore required. One can detect this happening by running '''vgdisplay''' and noticing that the STORAGE volume is unavailable.<br />
<br />
Because this discrepancy halts the boot-up process it must be done manually and the mount directive available via '''/etc/fstab''' should always be commented out. In any case, the manual command is very simple, so one just performs "reboot" on marvin, and once it is back on line (could take as long as 10 minutes), the following command should be invoked:<br />
<br />
vgchange -a y STORAGE<br />
<br />
one should check via "vgdisplay" that STORAGE is now available, and one can decomment the appropriate line in '''/etc/fstab''' and run<br />
<br />
mount /storage<br />
<br />
Of course this should then be followed by the commenting out (once again) of the storage line in '''/etc/fstab'''.<br />
<br />
<br />
<br />
NOTE: if you get many unfixable satle file handle problems ... which I have .. feel my pain (Im tired now - LOOOONg day):<br />
<br />
# on head node<br />
showmount -e marvin<br />
exportfs -f<br />
vgchange -a y STORAGE<br />
mount /storage<br />
<br />
# on the slaves<br />
showmount -e marvin<br />
service nfs restart<br />
service nfs restart<br />
mount -a<br />
df -h<br />
<br />
= Provisos =<br />
<br />
Restarting marvin is a major operation, as all running jobs are lost.<br />
<br />
It is therefore necessary to advise all users well in advance, as to when it might happen.<br />
<br />
= sungrid engine not running on the centos 7 nodes=<br />
restart the services. <br />
<br />
cd into <br />
cd/etc/init.d<br />
<br />
http://www.softpanorama.org/HPC/Grid_engine/Troubleshooting/starting_and_killing_sge_daemons.shtml<br />
<br />
cd /etc/init.d<br />
<br />
./sgeexecd.p6444 stop<br />
Shutting down Grid Engine execution daemon<br />
./sgemaster.p6444 stop<br />
shutting down Grid Engine qmaster<br />
<br />
service sgeexec.p6444 stop && sgeexec.p6444 start</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3495Kennedy manual2020-05-07T11:48:46Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]<br />
<br />
*[[samba like connection]]<br />
<br />
*[[Conda]]<br />
CONDA: http://stab.st-andrews.ac.uk/wiki/index.php/Conda#conda<br />
Training: github.com/peterthorpe5/Sys_admin/tree/master/cluster_course<br />
Note: on Kennedy, you may need to make programs executable to run. This is a know bug. Locate where the install have just gone, move to that directory and run <br />
chmod u+x prog(s)_of_interest. often in th elib folder<br />
<br />
The error that indicates this kind of error is:<br />
libptf77blas.so.3: '''cannot open shared object file<br />
'''<br />
<br />
==syntax highlighting in Nano==<br />
do the following<br />
cp /gpfs1/apps/kennedy_sys_admin/misc/.nanorc ./</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3494Kennedy manual2020-05-07T11:47:50Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]<br />
<br />
*[[samba like connection]]<br />
<br />
*[[Conda]]<br />
CONDA: http://stab.st-andrews.ac.uk/wiki/index.php/Conda#conda<br />
Training: github.com/peterthorpe5/Sys_admin/tree/master/cluster_course<br />
Note: on Kennedy, you may need to make programs executable to run. This is a know bug. Locate where the install have just gone, move to that directory and run chmod u+x prog_of_interest. The error that idicates something like this is:<br />
libptf77blas.so.3: '''cannot open shared object file<br />
'''<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
==syntax highlighting in Nano==<br />
do the following<br />
cp /gpfs1/apps/kennedy_sys_admin/misc/.nanorc ./</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3493Kennedy manual2020-05-06T12:35:06Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]<br />
<br />
*[[samba like connection]]<br />
<br />
*[[Conda]]<br />
CONDA: http://stab.st-andrews.ac.uk/wiki/index.php/Conda#conda<br />
Training: github.com/peterthorpe5/Sys_admin/tree/master/cluster_course<br />
Note: on Kennedy, you may need to make programs executable to run. This is a know bug. Locate where the install have just gone, move to that directory and run chmod u+x prog_of_interest<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
==syntax highlighting in Nano==<br />
do the following<br />
cp /gpfs1/apps/kennedy_sys_admin/misc/.nanorc ./</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3492Kennedy manual2020-05-06T12:32:54Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]<br />
<br />
*[[samba like connection]]<br />
<br />
*[[Conda]]<br />
<br />
<br />
<br />
<br />
<br />
<br />
==syntax highlighting in Nano==<br />
do the following<br />
cp /gpfs1/apps/kennedy_sys_admin/misc/.nanorc ./</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3491Kennedy manual2020-05-06T12:31:34Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]<br />
<br />
*[[samba like connection]]<br />
<br />
<br />
<br />
<br />
<br />
<br />
==syntax highlighting in Nano==<br />
do the following<br />
cp /gpfs1/apps/kennedy_sys_admin/misc/.nanorc ./</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Samba_like_connection&diff=3490Samba like connection2020-05-06T12:30:11Z<p>PeterThorpe: Created page with " == Windows net work connect in a samba like manner == http://stab.st-andrews.ac.uk/wiki/index.php/Windows_network_connect You probably don't know what samba is. This does..."</p>
<hr />
<div><br />
<br />
== Windows net work connect in a samba like manner ==<br />
<br />
http://stab.st-andrews.ac.uk/wiki/index.php/Windows_network_connect<br />
<br />
You probably don't know what samba is. This doesn't matter. ALl you need to know is - THIS IS COOL.<br />
<br />
install these on your windows or Linux PC. These are prebuilt installer and all you need to do it double click.<br />
sshfs-win: https://github.com/billziss-gh/sshfs-win <br />
Requires WinFsp (https://github.com/billziss-gh/winfsp/releases/latest ) <br />
# to be installed with Cygwin FUSE support ticked in the installer.<br />
<br />
Now open Windows Explorer and go to “This PC”.<br />
From the ribbon click “Map Network Drive”. Map this to a letter of your choice, e.g. k for kennedy<br />
Enter:<br />
<br />
\\sshfs.k\you@kennedy10<br />
<br />
Large file transfers, please use filezilla, I dont know how stable this is!!!!!!!! <br />
<br />
<br />
==Mount the scratch, you still have to navigate to it, like so:==<br />
\\sshfs.k\USERNAME@kennedy10/../../scratch/bioinf/USERNAME<br />
<br />
<br />
To map the scratch see the next slide.</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3489Kennedy manual2020-05-06T12:28:37Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]<br />
<br />
*[[samba like connection]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Submit_a_job_and_monitor_queues&diff=3488Submit a job and monitor queues2020-05-06T12:27:43Z<p>PeterThorpe: Created page with " == submit a job == to submit a shell which contains are the needed flags in the file. sbatch spades.sh ==Show information on the queues:== smap squeue (show the queue..."</p>
<hr />
<div><br />
== submit a job ==<br />
<br />
to submit a shell which contains are the needed flags in the file.<br />
sbatch spades.sh<br />
<br />
<br />
==Show information on the queues:==<br />
smap<br />
squeue (show the queue)<br />
squeue –p bigmem (show the queue for big mem)<br />
sview<br />
<br />
==Interactive mode (like qrsh):==<br />
srun --pty bash -p bigmem (Bioinf user : use the bigmem q)</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3487Kennedy manual2020-05-06T12:26:08Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]<br />
<br />
*[[submit a job and monitor queues]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Slurm_commands&diff=3486Slurm commands2020-05-06T12:25:14Z<p>PeterThorpe: </p>
<hr />
<div><br />
<br />
== slurm commands ==<br />
<br />
note the BIOINF community are supposed to use the -p bigmem queue. <br />
<br />
Sun Grid Engines (what we use on Marvin) to slurm command conversion:<br />
https://srcc.stanford.edu/sge-slurm-conversion<br />
<br />
<br />
<br />
The following are command line commands, or tags you can put in your shell script (or at the command line) to achieve certain functionality.<br />
<br />
request_48_thread_1.3TBRAM<br />
#!/bin/bash -l '''# not the -l is essential now'''<br />
#SBATCH -J fly_pilon #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=48<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --nodelist=kennedy150 # this is the specific node. This one has 1.5TB RAM<br />
#SBATCH --mem=1350GB<br />
<br />
<br />
test_conda_activate<br />
#!/bin/bash -l<br />
#SBATCH -J conda_test #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --tasks-per-node=1<br />
#SBATCH -p bigmem # big mem if for the BIOINF community<br />
#SBATCH --mail-type=END # email at the end of the job<br />
#SBATCH --mail-user=$USER@st-andrews.ac.uk # your email address<br />
<br />
cd /gpfs1/home/$USER/<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
# conda to activate the software<br />
<br />
echo $PATH<br />
<br />
conda activate spades<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
conda deactivate <br />
<br />
conda activate python27<br />
<br />
pyv="$(python2 -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
12threads_bigMeme_30G_RAM<br />
<br />
!/bin/bash -l # essential <br />
#SBATCH -J trimmo #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=12<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --mem=30GB<br />
<br />
<br />
Request an interactive job with one GPU:<br />
# srun --gres=gpu:1 -N 1 -p singlenode --pty /bin/bash</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=File:---C--Users-pjt6-Desktop-Picture1.png&diff=3485File:---C--Users-pjt6-Desktop-Picture1.png2020-05-06T12:22:33Z<p>PeterThorpe: PeterThorpe uploaded a new version of File:---C--Users-pjt6-Desktop-Picture1.png</p>
<hr />
<div></div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=File:---C--Users-pjt6-Desktop-Picture1.png&diff=3484File:---C--Users-pjt6-Desktop-Picture1.png2020-05-06T12:21:10Z<p>PeterThorpe: </p>
<hr />
<div></div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Filezilla_data_transfer&diff=3483Filezilla data transfer2020-05-06T12:20:24Z<p>PeterThorpe: Created page with "Media:file:///C:/Users/pjt6/Desktop/Picture1.png"</p>
<hr />
<div>[[Media:file:///C:/Users/pjt6/Desktop/Picture1.png]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3482Kennedy manual2020-05-06T12:18:59Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[logging on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[filezilla data transfer]]<br />
<br />
*[[slurm commands]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Pull_and_push_to_to_and_from_MARVIN&diff=3481Pull and push to to and from MARVIN2020-05-06T12:17:36Z<p>PeterThorpe: Created page with " == Copy data from marvin – or push to marvin for backup == users will want to copy data from the backed up area Marvin to the temp work/ scratch space on Kennedy, and visa..."</p>
<hr />
<div><br />
== Copy data from marvin – or push to marvin for backup ==<br />
<br />
users will want to copy data from the backed up area Marvin to the temp work/ scratch space on Kennedy, and visa versa. <br />
We recommend using rsync for this. It is surprisingly fast. <br />
<br />
https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/<br />
<br />
rsync -av $USER@marvin.st-andrews.ac.uk:path/ path/<br />
<br />
Type yes, then your marvin password<br />
<br />
PUSH FILES TO MARVIN (please look up the commands)<br />
<br />
rsync -avzhe ssh file_to_transfer $USER@marvin.st-andrews.ac.uk:/path_to/</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3480Kennedy manual2020-05-06T12:14:45Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[loggin on to Kennedy]]<br />
<br />
*[[pull and push to to and from MARVIN]]<br />
<br />
*[[slurm commands]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Mobaxterm_for_Windows&diff=3479Mobaxterm for Windows2020-05-06T12:13:54Z<p>PeterThorpe: </p>
<hr />
<div>If you use previously use MOBA and want to continue to use it. This is how. You can use any ssh client you wish<br />
<br />
NOTE: for myself, this was done before we changed the way people log in. This may not work for you until we test it. <br />
<br />
On mobaxterm: ( you may need to change the permission of the ssh key folder). <br />
<br />
ssh -i path_to_key username@kennedy.st-andrews.ac.uk<br />
<br />
On windows, mobaxterm required: <br />
<br />
/drives/c/ ….<br />
<br />
It is possible to save this session .. Google is your friend.</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Mobaxterm_for_Windows&diff=3478Mobaxterm for Windows2020-05-06T12:13:33Z<p>PeterThorpe: Created page with "If you use previously use MOBA and want to continue to use it. This is how. You can use any ssh client you wish NOTE: for myself, this was done before we changed the way peop..."</p>
<hr />
<div>If you use previously use MOBA and want to continue to use it. This is how. You can use any ssh client you wish<br />
<br />
NOTE: for myself, this was done before we changed the way people log in. This may not work for you until we test it. <br />
<br />
On mobaxterm: ( you may need to change the permission of the ssh key folder). <br />
<br />
ssh -i path_to_key username@kennedy.st-andrews.ac.uk<br />
<br />
On windows, mobaxterm required: /drives/c/ ….<br />
<br />
It is possible to save this session .. Google is your friend.</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Loggin_on_to_Kennedy&diff=3477Loggin on to Kennedy2020-05-06T12:12:42Z<p>PeterThorpe: </p>
<hr />
<div><br />
== Log onto Kennedy using a terminal e.g. putty for Windows or terminal for Linux/ MAC ==<br />
<br />
*[[Mobaxterm for Windows]]<br />
<br />
When you have received a password from the sys admins. <br />
<br />
You should be able to log onto kennedy from a terminal by doing: see next slide.<br />
<br />
ssh USERNAME@kennedy.st-Andrews.ac.uk<br />
<br />
PASSWORD:<br />
<br />
You will then be prompted for your password. <br />
<br />
Please change your password on 1st log in conforming to the university requirements: <br />
https://www.st-andrews.ac.uk/it-support/security/password/<br />
<br />
A strong password is around 10 characters, including uppercase and lowercase letters, numbers and symbols.<br />
<br />
Use the command:<br />
passwd</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Loggin_on_to_Kennedy&diff=3476Loggin on to Kennedy2020-05-06T12:12:26Z<p>PeterThorpe: </p>
<hr />
<div><br />
== Log onto Kennedy using a terminal e.g. putty for Windows or terminal for Linux/ MAC ==<br />
<br />
*[Mobaxterm for Windows]]<br />
<br />
When you have received a password from the sys admins. <br />
<br />
You should be able to log onto kennedy from a terminal by doing: see next slide.<br />
<br />
ssh USERNAME@kennedy.st-Andrews.ac.uk<br />
<br />
PASSWORD:<br />
<br />
You will then be prompted for your password. <br />
<br />
Please change your password on 1st log in conforming to the university requirements: <br />
https://www.st-andrews.ac.uk/it-support/security/password/<br />
<br />
A strong password is around 10 characters, including uppercase and lowercase letters, numbers and symbols.<br />
<br />
Use the command:<br />
passwd</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Loggin_on_to_Kennedy&diff=3475Loggin on to Kennedy2020-05-06T12:11:25Z<p>PeterThorpe: Created page with " == Log onto Kennedy using a terminal e.g. putty for Windows or terminal for Linux/ MAC == When you have received a password from the sys admins. You should be able to log..."</p>
<hr />
<div><br />
== Log onto Kennedy using a terminal e.g. putty for Windows or terminal for Linux/ MAC ==<br />
<br />
When you have received a password from the sys admins. <br />
<br />
You should be able to log onto kennedy from a terminal by doing: see next slide.<br />
<br />
ssh USERNAME@kennedy.st-Andrews.ac.uk<br />
<br />
PASSWORD:<br />
<br />
You will then be prompted for your password. <br />
<br />
Please change your password on 1st log in conforming to the university requirements: <br />
https://www.st-andrews.ac.uk/it-support/security/password/<br />
<br />
A strong password is around 10 characters, including uppercase and lowercase letters, numbers and symbols.<br />
<br />
Use the command:<br />
passwd</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3474Kennedy manual2020-05-06T12:10:23Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
#[[loggin on to Kennedy]]<br />
<br />
<br />
*[[slurm commands]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Windows_users&diff=3473Windows users2020-05-06T12:08:24Z<p>PeterThorpe: Created page with " == Windows: Create a public and private key using putty-gen == Type putty gen in the window search bar (download and install if needed) Click on generate and wiggle the mo..."</p>
<hr />
<div><br />
== Windows: Create a public and private key using putty-gen ==<br />
<br />
<br />
Type putty gen in the window search bar (download and install if needed)<br />
<br />
Click on generate and wiggle the mouse. <br />
<br />
Save private key to : <br />
C:\Users\username\.ssh\putty_priv.ppk<br />
<br />
(do not ever share you private key)<br />
<br />
Send the string in the upper grey part of the PuTTYgen window that says <br />
"Public key for pasting into OpenSSH authorized_keys file". <br />
Copy and paste this into a text file or just into the email to the sys admins.<br />
<br />
note for admins:<br />
The key should be called/renamed authorized_keys when put in the .ssh folder in the $HOME on kennedy – this will be done by the sys admin</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Apple_MAC_people_%2B_Linux&diff=3472Apple MAC people + Linux2020-05-06T12:05:35Z<p>PeterThorpe: Created page with " == create an ssh key using MAC or Linux platforms == open a terminal and enter the command "ssh-keygen". Accept all default filenames. Choose a passphrase when asked f..."</p>
<hr />
<div><br />
== create an ssh key using MAC or Linux platforms ==<br />
<br />
<br />
<br />
open a terminal and enter the command <br />
"ssh-keygen". Accept all default filenames. <br />
<br />
Choose a passphrase when asked for one. Then email the sys admins the file <br />
.ssh/id_rsa.pub<br />
<br />
Directories starting with a . are normally not visible, so it might be easiest to first copy that file into your home<br />
directory:<br />
<br />
cp ~/.ssh/id_rsa.pub</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Creating_ssh_keys_and_logging_on&diff=3471Creating ssh keys and logging on2020-05-06T12:05:06Z<p>PeterThorpe: </p>
<hr />
<div><br />
== the security we use requires ssh keys ==<br />
<br />
*[[Apple MAC people + Linux]]<br />
<br />
*[[Windows users]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Creating_ssh_keys_and_logging_on&diff=3470Creating ssh keys and logging on2020-05-06T12:03:57Z<p>PeterThorpe: Created page with " == the security we use requires ssh keys == '''Apple MAC people''' open a terminal and enter the command "ssh-keygen". Accept all default filenames. Choose a passphras..."</p>
<hr />
<div><br />
== the security we use requires ssh keys ==<br />
<br />
'''Apple MAC people'''<br />
<br />
open a terminal and enter the command <br />
"ssh-keygen". Accept all default filenames. <br />
<br />
Choose a passphrase when asked for one. Then email the sys admins the file <br />
.ssh/id_rsa.pub<br />
<br />
Directories starting with a . are normally not visible, so it might be easiest to first copy that file into your home<br />
directory:<br />
<br />
cp ~/.ssh/id_rsa.pub</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3469Kennedy manual2020-05-06T12:02:07Z<p>PeterThorpe: </p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
#[[creating ssh keys and logging on]]<br />
<br />
<br />
*[[slurm commands]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Quick_start&diff=3468Quick start2020-05-06T12:00:58Z<p>PeterThorpe: </p>
<hr />
<div><br />
== quick start ==<br />
<br />
You must either be on the University campus or logged on via a VPN see <br />
https://www.st-andrews.ac.uk/it-support/services/internet/vpn/<br />
<br />
<br />
<br />
Temporary quick start info can be found here. This is a work in progress until something official is made up.<br />
<br />
https://github.com/peterthorpe5/How_to_use_Kennedy_HPC<br />
<br />
<br />
'''Herbert’s excellent existing documentation<br />
'''<br />
Please also see here:<br />
<br />
<br />
https://universityofstandrews907.sharepoint.com/sites/chemistry/CurrentStudents/Shared%20Documents/Forms/AllItems.aspx?id=%2Fsites%2Fchemistry%2FCurrentStudents%2FShared%20Documents%2Fug%2FERCF%5FIntroduction%2Epdf&parent=%2Fsites%2Fchemistry%2FCurrentStudents%2FShared%20Documents%2Fug&p=true&originalPath=aHR0cHM6Ly91bml2ZXJzaXR5b2ZzdGFuZHJld3M5MDcuc2hhcmVwb2ludC5jb20vOmI6L3MvY2hlbWlzdHJ5L0N1cnJlbnRTdHVkZW50cy9FY1RrWjZzZ0xfaEZnZzFGY0dUZXRZNEJicHFCWHhvNF9Pc3cwRXF3NGlXNDFRP3J0aW1lPUEyLVl0YW5jMTBn</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Slurm_commands&diff=3467Slurm commands2020-05-06T11:58:07Z<p>PeterThorpe: /* slurm commands */</p>
<hr />
<div><br />
<br />
== slurm commands ==<br />
<br />
note the BIOINF community are supposed to use the -p bigmem queue. <br />
<br />
Sun Grid Engines (what we use on Marvin) to slurm command conversion:<br />
https://srcc.stanford.edu/sge-slurm-conversion<br />
<br />
<br />
<br />
The following are command line commands, or tags you can put in your shell script (or at the command line) to achieve certain functionality.<br />
<br />
request_48_thread_1.3TBRAM<br />
#!/bin/bash -l '''# not the -l is essential now'''<br />
#SBATCH -J fly_pilon #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=48<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --nodelist=kennedy150 # this is the specific node. This one has 1.5TB RAM<br />
#SBATCH --mem=1350GB<br />
<br />
<br />
test_conda_activate<br />
#!/bin/bash -l<br />
#SBATCH -J conda_test #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --tasks-per-node=1<br />
#SBATCH -p bigmem # big mem if for the BIOINF community<br />
#SBATCH --mail-type=END # email at the end of the job<br />
#SBATCH --mail-user=$USER@st-andrews.ac.uk # your email address<br />
<br />
cd /gpfs1/home/$USER/<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
# conda to activate the software<br />
<br />
echo $PATH<br />
<br />
conda activate spades<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
conda deactivate <br />
<br />
conda activate python27<br />
<br />
pyv="$(python2 -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
12threads_bigMeme_30G_RAM<br />
<br />
!/bin/bash -l # essential <br />
#SBATCH -J trimmo #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=12<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --mem=30GB</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Slurm_commands&diff=3466Slurm commands2020-05-06T11:56:44Z<p>PeterThorpe: Created page with " == slurm commands == note the BIOINF community are supposed to use the -p bigmem queue. The following are command line commands, or tags you can put in your shell scrip..."</p>
<hr />
<div><br />
<br />
== slurm commands ==<br />
<br />
note the BIOINF community are supposed to use the -p bigmem queue. <br />
<br />
<br />
<br />
The following are command line commands, or tags you can put in your shell script (or at the command line) to achieve certain functionality.<br />
<br />
request_48_thread_1.3TBRAM<br />
#!/bin/bash -l '''# not the -l is essential now'''<br />
#SBATCH -J fly_pilon #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=48<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --nodelist=kennedy150 # this is the specific node. This one has 1.5TB RAM<br />
#SBATCH --mem=1350GB<br />
<br />
<br />
test_conda_activate<br />
#!/bin/bash -l<br />
#SBATCH -J conda_test #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --tasks-per-node=1<br />
#SBATCH -p bigmem # big mem if for the BIOINF community<br />
#SBATCH --mail-type=END # email at the end of the job<br />
#SBATCH --mail-user=$USER@st-andrews.ac.uk # your email address<br />
<br />
cd /gpfs1/home/$USER/<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
# conda to activate the software<br />
<br />
echo $PATH<br />
<br />
conda activate spades<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
conda deactivate <br />
<br />
conda activate python27<br />
<br />
pyv="$(python2 -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
12threads_bigMeme_30G_RAM<br />
<br />
!/bin/bash -l # essential <br />
#SBATCH -J trimmo #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=12<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --mem=30GB</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Quick_start&diff=3465Quick start2020-05-06T11:55:20Z<p>PeterThorpe: Created page with " Temporary quick start info can be found here. This is a work in progress until something official is made up. https://github.com/peterthorpe5/How_to_use_Kennedy_HPC"</p>
<hr />
<div><br />
<br />
Temporary quick start info can be found here. This is a work in progress until something official is made up.<br />
<br />
https://github.com/peterthorpe5/How_to_use_Kennedy_HPC</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3464Kennedy manual2020-05-06T11:54:53Z<p>PeterThorpe: Replaced content with " *quick start *slurm commands"</p>
<hr />
<div><br />
*[[quick start]]<br />
<br />
<br />
*[[slurm commands]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Main_Page&diff=3463Main Page2020-05-06T11:52:15Z<p>PeterThorpe: </p>
<hr />
<div><br />
'''KENNEDY HPC for Bioinf community '''<br />
* [[Kennedy manual]]<br />
<br />
<br />
= Usage of Cluster=<br />
* [[Cluster Manual]]<br />
* [[Kennedy manual]]<br />
* [[Why a Queue Manager?]]<br />
* [[Available Software]]<br />
* [[how to use the cluster training course]]<br />
* [[windows network connect]]<br />
<br />
= Documented Programs =<br />
<br />
The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.<br />
<br />
{|style="width:85%"<br />
|* [[abacas]]<br />
|* [[albacore]]<br />
|* [[ariba]]<br />
|* [[aspera]]<br />
|* [[assembly-stats]]<br />
|* [[augustus]]<br />
|-<br />
|* [[BamQC]]<br />
|* [[bamtools]]<br />
|* [[banjo]]<br />
|* [[bcftools]]<br />
|* [[bedtools]]<br />
|* [[bgenie]]<br />
|-<br />
|* [[BLAST]]<br />
|* [[Blat]]<br />
|* [[blast2go: b2g4pipe]]<br />
|* [[bowtie]]<br />
|* [[bowtie2]]<br />
|* [[bwa]]<br />
|-<br />
|* [[BUSCO]]<br />
|* [[CAFE]]<br />
|* [[canu]]<br />
|* [[cd-hit]]<br />
|* [[cegma]]<br />
|* [[clustal]]<br />
|-<br />
|* [[cramtools]]<br />
|* [[conda]]<br />
|* [[deeptools]]<br />
|* [[detonate]]<br />
|* [[diamond]]<br />
|* [[ea-utils]]<br />
|* [[ensembl]]<br />
|-<br />
|* [[ETE]]<br />
|* [[FASTQC and MultiQC]]<br />
|* [[Archaeopteryx and Forester]]<br />
|* [[GapFiller]]<br />
|* [[GenomeTools]]<br />
|* [[gubbins]]<br />
|-<br />
|* [[JBrowse]]<br />
|* [[kallisto]]<br />
|* [[kentUtils]]<br />
|* [[last]]<br />
|* [[lastz]]<br />
|* [[macs2]]<br />
|-<br />
|* [[Mash]]<br />
|* [[mega]]<br />
|* [[meryl]]<br />
|* [[MUMmer]]<br />
|* [[NanoSim]]<br />
|* [[nseq]]<br />
|-<br />
|* [[OrthoFinder]]<br />
|* [[PASA]]<br />
|* [[perl]]<br />
|* [[PGAP]]<br />
|* [[picard-tools]]<br />
|* [[poRe]]<br />
|-<br />
|* [[poretools]]<br />
|* [[prokka]]<br />
|* [[pyrad]]<br />
|* [[python]]<br />
|* [[qualimap]]<br />
|* [[quast]]<br />
|-<br />
|* [[qiime2]]<br />
|* [[R]]<br />
|* [[RAxML]]<br />
|* [[Repeatmasker]]<br />
|* [[Repeatmodeler]]<br />
|* [[rnammer]]<br />
|-<br />
|* [[roary]]<br />
|* [[RSeQC]]<br />
|* [[samtools]]<br />
|* [[Satsuma]]<br />
|* [[sickle]]<br />
|* [[SPAdes]]<br />
|-<br />
|* [[squid]]<br />
|* [[sra-tools]]<br />
|* [[srst2]]<br />
|* [[SSPACE]]<br />
|* [[stacks]]<br />
|* [[Thor]]<br />
|-<br />
|* [[Tophat]]<br />
|* [[trimmomatic]]<br />
|* [[Trinity]]<br />
|* [[t-coffee]]<br />
|* [[Unicycler]]<br />
|* [[velvet]]<br />
|-<br />
|* [[ViennaRNA]]<br />
|}<br />
<br />
= Queue Manager Tips =<br />
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:<br />
* [[Queue Manager Tips]]<br />
* [[Queue Manager : shell script command]]<br />
* [[Queue Manager emailing when jobs run]]<br />
* [[General Command-line Tips]]<br />
* [[DRMAA for further Gridengine automation]]<br />
<br />
= Data Examples =<br />
* [[Two Eel Scaffolds]]<br />
<br />
= Procedures =<br />
(short sequence of tasks with a certain short-term goal, often, a simple script)<br />
* [[Calculating coverage]]<br />
* [[MinION Coverage sensitivity analysis]]<br />
<br />
= Navigating genomic data websites=<br />
* [[Patric]]<br />
* [[NCBI]]<br />
* [[IGSR/1000 Genomes]]<br />
<br />
= Explanations=<br />
* [[ITUcourse]]<br />
* [[VCF]]<br />
* [[Maximum Likelihood]]<br />
* [[SNP Analysis and phylogenetics]]<br />
* [[Normalization]]<br />
<br />
= Pipelines =<br />
(Workflow with specific end-goals)<br />
* [[Trinity_Protocol]]<br />
* [[STAR BEAST]]<br />
* [[callSNPs.py]]<br />
* [[pairwiseCallSNPs]]<br />
* [[mapping.py]]<br />
* [[Edgen RNAseq]]<br />
* [[Miseq Prokaryote FASTQ analysis]]<br />
* [[snpcallphylo]]<br />
* [[Bottlenose dolphin population genomic analysis]]<br />
* [[ChIP-Seq Top2 in Yeast]]<br />
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]<br />
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]<br />
* [[Bisulfite Sequencing]]<br />
* [[microRNA and Salmo Salar]]<br />
<br />
=Protocols=<br />
(Extensive workflows with different with several possible end goals)<br />
* [[Synthetic Long reads]]<br />
* [[MinION (Oxford Nanopore)]]<br />
* [[MinKNOW folders and log files]]<br />
* [[Research Data Management]]<br />
* [[MicroRNAs]]<br />
<br />
= Tech Reviews =<br />
* [[SWATH-MS Data Analysis]]<br />
<br />
= Cluster Administration =<br />
* [[StABDMIN]]<br />
* [[Hardware Issues]]<br />
* [[marvin and IPMI (remote hardware control)]]<br />
* [[restart a node]]<br />
* [[mounting drives]]<br />
* [[Admin Tips]]<br />
* [[RedHat]]<br />
* [[Globus_gridftp]]<br />
* [[Galaxy Setup]]<br />
* [[Son of Gridengine]]<br />
* [[Blas Libraries]]<br />
* [[CMake]]<br />
* [[conda bioconda]]<br />
* [[Users and Groups]]<br />
* [[Installing software on marvin]]<br />
* [[emailing]]<br />
* [[biotime machine]]<br />
* [[SCAN-pc laptop]]<br />
* [[node1 issues]]<br />
* [[6TB storage expansion]]<br />
* [[PIs storage sacrifice]]<br />
* [[SAN relocation task]]<br />
* [[Home directories max-out incident 28.11.2016]]<br />
* [[Frontend Restart]]<br />
* [[environment-modules]]<br />
* [[H: drive on cluster]]<br />
* [[Incident: Can't connect to BerkeleyDB]]<br />
* [[Bioinformatics Wordpress Site]]<br />
* [[Backups]]<br />
* [[users disk usage]]<br />
* [[Updating BLAST databases]]<br />
* [[Python DRMAA]]<br />
* [[message of the day]]<br />
* [[SAN disconnect incident 10.01.2017]]<br />
* [[Memory repair glitch 16.02.2017]]<br />
* [[node9 network failure incident 16-20.03.2017]]<br />
* [[Incorrect rebooting of marvin 19.09.2017]]<br />
* [[ansible]]<br />
* [[webstie and word press]]<br />
* [[allow user access to other peoples data]]<br />
* [[RAM and RAM slots]]<br />
* [[ldap is not ldap]]<br />
* [[reset a password]]<br />
* [[sending emails from command line examples]]<br />
<br />
= Courses =<br />
<br />
==I2U4BGA==<br />
* [[Original schedule]]<br />
* [[New schedule]]<br />
* [[Actual schedule]]<br />
* [[Course itself]]<br />
* [[Biolinux Source course]]<br />
* [[Directory Organization Exercise]]<br />
* [[Glossary]]<br />
* [[Key Bindings]]<br />
* [[one-liners]]<br />
* [[Cheatsheets]]<br />
* [[Links]]<br />
* [[pandoc modified manual]]<br />
* [[Command Line Exercises]]<br />
<br />
= hdi2u =<br />
<br />
The half-day linux course held on 20th April. Modified version of I2U4BGA.<br />
<br />
* [[hdi2u_intro]]<br />
* [[hdi2u_commandbased_exercises]]<br />
* [[hdi2u_dirorg_exercise]]<br />
* [[hdi2u_rendertotsv_exercise]]<br />
<br />
= RNAseq for DGE =<br />
* [[Theoretical background]]<br />
* [[Quality Control and Preprocessing]]<br />
* [[Mapping to Reference]]<br />
* [[Mapping Quality Exercise]]<br />
* [[Key Aspects of using R]]<br />
* [[Estimating Gene Count Exercise]]<br />
* [[Differential Expression Exercise]]<br />
* [[Functional Analysis Exercise]]<br />
<br />
= Introduction to Unix 2017 =<br />
* [[Introduction_to_Unix_2017]]<br />
<br />
<br />
<br />
==Templates==<br />
* [[edgenl2g]]</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3462Kennedy manual2020-05-06T11:48:25Z<p>PeterThorpe: </p>
<hr />
<div><br />
====slurm commands====<br />
<br />
<br />
Temporary quick start info can be found here. This is a work in progress until something official is made up.<br />
<br />
https://github.com/peterthorpe5/How_to_use_Kennedy_HPC<br />
<br />
<br />
The following are command line commands, or tags you can put in your shell script to achieve certain functionality.<br />
<br />
request_48_thread_1.3TBRAM<br />
#!/bin/bash -l '''# not the -l is essential now'''<br />
#SBATCH -J fly_pilon #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=48<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --nodelist=kennedy150 # this is the specific node. This one has 1.5TB RAM<br />
#SBATCH --mem=1350GB<br />
<br />
<br />
test_conda_activate<br />
#!/bin/bash -l<br />
#SBATCH -J conda_test #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --tasks-per-node=1<br />
#SBATCH -p bigmem # big mem if for the BIOINF community<br />
#SBATCH --mail-type=END # email at the end of the job<br />
#SBATCH --mail-user=$USER@st-andrews.ac.uk # your email address<br />
<br />
cd /gpfs1/home/$USER/<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
# conda to activate the software<br />
<br />
echo $PATH<br />
<br />
conda activate spades<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
conda deactivate <br />
<br />
conda activate python27<br />
<br />
pyv="$(python2 -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
12threads_bigMeme_30G_RAM<br />
<br />
!/bin/bash -l # essential <br />
#SBATCH -J trimmo #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=12<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --mem=30GB</div>PeterThorpehttp://stab.st-andrews.ac.uk/wiki/index.php?title=Kennedy_manual&diff=3461Kennedy manual2020-05-06T08:07:18Z<p>PeterThorpe: </p>
<hr />
<div><br />
Temporary quick start info can be found here. This is a work in progress until something official is made up.<br />
<br />
https://github.com/peterthorpe5/How_to_use_Kennedy_HPC<br />
<br />
<br />
The following are command line commands, or tags you can put in your shell script to achieve certain functionality.<br />
<br />
request_48_thread_1.3TBRAM<br />
#!/bin/bash -l '''# not the -l is essential now'''<br />
#SBATCH -J fly_pilon #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=48<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --nodelist=kennedy150 # this is the specific node. This one has 1.5TB RAM<br />
#SBATCH --mem=1350GB<br />
<br />
<br />
test_conda_activate<br />
#!/bin/bash -l<br />
#SBATCH -J conda_test #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --tasks-per-node=1<br />
#SBATCH -p bigmem # big mem if for the BIOINF community<br />
#SBATCH --mail-type=END # email at the end of the job<br />
#SBATCH --mail-user=$USER@st-andrews.ac.uk # your email address<br />
<br />
cd /gpfs1/home/$USER/<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
# conda to activate the software<br />
<br />
echo $PATH<br />
<br />
conda activate spades<br />
<br />
pyv="$(python -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
conda deactivate <br />
<br />
conda activate python27<br />
<br />
pyv="$(python2 -V 2>&1)"<br />
<br />
echo "$pyv"<br />
<br />
12threads_bigMeme_30G_RAM<br />
<br />
!/bin/bash -l # essential <br />
#SBATCH -J trimmo #jobname<br />
#SBATCH -N 1 #node<br />
#SBATCH --ntasks-per-node=12<br />
#SBATCH --threads-per-core=2<br />
#SBATCH -p bigmem<br />
#SBATCH --mem=30GB</div>PeterThorpe