Difference between revisions of "Main Page"

From wiki
Jump to: navigation, search
(→‎Cluster Administration)
 
(53 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
 +
'''KENNEDY HPC for Bioinf community '''
 +
* [[Kennedy manual]]
 +
 +
 
= Usage of Cluster=
 
= Usage of Cluster=
 
* [[Cluster Manual]]
 
* [[Cluster Manual]]
 +
* [[Kennedy manual]]
 
* [[Why a Queue Manager?]]
 
* [[Why a Queue Manager?]]
 
* [[Available Software]]
 
* [[Available Software]]
 +
* [[how to use the cluster training course]]
 +
* [[windows network connect]]
  
 
= Documented Programs =
 
= Documented Programs =
Line 18: Line 26:
 
|* [[BamQC]]
 
|* [[BamQC]]
 
|* [[bamtools]]
 
|* [[bamtools]]
 +
|* [[banjo]]
 
|* [[bcftools]]
 
|* [[bcftools]]
 
|* [[bedtools]]
 
|* [[bedtools]]
 
|* [[bgenie]]
 
|* [[bgenie]]
 +
|-
 
|* [[BLAST]]
 
|* [[BLAST]]
−
|-
 
 
|* [[Blat]]
 
|* [[Blat]]
 
|* [[blast2go: b2g4pipe]]
 
|* [[blast2go: b2g4pipe]]
Line 28: Line 37:
 
|* [[bowtie2]]
 
|* [[bowtie2]]
 
|* [[bwa]]
 
|* [[bwa]]
 +
|-
 
|* [[BUSCO]]
 
|* [[BUSCO]]
−
|-
 
 
|* [[CAFE]]
 
|* [[CAFE]]
 
|* [[canu]]
 
|* [[canu]]
Line 35: Line 44:
 
|* [[cegma]]
 
|* [[cegma]]
 
|* [[clustal]]
 
|* [[clustal]]
 +
|-
 
|* [[cramtools]]
 
|* [[cramtools]]
−
|-
+
|* [[conda]]
 
|* [[deeptools]]
 
|* [[deeptools]]
 
|* [[detonate]]
 
|* [[detonate]]
Line 42: Line 52:
 
|* [[ea-utils]]
 
|* [[ea-utils]]
 
|* [[ensembl]]
 
|* [[ensembl]]
 +
|-
 
|* [[ETE]]
 
|* [[ETE]]
−
|-
 
 
|* [[FASTQC and MultiQC]]
 
|* [[FASTQC and MultiQC]]
 
|* [[Archaeopteryx and Forester]]
 
|* [[Archaeopteryx and Forester]]
Line 49: Line 59:
 
|* [[GenomeTools]]
 
|* [[GenomeTools]]
 
|* [[gubbins]]
 
|* [[gubbins]]
 +
|-
 
|* [[JBrowse]]
 
|* [[JBrowse]]
−
|-
 
 
|* [[kallisto]]
 
|* [[kallisto]]
 
|* [[kentUtils]]
 
|* [[kentUtils]]
Line 56: Line 66:
 
|* [[lastz]]
 
|* [[lastz]]
 
|* [[macs2]]
 
|* [[macs2]]
 +
|-
 
|* [[Mash]]
 
|* [[Mash]]
−
|-
 
 
|* [[mega]]
 
|* [[mega]]
 +
|* [[meryl]]
 
|* [[MUMmer]]
 
|* [[MUMmer]]
 
|* [[NanoSim]]
 
|* [[NanoSim]]
 +
|* [[nseq]]
 +
|-
 
|* [[OrthoFinder]]
 
|* [[OrthoFinder]]
 +
|* [[PASA]]
 +
|* [[perl]]
 
|* [[PGAP]]
 
|* [[PGAP]]
 
|* [[picard-tools]]
 
|* [[picard-tools]]
 +
|* [[poRe]]
 
|-
 
|-
−
|* [[poRe]]
 
 
|* [[poretools]]
 
|* [[poretools]]
 
|* [[prokka]]
 
|* [[prokka]]
Line 71: Line 86:
 
|* [[python]]
 
|* [[python]]
 
|* [[qualimap]]
 
|* [[qualimap]]
 +
|* [[quast]]
 
|-
 
|-
−
|* [[quast]]
+
|* [[qiime2]]
 
|* [[R]]
 
|* [[R]]
 
|* [[RAxML]]
 
|* [[RAxML]]
 
|* [[Repeatmasker]]
 
|* [[Repeatmasker]]
 +
|* [[Repeatmodeler]]
 
|* [[rnammer]]
 
|* [[rnammer]]
 +
|-
 
|* [[roary]]
 
|* [[roary]]
−
|-
 
 
|* [[RSeQC]]
 
|* [[RSeQC]]
 
|* [[samtools]]
 
|* [[samtools]]
Line 84: Line 101:
 
|* [[sickle]]
 
|* [[sickle]]
 
|* [[SPAdes]]
 
|* [[SPAdes]]
 +
|-
 +
|* [[squid]]
 
|* [[sra-tools]]
 
|* [[sra-tools]]
−
|-
 
 
|* [[srst2]]
 
|* [[srst2]]
 
|* [[SSPACE]]
 
|* [[SSPACE]]
 
|* [[stacks]]
 
|* [[stacks]]
 
|* [[Thor]]
 
|* [[Thor]]
 +
|-
 
|* [[Tophat]]
 
|* [[Tophat]]
 
|* [[trimmomatic]]
 
|* [[trimmomatic]]
−
|-
 
 
|* [[Trinity]]
 
|* [[Trinity]]
 
|* [[t-coffee]]
 
|* [[t-coffee]]
 +
|* [[Unicycler]]
 
|* [[velvet]]
 
|* [[velvet]]
 +
|-
 +
|* [[ViennaRNA]]
 
|}
 
|}
  
Line 101: Line 122:
 
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:
 
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:
 
* [[Queue Manager Tips]]
 
* [[Queue Manager Tips]]
 +
* [[Queue Manager : shell script command]]
 +
* [[Queue Manager emailing when jobs run]]
 
* [[General Command-line Tips]]
 
* [[General Command-line Tips]]
 
* [[DRMAA for further Gridengine automation]]
 
* [[DRMAA for further Gridengine automation]]
Line 115: Line 138:
 
* [[Patric]]
 
* [[Patric]]
 
* [[NCBI]]
 
* [[NCBI]]
 +
* [[IGSR/1000 Genomes]]
  
 
= Explanations=
 
= Explanations=
Line 121: Line 145:
 
* [[Maximum Likelihood]]
 
* [[Maximum Likelihood]]
 
* [[SNP Analysis and phylogenetics]]
 
* [[SNP Analysis and phylogenetics]]
 +
* [[Normalization]]
  
 
= Pipelines =
 
= Pipelines =
−
(Workflow with a specific end-goal)
+
(Workflow with specific end-goals)
 
* [[Trinity_Protocol]]
 
* [[Trinity_Protocol]]
 
* [[STAR BEAST]]
 
* [[STAR BEAST]]
 
* [[callSNPs.py]]
 
* [[callSNPs.py]]
 +
* [[pairwiseCallSNPs]]
 
* [[mapping.py]]
 
* [[mapping.py]]
 
* [[Edgen RNAseq]]
 
* [[Edgen RNAseq]]
Line 132: Line 158:
 
* [[snpcallphylo]]
 
* [[snpcallphylo]]
 
* [[Bottlenose dolphin population genomic analysis]]
 
* [[Bottlenose dolphin population genomic analysis]]
−
* [[ChIP-Seq Top2 peak-calling]]
+
* [[ChIP-Seq Top2 in Yeast]]
 +
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]
 +
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]
 +
* [[Bisulfite Sequencing]]
 +
* [[microRNA and Salmo Salar]]
  
 
=Protocols=
 
=Protocols=
Line 140: Line 170:
 
* [[MinKNOW folders and log files]]
 
* [[MinKNOW folders and log files]]
 
* [[Research Data Management]]
 
* [[Research Data Management]]
 +
* [[MicroRNAs]]
  
 
= Tech Reviews =
 
= Tech Reviews =
Line 145: Line 176:
  
 
= Cluster Administration =
 
= Cluster Administration =
 +
* [[StABDMIN]]
 
* [[Hardware Issues]]
 
* [[Hardware Issues]]
 +
* [[marvin and IPMI (remote hardware control)]]
 +
* [[restart a node]]
 +
* [[mounting drives]]
 
* [[Admin Tips]]
 
* [[Admin Tips]]
 
* [[RedHat]]
 
* [[RedHat]]
Line 153: Line 188:
 
* [[Blas Libraries]]
 
* [[Blas Libraries]]
 
* [[CMake]]
 
* [[CMake]]
 +
* [[conda bioconda]]
 
* [[Users and Groups]]
 
* [[Users and Groups]]
 +
* [[Installing software on marvin]]
 
* [[emailing]]
 
* [[emailing]]
 
* [[biotime machine]]
 
* [[biotime machine]]
 +
* [[SCAN-pc laptop]]
 
* [[node1 issues]]
 
* [[node1 issues]]
 
* [[6TB storage expansion]]
 
* [[6TB storage expansion]]
 +
* [[PIs storage sacrifice]]
 +
* [[SAN relocation task]]
 
* [[Home directories max-out incident 28.11.2016]]
 
* [[Home directories max-out incident 28.11.2016]]
 
* [[Frontend Restart]]
 
* [[Frontend Restart]]
Line 165: Line 205:
 
* [[Bioinformatics Wordpress Site]]
 
* [[Bioinformatics Wordpress Site]]
 
* [[Backups]]
 
* [[Backups]]
 +
* [[users disk usage]]
 +
* [[Updating BLAST databases]]
 
* [[Python DRMAA]]
 
* [[Python DRMAA]]
 +
* [[message of the day]]
 
* [[SAN disconnect incident 10.01.2017]]
 
* [[SAN disconnect incident 10.01.2017]]
 
* [[Memory repair glitch 16.02.2017]]
 
* [[Memory repair glitch 16.02.2017]]
 
* [[node9 network failure incident 16-20.03.2017]]
 
* [[node9 network failure incident 16-20.03.2017]]
 +
* [[Incorrect rebooting of marvin 19.09.2017]]
 +
* [[ansible]]
 +
* [[webstie and word press]]
 +
* [[allow user access to other peoples data]]
 +
* [[RAM and RAM slots]]
 +
* [[ldap is not ldap]]
 +
* [[reset a password]]
 +
* [[sending emails from command line examples]]
 +
* [[disk management after shelf disk failure]]
 +
* [[firewall and iptables]]
  
 
= Courses =
 
= Courses =
Line 205: Line 258:
 
* [[Differential Expression Exercise]]
 
* [[Differential Expression Exercise]]
 
* [[Functional Analysis Exercise]]
 
* [[Functional Analysis Exercise]]
 +
 +
= Introduction to Unix 2017 =
 +
* [[Introduction_to_Unix_2017]]
  
  

Latest revision as of 18:14, 22 July 2020

KENNEDY HPC for Bioinf community


Usage of Cluster

Documented Programs

The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.

* abacas * albacore * ariba * aspera * assembly-stats * augustus
* BamQC * bamtools * banjo * bcftools * bedtools * bgenie
* BLAST * Blat * blast2go: b2g4pipe * bowtie * bowtie2 * bwa
* BUSCO * CAFE * canu * cd-hit * cegma * clustal
* cramtools * conda * deeptools * detonate * diamond * ea-utils * ensembl
* ETE * FASTQC and MultiQC * Archaeopteryx and Forester * GapFiller * GenomeTools * gubbins
* JBrowse * kallisto * kentUtils * last * lastz * macs2
* Mash * mega * meryl * MUMmer * NanoSim * nseq
* OrthoFinder * PASA * perl * PGAP * picard-tools * poRe
* poretools * prokka * pyrad * python * qualimap * quast
* qiime2 * R * RAxML * Repeatmasker * Repeatmodeler * rnammer
* roary * RSeQC * samtools * Satsuma * sickle * SPAdes
* squid * sra-tools * srst2 * SSPACE * stacks * Thor
* Tophat * trimmomatic * Trinity * t-coffee * Unicycler * velvet
* ViennaRNA

Queue Manager Tips

A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with q and with qsub being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:

Data Examples

Procedures

(short sequence of tasks with a certain short-term goal, often, a simple script)

Navigating genomic data websites

Explanations

Pipelines

(Workflow with specific end-goals)

Protocols

(Extensive workflows with different with several possible end goals)

Tech Reviews

Cluster Administration

Courses

I2U4BGA

hdi2u

The half-day linux course held on 20th April. Modified version of I2U4BGA.

RNAseq for DGE

Introduction to Unix 2017


Templates

Retrieved from "http://stab.st-andrews.ac.uk/wiki/index.php?title=Main_Page&oldid=3507"