2018-02-02T12:33:13Z

Rf:

= Introduction =

For example, when networking goes down on marvin and yet the machine is still up, ssh can no longer be used, so the administrator is stuck.

However all modern, industrial grade (i.e. non-consumer) server now include a separate subsystem on the machine which is independent of the main machine. This is known by various names, and there is a standard called IPMI, which is what Supermicor calls it. Dell calls it DRAC and HP calls it LightsOn (or something to that effect).

It can be seen as a hardware remote control system and the devices include a network connection so one can login to the IPMI module and send hardware commands (principally power-on and power-cycle).

Despite IPMI's isolation from the main system, it is not immune to faults of its own, in which case, the only cure is to ring Ian at Services and go to the datacentre. Unfortunately, IPMI tends not to cooperate exactly when the man machines has problems of its own, which is disappointing because that's exactly when it's needed. Nevertheless, IPMI is better than nothing and has proved useful on many occasions.

= Details =

Marvin's nodes can all be remotely controlled but only in marvin itself. So the usual exercise is to run firefox on marvin and connect to the IPMI IPs there.

When marvin's IPMI itself needs to be used, then this can be done from another computer within the University campus.

There is a standalone GUI application called IPMIconfig which does then same things as the IPMI web interface, but because it doesn't need a browser, can be faster.

The virtual console on IPMI's web interface uses JNLP (javaws) program and is the best implementation, but it can be patchy. It also allows the loading of a local Live Linux ISO file so that the machine may be booted from it, though this can be a bit tortuous. Certainly, it is very clear that Supermicro's IPMI interface is considerably inferior to Dell's DRAC interface used on the biotime machine. Nevertheless, it is possible to boot the recommended Linux Live ISO, [http://www.system-rescue-cd.org sysrescuecd]on Supermicro's IPMI.

Again it must be repeated that the virtual console's functioning is patchy. An even less dependable version of Virtual Console is SOL. It may seem silly to mention SOL when it is even worse than Virtual console, however it has one or two crucial advantages which make it the holy grail of remote hardware control:
* SOL is a raw terminal connection to the login screen of the main machine.
* it does not operate via buggy GUI's and web interfaces.
* one can connect via the command line and record all input and output via your local linux computer's "script" progam (see "man script").
* When it works, it is much faster than the alternatives.
* It behaves as if one really was sittigng down locally at the machine, looking at the login screen.

2018-01-29T18:24:03Z

Rf:

Qiime2

2018-01-29T18:16:47Z

Rf:

Qiime2

2018-01-29T18:16:37Z

Rf: Created page with "= Introduction = This is a large, self-contained bioinformatics environment confected to tackle in particular microbiome studies. Microbiome studies, do in fact pose special..."

Main Page

2018-01-29T18:10:05Z

Rf:

= Usage of Cluster=
* [[Cluster Manual]]
* [[Why a Queue Manager?]]
* [[Available Software]]

= Documented Programs =

The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.

{|style="width:85%"
|* [[abacas]]
|* [[albacore]]
|* [[ariba]]
|* [[aspera]]
|* [[assembly-stats]]
|* [[augustus]]
|-
|* [[BamQC]]
|* [[bamtools]]
|* [[banjo]]
|* [[bcftools]]
|* [[bedtools]]
|* [[bgenie]]
|-
|* [[BLAST]]
|* [[Blat]]
|* [[blast2go: b2g4pipe]]
|* [[bowtie]]
|* [[bowtie2]]
|* [[bwa]]
|-
|* [[BUSCO]]
|* [[CAFE]]
|* [[canu]]
|* [[cd-hit]]
|* [[cegma]]
|* [[clustal]]
|-
|* [[cramtools]]
|* [[deeptools]]
|* [[detonate]]
|* [[diamond]]
|* [[ea-utils]]
|* [[ensembl]]
|-
|* [[ETE]]
|* [[FASTQC and MultiQC]]
|* [[Archaeopteryx and Forester]]
|* [[GapFiller]]
|* [[GenomeTools]]
|* [[gubbins]]
|-
|* [[JBrowse]]
|* [[kallisto]]
|* [[kentUtils]]
|* [[last]]
|* [[lastz]]
|* [[macs2]]
|-
|* [[Mash]]
|* [[mega]]
|* [[meryl]]
|* [[MUMmer]]
|* [[NanoSim]]
|* [[nseq]]
|-
|* [[OrthoFinder]]
|* [[PASA]]
|* [[perl]]
|* [[PGAP]]
|* [[picard-tools]]
|* [[poRe]]
|-
|* [[poretools]]
|* [[prokka]]
|* [[pyrad]]
|* [[python]]
|* [[qualimap]]
|* [[quast]]
|-
|* [[qiime2]]
|* [[R]]
|* [[RAxML]]
|* [[Repeatmasker]]
|* [[Repeatmodeler]]
|* [[rnammer]]
|-
|* [[roary]]
|* [[RSeQC]]
|* [[samtools]]
|* [[Satsuma]]
|* [[sickle]]
|* [[SPAdes]]
|-
|* [[squid]]
|* [[sra-tools]]
|* [[srst2]]
|* [[SSPACE]]
|* [[stacks]]
|* [[Thor]]
|-
|* [[Tophat]]
|* [[trimmomatic]]
|* [[Trinity]]
|* [[t-coffee]]
|* [[Unicycler]]
|* [[velvet]]
|-
|* [[ViennaRNA]]
|}

= Queue Manager Tips =
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:
* [[Queue Manager Tips]]
* [[General Command-line Tips]]
* [[DRMAA for further Gridengine automation]]

= Data Examples =
* [[Two Eel Scaffolds]]

= Procedures =
(short sequence of tasks with a certain short-term goal, often, a simple script)
* [[Calculating coverage]]
* [[MinION Coverage sensitivity analysis]]

= Navigating genomic data websites=
* [[Patric]]
* [[NCBI]]
* [[IGSR/1000 Genomes]]

= Explanations=
* [[ITUcourse]]
* [[VCF]]
* [[Maximum Likelihood]]
* [[SNP Analysis and phylogenetics]]
* [[Normalization]]

= Pipelines =
(Workflow with specific end-goals)
* [[Trinity_Protocol]]
* [[STAR BEAST]]
* [[callSNPs.py]]
* [[pairwiseCallSNPs]]
* [[mapping.py]]
* [[Edgen RNAseq]]
* [[Miseq Prokaryote FASTQ analysis]]
* [[snpcallphylo]]
* [[Bottlenose dolphin population genomic analysis]]
* [[ChIP-Seq Top2 in Yeast]]
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]
* [[Bisulfite Sequencing]]
* [[microRNA and Salmo Salar]]

=Protocols=
(Extensive workflows with different with several possible end goals)
* [[Synthetic Long reads]]
* [[MinION (Oxford Nanopore)]]
* [[MinKNOW folders and log files]]
* [[Research Data Management]]
* [[MicroRNAs]]

= Tech Reviews =
* [[SWATH-MS Data Analysis]]

= Cluster Administration =
* [[Hardware Issues]]
* [[Admin Tips]]
* [[RedHat]]
* [[Globus_gridftp]]
* [[Galaxy Setup]]
* [[Son of Gridengine]]
* [[Blas Libraries]]
* [[CMake]]
* [[Users and Groups]]
* [[Installing software on marvin]]
* [[emailing]]
* [[biotime machine]]
* [[SCAN-pc laptop]]
* [[node1 issues]]
* [[6TB storage expansion]]
* [[SAN relocation task]]
* [[Home directories max-out incident 28.11.2016]]
* [[Frontend Restart]]
* [[environment-modules]]
* [[H: drive on cluster]]
* [[Incident: Can't connect to BerkeleyDB]]
* [[Bioinformatics Wordpress Site]]
* [[Backups]]
* [[Python DRMAA]]
* [[SAN disconnect incident 10.01.2017]]
* [[Memory repair glitch 16.02.2017]]
* [[node9 network failure incident 16-20.03.2017]]
* [[Incorrect rebooting of marvin 19.09.2017]]

= Courses =

==I2U4BGA==
* [[Original schedule]]
* [[New schedule]]
* [[Actual schedule]]
* [[Course itself]]
* [[Biolinux Source course]]
* [[Directory Organization Exercise]]
* [[Glossary]]
* [[Key Bindings]]
* [[one-liners]]
* [[Cheatsheets]]
* [[Links]]
* [[pandoc modified manual]]
* [[Command Line Exercises]]

= hdi2u =

The half-day linux course held on 20th April. Modified version of I2U4BGA.

* [[hdi2u_intro]]
* [[hdi2u_commandbased_exercises]]
* [[hdi2u_dirorg_exercise]]
* [[hdi2u_rendertotsv_exercise]]

= RNAseq for DGE =
* [[Theoretical background]]
* [[Quality Control and Preprocessing]]
* [[Mapping to Reference]]
* [[Mapping Quality Exercise]]
* [[Key Aspects of using R]]
* [[Estimating Gene Count Exercise]]
* [[Differential Expression Exercise]]
* [[Functional Analysis Exercise]]

= Introduction to Unix 2017 =
* [[Introduction_to_Unix_2017]]

==Templates==
* [[edgenl2g]]

Environment-modules

2018-01-29T18:00:44Z

Rf:

= Introduction =

Environment modules, to give them their full proper name, are quite and old package, but still extremely useful in cluster environments.

= Aspects =
* the manually compiled version of environment-modules in marvin is found in /usr/local/Modules. It is a little buggy. Principally it cannot unload modules from with a module file. Module unloading must therefore be carried manually.
* default modules are set in the following file
/usr/local/Modules/3.2.10/init/.bashrc

= Notes =
* Environment modules actually do alot of work when unloading a certain module. All its dependent modules are also unloaded.

= Script running trick =

There is a trick, which is ugly and does not work very well, bu enables a script to be run when a modules in loaded. The required line in the module file is:

puts stdout "source /usr/local/Modules/modulefiles/tools/cd-hit/gitv0_5acf038/thisis.sh"

While this does indeed run the script, it does also appears to corrupt the environment a little, so that "module list" suddenly throws and error of the following sort:

Currently Loaded Modulefiles:
ModuleCmd_List.c(146):FATAL:996: The environment variables LOADEDMODULES and _LMFILES_ have inconsistent lengths.

The only way to get out of this is to log out of the terminal session (or screen/tmux window)

Users and Groups

2018-01-29T14:06:30Z

Rf:

= Introduction=

Some, though not all, of the tips here are for setting up users and groups.

The tool of choice is smbldap.

= Usage =
==Users==
* To create a new user(s)
Root has a script in bin/creasu.sh, so as root:
sh bin/creasu.sh <user> <user1> <user2>
will create groups, accounts, home folder and all relevant files into the new home folder.
Then you need to setup passwords with:
smbldap-passwd <user>
for each of the users.

Then setup an ssh key for logging into the nodes.

Then, as root user, login a user via
su - <newuserid>

and execute
ssh-keygen

and just accept all the suggestions.
.ssh/id_rsa and .ssh/id_rsa.pub, then get created.

then
cp .ssh/id_rsa.pub .ssh/authorized_keys

and
chmod 600 .ssh/authorized_keys

then ssh node1 should log in to node1 without password (no need to test other nodes).

==Groups==
* To create a new group
smbldap-groupadd -a <newgrpname>

* To add users to a certain group (note that this seems to take some time to propagate, as well as only working on fresh logins)
smbldap-groupmod -m <list,of,users> <targetgroup>

MicroRNA and Salmo Salar

2018-01-26T13:37:11Z

Rf:

= Usage of Cluster=
* [[Cluster Manual]]
* [[Why a Queue Manager?]]
* [[Available Software]]

= Documented Programs =

The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.

{|style="width:85%"
|* [[abacas]]
|* [[albacore]]
|* [[ariba]]
|* [[aspera]]
|* [[assembly-stats]]
|* [[augustus]]
|-
|* [[BamQC]]
|* [[bamtools]]
|* [[banjo]]
|* [[bcftools]]
|* [[bedtools]]
|* [[bgenie]]
|-
|* [[BLAST]]
|* [[Blat]]
|* [[blast2go: b2g4pipe]]
|* [[bowtie]]
|* [[bowtie2]]
|* [[bwa]]
|-
|* [[BUSCO]]
|* [[CAFE]]
|* [[canu]]
|* [[cd-hit]]
|* [[cegma]]
|* [[clustal]]
|-
|* [[cramtools]]
|* [[deeptools]]
|* [[detonate]]
|* [[diamond]]
|* [[ea-utils]]
|* [[ensembl]]
|-
|* [[ETE]]
|* [[FASTQC and MultiQC]]
|* [[Archaeopteryx and Forester]]
|* [[GapFiller]]
|* [[GenomeTools]]
|* [[gubbins]]
|-
|* [[JBrowse]]
|* [[kallisto]]
|* [[kentUtils]]
|* [[last]]
|* [[lastz]]
|* [[macs2]]
|-
|* [[Mash]]
|* [[mega]]
|* [[meryl]]
|* [[MUMmer]]
|* [[NanoSim]]
|* [[nseq]]
|-
|* [[OrthoFinder]]
|* [[PASA]]
|* [[perl]]
|* [[PGAP]]
|* [[picard-tools]]
|* [[poRe]]
|-
|* [[poretools]]
|* [[prokka]]
|* [[pyrad]]
|* [[python]]
|* [[qualimap]]
|* [[quast]]
|-
|* [[R]]
|* [[RAxML]]
|* [[Repeatmasker]]
|* [[Repeatmodeler]]
|* [[rnammer]]
|* [[roary]]
|-
|* [[RSeQC]]
|* [[samtools]]
|* [[Satsuma]]
|* [[sickle]]
|* [[SPAdes]]
|* [[squid]]
|-
|* [[sra-tools]]
|* [[srst2]]
|* [[SSPACE]]
|* [[stacks]]
|* [[Thor]]
|* [[Tophat]]
|-
|* [[trimmomatic]]
|* [[Trinity]]
|* [[t-coffee]]
|* [[Unicycler]]
|* [[velvet]]
|* [[ViennaRNA]]
|}

= Queue Manager Tips =
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:
* [[Queue Manager Tips]]
* [[General Command-line Tips]]
* [[DRMAA for further Gridengine automation]]

= Data Examples =
* [[Two Eel Scaffolds]]

= Procedures =
(short sequence of tasks with a certain short-term goal, often, a simple script)
* [[Calculating coverage]]
* [[MinION Coverage sensitivity analysis]]

= Navigating genomic data websites=
* [[Patric]]
* [[NCBI]]
* [[IGSR/1000 Genomes]]

= Explanations=
* [[ITUcourse]]
* [[VCF]]
* [[Maximum Likelihood]]
* [[SNP Analysis and phylogenetics]]
* [[Normalization]]

= Pipelines =
(Workflow with specific end-goals)
* [[Trinity_Protocol]]
* [[STAR BEAST]]
* [[callSNPs.py]]
* [[pairwiseCallSNPs]]
* [[mapping.py]]
* [[Edgen RNAseq]]
* [[Miseq Prokaryote FASTQ analysis]]
* [[snpcallphylo]]
* [[Bottlenose dolphin population genomic analysis]]
* [[ChIP-Seq Top2 in Yeast]]
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]
* [[Bisulfite Sequencing]]
* [[microRNA and Salmo Salar]]

=Protocols=
(Extensive workflows with different with several possible end goals)
* [[Synthetic Long reads]]
* [[MinION (Oxford Nanopore)]]
* [[MinKNOW folders and log files]]
* [[Research Data Management]]
* [[MicroRNAs]]

= Tech Reviews =
* [[SWATH-MS Data Analysis]]

= Cluster Administration =
* [[Hardware Issues]]
* [[Admin Tips]]
* [[RedHat]]
* [[Globus_gridftp]]
* [[Galaxy Setup]]
* [[Son of Gridengine]]
* [[Blas Libraries]]
* [[CMake]]
* [[Users and Groups]]
* [[Installing software on marvin]]
* [[emailing]]
* [[biotime machine]]
* [[SCAN-pc laptop]]
* [[node1 issues]]
* [[6TB storage expansion]]
* [[SAN relocation task]]
* [[Home directories max-out incident 28.11.2016]]
* [[Frontend Restart]]
* [[environment-modules]]
* [[H: drive on cluster]]
* [[Incident: Can't connect to BerkeleyDB]]
* [[Bioinformatics Wordpress Site]]
* [[Backups]]
* [[Python DRMAA]]
* [[SAN disconnect incident 10.01.2017]]
* [[Memory repair glitch 16.02.2017]]
* [[node9 network failure incident 16-20.03.2017]]
* [[Incorrect rebooting of marvin 19.09.2017]]

= Courses =

==I2U4BGA==
* [[Original schedule]]
* [[New schedule]]
* [[Actual schedule]]
* [[Course itself]]
* [[Biolinux Source course]]
* [[Directory Organization Exercise]]
* [[Glossary]]
* [[Key Bindings]]
* [[one-liners]]
* [[Cheatsheets]]
* [[Links]]
* [[pandoc modified manual]]
* [[Command Line Exercises]]

= hdi2u =

The half-day linux course held on 20th April. Modified version of I2U4BGA.

* [[hdi2u_intro]]
* [[hdi2u_commandbased_exercises]]
* [[hdi2u_dirorg_exercise]]
* [[hdi2u_rendertotsv_exercise]]

= RNAseq for DGE =
* [[Theoretical background]]
* [[Quality Control and Preprocessing]]
* [[Mapping to Reference]]
* [[Mapping Quality Exercise]]
* [[Key Aspects of using R]]
* [[Estimating Gene Count Exercise]]
* [[Differential Expression Exercise]]
* [[Functional Analysis Exercise]]

= Introduction to Unix 2017 =
* [[Introduction_to_Unix_2017]]

==Templates==
* [[edgenl2g]]

SAN relocation task

2018-01-25T15:44:41Z

Rf:

= Introduction =

During Dec 2017 and Jan 2018, Ally Kirk rang up to say IT Services were doing some relocation on their SAN which is where marvin's STORAGE volume is located. The whole process is transparent, but it would be useful to have been supervise thigns operationally, just in case anything went wrong.

= Tasks =

Pretty similar to any changes that may occur on this volume:
* scan SCSI
* use multipath -ll to see what need device some up.

= Read-outs prior to change-over =

root@marvin:/storage/home/users/ramon > multipath -ll
mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 6:0:0:1 sdd 8:48 active ready running
|- 7:0:6:1 sdf 8:80 active ready running
|- 6:0:1:1 sde 8:64 active ready running
|- 7:0:7:1 sdg 8:96 active ready running
|- 6:0:4:1 sdh 8:112 active ready running
|- 7:0:1:1 sdk 8:160 active ready running
|- 6:0:5:1 sdi 8:128 active ready running
`- 7:0:0:1 sdj 8:144 active ready running
root@marvin:/storage/home/users/ramon > multipath -l
mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 6:0:0:1 sdd 8:48 active undef unknown
|- 7:0:6:1 sdf 8:80 active undef unknown
|- 6:0:1:1 sde 8:64 active undef unknown
|- 7:0:7:1 sdg 8:96 active undef unknown
|- 6:0:4:1 sdh 8:112 active undef unknown
|- 7:0:1:1 sdk 8:160 active undef unknown
|- 6:0:5:1 sdi 8:128 active undef unknown
`- 7:0:0:1 sdj 8:144 active undef unknown

== Things to check ==

How are the devices looking? Go to
/sys/block/sdg/device
for sdg (similarly for sdd, sde, sdf)

However, this exercise didn't really reveal anything

= After the change-over =

mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 6:0:0:1 sdd 8:48 failed faulty running
|- 7:0:6:1 sdf 8:80 failed faulty running
|- 6:0:1:1 sde 8:64 failed faulty running
|- 7:0:7:1 sdg 8:96 failed faulty running
|- 6:0:4:1 sdh 8:112 active ready running
|- 7:0:1:1 sdk 8:160 active ready running
|- 6:0:5:1 sdi 8:128 active ready running
`- 7:0:0:1 sdj 8:144 active ready running

Because all eight of these devices were

active ready running

previously, I wrote to Ally with my understanding of the process:

"SCSI id numbers coincide with previous scan output, new device names. Though a bit clueless about the process, it looks to me as though both sets of paths (4 devices each) were working simultaneously, and what happened was, in the changeover, the first set was pulled and we're seeing the second set stand on its own two feet (so to speak)."

He verified this, so that "failed faulty running" is entirely expected, as the old SAN multipaths are withdrawn.

SAN relocation task

2018-01-25T14:33:35Z

Rf:

= Introduction =

During Dec 2017 and Jan 2018, Ally Kirk rang up to say IT Services were doing some relocaation on their SAN which is where marvin's STORAGE volume is located.

= Tasks =

Pretty similar to any changes that may occur on this volume: scan SCSI!

= Read-outs =

root@marvin:/storage/home/users/ramon > multipath -ll
mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 6:0:0:1 sdd 8:48 active ready running
|- 7:0:6:1 sdf 8:80 active ready running
|- 6:0:1:1 sde 8:64 active ready running
|- 7:0:7:1 sdg 8:96 active ready running
|- 6:0:4:1 sdh 8:112 active ready running
|- 7:0:1:1 sdk 8:160 active ready running
|- 6:0:5:1 sdi 8:128 active ready running
`- 7:0:0:1 sdj 8:144 active ready running
root@marvin:/storage/home/users/ramon > multipath -l
mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 6:0:0:1 sdd 8:48 active undef unknown
|- 7:0:6:1 sdf 8:80 active undef unknown
|- 6:0:1:1 sde 8:64 active undef unknown
|- 7:0:7:1 sdg 8:96 active undef unknown
|- 6:0:4:1 sdh 8:112 active undef unknown
|- 7:0:1:1 sdk 8:160 active undef unknown
|- 6:0:5:1 sdi 8:128 active undef unknown
`- 7:0:0:1 sdj 8:144 active undef unknown

= Things to check =

How are the devices looking? Go to
/sys/block/sdg/device
for sdg (similarly for sdd, sde, sdf)

However, this exercise didn't really reveal anything

= After the "changeover" =

mpathd (36000d3100051f800000000000000016e) dm-6 COMPELNT,Compellent Vol
size=62T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 6:0:0:1 sdd 8:48 failed faulty running
|- 7:0:6:1 sdf 8:80 failed faulty running
|- 6:0:1:1 sde 8:64 failed faulty running
|- 7:0:7:1 sdg 8:96 failed faulty running
|- 6:0:4:1 sdh 8:112 active ready running
|- 7:0:1:1 sdk 8:160 active ready running
|- 6:0:5:1 sdi 8:128 active ready running
`- 7:0:0:1 sdj 8:144 active ready running

Because all eight of these devices were

active ready running

previously, I wrote to Ally with my understanding of the process:

"SCSI id numbers coincide with previous scan output, new device names. Though a bit clueless about the process, it looks to me as though both sets of paths (4 devices each) were working simultaneously, and what happened was, in the changeover, the first set was pulled and we're seeing the second set stand on its own two feet (so to speak)."

SAN relocation task

2018-01-25T14:10:27Z

Rf: Created page with "= Introduction = During Dec 2017 and Jan 2018, Ally Kirk rang up to say IT Services were doing some relocaation on their SAN which is where marvin's STORAGE volume is located..."

Main Page

2018-01-25T14:07:51Z

Rf:

= Usage of Cluster=
* [[Cluster Manual]]
* [[Why a Queue Manager?]]
* [[Available Software]]

= Documented Programs =

The following can be seen as extra notes referring to these programs usage on the marvin cluster, with an emphais on example use-cases. Most, if not all, will have their own special websites, with more detailed manuals and further information.

{|style="width:85%"
|* [[abacas]]
|* [[albacore]]
|* [[ariba]]
|* [[aspera]]
|* [[assembly-stats]]
|* [[augustus]]
|-
|* [[BamQC]]
|* [[bamtools]]
|* [[banjo]]
|* [[bcftools]]
|* [[bedtools]]
|* [[bgenie]]
|-
|* [[BLAST]]
|* [[Blat]]
|* [[blast2go: b2g4pipe]]
|* [[bowtie]]
|* [[bowtie2]]
|* [[bwa]]
|-
|* [[BUSCO]]
|* [[CAFE]]
|* [[canu]]
|* [[cd-hit]]
|* [[cegma]]
|* [[clustal]]
|-
|* [[cramtools]]
|* [[deeptools]]
|* [[detonate]]
|* [[diamond]]
|* [[ea-utils]]
|* [[ensembl]]
|-
|* [[ETE]]
|* [[FASTQC and MultiQC]]
|* [[Archaeopteryx and Forester]]
|* [[GapFiller]]
|* [[GenomeTools]]
|* [[gubbins]]
|-
|* [[JBrowse]]
|* [[kallisto]]
|* [[kentUtils]]
|* [[last]]
|* [[lastz]]
|* [[macs2]]
|-
|* [[Mash]]
|* [[mega]]
|* [[meryl]]
|* [[MUMmer]]
|* [[NanoSim]]
|* [[nseq]]
|-
|* [[OrthoFinder]]
|* [[PASA]]
|* [[perl]]
|* [[PGAP]]
|* [[picard-tools]]
|* [[poRe]]
|-
|* [[poretools]]
|* [[prokka]]
|* [[pyrad]]
|* [[python]]
|* [[qualimap]]
|* [[quast]]
|-
|* [[R]]
|* [[RAxML]]
|* [[Repeatmasker]]
|* [[Repeatmodeler]]
|* [[rnammer]]
|* [[roary]]
|-
|* [[RSeQC]]
|* [[samtools]]
|* [[Satsuma]]
|* [[sickle]]
|* [[SPAdes]]
|* [[squid]]
|-
|* [[sra-tools]]
|* [[srst2]]
|* [[SSPACE]]
|* [[stacks]]
|* [[Thor]]
|* [[Tophat]]
|-
|* [[trimmomatic]]
|* [[Trinity]]
|* [[t-coffee]]
|* [[Unicycler]]
|* [[velvet]]
|* [[ViennaRNA]]
|}

= Queue Manager Tips =
A cluster is a shared resource with different users running different types of analyses. Nearly all clusters use a piece of software called a queue manager to fairly share out the resource. The queue manager on marvin is called Grid Engine, and it has several commands available, all beginning with '''q''' and with '''qsub''' being the most commonly used as it submits a command via a jobscript to be processed. Here are some tips:
* [[Queue Manager Tips]]
* [[General Command-line Tips]]
* [[DRMAA for further Gridengine automation]]

= Data Examples =
* [[Two Eel Scaffolds]]

= Procedures =
(short sequence of tasks with a certain short-term goal, often, a simple script)
* [[Calculating coverage]]
* [[MinION Coverage sensitivity analysis]]

= Navigating genomic data websites=
* [[Patric]]
* [[NCBI]]
* [[IGSR/1000 Genomes]]

= Explanations=
* [[ITUcourse]]
* [[VCF]]
* [[Maximum Likelihood]]
* [[SNP Analysis and phylogenetics]]
* [[Normalization]]

= Pipelines =
(Workflow with specific end-goals)
* [[Trinity_Protocol]]
* [[STAR BEAST]]
* [[callSNPs.py]]
* [[pairwiseCallSNPs]]
* [[mapping.py]]
* [[Edgen RNAseq]]
* [[Miseq Prokaryote FASTQ analysis]]
* [[snpcallphylo]]
* [[Bottlenose dolphin population genomic analysis]]
* [[ChIP-Seq Top2 in Yeast]]
* [[ChIP-Seq Top2 in Yeast 12.09.2017]]
* [[ChIP-Seq Top2 in Yeast 07.11.2017]]
* [[Bisulfite Sequencing]]

=Protocols=
(Extensive workflows with different with several possible end goals)
* [[Synthetic Long reads]]
* [[MinION (Oxford Nanopore)]]
* [[MinKNOW folders and log files]]
* [[Research Data Management]]
* [[MicroRNAs]]

= Tech Reviews =
* [[SWATH-MS Data Analysis]]

= Cluster Administration =
* [[Hardware Issues]]
* [[Admin Tips]]
* [[RedHat]]
* [[Globus_gridftp]]
* [[Galaxy Setup]]
* [[Son of Gridengine]]
* [[Blas Libraries]]
* [[CMake]]
* [[Users and Groups]]
* [[Installing software on marvin]]
* [[emailing]]
* [[biotime machine]]
* [[SCAN-pc laptop]]
* [[node1 issues]]
* [[6TB storage expansion]]
* [[SAN relocation task]]
* [[Home directories max-out incident 28.11.2016]]
* [[Frontend Restart]]
* [[environment-modules]]
* [[H: drive on cluster]]
* [[Incident: Can't connect to BerkeleyDB]]
* [[Bioinformatics Wordpress Site]]
* [[Backups]]
* [[Python DRMAA]]
* [[SAN disconnect incident 10.01.2017]]
* [[Memory repair glitch 16.02.2017]]
* [[node9 network failure incident 16-20.03.2017]]
* [[Incorrect rebooting of marvin 19.09.2017]]

= Courses =

==I2U4BGA==
* [[Original schedule]]
* [[New schedule]]
* [[Actual schedule]]
* [[Course itself]]
* [[Biolinux Source course]]
* [[Directory Organization Exercise]]
* [[Glossary]]
* [[Key Bindings]]
* [[one-liners]]
* [[Cheatsheets]]
* [[Links]]
* [[pandoc modified manual]]
* [[Command Line Exercises]]

= hdi2u =

The half-day linux course held on 20th April. Modified version of I2U4BGA.

* [[hdi2u_intro]]
* [[hdi2u_commandbased_exercises]]
* [[hdi2u_dirorg_exercise]]
* [[hdi2u_rendertotsv_exercise]]

= RNAseq for DGE =
* [[Theoretical background]]
* [[Quality Control and Preprocessing]]
* [[Mapping to Reference]]
* [[Mapping Quality Exercise]]
* [[Key Aspects of using R]]
* [[Estimating Gene Count Exercise]]
* [[Differential Expression Exercise]]
* [[Functional Analysis Exercise]]

= Introduction to Unix 2017 =
* [[Introduction_to_Unix_2017]]

==Templates==
* [[edgenl2g]]

RedHat

2018-01-24T14:09:19Z

Rf:

The StABU cluster subscribes to Red Hat, and uses version 6, which is one behind the latest version, version 7.

As of November 2017, all nodes were on the latest version of version 6.9 "Santiago".

Being one version behind is typicaly not a problem, as it continues to be fully supported. However, there are software packages that like to use the bleeding edge, and there are admitted problems installing these programs.

= Installation of devtoolset-2 =

There are two ways apparently via RHN Classic or RH Subscription Management, and we choose the latter because there is a useful command-line tool for it.

To install on a node:
ssh node4 'subscription-manager subscribe --pool=8a85f9814ed67b98014eda13ed983c7d'

Note here that we have previously been able to verify the pool ID. Documentation for this can be found at

[https://access.redhat.com/documentation/en-US/Red_Hat_Developer_Toolset/1/html/User_Guide/sect-Red_Hat_Developer_Toolset-Subscribe.html Click here or google Red Hat Developer Toolset]

This will also enable the appropriate repos for you (though *-source-rpms and *-debug-rpms may be left out).

= Problems =

== outdated repomd.xml warnings ==
You can get rid of them (i.e. update the repomd.xml) by:
yum clean all
yum update yum*

== PYCURL error 7 ==

Note that this could easily occur is the squid server on marvin has failed. Make sure you at least try and restart the squid server on marvin.

[https://access.redhat.com/solutions/291643 This link] has a fairly exhaustive for checking errors using '''tcpdump''' and '''nc'''.

= RedHat service types =

* A service is usually under one or more of these.
* Some of them are a little old.
* it's probably best to use subscription manager.
man subscription-manager
will work.

The wiki entry on problems with node1 can also be helpful: [http://stab.st-andrews.ac.uk/wiki/index.php/Node1_issues node 1 issues]

In November, marvin was migrated from RHN to RHSM. This apparently looked difficult to do, but then it turned out that RHN was being decomissioned, so there was no choice. In the end, it was not such a difficult thing to do with the
rhn-migrate-classic-to-rhsm

command.

= ssl connection errors =

The nodes are connected to the internet via the squid proxy service running on marvin. The proxy was installed due to pressure from IT Services, as the NAT option has been discredited by them. In Miguel-era StABU, the NAT would be set up during updates or short temporary periods (a hour or two) and then closed. However, this would mean that generally, the nodes woul dnot have access to internet.

The Squid proxy was installed so that nodes would have full internet access and to satisfy IT Services. The relevant IP and port numbers are

172.25.64.1:3128

This has always worked quite well since 2016 when it was installed. And the big September 2017 update went through without a hitch. However in November 2017, it started failing with SSL connection errors. Not to EPEL or anything that was not REd Hat, but criticially, to

https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/optional/os/repodata/repomd.xml

This is nasty because it's the most important connection. A longish, but somewhat raw troubleshooting page from Red Hat is up at [https://access.redhat.com/solutions/189533 this link.

One vital test is working out whether you can get through to Redhat's SSL https servers. Try this with

curl -v --proxy 172.25.64.1:3128 https://subscription.rhn.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem

or, because it seems to amount to the same thing:

curl -v -k --proxy 172.25.64.1:3128 https://subscription.rhn.redhat.com/subscription/ --cacert /etc/rhsm/ca/redhat-uep.pem

Output from this, with this problem still present is

* Trying 172.25.64.1...
* TCP_NODELAY set
* Connected to 172.25.64.1 (172.25.64.1) port 3128 (#0)
* Establish HTTP proxy tunnel to subscription.rhn.redhat.com:443
> CONNECT subscription.rhn.redhat.com:443 HTTP/1.1
> Host: subscription.rhn.redhat.com:443
> User-Agent: curl/7.54.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
<
* Proxy replied OK to CONNECT request
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/rhsm/ca/redhat-uep.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* Server certificate:
* subject: C=US; ST=North Carolina; O=Red Hat, Inc.; OU=Red Hat Network; CN=subscription.rhn.redhat.com; emailAddress=customerservice@redhat.com
* start date: May 18 16:24:18 2017 GMT
* expire date: May 18 16:24:18 2019 GMT
* issuer: C=US; ST=North Carolina; O=Red Hat, Inc.; OU=Red Hat Network; CN=Red Hat Entitlement Operations Authority; emailAddress=ca-support@redhat.com
* SSL certificate verify ok.
> GET /subscription/ HTTP/1.1
> Host: subscription.rhn.redhat.com
> User-Agent: curl/7.54.1
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: application/json
< Server: BigIP
* HTTP/1.0 connection set to keep alive!
< Connection: Keep-Alive
< Content-Length: 1224
<
[ {"rel":"","href":"/"},{"rel":"deleted_consumers","href":"/deleted_consumers"},{"rel":"consumertypes","href":"/consumertypes"},{"rel":"crl","href":"/crl"},{"rel":"entitlements","href":"/entitlements"},{"rel":"guestids","href":"/consumers/{consumer_uuid}/guestids"},
{"rel":"pools","href":"/pools"},{"rel":"products","href":"/products"},{"rel":"activation_keys","href":"/activation_keys"},{"rel":"statistics/generate","href":"/statistics/generate"},{"rel":"distributor_versions","href":"/distributor_versions"},{"rel":"owners","href":"/owners"},
{"rel":"cdn","href":"/cdn"},{"rel":"roles","href":"/roles"},{"rel":"events","href":"/events"},{"rel":"consumers","href":"/consumers"},{"rel":"atom","href":"/atom"},{"rel":"content_overrides","href":"/consumers/{consumer_uuid}/content_overrides"},{"rel":"rules","href":"/rules"},
{"rel":"content","href":"/content"},{"rel":"hypervisors","href":"/hypervisors"},{"rel":"users","href":"/users"},{"rel":"migrations","href":"/migrations"},{"rel":"status","href":"/status"},{"rel":"admin"* Connection #0 to host 172.25.64.1 left intact
,"href":"/admin"},{"rel":"serials","href":"/serials"},{"rel":"jobs","href":"/jobs"},{"rel":"subscriptions","href":"/subscriptions"} ,{"href": "/consumers/{consumer_uuid}/packages","rel": "packages"} ]

This all looks really positive. Is the error bona-fide? Well, it does refer to optional-rpms only? Yes, in fact, this is the nature of the problem. OK, disable it

subscription-manager repos --disable rhel-6-server-optional-rpms

However, this gets the following error:

/var/log/rhsm/rhsm.log

== Work around ==

The Redhat webpage helping with this issue mentioned above (#189533) rambles on a bit, it's not quite clear how to follow it. Problem is, it's often quoted as the go-to webpage for thes e issues! In any case, none of its checks gave any joy. Over and over again, "yum check-update" would fail continuously with the Error 14 no ssl connection.

Because updating the nodes is rather import, the workaroudn consists of
* opening up the NAT temporarily from marvin
sudo iptables -t nat -A POSTROUTING -o bond0.506 -j MASQUERADE
* on each node, commenting out proxy IP and port number in the
/etc/rhsm/rhsm.conf
file
* updating yum as usual, and rebooting node
* (optional because root cause is not resolved) uncomment the proxy IP and PORT settings in /etc/rhsm/rhsm.conf.
* If yu have finished with the nodes, then remove the NAT rule on marvin:
sudo iptables -t nat -D POSTROUTING 1
(note this command assumes the rule to be the only rule and numbered "1")

= Considering upgrading to RHEL7 =

A link for this is [https://access.redhat.com/solutions/637583 solution number 637583]

== Thoughts ==
* This could be one heck of an epic upgrade, especially as regards the manually installed software programs (150 or them, some of them very big)
* Most likely it will be the different C library that will cause problems
* Library and linking problems are big ... fear them!

== Test with node10 ==

Following solution 637583, we install the upgrade assessemnt software and and run

preupg

which, happily enough, will not install the upgrade but report on what problems are likely to be encountered.

Note, that it takes a while to run, may be 30 mins.