Difference between revisions of "Biolinux Source course"

From wiki
Jump to: navigation, search
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
Part One: Introduction to the Bio-Linux 8 System
+
= Starting with an empty open terminal =
Logging in and exploring the Bio-Linux desktop
 
You can log into your Bio-Linux machine locally or over the network, on a fully installed system or a Virtual
 
Machine or on a system running Live from a USB memory stick or a DVD.
 
These course notes are written from the perspective of someone running the Live version of the system – that
 
is, having booted a PC directly from a USB memory stick and selected "Try Bio-Linux". The main
 
differences for people working on an installed system will be the name of the account you are logged into
 
and what privileges that particular user account has. For example, the user of the Live system always has full
 
administrative privileges. So don't worry if you find small differences between what is described here and
 
what you see on your system.
 
Please refer to our on-line document about various ways you can set up a Bio-Linux system:
 
http://environmentalomics.org/bio-linux-installation
 
If you are booting the machine from a DVD or a USB memory stick, when prompted, select
 
Option 1: Try Bio-Linux
 
After the system has started up, you will see the Bio-Linux desktop (Figure 1).
 
  
Figure 1: A view of the Bio-Linux 8 desktop
+
Open a new terminal by clicking the icon in the dash --->
 +
Type the following at the command prompt and press the enter key :
  
1
+
tar -xz -f bioinf_files.tar.gz
  
�There are three icons on the desktop
+
This command uncompresses and unpacks the contents of the tar file into your current working directory,
+
which in this case is your home folder. You should then see a new prompt, just like this:
  
Install Bio-Linux 8
+
If you see an error, try typing the command again, making sure it is exactly as shown above including
 +
spaces, hyphens, underscores, etc. If the error says "No such file or directory " then check you really did
 +
copy the file in step (b) above. You can confirm the extraction worked by looking in the file browser or
 +
using the ls command.
  
+
== Re-visiting the command above ==
 +
Press the up arrow key while in the terminal. The previous command should re-appear for you to edit.  You can move the cursor left and right using the keyboard but don't try to move it with the mouse – that won't work.
  
Bio-Linux Documentation Opens a menu of links as follows:
+
* Edit the command by adding an extra 'v' righ after '-xz' so that the full command reads:
 +
tar -xzv -f bioinf_files.tar.gz
 +
Hit the enter key to run it. You don't need to scroll the cursor back the end before you do this. What is the result this time?
  
+
The letters after the hyphens are parameters of the tar command: x means “unpack/extract”, the z means
 +
“the file should be uncompressed with gzip”, the f indicates the file to unpack, and the v you just added
 +
means "be verbose". Therefore on this occasion you should have seen a list of the files being unpacked.
 +
This is a common behavior for many Linux commands. If the command runs successfully without errors
 +
it says nothing and just goes right back to the prompt. If you want the command to tell you what it is
 +
doing, adding -v makes it verbose, otherwise you may assume that "no news is good news".
 +
The use of the cursor keys to re-visit commands is a major time-saver in the terminal and you must get in
 +
the habit of doing this. The other major time-saver is Tab completion which we will come to soon.
  
On the Live System only – click this icon to start the Bio-Linux installer
+
== Removing the compressed tarball ==
  
+
The unpacked files that you will be working with in this tutorial are now in a directory called bioinf_files.
 +
You can remove the compressed tar file now if you wish. Again, this can be done via the command line or
 +
using the graphical file browser but we'll stick with the command line version. More details about how to
 +
remove files from the system are covered in the Removing Files and Directories part of this tutorial.
 +
* Open a terminal window if you don't have one already.
 +
* Type the following into the terminal, then press Enter:
 +
rm bioinf_files.tar.gz
  
NEBC Homepage
+
Enter “y” to agree when you are asked if you wish to delete the file.
  
Opens the NEBC home page in a web browser
+
== Finding your way on the system ==
  
+
In Linux/Unix systems, documents are usually referred to as files, and file folders are referred to as directories.
  
User Guide
+
Your Bio-Linux file system can be thought of as a huge file folder (directory), inside of which are many other file folders (directories). Inside these there are more nested file folders (directories), and so on. As in the real world, where file folders can contain documents and other file folders, in Linux directories can contain files and other directories. The hierarchy of folders is called the directory tree.
  
Opens the Bio-Linux Userguide – a basic introduction to system admin
+
Your personal Home folder is one directory within the tree of directories that make up your Bio-Linux machine. In your account, you can create other directories, store data, run programs, etc. A graphical view of your home directory is available by clicking on the file cabinet Files icon in the Dash toolbar (Figure 5). This opens up a window that shows the files and directories in your Home. The full name of this folder on the system is /home/live, ie. a directory named after the login account, live, within the top-level directory named /home, but the graphical file browser just shows it as Home.
  
+
Linux enforces file permissions depending on the login account. By default on Bio-Linux, your account has the right to create, delete and edit files in your own Home folder, but not in other people’s accounts or in system directories. You can be given permission (or give yourself permission, if it's your system) to work on files in such areas, and some information on setting file permissions is given later in this course. Your system administrator or local IT support should be able to help you with sharing files if they are on a shared server.
  
Introductory Tutorial Opens the folder of Introductory Bio-Linux tutorials and data files
+
You can use the graphical file browser to explore directory areas on the machine, and to move around in your own files. It allows you to accomplish most typical file operation, including opening files and copying, moving or deleting files using drag and drop or copy/cut/paste. To view areas of the system outside your Home directory, click on Computer under Devices in the left hand pane to see the root directory of the system.
  
+
== Exercise 1-2 ==
  
Bioinformatics Docs Shows the NEBC Bio-Linux Bioinformatics Documentation System
+
* If you have not done so already, click on the filing cabinet Files icon near the top of the Dash
 +
* Double-click on the bioinf_files directory that you unpacked in Exercise 1-1, to view the contents
  
Sample Data
+
Investigate the options under the file browser menus. These appear on the bar at the very top of the screen.
  
Provides access to much sample data to help you in trying out new
+
* Click on the Computer icon in the left panel. This allows you to see the root directory – the base of the whole filesystem hierarchy.
 +
* Find the folder called home and double click on it.
  
software
+
You should see a single folder called live listed. Select this to get back to your Home folder. If you are not working on a live-booted system you should see a folder with your username, and other user folders may also listed. A lock symbol on a folder would inform you that you do not have permission to view the contents of that folder.
On the left of the screen you will see the Dash, which is used to launch and organize applications. The
 
dash is populated by a column of large button icons. The Dash Button at the top with the Ubuntu logo
 
brings up the main Dash panel to find files and applications (see below). The other icons are, by
 
default, from the top:
 
1.
 
  
Open your home folder
+
* The Root Folder
 +
The name of the base directory of the whole system, the one within which every file on the system is
 +
contained, is the root directory. It is referred to by a single forward slash “ / ”.
 +
When you work in the graphical file browser it shows your location relative to your Home folder, unless you
 +
are looking at files outside your Home in which case it shows the location relative to the root. You should
 +
have seen how the location changed as you browsed folders in exercise 1-2.
 +
Figure 6: Location path for Templates folder in File Browser view.
  
8. Shell Terminal
+
Your personal home folder (actually called live but labeled as Home), sits within the directory called home (with a small h), that contains homes for all users. This directory home is under the root directory, represented by a tiny picture of a disk in the graphical view or a single forward slash in the terminal. In other words, this information tells you where you are in system.
  
2.
+
The location of a file or directory within the system is its path. If you are asked for the full path or absolute path to a file, you need to provide a complete listing of all the directories traversed on the system to get to that file. That is, you need to give the full path from the root directory to that file. The path is written by starting with a forward slash “/” then listing the names of the directories you need to traverse in the system to find that file, with each directory name separated with another forward slash.
  
Launch Firefox web browser
+
To see the full path in the conventional format most command-line programs would expect you to provide,
 +
press Ctrl-L while viewing a File Browser window. You should see something like this:
  
9. Ubuntu Software Centre (find and install
+
Figure 7: Location in graphical file browser given in text; this is the the full
 +
path to the Templates folder in the home directory of the live user account.
  
3.
+
To summarize the syntax provided in Figures 9 and 10:
 +
/home
 +
/home/live
  
Launch Evolution mail reader
+
home is a directory located within the root directory
 +
live is a directory within the directory home which is within the root
 +
directory. This special directory will sometimes be shown as
  
4.
+
Home, with a capital H, because it is the home folder for the live user.
  
LibreOffice Writer word processor
+
As another example: the full path to the file capsall.fasta, in the bioinf_files directory within the home directory of the live user:
 +
/home/live/bioinf_files/capsall.fasta
  
10. System Settings and User Preferences
+
Often you can provide just the route from where you are on the system to where your file is; this is referred to as a relative path. For example, if you are working in your home directory, the relative path to the file mentioned above would be bioinf_files/capsall.fasta.
  
5.
+
= Keeping things organised =
  
LibreOffice Calc spreadsheet
+
Everyone knows it, but it's worth restating: if you start by creating a folder structure with meaningfully named subfolders, name your files so that the names indicate the contents (or follow some defined naming convention), and store your files in the right place, your life will be much, much easier!
  
11. Virtual Desktop Switcher
+
= Using the command shell =
  
6.
+
The real power of Linux/Unix systems is the command line.
  
LibreOffice Impress presentation editor
+
A list of common Linux commands is provided in Appendix D of this document for reference.
  
12. Disks and USB removable media
+
Many programs and facilities are available through graphical options on Linux, but all programs and facilities can be accessed by the command line, also known as the shell. Some tasks are easier, or more appropriately done using graphical interfaces. Equally though, other things are easier or more appropriately done using the command line. Obvious examples include when you need to work with large numbers of files or want to automate processes. First steps on the command line can be hard but the rewards are worth it (we promise!)
  
apps)
+
Access to the command line is done through a terminal window.
 +
You can open a new terminal by:
  
13. Rubbish Bin (deleted files area)
+
* clicking the middle button on the terminal icon on the Dash toolbar
On the top of the screen you will see the menu and panel bar (Figure 2).
+
or, going into an already open terminal and typing a command to open a second terminal:
Figure 2: The menu and panel bar, found at the top of the screen.
+
gnome-terminal &
  
If you open an application window, the name of the active application will appear in the left portion of this
+
= Anatomy of a Command =
bar. If you move the mouse over it, a context menu for the active window will appear (like on Apple Mac).
+
Linux/Unix commands usually take the form shown in Figure 11. You've already seen a good example in
The right portion of the bar has a panel of icons to control some system settings.
+
Exercise 1-1 part c.
From left to right, the things you see in the panel area above are:
+
command
1. Network monitor and setup (the icon shown
+
what I want to do
indicates WiFi is active – you may see others)
+
eg: tar
2. Keyboard selector (defaults to UK keyboard)
 
3. Battery monitor (on laptops only)
 
  
2
+
parameters
 +
how I want to do it
 +
-xvz -f
  
4. Audio volume control
+
arguments
5. Wall clock (click it for a calendar)
+
on what do I want to do it
6. System menu (includes access to system
+
bioinf_files.tar.gz
settings and options to lock screen, switch
 
user, shut down, etc.)
 
  
�Running applications
+
Figure 8: The Linux/Unix command line structure. Each part of a command is separated by
Clicking the Dash Button at the top left of the screen opens a panel where you can search for applications
+
one or more spaces.
and files on the system. This includes bioinformatics tools and any other applications you have installed.
 
Start typing either the application name or a keyword, or select the DNA icon at the bottom (circled in the
 
image) to see a list of bioinformatics tools and resources.
 
  
Figure 3: Searching for applications in the Dash
+
The first word you supply on the command line is interpreted by the system as a command; that is –
 
+
something the system should do or a program to be run. Items that appear after that on on the same line are
The applications found in the menu are by no means all the means all those found on the system. Most
+
separated by spaces. The additional input on the command line indicates to the system how the command
bioinformatics applications need to be run from the terminal as detailed at length in this tutorial.
+
should work. For example, what file you want the command to work on, or the format for the information
 
+
that should be returned to you.
Finding files and drives
+
Most commands have options available that will alter the way the command functions. You make use of
The file cabinet icon near the top of the Dash takes you directly to your Home folder.
+
these options by providing the command with parameters, some of which will take arguments. Examples in
 
+
the following sections should make it clear how this works. With some commands you don't need to issue
Figure 4: Your home folder
+
any parameters or arguments. Occasionally this is because there are none available, but usually this is
 
+
because the command will use default settings if nothing is specified.
3
+
If a command runs successfully, it will usually not report anything back to you, unless reporting to you was
 
+
the purpose of the command (eg. ls). If the command does not execute properly, you will see an error
�Your personal Desktop, and folders in your Home area called Documents, Pictures, Videos, etc. are listed.
+
message returned. Some of these messages are hard to decipher until you have a bit of Linux experience but
You can use these or else create your own folders as you wish.
+
ultimately they should tell you what has gone wrong.
The file browser provides convenient shortcuts to these directories in the left pane, even if you are viewing
+
Note: Items supplied on the command line separated by spaces are interpreted as individual pieces of
another folder in the main panel.
+
information for the system. For this reason, a filename with a space in it will be interpreted as two filenames
Devices recognized by your system such as the disk drives, CD/DVD devices, USB sticks, etc. are listed at
+
by default. How to get around this is is addressed in more detail later in the course.
the bottom of the left pane. Removable media can be ejected by clicking the icon next to the device name.
+
Note 2: The use of the ampersand in the previous example, gnome-terminal &, is explained in a few pages
Networks resources can be accessed through the Browse Network icon. This includes Windows network
+
time. You would not put an ampersand on the end of most shell commands.
shares using the CIFS protocol and files on other Bio-Linux machines if you can access them via the SFTP
 
protocol. Browsing regular FTP servers is also supported.
 
Note: The Dash also has a file and media finder, as seen on the previous page, selected by clicking the
 
Ubuntu button at the top left to bring up the Dash console and then selecting one of the little white icons
 
from along the bottom of the window.
 
 
 
Setting things up
 
The System settings icon
 
 
 
allows you to customise
 
 
 
and administer your system (Figure 6) in various ways.
 
The Personal area is used for customising a variety of
 
attributes relating to your personal preferences.
 
The Hardware and System areas allow you to do things such
 
as configuring hardware drivers, changing firewall settings,
 
administering users and groups, and managing the packages on
 
your system.
 
Other features - Virtual Desktops etc.
 
  
The icon that looks like this:
+
= Listing files in a directory =
 +
The command ls lists files in a directory.
 +
By default, the command will list the filenames of the files in your current working directory. When you first
 +
open a shell this is your home directory.
  
Figure 5: The System Settings Window
+
If you add a space followed by a –l (that is, a hyphen and a small letter L), after the ls command, it alters the behavior of the command: it will now list the files in your current directory, but with details about them including who owns them, what the size is, and what kind of file it is. Information about this is shown in
  
allows you to switch
+
Date and time
 +
modified
  
“virtual desktops”. Unlike Windows, Linux by default gives you access to multiple desktop areas. This
+
twilliams
allows you to have windows open for different things in different virtual desktops. For example, if you were
+
hybInfo.txt
working on writing an article, you could have programs relevant to that work open and visible via one of
+
targets_v1.txt
these desktops. Meanwhile, you could have programs related to sequence analysis open on another desktop,
+
targets_v2.txt
and so on. This is a great tool for keeping things organised during your working day. Clicking the icon will
 
zoom out to show an overview of all desktops. You can also switch quickly by holding down Ctrl+Alt and
 
tapping the arrow keys on the keyboard.
 
  
The Deleted Items Folder icon
+
Filename
  
(also commonly referred to as a Rubbish Bin or Trashcan) is the
+
Figure 9: The detailed output of the command ls when run with the -l flag
  
bottom icon the Dash. This is where files deleted in the file browser usually end up. This gives you a chance
+
== Exercise ==
to salvage them if you deleted them by mistake. Deleting files on the system is covered in more detail in the
+
a) Try browsing files in both the terminal and the graphical file browser:
Removing Files and Directories section of this tutorial.
+
* Open a new terminal by clicking the terminal icon
 +
 
 +
In the terminal, type the command ls. Compare what you see listed with what you see in the graphical
 +
representation of your Home directory.
 +
* Type the command ls –l and note the kind of information being provided and how it compares to the
 +
graphical representation of your files.
 +
* In the graphical File Browser, click on the List option under the View menu, and compare this
 +
information to that provided using the ls –l command.
 +
* In the console, type ls –l bioinf_files and also click on the bioinf_files folder in the graphical file
 +
browser and compare what you are seeing.
 +
* You can also use glob patterns to identify file names by pattern.
 +
*
 +
?
 +
[]
  
4
+
an asterisk means any string of characters
 +
a question mark means a single character
 +
square brackets can be used to designate a group of characters
  
�Exercise 1-1
+
More details about this are given in the Linux shorthand and shortcuts section below.
a) Exploring the desktop
 
Take some time to explore the desktop. Look at the options under each of the icons covered in the previous
 
section, and try the various subsections in the Dash console. Try clicking the icons on the desktop. Also try
 
using the right and middle mouse buttons when the mouse pointer is over the icons in the Dash and explore
 
the menus presented to you.
 
Try going to a different virtual desktop and starting up some windows/applications there. Try moving
 
windows off one desktop area and onto another.
 
  
b) Obtaining the example files for this tutorial
+
b) Try these commands that use wildcards to match multiple files:
The sample files referred to in this tutorial can be found on the system as a compressed package file. You'll
+
* List all the files in the directory bioinf_files. that start with the letters tes
need to copy and unpack them before proceeding.
+
ls bioinf_files/tes*
Copying the compressed file from the tutorials folder on the system
+
 
Double-click the Bio-Linux Documentation icon on the desktop
+
* List all the files in your directory that start with tes, and end in 1.embl, 2.embl or 3.embl
Open the Introductory Tutorial
+
ls bioinf_files/tes*[123].embl
Drag the bioinf_files.tar.gz file to the left and drop it over the word Home to copy it to your home
+
 
folder.
+
= Learning about Linux commands =
+
 
+
Most Linux commands have a manual page that provides information about the command and options that
+
can alter its behaviour. Many tasks can be made easier by using command options. A good rule of thumb is
 +
to ask yourself whether what you want to do is something many others may have wanted to do. If the answer
 +
is yes, then there may well be commands and options available to do that task.
 +
Linux manual pages are referred to as man pages. To open the man page for a particular command, you just
 +
need to type man followed by the name of the command you are interested in. To browse through a man
 +
page, use the cursor keys (↓ and ↑). To close the man page simply hit the q key on your keyboard.
 +
If you do not know the name of a command to use for a particular job, you can search using man –k
 +
followed by the type of thing you are trying to do. An example of this is in exercise 1-3, part c).
  
Note that a copy of this file can also be found online if you need it for some reason.
+
c)
http://nebc.nerc.ac.uk/downloads/courses/Bio-Linux/bioinf_files.tar.gz
+
* Look up the manual information for the ls command by typing the following in a terminal:
 +
man ls
  
c) Extracting the files from the compressed tarball
+
Skim through the man page. You can scroll forward using the up and down arrow keys on your
The file you just downloaded is referred to as a tar file or tarball. Tar is a utility similar to Winzip; it
+
keyboard. You can go forward a page by using the space bar, and move backwards a page by using the b
makes package of files. The extra .gz extension shows that the gzip method has been used to compress the
+
key.
tar file.
+
* * What does the -h option do? What about the -a option? What would running ls -lrt do?
Here are two equivalent options for how to unpack these files, one on the command line and one graphical.
 
Both should produce the same result.
 
Option 1 – extracting via the command line
 
 
 
  
Open a new terminal by clicking the icon in the dash --->
+
* Press the q key when you want to quit reading the man page.
Type the following at the command prompt and press the enter key :
 
tar -xz -f bioinf_files.tar.gz
 
  
This command uncompresses and unpacks the contents of the tar file into your current working directory,
+
* Try running ls using some of the options mentioned above.
which in this case is your home folder. You should then see a new prompt, just like this:
 
  
5
+
* Look up some programs with man pages with the keywords “list directory”
 +
man –k “list directory”
  
�(exercise 1-1 continued)
+
= Basic Linux tips for filenames =
If you see an error, try typing the command again, making sure it is exactly as shown above including
 
spaces, hyphens, underscores, etc. If the error says "No such file or directory " then check you really did
 
copy the file in step (b) above. You can confirm the extraction worked by looking in the file browser or
 
using the ls command.
 
Option 2 – extracting via a graphical interface
 
But don't use this version – we're trying to learn about the command line here!!
 
 
 
  
Open your Home Folder by clicking the file cabinet icon in the Dash.
+
* Linux does not deal well with spaces in filenames!
Click the right mouse button over the bioinf_files.tar.gz file and select Extract Here.
 
  
d) Re-visiting the command above
+
Or to be more precise, Linux itself deals perfectly well with spaces and all manner of special characters in filenames but many programs you'll want to run on Linux do not, and if you're talking about those files in the terminal you'll need to remember to quote them as described below. If you stick with letters, numbers, hyphens, underscores and full stops, you will be fine.
Press the up arrow key while in the terminal. The previous command should re-appear for you to edit.
 
You can move the cursor left and right using the keyboard but don't try to move it with the mouse – that
 
won't work.
 
Edit the command by adding an extra 'v' righ after '-xz' so that the full command reads:
 
tar -xzv -f bioinf_files.tar.gz
 
Hit the enter key to run it. You don't need to scroll the cursor back the end before you do this. What is
 
the result this time?
 
The letters after the hyphens are parameters of the tar command: x means “unpack/extract”, the z means
 
“the file should be uncompressed with gzip”, the f indicates the file to unpack, and the v you just added
 
means "be verbose". Therefore on this occasion you should have seen a list of the files being unpacked.
 
This is a common behavior for many Linux commands. If the command runs successfully without errors
 
it says nothing and just goes right back to the prompt. If you want the command to tell you what it is
 
doing, adding -v makes it verbose, otherwise you may assume that "no news is good news".
 
The use of the cursor keys to re-visit commands is a major time-saver in the terminal and you must get in
 
the habit of doing this. The other major time-saver is Tab completion which we will come to soon.
 
  
e) Removing the compressed tarball
+
Filenames with spaces in them are a common problem when transferring files to Linux from computers running Windows, or Mac operating systems. Normally the simplest thing is to rename the files before you work with them.
The unpacked files that you will be working with in this tutorial are now in a directory called bioinf_files.
 
You can remove the compressed tar file now if you wish. Again, this can be done via the command line or
 
using the graphical file browser but we'll stick with the command line version. More details about how to
 
remove files from the system are covered in the Removing Files and Directories part of this tutorial.
 
 
  
Open a terminal window if you don't have one already.
+
If you want to reference filenames with spaces in them, you will need to enclose the entire filename in quotation marks so that Linux understands that the space is part of one single name.  Alternatively, you can “escape” the space using a backslash. For example, if I have a file called my document Linux will see this as two words, “my” and “document”.
  
+
But you could write either of the following to make it understand you mean a single file:
 +
"my document"
 +
my\ document
  
Type the following into the terminal, then press Enter:
+
To avoid worrying about this, a common practice is to replace the space with an underscore. For example:
rm bioinf_files.tar.gz
+
mv “my document” my_document
  
+
* Everything is case sensitive
  
6
+
Linux systems consider capital letters different from lower case letters. The filename myFile is not the same
 +
as the filename Myfile or myfile. You could have all three of these in the same folder.
 +
There are some common naming conventions in place for biological data that you should try to follow. More
 +
is said on this in the second part of this tutorial.
  
Enter “y” to agree when you are asked if you wish to delete the file.
+
= Getting the prompt back when running graphical applications from the terminal =
 +
On an earlier page the command gnome-terminal & was suggested as a way to start a new terminal, but the
 +
ampersand symbol was not explained. By default, when you run a command the shell expects that the
 +
command will want to display text in the terminal window so it gets out fo the way until the command is
 +
finished. Ending a command with & tells the shell to go immediately back to the prompt, not waiting for the
 +
command to complete. This makes most sense when you expect the command to open up a new graphical
 +
window. It is also possible, though more fiddly, to change your mind and get the prompt back while the
 +
command is running.
 +
Confusingly, some graphical programs will always signal the shell to keep going even if you omit the &
 +
from the command. To demonstrate the default behavior we can use a very simple program called xcalc.
 +
The following exercise will hopefully help you understand how all this works.
  
�Finding your way on the system
+
== Exercise – understanding the function of "&": ==
In Linux/Unix systems, documents are usually referred to as files, and file folders are referred to as
+
1. In a terminal, type the command xcalc
directories.
+
1. A basic calculator should appear. Try it out.
Your Bio-Linux file system can be thought of as a huge file folder (directory), inside of which are many
+
2. Try to type another command (eg. pwd) back in your terminal window.
other file folders (directories). Inside these there are more nested file folders (directories), and so on. As in
+
3. Close the xcalc window and now see what happens back in the terminal.
the real world, where file folders can contain documents and other file folders, in Linux directories can
+
2. Run xcalc again and leave it running. Now we're going to get the terminal prompt back...
contain files and other directories. The hierarchy of folders is called the directory tree.
+
1. Back at the terminal, type Ctrl-z (ie. hold down Ctrl and tap z).
Your personal Home folder is one directory within the tree of directories that make up your Bio-Linux
+
2. What message do you see? Hopefully you can run commands again.
machine. In your account, you can create other directories, store data, run programs, etc. A graphical view of
+
3. Try using the calculator.
your home directory is available by clicking on the file cabinet Files icon in the Dash toolbar (Figure 5). This
+
4. In the terminal, give the command bg and try using the calculator again.
opens up a window that shows the files and directories in your Home. The full name of this folder on the
+
3. Run xcalc once again with an ampersand after the command – xcalc &
system is /home/live, ie. a directory named after the login account, live, within the top-level directory named
+
 
/home, but the graphical file browser just shows it as Home.
+
= Linux shorthand and shortcuts =
Linux enforces file permissions depending on the login account. By default on Bio-Linux, your account has
+
Understanding Linux commands can seem daunting at first. This is in part due to particular characters (full
the right to create, delete and edit files in your own Home folder, but not in other people’s accounts or in
+
stops, question marks, etc.) having special meaning in commands. Once you learn the basics, these shorthand
system directories. You can be given permission (or give yourself permission, if it's your system) to work on
+
characters are extremely useful and time saving.
files in such areas, and some information on setting file permissions is given later in this course. Your system
+
The following incomplete list covers the symbols you will see most often today and describes their meanings
administrator or local IT support should be able to help you with sharing files if they are on a shared server.
+
as you will most likely encounter them in this course.
You can use the graphical file browser to explore directory areas on the machine, and to move around in your
 
own files. It allows you to accomplish most typical file operation, including opening files and copying,
 
moving or deleting files using drag and drop or copy/cut/paste. To view areas of the system outside your
 
Home directory, click on Computer under Devices in the left hand pane to see the root directory of the
 
system.
 
 
 
Exercise 1-2
 
 
 
 
If you have not done so already, click on the filing cabinet Files icon near the top of the Dash
 
 
 
 
  
Double-click on the bioinf_files directory that you unpacked in Exercise 1-1, to view the contents
+
* "*", matches any character appearing 0 or more times, also known as a wildcard
 +
ls mydir/*
 +
ls cat*
 +
ls cat*hat
  
Investigate the options under the file browser menus. These appear on the bar at the very top of the
+
* "?"
screen.
 
  
+
  list all the files under the directory mydir
 +
  list all files starting with the letters cat
 +
  list all files starting with the letters cat and ending in hat
  
Click on the Computer icon in the left panel. This allows you to see the root directory – the base of the
+
matches a single character
whole filesystem hierarchy.
+
ls cat??hat
  
+
list all files starting with the letters cat followed by any 2 letters,
+
and then hat
  
Find the folder called home and double click on it.
+
.
  
You should see a single folder called live listed. Select this to get back to your Home folder. If you
+
the directory you are currently in – ie. the last one you moved to using cd
are not working on a live-booted system you should see a folder with your username, and other user
 
folders may also listed. A lock symbol on a folder would inform you that you do not have permission to
 
view the contents of that folder.
 
  
+
..
  
The Root Folder
+
the directory one level above the one you are currently in, aka. the parent directory
The name of the base directory of the whole system, the one within which every file on the system is
 
contained, is the root directory. It is referred to by a single forward slash “ / ”.
 
When you work in the graphical file browser it shows your location relative to your Home folder, unless you
 
are looking at files outside your Home in which case it shows the location relative to the root. You should
 
have seen how the location changed as you browsed folders in exercise 1-2.
 
Figure 6: Location path for Templates folder in File Browser view.
 
  
7
+
* "~" shorthand for your home directory, eg. /home/live
  
�Your personal home folder (actually called live but labeled as Home), sits within the directory called home
+
* $var dollar sign indicates a variable substitution, even within double quotes
(with a small h), that contains homes for all users. This directory home is under the root directory,
+
– see the section on environment variables
represented by a tiny picture of a disk in the graphical view or a single forward slash in the terminal.
 
In other words, this information tells you where you are in system.
 
The location of a file or directory within the system is its path. If you are asked for the full path or absolute
 
path to a file, you need to provide a complete listing of all the directories traversed on the system to get to
 
that file. That is, you need to give the full path from the root directory to that file. The path is written by
 
starting with a forward slash “/” then listing the names of the directories you need to traverse in the system
 
to find that file, with each directory name separated with another forward slash.
 
To see the full path in the conventional format most command-line programs would expect you to provide,
 
press Ctrl-L while viewing a File Browser window. You should see something like this:
 
  
Figure 7: Location in graphical file browser given in text; this is the the full
+
* "!" used for history substitution – not covered in this course
path to the Templates folder in the home directory of the live user account.
 
  
To summarize the syntax provided in Figures 9 and 10:
+
* "-" often seen preceding a parameter (eg. ls -l)
/home
+
also, the command cd - is a special case meaning “cd to previous directory”
/home/live
 
  
home is a directory located within the root directory
+
* ";" a semicolon can be used to separate two commands on the same line;
live is a directory within the directory home which is within the root
+
it is also used when writing loops – see p59
directory. This special directory will sometimes be shown as
 
  
Home, with a
+
= Changing directories =
capital H, because it is the home folder for the live user.
+
The command used to change directories is cd
As another example: the full path to the file capsall.fasta, in the bioinf_files directory within the home
 
directory of the live user:
 
/home/live/bioinf_files/capsall.fasta
 
Often you can provide just the route from where you are on the system to where your file is; this is referred
 
to as a relative path. For example, if you are working in your home directory, the relative path to the file
 
mentioned above would be bioinf_files/capsall.fasta.
 
  
Keeping things organised
+
If you think of your directory structure, (i.e. this set of nested file folders you are in), as a tree structure, then the simplest directory change you can do is move into a directory directly above or below the one you are in.
Everyone knows it, but it's worth restating: if you start by creating a folder structure with meaningfully
 
named subfolders, name your files so that the names indicate the contents (or follow some defined naming
 
convention), and store your files in the right place, your life will be much, much easier!
 
  
Using the command shell
+
To change to a directory one below you are in, just use the cd command followed by the subdirectory name:
The real power of Linux/Unix systems is the command line.
+
cd subdir_name
A list of common Linux commands is provided in Appendix D of this document for reference.
+
To change directory to the one above your are in, use the shorthand for “the directory above” ..
Many programs and facilities are available through graphical options on Linux, but all programs and
+
cd ..
facilities can be accessed by the command line, also known as the shell. Some tasks are easier, or more
+
If you need to change directory without worrying where you are now, you could explicitly state the full path:
appropriately done using graphical interfaces. Equally though, other things are easier or more appropriately
+
cd /usr/local/bin
8
+
If you wish to return to your home directory at any time, just type cd by itself.
 +
cd
 +
And finally, you can type
 +
cd –
 +
This returns you to the last directory you were working in before this one.
 +
If you get lost and want to confirm where you are in the directory structure , use the pwd command (print
 +
working directory). This will return the full path of the directory you are currently in. Also by default in BioLinux, you see the name of the current directory you are working in as part of your prompt.
 +
For example, when you first opened the terminal in a live session you should see the prompt:
 +
live@biolinux[live]
 +
This means you are logged in as the user live on the machine named biolinux, and you are in a directory
 +
called live. (Recall that the full path of your home directory is /home/live.)
 +
If you move into the bioinf_files directory
 +
cd bioinf_files
 +
you would see the prompt:
 +
live@biolinux[bioinf_files]
  
�done using the command line. Obvious examples include when you need to work with large numbers of files
+
14
or want to automate processes. First steps on the command line can be hard but the rewards are worth it (we
 
promise!)
 
Access to the command line is done through a terminal window.
 
You can open a new terminal by:
 
 
 
  
clicking the middle button on the terminal icon on the Dash toolbar
+
== Exercise 1-4 ==
or, going into an already open terminal and typing a command to open a second terminal:
+
Ensure you start in your home directory by using the cd command on its own. Change directory from
gnome-terminal &
+
your home directory to the directory bioinf_files by typing
 +
* cd bioinf_files
 +
* Find the full path to where you are by typing
 +
pwd
 +
 
 +
* Type cd bioinf_files a second time. Why doesn't this work?
  
Anatomy of a Command
+
* Change directory into the /usr/bin directory by typing
Linux/Unix commands usually take the form shown in Figure 11. You've already seen a good example in
+
cd /usr/bin
Exercise 1-1 part c.
 
command
 
what I want to do
 
eg: tar
 
  
parameters
+
* List the files in this directory.
how I want to do it
+
This is the main directory of runnable programs on the system.
-xvz -f
+
Some bioinformatics software can be found in here. Others are in /usr/local/bin
  
arguments
+
How can you get back to the bioinf_files folder from here? Can you work out how to do it with a
on what do I want to do it
+
single command?
bioinf_files.tar.gz
+
* Tab completion
 +
Tab completion is an incredibly useful facility for working on the command line.
 +
The main thing tab completion does is complete the filename or program name you have started typing,
 +
saving you typing time and reducing spelling errors.
 +
For example, from your home directory, you could type:
 +
cd bio
 +
and hit the tab key.
 +
If there is only one directory with a name starting with the letters “bio”, the rest of the name will be
 +
completed for you. Here this would give you:
 +
cd bioinf_files
 +
The terminal environment on Bio-Linux is set up such that if there is more than one file with that
 +
combination of letters, all the files will be shown to you. You can choose the one you want by typing more of
 +
the filename, or by continuing to hit the tab key multiple times.
  
Figure 8: The Linux/Unix command line structure. Each part of a command is separated by
+
15
one or more spaces.
 
  
The first word you supply on the command line is interpreted by the system as a command; that is –
+
== Exercise 1-5 ==
something the system should do or a program to be run. Items that appear after that on on the same line are
+
* Return to your home directory if you are not already there by typing cd
separated by spaces. The additional input on the command line indicates to the system how the command
 
should work. For example, what file you want the command to work on, or the format for the information
 
that should be returned to you.
 
Most commands have options available that will alter the way the command functions. You make use of
 
these options by providing the command with parameters, some of which will take arguments. Examples in
 
the following sections should make it clear how this works. With some commands you don't need to issue
 
any parameters or arguments. Occasionally this is because there are none available, but usually this is
 
because the command will use default settings if nothing is specified.
 
If a command runs successfully, it will usually not report anything back to you, unless reporting to you was
 
the purpose of the command (eg. ls). If the command does not execute properly, you will see an error
 
message returned. Some of these messages are hard to decipher until you have a bit of Linux experience but
 
ultimately they should tell you what has gone wrong.
 
Note: Items supplied on the command line separated by spaces are interpreted as individual pieces of
 
information for the system. For this reason, a filename with a space in it will be interpreted as two filenames
 
by default. How to get around this is is addressed in more detail later in the course.
 
Note 2: The use of the ampersand in the previous example, gnome-terminal &, is explained in a few pages
 
time. You would not put an ampersand on the end of most shell commands.
 
  
9
+
* Type cd bio and use tab completion for the rest of the command. Only then press the return key.
  
�Listing files in a directory
+
* You will now be in the bioinf_files directory.
The command ls lists files in a directory.
 
By default, the command will list the filenames of the files in your current working directory. When you first
 
open a shell this is your home directory.
 
If you add a space followed by a –l (that is, a hyphen and a small letter L), after the ls command, it alters the
 
behavior of the command: it will now list the files in your current directory, but with details about them
 
including who owns them, what the size is, and what kind of file it is. Information about this is shown in
 
Figure 11.
 
  
drwxr-xr-x 6
+
* Type ls testseq and use tab completion. This will show you a list of files that start with testseq.
-rw-r--r-- 1
+
You now have the option of completing the filename yourself, or “tabbing” through the filenames
-rw-r--r-- 1
+
available.
-rw-r--r-- 1
 
  
File
+
* Press the tab key a number of times to see what happens.
type
 
  
manager
+
* Type ls c and press tab once to view the files available.
manager
 
manager
 
manager
 
  
File
+
* Type a further a such that you now have ls ca on the command line.
permissions
 
  
User
+
* Now press the tab key again.
 +
As you get faster with this, it will save you a lot of typing effort. Also, tab completion knows how to
 +
escape spaces and other non-standard characters in file names for you.
  
users
+
== Exercise 1-6 ==
users
+
In the previous exercise tab completion was finding files in the working directory, but it can also help
users
+
you find command and program names because the system knows that the first word you type is going
users
+
to be a command name.
 +
* Type a on the command line and then press the tab key.
  
Group
+
* Add rte to the a so that you now have arte on the command line. Press the tab key again.
  
4096
+
* You will see that there is only one command that starts with these letters: artemis
9784
+
For programs that might contain case sensitive names, tab completion can be especially useful.
9784
 
7793
 
  
2008-08-21
+
* Type bl on the command line and press the tab key. You will see a number of program names listed.
2007-03-19
 
2007-03-19
 
2007-03-19
 
  
File
+
* Keep pressing the tab key to see how the filenames will cycle through on the command line.
size
 
  
09:26
+
16
14:09
 
14:09
 
14:14
 
  
Date and time
+
= Command history =
modified
 
  
twilliams
+
Previous commands you have used are stored in your history. You can save a lot of typing by using your command history effectively.  
hybInfo.txt
 
targets_v1.txt
 
targets_v2.txt
 
  
Filename
+
If you use the up arrow key when you are at the prompt in your terminal, you can see previous commands you have run. This is particularly useful if you have mistyped something and want to edit the command without writing the whole command out again.
 +
You can also view past commands using the command history.
  
Figure 9: The detailed output of the command ls when run with the -l flag
+
By default, history will return a list of the last 15 commands run. You can add a number as a parameter to the command to ask for longer or shorter lists. For example, to return the last 30 commands run, you would type:
 +
history -30
  
Exercise 1-3
+
It is possible to "speed search" previously-executed commands by pressing the key combination:
a) Try browsing files in both the terminal and the graphical file browser:
+
Ctrl-r (ie. hold down Ctrl and tap the R key)
+
Then start to type. The command history will be scanned and the last matching command will be displayed on the console. Type Ctrl-r repeatedly to cycle through the entire list of matching commands.
  
Open a new terminal by clicking the terminal icon
+
== Exercise 1-7 ==
 +
* Type history -n 10 on the command line.
 +
* Type Ctrl-r, then start typing ist.
  
In the terminal, type the command ls. Compare what you see listed with what you see in the graphical
+
= Making a directory =
representation of your Home directory.
 
 
  
Type the command ls –l and note the kind of information being provided and how it compares to the
+
To make a new directory, use the command mkdir (make directory). For example:
graphical representation of your files.
+
mkdir newdir
+
would create a new directory called newdir.
  
In the graphical File Browser, click on the List option under the View menu, and compare this
+
== Exercise 1-8 ==
information to that provided using the ls –l command.
+
* Start in your bioinf_files directory.
+
* Make a new directory called testdir
 +
The graphical view of your account should immediately update to show this new directory.
  
In the console, type ls –l bioinf_files and also click on the bioinf_files folder in the graphical file
+
* Move into the new directory testdir
browser and compare what you are seeing.
 
 
  
You can also use glob patterns to identify file names by pattern.
+
Move straight back into the bioinf_files directory using a single command. (see the shorthand and
*
+
shortcuts section above for a hint)
?
 
[]
 
  
an asterisk means any string of characters
+
= Office software =
a question mark means a single character
+
Leaving the command line for a short while... There are a number of word processors and spreadsheet
square brackets can be used to designate a group of characters
+
programs available for your system. In this course we will look at the LibreOffice suite of programs,
 +
previously known as OpenOffice. This is an open source alternative to Microsoft Office and can be run on
 +
both Linux and Windows.
 +
The programs within LibreOffice can be run graphically from the icons in the Dash toolbar.
  
More details about this are given in the Linux shorthand and shortcuts section below.
+
Word processor
  
10
+
== Spreadsheet ==
  
�(Exercise 1-3, continued)
+
Presentation editor
 +
Figure 10: LibreOffice Applications in the dash toolbar
  
b) Try these commands that use wildcards to match multiple files:
+
== Exercise 1-9 ==
+
* Click on the LibreOffice Calc Spreadsheet icon.
  
List all the files in the directory bioinf_files. that start with the letters tes
+
* Under the File menu, click on Open.
ls bioinf_files/tes*
 
  
+
* Look inside the bioinf_files directory.
  
List all the files in your directory that start with tes, and end in 1.embl, 2.embl or 3.embl
+
* Open the file called example.xls.
ls bioinf_files/tes*[123].embl
 
  
Learning about Linux commands
+
* Make a few changes and save the file using the Save or Save As… options under the File menu.
Most Linux commands have a manual page that provides information about the command and options that
 
can alter its behaviour. Many tasks can be made easier by using command options. A good rule of thumb is
 
to ask yourself whether what you want to do is something many others may have wanted to do. If the answer
 
is yes, then there may well be commands and options available to do that task.
 
Linux manual pages are referred to as man pages. To open the man page for a particular command, you just
 
need to type man followed by the name of the command you are interested in. To browse through a man
 
page, use the cursor keys (↓ and ↑). To close the man page simply hit the q key on your keyboard.
 
If you do not know the name of a command to use for a particular job, you can search using man –k
 
followed by the type of thing you are trying to do. An example of this is in exercise 1-3, part c).
 
(Exercise 1-3, continued)
 
  
c)
+
* Close LibreOffice Calc by choosing Exit from under the File menu.
 
  
Look up the manual information for the ls command by typing the following in a terminal:
+
= Text files, Word Processors and Bioinformatics =
man ls
 
  
Skim through the man page. You can scroll forward using the up and down arrow keys on your
+
Documents written using a word processor such as Microsoft Word or LibreOffice Write are not plain text documents. If your filename has an extension such as .doc or .odt, it is unlikely to be a plain text document.  (Try opening a Word document in notepad on Windows if you want proof of this.)
keyboard. You can go forward a page by using the space bar, and move backwards a page by using the b
 
key.
 
 
  
+
Word processors are very useful for preparing printed documents, but we recommend you do not use them when working with bioinformatics data files.
  
What does the -h option do? What about the -a option? What would running ls -lrt do?
+
There is a handy command called simply file that will inspect a file and tell you what it looks like. If you run this on a FASTA file it will say "ASCII text" because FASTA is a plain text format. If it says "binary data" or "HTML" or "OpenDocument Text" or whatever then this is not actually a FASTA file, even if it resembles one when viewed in soem applications.
  
+
= Using text editors =
  
Press the q key when you want to quit reading the man page.
+
Plain text files are important, both as input to bioinformatics programs and as input or configuration files for system programs. We highly recommend that you learn to use a text editor to prepare and edit plain text files.
  
+
There are a number of different text editors available on Bio-Linux. These range in ease of use, and each has its pros and cons. In this practical we will briefly look at two editors, nano and gedit.
  
Try running ls using some of the options mentioned above.
+
== Nano ==
 +
* Pros
 +
very simple – for example, most command options are visible at the bottom of the window
 +
can be used right in the terminal without graphical support
 +
fast to start up and use
 +
supports syntax highlighting
 +
 +
* Cons
 +
 due to simplicity, lacks some advanced
 +
features – eg. line numbering, search by
 +
pattern
 +
 it is not completely intuitive for people who
 +
are used to graphical word processors
  
+
== Gedit ==
 +
* Pros
 +
very easy to start using
 +
supports syntax highlighting
 +
looks similar to a word processor, but is in
 +
fact a powerful text editor.
 +
has many useful plugins that you can easily
 +
install
  
Look up some programs with man pages with the keywords “list directory”
+
Cons:
man –k “list directory”
+
 it is a graphical program and cannot be run
 +
from a text-only environment
 +
 it is slightly slower to start up than nongraphical editors
 +
 for real power users, it's not a match for Vim or Emacs
  
11
 
  
�Basic Linux tips for filenames
+
== Exercise: Editing a file with Gedit ==
 
  
Linux does not deal well with spaces in filenames!
+
To start up Gedit, you can use the command line, or find it in the Dash menu. Choose one of the two
 +
methods to open gedit:
 +
Command line
 +
Type gedit &
 +
Graphical menu
 +
Click the Dash Home at the top left of the screen, then type edit and click the Text Editor icon.
 +
* Type three or four lines of text into the gedit window.
  
Or to be more precise, Linux itself deals perfectly well with spaces and all manner of special characters in
+
* Save your file using the save option under the File menu (note, you have to move your mouse right to
filenames but many programs you'll want to run on Linux do not, and if you're talking about those files in
+
the top of the screen to see this) or simply click the Save button on the Toolbar. Save it as
the terminal you'll need to remember to quote them as described below. If you stick with letters, numbers,
+
myfirstfile.txt in your testdir directory.
hyphens, underscores and full stops, you will be fine.
 
Filenames with spaces in them are a common problem when transferring files to Linux from computers
 
running Windows, or Mac operating systems. Normally the simplest thing is to rename the files before you
 
work with them.
 
If you want to reference filenames with spaces in them, you will need to enclose the entire filename in
 
quotation marks so that Linux understands that the space is part of one single name.
 
Alternatively, you can “escape” the space using a backslash. For example, if I have a file called
 
my document
 
Linux will see this as two words, “my” and “document”.
 
But you could write either of the following to make it understand you mean a single file:
 
“my document”
 
my\ document
 
To avoid worrying about this, a common practice is to replace the space with an underscore. For example:
 
mv “my document” my_document
 
 
  
Everything is case sensitive
+
To save a file under the testdir directory, you may have to click on the drop down arrow to Browse for
 +
other folders. This will expand this section into a File Browser like the one you've seen in past exercises.
 +
Simply browse through to the location testdir is in and click the Save button.
 +
Add a new line to your file and save the file again using the Save As… option under the File menu.
 +
Save this file as mysecondfile.txt in the testdir directory.
 +
* Add more functionality to gedit by choosing the menu options; Edit → Preferences. A pop-up box
 +
will appear with 4 tabs:
 +
* View
  
Linux systems consider capital letters different from lower case letters. The filename myFile is not the same
+
Editor
as the filename Myfile or myfile. You could have all three of these in the same folder.
 
There are some common naming conventions in place for biological data that you should try to follow. More
 
is said on this in the second part of this tutorial.
 
  
Getting the prompt back when running graphical applications from the
+
Font & Colours
terminal
 
On an earlier page the command gnome-terminal & was suggested as a way to start a new terminal, but the
 
ampersand symbol was not explained. By default, when you run a command the shell expects that the
 
command will want to display text in the terminal window so it gets out fo the way until the command is
 
finished. Ending a command with & tells the shell to go immediately back to the prompt, not waiting for the
 
command to complete. This makes most sense when you expect the command to open up a new graphical
 
window. It is also possible, though more fiddly, to change your mind and get the prompt back while the
 
command is running.
 
Confusingly, some graphical programs will always signal the shell to keep going even if you omit the &
 
from the command. To demonstrate the default behavior we can use a very simple program called xcalc.
 
The following exercise will hopefully help you understand how all this works.
 
  
12
+
Plugins
  
�Exercise – understanding the function of "&":
+
Seeing the line numbers in a file helps to keep track of your position in that file. We will enable line
1. In a terminal, type the command xcalc
+
numbers here.
1. A basic calculator should appear. Try it out.
+
* On the View tab enable Display line numbers. Now you can see the line numbers on the left.
2. Try to type another command (eg. pwd) back in your terminal window.
 
3. Close the xcalc window and now see what happens back in the terminal.
 
2. Run xcalc again and leave it running. Now we're going to get the terminal prompt back...
 
1. Back at the terminal, type Ctrl-z (ie. hold down Ctrl and tap z).
 
2. What message do you see? Hopefully you can run commands again.
 
3. Try using the calculator.
 
4. In the terminal, give the command bg and try using the calculator again.
 
3. Run xcalc once again with an ampersand after the command – xcalc &
 
  
Linux shorthand and shortcuts
+
Next, click on the Plugins tab and enable the Change Case and the Document Statistics plugins.
Understanding Linux commands can seem daunting at first. This is in part due to particular characters (full
+
Browse around the other plugins and see what functionality they provide.
stops, question marks, etc.) having special meaning in commands. Once you learn the basics, these shorthand
+
* Under the Tools menu, click on Document Statistics.
characters are extremely useful and time saving.
 
The following incomplete list covers the symbols you will see most often today and describes their meanings
 
as you will most likely encounter them in this course.
 
*
 
  
matches any character appearing 0 or more times, also known as a wildcard
+
Try out the other newly added plugin, by selecting a piece of text from the document you are editing
ls mydir/*
+
with the mouse and click on the Edit menu. Hover the mouse over the Change Case menu and choose one
ls cat*
+
of the options you are presented with.
ls cat*hat
+
* Change part of one of the lines in this file and save it again using the Save As… option under the File
 +
menu. This time save it as mythirdfile.txt in the testdir directory.
 +
* * Quit gedit by choosing the option Quit under the File menu.
  
?
+
= Reading text files =
  
list all the files under the directory mydir
+
There are many commands available for reading text files on Linux/Unix. These are useful when you want to look at the contents of a file, but not edit it. Among the most common of these commands are cat, more, and less.
list all files starting with the letters cat
 
list all files starting with the letters cat and ending in hat
 
  
matches a single character
+
cat simply prints out a whole file in the terminal, which is often a very useful thing to do. However, cat
ls cat??hat
+
streams the entire contents of a file to your terminal at once and is thus not that useful for reading long files
 +
as the text streams past too quickly to read. (Note – cat is short for concatenate because if you give it
 +
multiple files it will string them together in order before printing them.)
 +
more and less are commands that show the contents of a file one screenful at a time. less has more
 +
functionality than more; specifically it can scroll backwards, hence the name. With both more and less, you
 +
can use the space bar to scroll down the page, and typing the letter q causes the program to quit – returning
 +
you to your command line prompt.
 +
Once you are reading a document with more or less, typing a forward slash / will start a prompt at the
 +
bottom of the page, and you can then type in text that is searched for below the point in the document you
 +
were at. Typing in a ? also searches for a text string you enter, but it searches in the document above the
 +
point you were at. Hitting the n key during a search looks for the next instance of that text in the file.
  
list all files starting with the letters cat followed by any 2 letters,
+
With less (but not more), you can use the arrow keys to scroll up and down the page, and the b key to move back up the document if you wish to.
and then hat
 
  
.
+
Exercise 1-11a
 +
* Move into the bioinf_files directory.
  
the directory you are currently in – ie. the last one you moved to using cd
+
* Read the file hsy14768.embl using the commands cat, more and less.
  
..
+
Don’t forget that tab completion can save you typing effort.
 +
cat hsy14768.embl
 +
more hsy14768.embl
 +
less hsy14768.embl
  
the directory one level above the one you are currently in, aka. the parent directory
+
Use the spacebar to scroll down
 +
Press q to quit.
 +
Use the spacebar to scroll down, b to go up a page, and the up and
 +
down arrow keys to move up and down the file line by line.
 +
Press the / key and search for the letters sequen in the file.
 +
Press the ? key and search for the letters gene in the file.
 +
Press the n key to search for other instances of gene in the file.
  
~
+
In almost all cases, if you want to look at a file in the terminal you want to use less. The cat command is more usually used in conjunction with other commands or when you actually want to concatenate files. The more command does nothing that less can't do.
  
shorthand for your home directory, eg. /home/live
+
Remember the man pages
 +
There are many command line options available for each of the above commands, as well as
 +
functionality we do not cover here. To read more about them, consult the manual pages:
 +
man cat
 +
man less
  
$var
+
As you'll see, the manual pages are actually displayed for you using less.
  
dollar sign indicates a variable substitution, even within double quotes
+
== An important note on line endings CR and LF ==
see the section on environment variables
 
  
!
+
There is one major gotcha when working with text files, and it stems from a decision made way back in the olden days of line printers. To print a text file on such a device, you would send the raw text file directly down the serial line to the printer and at the end of each line you sent two control codes, one to advance the paper (line feed) and the other to move the print carriage back to the start (carriage return).
  
used for history substitution – not covered in this course
+
In MS-DOS, later Windows, both these codes were embedded in standard text files at the end of every line.
  
-
+
In UNIX, and later Linux, a single LF character is used to indicate a newline. On old Macs it was a single LF. New Macs use the UNIX convention, so text files with single LF newlines are rare.
  
often seen preceding a parameter (eg. ls -l)
+
Many programs on Linux are written to deal with all these conventions – they just helpfully regard any
also, the command cd - is a special case meaning “cd to previous directory”
+
combination of CR and LF as meaning "next line". Others are not, and will either complain the file is invalid
 +
or worse will try to process the extra characters as meaningful data and produce nonsense results. You don't
 +
need this hassle so, much like we recommended removing spaces from filenames above, we also recommend
 +
ensuring all your text files are in order before attempting any bioinformatics on them. The next exercise
 +
shows how you might do this.
  
;
+
=== Exercise 1-11b ===
 
+
In Gedit, open the file hexaseqs.list which is provided in bioinf_files.
a semicolon can be used to separate two commands on the same line;
+
Without editing the file, save it as a new file named hexaseqs_crlf.list but on the Save As dialog switch
it is also used when writing loops – see p59
+
the Line Ending option to Windows.
 
+
● Try these commands in order:
More Basic Linux Commands
+
○ file hexaseqs.list hexaseqs_crlf.list
13
+
○ ls -l hexaseqs.list hexaseqs_crlf.list
 +
Note the difference in file sizes in the fourth column
 +
 +
* ○
 +
  
�A list of common Linux commands is provided in Appendix D of this document for reference.
+
cat hexaseqs.list
 +
cat hexaseqs_crlf.list
  
Changing directories
+
The command used to change directories is cd
+
If you think of your directory structure, (i.e. this set of nested file folders you are in), as a tree structure, then
 
the simplest directory change you can do is move into a directory directly above or below the one you are in.
 
To change to a directory one below you are in, just use the cd command followed by the subdirectory name:
 
cd subdir_name
 
To change directory to the one above your are in, use the shorthand for “the directory above” ..
 
cd ..
 
If you need to change directory without worrying where you are now, you could explicitly state the full path:
 
cd /usr/local/bin
 
If you wish to return to your home directory at any time, just type cd by itself.
 
cd
 
And finally, you can type
 
cd –
 
This returns you to the last directory you were working in before this one.
 
If you get lost and want to confirm where you are in the directory structure , use the pwd command (print
 
working directory). This will return the full path of the directory you are currently in. Also by default in BioLinux, you see the name of the current directory you are working in as part of your prompt.
 
For example, when you first opened the terminal in a live session you should see the prompt:
 
live@biolinux[live]
 
This means you are logged in as the user live on the machine named biolinux, and you are in a directory
 
called live. (Recall that the full path of your home directory is /home/live.)
 
If you move into the bioinf_files directory
 
cd bioinf_files
 
you would see the prompt:
 
live@biolinux[bioinf_files]
 
  
14
+
cat -A hexaseqs.list
 +
cat -A hexaseqs_crlf.list
  
�Exercise 1-4
+
Now run these. Remember that the * in a filename is a shorthand to match multiple files at once. Don't
Ensure you start in your home directory by using the cd command on its own. Change directory from
+
worry about the specific meaning of the sed command but do ensure you type it exactly like as shown.
your home directory to the directory bioinf_files by typing
+
* ○
+
  
cd bioinf_files
+
sed -i "s/\r//" hexaseqs*.list
+
file hexaseqs*.list
  
Find the full path to where you are by typing
+
== In summary ==
pwd
+
○ The line endings problem is a historical annoyance that won't go away.
 +
○ The file and cat -A commands are the quickest ways to detect troublesome CRLF line endings.
 +
○ Using Gedit and saving with the Unix/Linux mode is the simplest and safest way to remove
 +
them.
 +
○ The command shown above using sed (sed is a handy tool but we don't really have time to cover
 +
it in this course) can quickly strip all the CR characters from multiple files in one go. It's safe to
 +
run this on any regular text file, but if you run it on, say, and Excel file or an image or a .zip or
 +
.tar.gz file then the file will effectively be destroyed.
  
+
= Copying files =
 +
The basic command used to copy files using the command line is cp. At a minimum, you must specify two
 +
arguments: the name of the file to be copied, and where you wish to copy the file to.
 +
The main things to know about using the cp command are:
  
Type cd bioinf_files a second time. Why doesn't this work?
+
* if you provide the name of an existing directory as the second argument, the file named in the first
 +
argument will be copied into that directory.
 +
otherwise, it will be assumed that the second argument is the new name to be used for the copy you
 +
are making, whether the name corresponds to an existing file or not
 +
if you provide more than two arguments to cp, the final argument needs to be the name of a directory
 +
that already exists and all the preceding arguments need to be files that will be copied to the
 +
directory
  
+
Examples (try these in the bioinf_files folder if you like, or go straight on to 1-12):
 +
cp unknown.fasta my_new_file.fasta - clones unknown.fasta with the new name my_new_file.fasta
  
Change directory into the /usr/bin directory by typing
+
== cp unknown.fasta my_new_directory - probably not what you wanted! It just makes another file. ==
cd /usr/bin
+
mkdir an_actual_directory
 
+
cp unknown.fasta an_actual_directory - copy unknown.fasta into an_actual_directory you just made
+
cp *.embl an_actual_directory - copy all the .embl files into the new directory in one go
 
+
To copy whole directories, with all the subfiles and subdirectories, use the –R option, (meaning recursive).
List the files in this directory.
+
cp –R an_actual_directory foo - copy directory and its contents as a new directory, foo
This is the main directory of runnable programs on the system.
+
The Linux shorthand for “this directory right here” (a dot . ) and "the parent directory" ( .. ) comes in handy
Some bioinformatics software can be found in here. Others are in /usr/local/bin
+
when copying:
 +
cd foo
 +
cp –R ../blastdb .
  
How can you get back to the bioinf_files folder from here? Can you work out how to do it with a
+
copy blastdb from the directory above and put the copy here in foo
single command?
 
 
  
Tab completion
+
Make sure you leave a space between the directory name and the final dot.
Tab completion is an incredibly useful facility for working on the command line.
 
The main thing tab completion does is complete the filename or program name you have started typing,
 
saving you typing time and reducing spelling errors.
 
For example, from your home directory, you could type:
 
cd bio
 
and hit the tab key.
 
If there is only one directory with a name starting with the letters “bio”, the rest of the name will be
 
completed for you. Here this would give you:
 
cd bioinf_files
 
The terminal environment on Bio-Linux is set up such that if there is more than one file with that
 
combination of letters, all the files will be shown to you. You can choose the one you want by typing more of
 
the filename, or by continuing to hit the tab key multiple times.
 
  
15
+
Also useful is the shorthand for someone’s home account. e.g. instead of having to know and type the
 +
location of their account, you can use ~username In the case of your own account, you use just the ~
 +
symbol, followed by a / if you want to specify any subdirectories in your account.
 +
(note the next two examples don't work on the demo system as the files are not in place)
 +
cp ~user2/somefile .
  
�Exercise 1-5
+
copy the file somefile from user2’s home directory to my current working directory. Note that you need the appropriate permissions to do this!
 
  
Return to your home directory if you are not already there by typing cd
+
cp ~/Documents/mytext .
  
+
copy the file or directory called mytext from within my Documents
 +
directory to my current working directory.
  
Type cd bio and use tab completion for the rest of the command. Only then press the return key.
+
== Exercise 1-12 ==
 +
* Move into your directory testdir from exercise 1-8.
  
+
* List the files in this directory.
  
You will now be in the bioinf_files directory.
+
* Make a copy of myfirstfile.txt called test.txt
  
+
* Make a copy of mythirdfile.txt called myfourthfile.txt.
  
Type ls testseq and use tab completion. This will show you a list of files that start with testseq.
+
* Make a directory called subdir.
You now have the option of completing the filename yourself, or “tabbing” through the filenames
 
available.
 
  
+
* Copy mysecondfile.txt into subdir
  
Press the tab key a number of times to see what happens.
+
* Copy all the files that have the letters fil in the name into the subdir directory.
  
+
* Move back into the bioinf_files directory
  
Type ls c and press tab once to view the files available.
+
* Copy all the files that start with the letters tes and end in .embl into the directory subdir.
  
+
= Linking to files =
 +
Sometimes you want to access a file or directory at a different location but you don't actually want to copy it.
 +
For example if you have a data file in a system folder or network drive that you want to be able to access
 +
quickly from your desktop, but you don't actually want the entire file to be copied to your desktop folder:
  
Type a further a such that you now have ls ca on the command line.
+
ln -s /usr/local/bioinf/sampledata/nucleotide_seqs/multiple_seqs.fasta ~/Desktop/multiple.fasta
  
+
If you now try to open multiple.fasta in any application (eg. Gedit), you will see the data from the linked file as if you accessed it directly. If you write to the link you will be writing data straight to the original file (but in this case you will not have permission to do so).
  
Now press the tab key again.
+
You can examine links using the long output mode of ls.
As you get faster with this, it will save you a lot of typing effort. Also, tab completion knows how to
+
ls -l ~/Desktop/multiple.fasta
escape spaces and other non-standard characters in file names for you.
+
lrwxrwxrwx 1 live live 35 2011-05-12 11:46
 +
/home/live/Desktop/multiple.fasta ->
 +
/usr/local/bioinf/sampledata/nucleotide_seqs/file1.fasta
  
Exercise 1-6
+
The initial letter 'l' shows we are dealing with a link. Links do not have their own permission settings so ls shows them all as enabled, but links do have an owner depending on who created them. The target of the link is shown last. The target can be any file, directory or even another link. Note that Linux will not stop you from making a link where the target is non-existent or inaccessible, but ls will help you to spot these "dangling links" by colouring them in red.
In the previous exercise tab completion was finding files in the working directory, but it can also help
 
you find command and program names because the system knows that the first word you type is going
 
to be a command name.
 
 
  
Type a on the command line and then press the tab key.
+
= Removing files and directories =
  
+
The key difference between deleting something from the command line and using the graphical file browser is that in the first case the file vanishes immediately, but in the second it will be stored for a while in the Rubbish Bin and can be retrieved.
  
Add rte to the a so that you now have arte on the command line. Press the tab key again.
+
Option 1: Using the command line (effect: deletes files from the system)
 +
To remove a file or files, use the rm command followed by the name of the file(s) you wish to delete.
 +
rm file1
 +
rm file2 file3 file4
 +
rm foo/*
  
+
remove all files in foo but not the directory itself
  
You will see that there is only one command that starts with these letters: artemis
+
To remove an empty directory, you can use the rmdir command:
For programs that might contain case sensitive names, tab completion can be especially useful.
+
rmdir thisdir
 +
If that directory contains any files, you will not able to delete the directory using rmdir until you have
 +
deleted all the files within it. To delete a directory and all the files in it at the same time, use the rm
 +
command with the option -r (for recursive)
 +
rm –r fulldir
 +
If you use the above command on Bio-Linux, you will be prompted to confirm that you wish to delete each
 +
file. While sometimes useful, this can be tedious. If you are certain that you want to delete all the files in that
 +
directory, as well as the directory itself, then you can combine the recursive flag with the force (-f) flag
 +
rm -rf anydir
 +
So if you are 100% confident that you will never make a mistake, you can use rm -rf for all deletions, but
 +
for mere mortals it is good practice to use the more specific commands, as this can mitigate mistakes.
 +
Option 2: Using the File Browser (effect: moves files into the Rubbish Bin)
 +
If you are in the graphical file browser, just find the file you wish to remove, right click on it and choose the
 +
Move to Rubbish Bin option or else press the Delete key on the keyboard. Note that this file will not be removed from your system, only hidden, and can be retrieved via the Rubbish Bin icon in the bottom right of the screen.
  
+
If you were deleting the file to make space, you now have to empty it from the Rubbish Bin to actually get
 +
the disk space back. You can remove the file permanently in one go by holding down the Shift key on your
 +
keyboard and while keeping this key depressed, pressing the Delete key. A message box will pop up asking
 +
you to confirm that you really wish to permanently delete your file.
  
Type bl on the command line and press the tab key. You will see a number of program names listed.
+
= Exercise 1-13 =
  
+
* Move into the testdir directory.
 +
* Delete mythirdfile.txt using the command line
 +
* Delete myfourthfile.txt using the graphical file browser. Is the files now sitting in the Rubbish Bin?
 +
* Back on the command line, move back into your Home directory.
 +
* Then delete myfirstfile.txt from testdir without moving back to the testdir directory.
  
Keep pressing the tab key to see how the filenames will cycle through on the command line.
+
Delete the entire testdir/subdir directory without being prompted about the deletion of each file
 +
individually.
  
16
+
== Notes on Reading, Copying and Removing Files and Directories ==
  
�Command history
+
On Bio-Linux the commands cp, mv and rm have been aliased to cp –i , mv –i and rm –i respectively.
Previous commands you have used are stored in your history. You can save a lot of typing by using your
+
This means the system will ask you if you really mean to overwrite files should the situation arise with cp or
command history effectively. If you use the up arrow key when you are at the prompt in your terminal, you
+
mv, or delete the file you have just asked to delete when using rm. You must respond with a y or Y if you do
can see previous commands you have run. This is particularly useful if you have mistyped something and
+
wish to proceed. Hitting any other key will cause the action you requested to be ignored.
want to edit the command without writing the whole command out again.
+
You cannot assume that any other Linux/Unix systems you work on will be configured this way, but you can
You can also view past commands using the command history. By default, history will return a list of the
+
always set these settings yourself.
last 15 commands run. You can add a number as a parameter to the command to ask for longer or shorter
 
lists. For example, to return the last 30 commands run, you would type:
 
history -30
 
It is possible to "speed search" previously-executed commands by pressing the key combination:
 
Ctrl-r (ie. hold down Ctrl and tap the R key)
 
Then start to type. The command history will be scanned and the last matching command will be displayed
 
on the console. Type Ctrl-r repeatedly to cycle through the entire list of matching commands.
 
  
Exercise 1-7
+
Redirecting output to files
+
You have seen how the cat command can take the contents of a file and put it straight into the terminal, but
 +
we can also do what is essentially the opposite and capture output that would normally go to the terminal and
 +
put it in a file. This is done by the redirection operator >. For example:
 +
ls > file_list.txt
 +
In this case the output of ls will not appear on the screen but you will see a new file called file_list.txt. If
 +
you cat this file or open it in gedit you'll see the file list. Note that the result is no longer coloured, as there
 +
is no way to represent colour information in a plain text file, and has been formatted into a single column list,
 +
but otherwise is identical.
  
Type history -n 10 on the command line.
+
25
  
+
== Piping output between applications ==
 +
A remarkably powerful facility on the Linux command line is the ability to take the output of one command
 +
and use it directly as the input to another command. This is referred to as piping the output of one command
 +
into another command.
 +
The vertical bar symbol used for this is called a pipe and looks like:
  
Type Ctrl-r, then start typing ist.
+
|
  
Making a directory
+
Standard UK PC keyboards have the pipe symbol on the same key as the backslash symbol, at the bottom,
To make a new directory, use the command mkdir (make directory). For example:
+
left hand side of the keyboard. So pressing the Shift key and the backslash key together will give you the
mkdir newdir
+
pipe symbol.
would create a new directory called newdir.
+
On some keyboards, the pipe symbol is at the top left hand side, on the same key as the backtick. To type a
 
+
pipe symbol on such keyboards, hold down the key Alt Gr and hit the back tick ( ` ) key (left of the number
Exercise 1-8
+
1 key).
+
An example of when you want to use a pipe would be if you wanted to list all the files in a directory, but
 +
there are too many to fit on a single page. You probably saw this when you listed the contents of /usr/bin
 +
back in Ex. 1-4.
 +
You can pipe the output of the ls command (a list of files) into the less command, which will allow you to
 +
view the list page by page. To list the files in /usr/bin and view them page by page, the command would be:
 +
ls /usr/bin | less
 +
Another useful command to use with pipes is the wc command, which stands for wordcount. By default, wc
 +
returns the number of newlines, words and bytes in a file. Or you can tell wc to return just the number of
 +
lines by using the -l parameter (see the manpage for wc).
 +
For example, you could find out how many files you had in a directory by typing:
 +
ls | wc -l
  
Start in your bioinf_files directory.
+
= Diff, Grep and Sort =
  
+
In this section, we look briefly at three very useful commands: diff, grep and sort. As with all the commands covered today, we recommend that you read the manual page for more information about how these work and what options are available.
  
Make a new directory called testdir
+
== Diff ==
The graphical view of your account should immediately update to show this new directory.
+
diff compares files line by line and reports the differences between the files. In fact, diff can be used for
 +
more involved tasks as well, like comparing the contents of directories. This can be very useful when you are
 +
looking for changes that you or someone else has made.
  
+
=== Exercise 1-14 ===
 +
* Move into the testdir directory.
  
Move into the new directory testdir
+
* Type diff test.txt mysecondfile.txt to see what diff reports to you.
  
Move straight back into the bioinf_files directory using a single command. (see the shorthand and
+
* Type cat mysecondfile.txt | diff - test.txt
shortcuts section above for a hint)
 
 
  
17
+
In the above command the hyphen (-) refers to the information being given to diff through the pipe. That is,
 +
the information resulting from the command cat mysecondfile.txt is put directly into the diff command.
 +
Obviously, in this instance it would be easier just to give the name of the file, mysecondfile.txt, but there
 +
are many instances where being able to use – to mean “what I am sending in via the pipe” can be useful.
  
�Office software
+
== Grep ==
Leaving the command line for a short while... There are a number of word processors and spreadsheet
+
grep stands for global regular expression print; you use this command to search for text patterns in a file
programs available for your system. In this course we will look at the LibreOffice suite of programs,
+
(or any stream of text). Eg try this.
previously known as OpenOffice. This is an open source alternative to Microsoft Office and can be run on
+
grep "adge" /usr/share/dict/words
both Linux and Windows.
+
You can also use flexible search terms, known as regular expressions, in your grep searches. You have
The programs within LibreOffice can be run graphically from the icons in the Dash toolbar.
+
already used glob pattern expressions in this practical, but regular expressions are somewhat different and
 
+
more powerful. For example, when you listed all files with the pattern tes*embl* you were using a glob
Word processor
+
pattern comprising explicit characters (e.g. tes) and special symbols (* meaning any character or characters).
 +
The equivalent in grep would be “tes.*embl.*” where the period signifies any single character and the *
 +
signifies any number of repeats.
 +
Therefore to convert from a shell glob pattern to a regular expression replace each * with .* and each ? with .
 +
. You also need to enclose the expression in quotes to tell the shell not to try and interpret it as a glob.
 +
Unmodified glob patterns fed to grep but will not work as intended. For example the pattern tes* in grep
 +
means te followed by any number of s characters in sequence (te, tes, tess, tesss, ...). The question mark
 +
now signifies optionality – so tes? means te followed by zero or one s character (te, tes). Regular
 +
expressions are found in several places other than grep, most notably in the Perl scripting language. The full
 +
syntax is extensive and powerful but is beyond the scope of this course, so back to the grep command itself...
 +
grep requires a regular expression pattern as a parameter, and prints all the lines in a file containing that
 +
pattern.
 +
grep is especially useful in combination with pipes as you can filter the results of other commands.
 +
For example, perhaps you only want to see only the information in an EMBL file relating to the origin of the
 +
sequence, that is, the DE line. You do not need to search the file in an editor, you can just grep for lines
 +
beginning in DE, as in the next exercise.
 +
27
  
Spreadsheet
+
=== Exercise 1-15 ===
 +
* While in the bioinf_files directory, type the command: grep "DE" hsy14768.embl
 +
What is this command doing?
  
Presentation editor
+
Can you see why the above command results in the output you see?
Figure 10: LibreOffice Applications in the dash toolbar
+
An explanation of this command can be found below this exercise box.
 
+
* Try the commands: grep "^DE" hsy14768.embl and grep -x "DE.*" hsy14768.embl
Exercise 1-9
+
What are the ^ symbol and the -x parameter in these commands doing?
+
Check the manpage for grep to be sure.
  
Click on the LibreOffice Calc Spreadsheet icon.
+
* Try the command: cat hsy14768.embl | grep "^DE". Does that do what you expected?
  
+
* Move to your home directory and type ls –lR
 +
Read the manual page for ls if it is not clear what this command returns.
  
Under the File menu, click on Open.
+
* Use the above command with a pipe and a grep command to search for files created or
  
+
modified today.
 +
* List the files in the bioinf_files directory and use the grep command to look for those containing the
  
Look inside the bioinf_files directory.
+
characters d4.
 
+
The first command in the previous exercise searches all the text in the hsy14768.embl file and returns the
+
lines in which it finds the letter D followed by the letter E.
 +
The second command in the exercise also returns lines in the file that have a letter D followed by a letter E,
 +
but only where DE is found at the beginning of a line. This is because the ^ symbol means “match at the
 +
beginning of a line”. The $ symbol can be used similarly to mean “at the end of a line”. These are known as
 +
anchors. Passing the -x flag to grep tells it to automatically anchor both ends of the search pattern.
 +
What this anchoring does in the example above is return to you just the organism information in the embl
 +
file. This is because none of the other lines returned in the previous command started with DE, they just
 +
contained DE somewhere in them. This is an example where knowing how information is stored in an given
 +
file, along with a few basic Linux commands, allows you to retrieve information quickly.
 +
Another common example is counting how many sequences are in a set of multi-fasta files. We can do this
 +
with pipes between the commands cat, grep and the ever-handy wc, which here we use to count lines found
 +
by grep.
 +
cat *seqs.fasta | grep "^>" | wc -l
 +
Each sequence in a fasta file starts with a header line that begins with a > . The above command streams the
 +
contents of all files matching the glob pattern *seqs.fasta through a search with grep looking for lines that
 +
start with the symbol > . The quotes around the pattern ^> are necessary, as otherwise it is interpreted as a
 +
request for redirection of output to a file, rather than as a character to look for. As before, the ^ symbol
 +
means “match only at the beginning of the line”.
 +
The output of this grep search is sent to the wc command, with the -l indicating that you want to know the
 +
number of lines – ie. the number of headers and by implication the number of sequences.
 +
So a synopsis of the command above is: Read through all files with names ending seqs.fasta and look for all
 +
the header lines in the combined output, then count up those lines that matched and return the number to
 +
screen.
 +
We cover sequence formats later on in part 2 of the tutorial.
 +
28
  
Open the file called example.xls.
+
=== Environment Variables ===
 +
We have seen that the way commands run can be modified by the options passed on the command line.
 +
Some commands also read values called environment variables which affect their behaviour. Environmental
 +
variables are set within the shell via the export command and are passed to any processes you run. This is
 +
useful when you want to set some parameter that is common to all invocations of a command, or applies
 +
across several commands. For example, your favourite text editor may be, say, Gedit, or Nano, or Vim, or
 +
Emacs. In the shell you can say:
 +
export EDITOR=vim
 +
Now any command that wants to run a text editor knows what your preferred editor is. Within the shell you
 +
can get at the current value of en environment variable by prefixing it with a $ sign, eg.
 +
echo $EDITOR
  
+
prints the current value of the EDITOR environment variable to the screen
  
Make a few changes and save the file using the Save or Save As… options under the File menu.
+
The printenv command dumps all environment variables. Note that environment variables are only set in
 
+
the current shell and are not saved by default, so if you run a command in another terminal or close and
+
restart the terminal any values you set will be lost. For information on making the settings permanent by
 +
editing your .zshrc file see the user guide under Supported Shells.
  
Close LibreOffice Calc by choosing Exit from under the File menu.
+
== Exercise 1-16 ==
  
Text files, Word Processors and Bioinformatics
+
* Give the command: export VAR1=hello (with no spaces around the = sign) then:
Documents written using a word processor such as Microsoft Word or LibreOffice Write are not plain text
+
◦ echo $VAR1
documents. If your filename has an extension such as .doc or .odt, it is unlikely to be a plain text document.
+
◦ echo $ VAR1
(Try opening a Word document in notepad on Windows if you want proof of this.)
+
◦ echo "$VAR1"
Word processors are very useful for preparing printed documents, but we recommend you do not use them
+
◦ echo '$VAR1'
when working with bioinformatics data files.
 
There is a handy command called simply file that will inspect a file and tell you what it looks like. If you
 
run this on a FASTA file it will say "ASCII text" because FASTA is a plain text format. If it says "binary
 
data" or "HTML" or "OpenDocument Text" or whatever then this is not actually a FASTA file, even if it
 
resembles one when viewed in soem applications.
 
  
18
+
* Start a new terminal window by typing: gnome-terminal &
 +
◦ Within this new terminal: echo $VAR1
  
�Using text editors
+
* Start a second new terminal by right-clicking the icon in the Dash and selecting New Terminal
Plain text files are important, both as input to bioinformatics programs and as input or configuration files for
+
◦ Within this new shell: echo $VAR1
system programs. We highly recommend that you learn to use a text editor to prepare and edit plain text
 
files.
 
There are a number of different text editors available on Bio-Linux. These range in ease of use, and each has
 
its pros and cons. In this practical we will briefly look at two editors, nano and gedit.
 
  
Nano
+
* Go back to the original shell window
Pros:
+
unset VAR1
+
echo $VAR1
  
very simple – for example, most command
+
Has this affected either of the other two shells you started? Check them:
options are visible at the bottom of the
+
echo $VAR1
window
 
can be used right in the terminal without
 
graphical support
 
fast to start up and use
 
supports syntax hilighting
 
  
+
Environment variables are inherited when one process starts another, much like genetic material is inherited
+
when a cell divides. Hopefully this explains the behaviour you see in the exercise above. When you start a
+
terminal from en existing shell it inherits the environment from that shell. When you start one from the
 +
system menu it inherits just the base system environment. Furthermore, once a program is running no
 +
external program can modify its environment variables.
  
Cons:
+
=== Changing permissions on files and directories ===
 due to simplicity, lacks some advanced
+
Every file on the system has a set of permissions on it that dictate who on the system can read, change or
features – eg. line numbering, search by
+
delete, or execute the file. By default, all the files you create in your account are readable, changeable or
pattern
+
executable by you. However, you can grant other users permissions to access parts of your account if you
 it is not completely intuitive for people who
+
wish.
are used to graphical word processors
+
Below is some basic information about file permissions. Since there is only one user on the live system this
 
+
isn't really relevant to your current setup. If you are working on a shared system and want to set up access to
Gedit
+
your files for other people on the system, please get advice from your system administrator.
Pros:
+
The command to change permissions is chmod. You have to specify who you are modifying the permissions
+
of, what the new permissions are, and what file or directory to act on.
+
The format of the chmod command is:
+
chmod who ± permissions filename(s)
+
who can be:
  
very easy to start using
+
* "u" means user and refers to the owner of the file
supports syntax hilighting
+
* "g" means group, and refers to the group the file belongs to
looks similar to a word processor, but is in
+
* "o" means others, everyone on your systems apart from those above
fact a powerful text editor.
+
* "a" means all three, i.e. user, group and others
has many useful plugins that you can easily
 
install
 
  
Cons:
+
permissions can be:
 it is a graphical program and cannot be run
 
from a text-only environment
 
 it is slightly slower to start up than nongraphical editors
 
 for real power users, it's not a match for Vim
 
or Emacs
 
  
As most users will work on Bio-Linux using a graphical environment, we will only use Gedit in the exercise
+
* "r" means read permission
for this section.
+
* "w" means write permission
 +
* "x" means execute permission
  
Exercise 1-10
+
Each user has a default group and possibly extra group memberships. Use the id command to view your group memberships. When you create a new file it will be owned by you and by your default group. If you are a member of additional groups, you can switch the file to any of those groups using the chgrp command.
Editing a file with Gedit
 
To start up Gedit, you can use the command line, or find it in the Dash menu. Choose one of the two
 
methods to open gedit:
 
Command line
 
Type gedit &
 
Graphical menu
 
Click the Dash Home at the top left of the screen, then type edit and click the Text Editor icon.
 
 
  
Type three or four lines of text into the gedit window.
+
(Please refer to the manual pages for the commands chown, chgrp and chmod for more on this topic.)
  
+
For simplicity, let us assume that you and a co-worker have both been put in the default group labusers and
 +
wish to share your data files found in ~/bioinf_files.
 +
chmod a+x ~
 +
chmod g+rx ~/bioinf_files
 +
chmod g+r ~/bioinf_files/*
 +
directory
  
Save your file using the save option under the File menu (note, you have to move your mouse right to
+
give permission to anyone to execute, in this case, so
the top of the screen to see this) or simply click the Save button on the Toolbar. Save it as
+
that they can move through, your home directory.
myfirstfile.txt in your testdir directory.
+
give permission to people in the group to access files in the
 
+
bioinf_files directory under your home directory, including
19
+
listing the files with ls
 +
give permission to people in the group to read the files in the
  
�Exercise 1-10 continued
+
The first command could have been “chmod g+x ~”. This would unlock your home directory only to users
To save a file under the testdir directory, you may have to click on the drop down arrow to Browse for
+
in the labusers group. However, enabling access for anyone is generally safe, as long as permissions on the
other folders. This will expand this section into a File Browser like the one you've seen in past exercises.
+
files and subfolders prevent anyone from actually accessing them, and unless you set a+w in addition to a+x
Simply browse through to the location testdir is in and click the Save button.
+
nobody but you will be able to list the files in your home directory.
Add a new line to your file and save the file again using the Save As… option under the File menu.
 
Save this file as mysecondfile.txt in the testdir directory.
 
 
  
Add more functionality to gedit by choosing the menu options; Edit → Preferences. A pop-up box
+
=== Some other useful information ===
will appear with 4 tabs:
+
Copying and pasting text
+
Most Linux applications, including the shell terminal windows, have Copy and Paste options in the Edit
 +
menu or available in the pop-up menu when you click the right mouse button. You can copy text within
 +
the application or between different applications. There is also a quick way to copy text within the
 +
terminal by highlighting text to select it, and using the middle mouse button to paste the text.
 +
The exact way to select, copy and paste text from within a terminal windows depends on how your mouse
 +
has been set up. Normally you would highlight text by dragging the mouse across it with your left mouse
 +
button depressed to copy the text, and paste by clicking the middle mouse button (or the two outer mouse
 +
buttons pressed simultaneously). Note that within the terminal it doesn't matter where you click the middle
 +
mouse button – the text will always be inserted at the current cursor position.
  
View
+
= The simple way to stop a process =
 +
Sometimes a command or program you run in the terminal goes on too long, or is obviously doing something
 +
you did not plan. If there is no obvious way (such as a menu option or button) to stop the program running,
 +
try using Control and c (more commonly written as Ctrl-c). i.e. hold down the Control key and hit the c
 +
key. This requests the program to stop immediately, though the program may ignore the request.
 +
Note that this is the same key combination used in most graphical applications for copying text. Remember
 +
that highlighting text in a Linux terminal automatically copies it into the buffer – you don't need to press
 +
Ctrl-c before pasting with the middle button.
  
Editor
+
= Putting a command to one side =
 
+
Sometimes, you are in the middle of typing a long command, and you suddenly realise you need to do
Font & Colours
+
something else in the terminal, like list the current directory contents or check the manpage, before you run
 +
the command. Z-shell provides a handy shortcut for this: Alt-q. When you press Alt-q the current
 +
command disappears and you have a new empty prompt, but the unfinished command has been remembered
 +
and will reappear with the next prompt ready for you to edit and run it.
 +
An alternative is to hit Ctrl-c. Within the shell, Ctrl-c does not cause the shell to exit but it does cause the
 +
current command to be abandoned and a fresh prompt to appear. Unlike with Alt-q the unfinished command
 +
will still be visible in the terminal display so you can select it and paste it back in with the middle button if
 +
you decide you want it after all. (Try it!)
  
Plugins
+
== Logging out of a session ==
 +
To logout, you can press the Power Icon on the far right of the top taskbar (Figure 2) and choose the Log
 +
Out option.
 +
To shut down the machine, you can choose the Shut Down option on the same menu. If you are working on
 +
the console of a machine with users apart from you, then please check with your system administrator before
 +
powering down the machine. Other people might want to log in remotely.
  
Seeing the line numbers in a file helps to keep track of your position in that file. We will enable line
+
== Clearing your terminal of text ==
numbers here.
 
 
  
On the View tab enable Display line numbers. Now you can see the line numbers on the left.
+
Your terminal windows can fill up with lots of text, and it can become difficult to see the information you
 +
want because of all the clutter. You can clear the terminal window by typing clear
  
Next, click on the Plugins tab and enable the Change Case and the Document Statistics plugins.
+
= Accessing a running program or working with others interactively =
Browse around the other plugins and see what functionality they provide.
 
 
  
+
If you just run a job and then close down the terminal you ran it from, normally the job will be terminated. It would be nice to be able to leave a long job running and be able to log out and then log back in again to see how it is progressing. This is especially true if you log in remotely via SSH and experience network disruptions, or if you run programs that can take quite a long time, but ask you for input periodically.
  
Under the Tools menu, click on Document Statistics.
+
Luckily, there is a tool that makes it possible to leave programs running with no danger of them terminating if you log off or your terminal is closed. In addition, when you log back into your system, either locally or remotely, you can “re-attach” to your earlier session so it feels like you are picking up where you left off, in the same window you were running your program from.
  
Try out the other newly added plugin, by selecting a piece of text from the document you are editing
+
The utility that allows you to do this is called screen. It must be run before you start running other programs in your window. Screen can also allow two people on different machines to work in the same session – i.e. Real time collaborative editing is possible with screen.
with the mouse and click on the Edit menu. Hover the mouse over the Change Case menu and choose one
 
of the options you are presented with.
 
 
  
Change part of one of the lines in this file and save it again using the Save As… option under the File
+
Unfortunately, how to work with screen is beyond the scope of this course. However, the link below provides a useful beginners tutorial about screen and multi-user sessions:
menu. This time save it as mythirdfile.txt in the testdir directory.
+
https://www.linode.com/docs/networking/ssh/using-gnu-screen-to-manage-persistent-terminalsessions#screen-basics
 
  
+
An extensive list of command options can be found in the screen manpage (ie. type man screen).
 +
There are many useful commands available on Linux and we cannot begin to cover them in this course. We
 +
recommend that you consider buying a book to help you learn how to use Linux efficiently.
  
Quit gedit by choosing the option Quit under the File menu.
+
Accessing your machine – including a full graphical desktop - remotely
 
+
Bio-Linux is set up for secure remote access. We can't demonstrate this on the Live system but it is well
Reading text files
+
worth knowing that if you have an installed Bio-Linux system you can connect to it securely over the
There are many commands available for reading text files on Linux/Unix. These are useful when you want to
+
network, so long as your account is enabled in the ssh group and you have network access to the machine (ie.
look at the contents of a file, but not edit it. Among the most common of these commands are cat, more, and
+
not blocked by a site firewall)
less.
+
You can connect to your (installed) Bio-Linux system remotely using X2Go software. If you download an
cat simply prints out a whole file in the terminal, which is often a very useful thing to do. However, cat
+
X2Go client to another Windows, Linux or Mac system, you can connect to an installed Bio-Linux system
streams the entire contents of a file to your terminal at once and is thus not that useful for reading long files
+
and run a full, graphical, desktop session remotely. Further details on how to do this can be found on the
as the text streams past too quickly to read. (Note – cat is short for concatenate because if you give it
+
website at:
multiple files it will string them together in order before printing them.)
+
http://environmentalomics.org/bio-linux-remote-access
more and less are commands that show the contents of a file one screenful at a time. less has more
+
Note that due to limitations of the remote protocol, X2Go will use a fallback desktop “MATE” session which
functionality than more; specifically it can scroll backwards, hence the name. With both more and less, you
+
is slightly different to the default “Unity” desktop environment described in this tutorial.
can use the space bar to scroll down the page, and typing the letter q causes the program to quit – returning
 
you to your command line prompt.
 
Once you are reading a document with more or less, typing a forward slash / will start a prompt at the
 
bottom of the page, and you can then type in text that is searched for below the point in the document you
 
were at. Typing in a ? also searches for a text string you enter, but it searches in the document above the
 
point you were at. Hitting the n key during a search looks for the next instance of that text in the file.
 
  
20
+
= Part Two: Introduction to Bioinformatics on Bio-Linux =
  
With less (but not more), you can use the arrow keys to scroll up and down the page, and the b key to move back up the document if you wish to.
+
This section of the tutorial introduces you to running bioinformatics software on Bio-Linux, including how to find out what is available for particular types of bioinformatics tasks, some options you have for running programs on the system, and where to find documentation about the software on the system. This course
 +
does not cover the detailed use or understanding of any particular piece of software.
 +
You should read through the general information in the next few pages, then look at which specific programs
 +
are of most interest to you.
  
Exercise 1-11a
+
The main points we hope you take away after completing this section of the tutorial are:
+
a) You can discover and run bioinformatics tools even if you have not explicitly been taught how to use them.
 +
b) If you have repetitive tasks to carry out, chances are there are ways of fully or partially automating them.
 +
c) Web interfaces are easy, and have certain benefits, but a competence with the command line gives you access to more possibilities and sometimes these will suit your needs better.
  
Move into the bioinf_files directory.
+
* local copies of software documentation – look in /usr/share/doc
 +
* options under the help menus in some graphical programs
 +
* web pages
 +
* journal articles.
  
+
== Help Functions within the Programs ==
 +
Documentation is available from within many programs. For example, many graphical programs have a Help
 +
menu or button; many command line programs provide help if you type the name of the program followed
 +
by –h, –help or --help. Some programs even have their own manual pages that can be accessed by typing
 +
man followed by the program name.
  
Read the file hsy14768.embl using the commands cat, more and less.
+
== Example data for this tutorial ==
  
Don’t forget that tab completion can save you typing effort.
+
The sequences referred to in this tutorial can be unpacked from the file
cat hsy14768.embl
+
/usr/local/bioinf/documentation/bio-linux/intro_course/bioinf_files.tar.gz.
more hsy14768.embl
 
less hsy14768.embl
 
  
Use the spacebar to scroll down
+
If you have just done the associated Introduction to Linux tutorial, you will already have these files – please
Press q to quit.
+
move on to the next section of the tutorial.
Use the spacebar to scroll down, b to go up a page, and the up and
+
If you have joined the tutorial at this point, please refer to Exercise 1-1, parts b, c and d to download and
down arrow keys to move up and down the file line by line.
+
unpack the necessary sample data files.
Press the / key and search for the letters sequen in the file.
+
For some parts you will also need qiime_tutorial_data.tar.gz, mothur_tutorial_data.tar.gz and
Press the ? key and search for the letters gene in the file.
+
assembly_taster.tar.xz which are available in the same directory.
Press the n key to search for other instances of gene in the file.
+
 
 +
34
 +
 
 +
=== Interface choices ===
 +
Software can be run on the command line, via graphical programs on your computer, via web interfaces, via
 +
web services and/or via scripts. Bioinformatics programs can often be run using more than one of these
 +
options. Each type of interface has pros and cons. We have summarised some of these for reference below.
 +
Interface
 +
Command line
 +
 
 +
Pros
  
In almost all cases, if you want to look at a file in the terminal you want to use less. The cat command is
+
= Cons =
more usually used in conjunction with other commands or when you actually want to concatenate files. The
 
more command does nothing that less can't do.
 
Remember the man pages
 
There are many command line options available for each of the above commands, as well as
 
functionality we do not cover here. To read more about them, consult the manual pages:
 
man cat
 
man less
 
  
As you'll see, the manual pages are actually displayed for you using less.
+
= Fast to run once you know the program =
  
An important note on line endings – CR and LF
+
Have to learn the syntax
There is one major gotcha when working with text files, and it stems from a decision made way back in the
 
olden days of line printers. To print a text file on such a device, you would send the raw text file directly
 
down the serial line to the printer and at the end of each line you sent two control codes, one to advance the
 
paper (line feed) and the other to move the print carriage back to the start (carriage return).
 
In MS-DOS, later Windows, both these codes were embedded in standard text files at the end of every line.
 
In UNIX, and later Linux, a single LF character is used to indicate a newline. On old Macs it was a single
 
LF. New Macs use the UNIX convention, so text files with single LF newlines are rare.
 
Many programs on Linux are written to deal with all these conventions – they just helpfully regard any
 
combination of CR and LF as meaning "next line". Others are not, and will either complain the file is invalid
 
or worse will try to process the extra characters as meaningful data and produce nonsense results. You don't
 
need this hassle so, much like we recommended removing spaces from filenames above, we also recommend
 
ensuring all your text files are in order before attempting any bioinformatics on them. The next exercise
 
shows how you might do this.
 
21
 
  
�Exercise 1-11b
+
Very flexible; usually many options
In Gedit, open the file hexaseqs.list which is provided in bioinf_files.
 
Without editing the file, save it as a new file named hexaseqs_crlf.list but on the Save As dialog switch
 
the Line Ending option to Windows.
 
● Try these commands in order:
 
○ file hexaseqs.list hexaseqs_crlf.list
 
○ ls -l hexaseqs.list hexaseqs_crlf.list
 
Note the difference in file sizes in the fourth column
 
 
 
  
+
Have to find out what options are available
 
  
cat hexaseqs.list
+
Type out the command Repetitive tasks are easy to run or automate
cat hexaseqs_crlf.list
+
and press enter
 +
Easy to log in remotely and carry out tasks
 +
Easy to run; don't have to remember the
 +
Prompted command command line syntax
 +
line
 +
Easy to log in remotely and carry out tasks
  
+
Easy to forget the diversity of options for a
+
program because of the temptation to just
 +
reply to prompts provided
  
cat -A hexaseqs.list
+
Type out the command
cat -A hexaseqs_crlf.list
+
and respond to
 +
prompts on screen
  
Now run these. Remember that the * in a filename is a shorthand to match multiple files at once. Don't
+
Slower to get running than “pure” command
worry about the specific meaning of the sed command but do ensure you type it exactly like as shown.
+
line
 
  
+
Graphical interface
 
  
sed -i "s/\r//" hexaseqs*.list
+
= Often more intuitive and visually pleasing =
file hexaseqs*.list
+
than the command line
  
In summary:
+
Can be slower to use than the command line,
○ The line endings problem is a historical annoyance that won't go away.
+
especially for repetitive tasks
○ The file and cat -A commands are the quickest ways to detect troublesome CRLF line endings.
 
○ Using Gedit and saving with the Unix/Linux mode is the simplest and safest way to remove
 
them.
 
○ The command shown above using sed (sed is a handy tool but we don't really have time to cover
 
it in this course) can quickly strip all the CR characters from multiple files in one go. It's safe to
 
run this on any regular text file, but if you run it on, say, and Excel file or an image or a .zip or
 
.tar.gz file then the file will effectively be destroyed.
 
  
Copying files
+
Extensive help is often available via a menu
The basic command used to copy files using the command line is cp. At a minimum, you must specify two arguments: the name of the file to be copied, and where you wish to copy the file to.
+
option or button
The main things to know about using the cp command are:
 
 
 
 
  
if you provide the name of an existing directory as the second argument, the file named in the first
+
For some programs, the command line
argument will be copied into that directory.
+
version provides more functionality.
otherwise, it will be assumed that the second argument is the new name to be used for the copy you
 
are making, whether the name corresponds to an existing file or not
 
if you provide more than two arguments to cp, the final argument needs to be the name of a directory
 
that already exists and all the preceding arguments need to be files that will be copied to the
 
directory
 
  
Examples (try these in the bioinf_files folder if you like, or go straight on to 1-12):
+
Some programs (not all!) can be run by
cp unknown.fasta my_new_file.fasta - clones unknown.fasta with the new name my_new_file.fasta
+
Start the program and clicking an icon in the Applications |
 +
Bioinformatics menu on your system.
 +
interact via menus
  
22
+
You may need your system admin to set up
 +
programs so that you can run graphical
 +
programs when logging in remotely
  
�cp unknown.fasta my_new_directory - probably not what you wanted! It just makes another file.
+
Appropriate for visual tasks such as
mkdir an_actual_directory
+
alignment editing, detailed annotation
cp unknown.fasta an_actual_directory - copy unknown.fasta into an_actual_directory you just made
+
checking, etc.
cp *.embl an_actual_directory - copy all the .embl files into the new directory in one go
+
Usually intuitive
To copy whole directories, with all the subfiles and subdirectories, use the –R option, (meaning recursive).
 
cp –R an_actual_directory foo - copy directory and its contents as a new directory, foo
 
The Linux shorthand for “this directory right here” (a dot . ) and "the parent directory" ( .. ) comes in handy
 
when copying:
 
cd foo
 
cp –R ../blastdb .
 
  
copy blastdb from the directory above and put the copy here in foo
+
== Web interface ==
  
Make sure you leave a space between the directory name and the final dot.
+
Can provide functionality not available via
Also useful is the shorthand for someone’s home account. e.g. instead of having to know and type the
+
locally-run programs such as access to
location of their account, you can use ~username In the case of your own account, you use just the ~
+
important data resources or results presented
symbol, followed by a / if you want to specify any subdirectories in your account.
+
in useful formats, e.g. including links to
(note the next two examples don't work on the demo system as the files are not in place)
+
related data resources, graphics, etc.
cp ~user2/somefile .
 
  
copy the file somefile from user2’s home directory to my
+
== Run via a web browser ==
current working directory. Note that you need the appropriate
+
window, usually at a Some websites allow a certain degree of
permissions to do this!
+
“pipelining”, where the outputs of one
 +
remote site
 +
program can intuitively be supplied as input
 +
to another.
  
cp ~/Documents/mytext .
+
Can be slow to use relative to the command
 
+
line, especially for repetitive tasks
copy the file or directory called mytext from within my Documents
+
You are subject to the rules and restrictions
directory to my current working directory.
+
of the site you are working on (e.g. data
 
+
volume, number of tasks, options available,
Exercise 1-12
+
etc.)
+
You may not want to send private data over
 +
the internet (e.g. if you are applying for a
 +
patent?)
 +
You can be subject to the whims of network
 +
connectivity
  
Move into your directory testdir from exercise 1-8.
+
=== Web services ===
 +
Runs tasks over the
 +
internet from a
 +
program, usually
 +
locally installed or run
 +
via java webstart.
  
+
Can bring together the ease of a locally run
 +
program with the data and computing
 +
resources of a remote site
  
List the files in this directory.
+
You are dependent on the consistency of the
 +
remote server where the functions you need
 +
Can be used via graphical programs or scripts are running
 +
You are dependent on the functionality the
 +
remote site offers; this may not be as
 +
extensive as the functionality you get locally
 +
for some programs.
 +
Very flexible
  
+
= Scripts =
  
Make a copy of myfirstfile.txt called test.txt
+
You are dependent on network connectivity
  
+
Great for automating tasks
  
Make a copy of mythirdfile.txt called myfourthfile.txt.
+
Using a small
 +
Great for carrying out customised tasks
 +
program that runs a
 +
program or programs Straightforward to learn enough to alter
 +
existing scripts to do exactly the task you
 +
for you
 +
want.
  
+
You have to write the script or find a script
 +
that does the job. This means learning a
 +
programming language (or asking someone
 +
who knows one to help you)
  
Make a directory called subdir.
+
For repetitive tasks, we highly recommend the use of the command line, workflow software and/or scripting.
  
+
General points about working with bioinformatics programs
 +
Sequence formats
 +
A simple thing that often trips people up is sequence formats. There are many different sequence formats;
 +
the reasons for this are both historical and functional.
 +
Historically, when people first started writing analysis programs for molecular data, they designed a format
 +
that they felt suited their needs. As time went on, numerous formats came into existence. We live with the
 +
legacy of this. We must know what format our data is in, and whether the program we want to run can use
 +
data in that format.
 +
Functionally, a program may require information that can be included with data held in certain formats, but
 +
not others. For example, EMBL format files can, in addition to the sequence data itself, contain descriptive
 +
information about a sequence, such as its features. In contrast, plain format contains nothing inside the file
 +
except the sequence data, while FASTA format allows a small amount of information about a sequence to be
 +
given in a header line and FASTQ adds read quality information alongside the sequence. Clustal and msf
 +
formats handle multiple aligned sequences, while phylip and nexus format files contain aligned sequences as
 +
well as information relevant to phylogenetic analysis programs.
  
Copy mysecondfile.txt into subdir
+
To analyse data, it must be presented to the analysis program in a format the progam
 +
understands.
 +
This seems obvious, but frequent errors (or worse, misleading results) occur when the data entered into
 +
a program is not appropriate.
  
+
36
  
Copy all the files that have the letters fil in the name into the subdir directory.
+
== Converting files to different sequence formats used to be a frequent, and often time consuming, task in ==
 +
bioinformatics. Luckily there are file conversion programs that take care of this easily for many formats. In
 +
addition, many program understand more than one format.
 +
Some common bioinformatics sequence formats, along with common filename conventions used for those
 +
formats, are listed in the table that follows the next section.
 +
We recommend the following page for more information and examples of common bioinformatics file
 +
formats:
 +
http://www.molecularevolution.org/resources/fileformats
  
+
= File naming conventions in bioinformatics =
 
+
The suffix, (the part of the filename after the final dot), is often used to denote to you, and other people, what
Move back into the bioinf_files directory
+
the format of the data inside the file is.
 +
For example, the common suffix for clustal formatted alignments is aln. .A bioinformatics file that ends in
 +
.aln is usually assumed to be a clustal formatted alignment file.
 +
Another multiple sequence alignment format is phylip. A common suffix used on files containing sequences
 +
in phylip format is phy.
 +
Common suffices used for files containing data in particular formats are listed in the table following this
 +
section. We highly recommend that you follow conventions when naming your data files.
 +
Benefits to following the convention for filename endings include:
 +
* You will know your data format just by looking at the name of the file.
  
+
Following standard conventions, (rather than making up your own naming system), makes it
 +
easier for other people looking at your files, (e.g. collaborators, or people helping you); they will
 +
know the data format just by looking at the name.
 +
* Some graphical programs have filters set so that only files with particular suffices will be
 +
listed in the file browser window when you try to load some data. If you use conventional
 +
filename endings, this is less likely to cause problems for you.
 +
* Certain programs use information in the filename to interpret aspects of the data, (not just the data format).
 +
Such programs have strict naming conventions for the whole filename. For example, some sequence
 +
assembly programs either require, or are benefited by, defined naming schemes for sequence traces. The
 +
filename will inform them about which sequences are read pairs, what direction sequence reads are in, and
 +
other information relevant to assembly or visualisation. You will need to read the program documentation to
 +
find out what is required in such instances.
  
Copy all the files that start with the letters tes and end in .embl into the directory subdir.
+
You are not restricted to naming your files in any particular way but we highly recommend that you
 +
follow the convention for the type of file you are generating/saving.
 +
Following file naming conventions from the beginning will save you, and your collaborators,
 +
a lot of time!
  
Linking to files
+
37
Sometimes you want to access a file or directory at a different location but you don't actually want to copy it.
 
For example if you have a data file in a system folder or network drive that you want to be able to access
 
quickly from your desktop, but you don't actually want the entire file to be copied to your desktop folder:
 
23
 
  
�ln -s /usr/local/bioinf/sampledata/nucleotide_seqs/multiple_seqs.fasta ~/Desktop/multiple.fasta
+
== Common bioinformatics file formats ==
If you now try to open multiple.fasta in any application (eg. Gedit), you will see the data from the linked file
+
Format
as if you accessed it directly. If you write to the link you will be writing data straight to the original file (but
+
Embl or
in this case you will not have permission to do so).
+
swissprot
You can examine links using the long output mode of ls.
 
ls -l ~/Desktop/multiple.fasta
 
lrwxrwxrwx 1 live live 35 2011-05-12 11:46
 
/home/live/Desktop/multiple.fasta ->
 
/usr/local/bioinf/sampledata/nucleotide_seqs/file1.fasta
 
  
The initial letter 'l' shows we are dealing with a link. Links do not have their own permission settings so ls
+
= Some common =
shows them all as enabled, but links do have an owner depending on who created them. The target of the
+
filename endings
link is shown last. The target can be any file, directory or even another link. Note that Linux will not stop
+
.dat
you from making a link where the target is non-existent or inaccessible, but ls will help you to spot these
+
.embl
“dangling links” by colouring them in red.
+
.sprot
 +
.swiss
  
Removing files and directories
+
= Comments =
The key difference between deleting something from the command line and using the graphical file browser
+
Usually these files, along with genbank files, contain feature information
is that in the first case the file vanishes immediately, but in the second it will be stored for a while in the
+
as well as sequence.
Rubbish Bin and can be retrieved.
+
Embl and Swisprot (or Uniprot) format are the same. Embl files contains
Option 1: Using the command line (effect: deletes files from the system)
+
nucleotide sequences and Uniprot files contain peptide sequences.
To remove a file or files, use the rm command followed by the name of the file(s) you wish to delete.
+
Files downloaded from EMBL or Uniprot websites use the suffix .dat.
rm file1
+
Often these are compressed with gzip, and so end in .dat.gz
rm file2 file3 file4
+
Files generated by individuals in embl format will tend to end in .embl.
rm foo/*
 
  
remove all files in foo but not the directory itself
+
= Genbank =
  
To remove an empty directory, you can use the rmdir command:
+
.seq
rmdir thisdir
+
.gb
If that directory contains any files, you will not able to delete the directory using rmdir until you have
+
.genbank
deleted all the files within it. To delete a directory and all the files in it at the same time, use the rm
 
command with the option -r (for recursive)
 
rm –r fulldir
 
If you use the above command on Bio-Linux, you will be prompted to confirm that you wish to delete each
 
file. While sometimes useful, this can be tedious. If you are certain that you want to delete all the files in that
 
directory, as well as the directory itself, then you can combine the recursive flag with the force (-f) flag
 
rm -rf anydir
 
So if you are 100% confident that you will never make a mistake, you can use rm -rf for all deletions, but
 
for mere mortals it is good practice to use the more specific commands, as this can mitigate mistakes.
 
Option 2: Using the File Browser (effect: moves files into the Rubbish Bin)
 
If you are in the graphical file browser, just find the file you wish to remove, right click on it and choose the
 
Move to Rubbish Bin option or else press the Delete key on the keyboard. Note that this file will not be
 
24
 
  
�removed from your system, only hidden, and can be retrieved via the Rubbish Bin icon in the bottom right of
+
These files, along with embl and swissprot files, usually contain feature
the screen.
+
information as well as sequence.
If you were deleting the file to make space, you now have to empty it from the Rubbish Bin to actually get
+
Individuals using this format, usually use the .gb or .genbank suffix. The
the disk space back. You can remove the file permanently in one go by holding down the Shift key on your
+
NCBI usually uses .seq for genbank sections.
keyboard and while keeping this key depressed, pressing the Delete key. A message box will pop up asking
 
you to confirm that you really wish to permanently delete your file.
 
  
Exercise 1-13
+
= FASTA =
 
  
Move into the testdir directory.
+
= FASTQ =
  
+
Plain
  
Delete mythirdfile.txt using the command line
+
.fasta
 +
.fsa
 +
.fa
  
+
Possibly the most common sequence format.
  
Delete myfourthfile.txt using the graphical file browser. Is the files now sitting in the Rubbish Bin?
+
.fastq
 +
.fq
  
+
Very common for NextGen reads. Like FASTA with extra quality info
 +
per sequence.
 +
Alternative extensions may indicate the type of sequencing technology
 +
- .fastqsanger, .fastqsolexa, etc.
  
Back on the command line, move back into your Home directory.
+
.pln
 +
.staden
 +
.sdn
  
+
Not commonly used, as the file contents contain nothing but the sequence
 +
itself; the only identifier of the sequence is in the filename.
  
Then delete myfirstfile.txt from testdir without moving back to the testdir directory.
+
It may contain nucleotide or peptide sequence(s) and a single-line header
 +
per sequence.
  
Delete the entire testdir/subdir directory without being prompted about the deletion of each file
+
Staden programs use the plain format, accounting for the last two of the
individually.
+
file suffices given.
+
Clustal
 +
 
 +
.aln
  
Notes on Reading, Copying and Removing Files and Directories
+
= Multiple sequence alignment format =
On Bio-Linux the commands cp, mv and rm have been aliased to cp –i , mv –i and rm –i respectively.
+
Originally from the clustalw program, but now recognised by many
This means the system will ask you if you really mean to overwrite files should the situation arise with cp or
+
programs that accept or output multiple sequence alignments.
mv, or delete the file you have just asked to delete when using rm. You must respond with a y or Y if you do
 
wish to proceed. Hitting any other key will cause the action you requested to be ignored.
 
You cannot assume that any other Linux/Unix systems you work on will be configured this way, but you can
 
always set these settings yourself.
 
  
Redirecting output to files
+
Phylip
You have seen how the cat command can take the contents of a file and put it straight into the terminal, but
 
we can also do what is essentially the opposite and capture output that would normally go to the terminal and
 
put it in a file. This is done by the redirection operator >. For example:
 
ls > file_list.txt
 
In this case the output of ls will not appear on the screen but you will see a new file called file_list.txt. If
 
you cat this file or open it in gedit you'll see the file list. Note that the result is no longer coloured, as there
 
is no way to represent colour information in a plain text file, and has been formatted into a single column list,
 
but otherwise is identical.
 
  
25
+
.phy
 +
.phylip
  
�Piping output between applications
+
= Multiple sequence alignment format =
A remarkably powerful facility on the Linux command line is the ability to take the output of one command
+
Used by the Phylip suite of programs and many others, especially those
and use it directly as the input to another command. This is referred to as piping the output of one command
+
associated with phylogenetic analysis.
into another command.
 
The vertical bar symbol used for this is called a pipe and looks like:
 
  
|
+
Msf
  
Standard UK PC keyboards have the pipe symbol on the same key as the backslash symbol, at the bottom,
+
.msf
left hand side of the keyboard. So pressing the Shift key and the backslash key together will give you the
 
pipe symbol.
 
On some keyboards, the pipe symbol is at the top left hand side, on the same key as the backtick. To type a
 
pipe symbol on such keyboards, hold down the key Alt Gr and hit the back tick ( ` ) key (left of the number
 
1 key).
 
An example of when you want to use a pipe would be if you wanted to list all the files in a directory, but
 
there are too many to fit on a single page. You probably saw this when you listed the contents of /usr/bin
 
back in Ex. 1-4.
 
You can pipe the output of the ls command (a list of files) into the less command, which will allow you to
 
view the list page by page. To list the files in /usr/bin and view them page by page, the command would be:
 
ls /usr/bin | less
 
Another useful command to use with pipes is the wc command, which stands for wordcount. By default, wc
 
returns the number of newlines, words and bytes in a file. Or you can tell wc to return just the number of
 
lines by using the -l parameter (see the manpage for wc).
 
For example, you could find out how many files you had in a directory by typing:
 
ls | wc -l
 
  
26
+
Multiple sequence alignment format
 
+
This was the standard output format from some of the suite of programs
�Diff, Grep and Sort
+
called GCG. The format is still sometimes used.
In this section, we look briefly at three very useful commands: diff, grep and sort. As with all the commands
+
Other multiple alignment formats are more generally used and thus are
covered today, we recommend that you read the manual page for more information about how these work
+
often a better option to choose if you have a choice.
and what options are available.
 
  
Diff
+
= Nexus =
diff compares files line by line and reports the differences between the files. In fact, diff can be used for
 
more involved tasks as well, like comparing the contents of directories. This can be very useful when you are
 
looking for changes that you or someone else has made.
 
  
Exercise 1-14
+
.nxs
+
.nex
  
Move into the testdir directory.
+
Multiple sequence alignment format
 +
Used by a number of phylogenetics programs.
  
+
GFF
  
Type diff test.txt mysecondfile.txt to see what diff reports to you.
+
38
  
+
.gff
  
Type cat mysecondfile.txt | diff - test.txt
+
A format for describing genes and other features associated with DNA,
 +
RNA and Protein sequences. Not generally used as input for analyses.
  
In the above command the hyphen (-) refers to the information being given to diff through the pipe. That is,
+
== Naming files and the danger of over-writing previous results ==
the information resulting from the command cat mysecondfile.txt is put directly into the diff command.
+
Many programs will suggest a name for your results file. Sometimes this name is generated by taking the
Obviously, in this instance it would be easier just to give the name of the file, mysecondfile.txt, but there
+
beginning of the name of your input file, and adding a new suffix. However, sometimes it is just a generic
are many instances where being able to use – to mean “what I am sending in via the pipe” can be useful.
+
name like prettyplot.ps or clustalw.aln. We encourage you to change generic names to something
 +
meaningful.
 +
Apart from the fact that filenames like prettyplot.ps give you little idea what is in the file, if you do not
 +
change the name, the next time a file of the same name is generated, you will overwrite previous results.
  
Grep
+
A common problem: what is a text file and what is not
grep stands for global regular expression print; you use this command to search for text patterns in a file
+
If you didn't work through the section on text files in part 1 we suggest you do so now. This part reiterates
(or any stream of text). Eg try this.
+
the key points.
grep "adge" /usr/share/dict/words
+
Sequence data are usually stored in text or binary files. Text files contain data you can look at in a text editor.
You can also use flexible search terms, known as regular expressions, in your grep searches. You have
+
Binary files are not human readable. The file formats referred to in the table above are all text formats.
already used glob pattern expressions in this practical, but regular expressions are somewhat different and
+
Examples of binary formats include ABI sequences and SFF sequence files.
more powerful. For example, when you listed all files with the pattern tes*embl* you were using a glob
+
Word documents may look like text, but they aren’t. The letters you see on the page of a Word document
pattern comprising explicit characters (e.g. tes) and special symbols (* meaning any character or characters).
+
(or OpenOffice Write, or other word processing programs) are stored along with layout data in a binary
The equivalent in grep would be “tes.*embl.*” where the period signifies any single character and the *
+
format.
signifies any number of repeats.
+
Most sequence analysis programs expect text. Plain old, nothing fancy, text.
Therefore to convert from a shell glob pattern to a regular expression replace each * with .* and each ? with .
+
It is an unusual situation to need to use sequence data that has been stored as a Word document (if it is not
. You also need to enclose the expression in quotes to tell the shell not to try and interpret it as a glob.
+
unusual to you, you are probably doing things the hard way!). To get a text document when using Word,
Unmodified glob patterns fed to grep but will not work as intended. For example the pattern tes* in grep
+
save it as text only.
means te followed by any number of s characters in sequence (te, tes, tess, tesss, ...). The question mark
 
now signifies optionality – so tes? means te followed by zero or one s character (te, tes). Regular
 
expressions are found in several places other than grep, most notably in the Perl scripting language. The full
 
syntax is extensive and powerful but is beyond the scope of this course, so back to the grep command itself...
 
grep requires a regular expression pattern as a parameter, and prints all the lines in a file containing that
 
pattern.
 
grep is especially useful in combination with pipes as you can filter the results of other commands.
 
For example, perhaps you only want to see only the information in an EMBL file relating to the origin of the
 
sequence, that is, the DE line. You do not need to search the file in an editor, you can just grep for lines
 
beginning in DE, as in the next exercise.
 
27
 
  
�Exercise 1-15
+
= Rule of thumb =
+
If you are using Word or any other word processing program at any stage your work with sequences, then it is
 +
very likely that your life could be made a lot easier.
 +
Please seek advice about other ways to handle your data. You will almost certainly save yourself time and
 +
frustration. Honest.
  
While in the bioinf_files directory, type the command: grep "DE" hsy14768.embl
+
== Exercise 2-2a ==
What is this command doing?
+
A useful Linux command to find out what type of file you are dealing with is file. This does not
 
+
look at the filename but interrogates the file contents directly.
Can you see why the above command results in the output you see?
+
In your bioinf_files directory is the file example.xls. Move into your bioinf_files directory
An explanation of this command can be found below this exercise box.
+
if you are not already there and try running the command
+
* file example.xls
 +
* In the bioinf_files directory is a file called testseq1.embl. Try running the command
 +
file testseq1.embl
  
Try the commands: grep "^DE" hsy14768.embl and grep -x "DE.*" hsy14768.embl
+
= GZipped files in bioinformatics =
What are the ^ symbol and the -x parameter in these commands doing?
+
gzip is a simple compression program, which you met right at the start of this course when you unpacked a
Check the manpage for grep to be sure.
+
.tar.gz file. Any file can be compressed with gzip and .fastq.gz is now particularly popular as it saves a lot of
 
+
disk space. Some programs deal with .fastq.gz files directly, but for others you have to gunzip them first.
+
You can unpack the file on disk or use pipe syntax to feed it directly to your application. The zcat command
 
+
prints out the uncompressed contents of a gzipped file, so something like
Try the command: cat hsy14768.embl | grep "^DE". Does that do what you expected?
+
zcat some_file.fastq.gz | some_app will work in many situations. Remember that the "" by convention tells the application to process the data
 
+
received via the pipe. This way you never have to store the big uncompressed file on disk.
+
bzip2 and xz are similar compression programs. The tools bunzip2/bzcat and unxz/xzcat are provided to
 
+
unpack these files from the command line, but if in doubt just click on the file in the File Browser. The
Move to your home directory and type ls –lR
+
graphical File Roller application will know how to unpack these and more file types.
Read the manual page for ls if it is not clear what this command returns.
 
 
 
 
 
 
Use the above command with a pipe and a grep command to search for files created or
 
 
 
modified today.
 
 
 
 
List the files in the bioinf_files directory and use the grep command to look for those containing the
 
 
 
characters d4.
 
The first command in the previous exercise searches all the text in the hsy14768.embl file and returns the
 
lines in which it finds the letter D followed by the letter E.
 
The second command in the exercise also returns lines in the file that have a letter D followed by a letter E,
 
but only where DE is found at the beginning of a line. This is because the ^ symbol means “match at the
 
beginning of a line”. The $ symbol can be used similarly to mean “at the end of a line”. These are known as
 
anchors. Passing the -x flag to grep tells it to automatically anchor both ends of the search pattern.
 
What this anchoring does in the example above is return to you just the organism information in the embl
 
file. This is because none of the other lines returned in the previous command started with DE, they just
 
contained DE somewhere in them. This is an example where knowing how information is stored in an given
 
file, along with a few basic Linux commands, allows you to retrieve information quickly.
 
Another common example is counting how many sequences are in a set of multi-fasta files. We can do this
 
with pipes between the commands cat, grep and the ever-handy wc, which here we use to count lines found
 
by grep.
 
cat *seqs.fasta | grep "^>" | wc -l
 
Each sequence in a fasta file starts with a header line that begins with a > . The above command streams the
 
contents of all files matching the glob pattern *seqs.fasta through a search with grep looking for lines that
 
start with the symbol > . The quotes around the pattern ^> are necessary, as otherwise it is interpreted as a
 
request for redirection of output to a file, rather than as a character to look for. As before, the ^ symbol
 
means “match only at the beginning of the line”.
 
The output of this grep search is sent to the wc command, with the -l indicating that you want to know the
 
number of lines – ie. the number of headers and by implication the number of sequences.
 
So a synopsis of the command above is: Read through all files with names ending seqs.fasta and look for all
 
the header lines in the combined output, then count up those lines that matched and return the number to
 
screen.
 
We cover sequence formats later on in part 2 of the tutorial.
 
28
 
 
 
�Environment Variables
 
We have seen that the way commands run can be modified by the options passed on the command line.
 
Some commands also read values called environment variables which affect their behaviour. Environmental
 
variables are set within the shell via the export command and are passed to any processes you run. This is
 
useful when you want to set some parameter that is common to all invocations of a command, or applies
 
across several commands. For example, your favourite text editor may be, say, Gedit, or Nano, or Vim, or
 
Emacs. In the shell you can say:
 
export EDITOR=vim
 
Now any command that wants to run a text editor knows what your preferred editor is. Within the shell you
 
can get at the current value of en environment variable by prefixing it with a $ sign, eg.
 
echo $EDITOR
 
 
 
prints the current value of the EDITOR environment variable to the screen
 
 
 
The printenv command dumps all environment variables. Note that environment variables are only set in
 
the current shell and are not saved by default, so if you run a command in another terminal or close and
 
restart the terminal any values you set will be lost. For information on making the settings permanent by
 
editing your .zshrc file see the user guide under Supported Shells.
 
 
 
Exercise 1-16
 
 
 
 
Give the command: export VAR1=hello (with no spaces around the = sign) then:
 
◦ echo $VAR1
 
◦ echo $ VAR1
 
◦ echo "$VAR1"
 
◦ echo '$VAR1'
 
 
 
 
 
 
Start a new terminal window by typing: gnome-terminal &
 
◦ Within this new terminal: echo $VAR1
 
 
 
 
 
 
Start a second new terminal by right-clicking the icon in the Dash and selecting New Terminal
 
◦ Within this new shell: echo $VAR1
 
 
 
 
 
 
Go back to the original shell window
 
◦ unset VAR1
 
◦ echo $VAR1
 
 
 
 
 
 
Has this affected either of the other two shells you started? Check them:
 
◦ echo $VAR1
 
 
 
Environment variables are inherited when one process starts another, much like genetic material is inherited
 
when a cell divides. Hopefully this explains the behaviour you see in the exercise above. When you start a
 
terminal from en existing shell it inherits the environment from that shell. When you start one from the
 
system menu it inherits just the base system environment. Furthermore, once a program is running no
 
external program can modify its environment variables.
 
 
 
29
 
 
 
�Changing permissions on files and directories
 
Every file on the system has a set of permissions on it that dictate who on the system can read, change or
 
delete, or execute the file. By default, all the files you create in your account are readable, changeable or
 
executable by you. However, you can grant other users permissions to access parts of your account if you
 
wish.
 
Below is some basic information about file permissions. Since there is only one user on the live system this
 
isn't really relevant to your current setup. If you are working on a shared system and want to set up access to
 
your files for other people on the system, please get advice from your system administrator.
 
The command to change permissions is chmod. You have to specify who you are modifying the permissions
 
of, what the new permissions are, and what file or directory to act on.
 
The format of the chmod command is:
 
chmod who ± permissions filename(s)
 
who can be:
 
u
 
g
 
o
 
a
 
 
 
means user and refers to the owner of the file
 
means group, and refers to the group the file belongs to
 
means others, everyone on your systems apart from those above
 
means all three, i.e. user, group and others
 
 
 
permissions can be:
 
r
 
w
 
x
 
 
 
means read permission
 
means write permission
 
means execute permission
 
 
 
Each user has a default group and possibly extra group memberships. Use the id command to view your
 
group memberships. When you create a new file it will be owned by you and by your default group. If you
 
are a member of additional groups, you can switch the file to any of those groups using the chgrp command.
 
(Please refer to the manual pages for the commands chown, chgrp and chmod for more on this topic.)
 
For simplicity, let us assume that you and a co-worker have both been put in the default group labusers and
 
wish to share your data files found in ~/bioinf_files.
 
chmod a+x ~
 
chmod g+rx ~/bioinf_files
 
 
 
chmod g+r ~/bioinf_files/*
 
directory
 
 
 
give permission to anyone to execute, in this case, so
 
that they can move through, your home directory.
 
give permission to people in the group to access files in the
 
bioinf_files directory under your home directory, including
 
listing the files with ls
 
give permission to people in the group to read the files in the
 
 
 
The first command could have been “chmod g+x ~”. This would unlock your home directory only to users
 
in the labusers group. However, enabling access for anyone is generally safe, as long as permissions on the
 
files and subfolders prevent anyone from actually accessing them, and unless you set a+w in addition to a+x
 
nobody but you will be able to list the files in your home directory.
 
 
 
30
 
 
 
�Some other useful information
 
Copying and pasting text
 
Most Linux applications, including the shell terminal windows, have Copy and Paste options in the Edit
 
menu or available in the pop-up menu when you click the right mouse button. You can copy text within
 
the application or between different applications. There is also a quick way to copy text within the
 
terminal by highlighting text to select it, and using the middle mouse button to paste the text.
 
The exact way to select, copy and paste text from within a terminal windows depends on how your mouse
 
has been set up. Normally you would highlight text by dragging the mouse across it with your left mouse
 
button depressed to copy the text, and paste by clicking the middle mouse button (or the two outer mouse
 
buttons pressed simultaneously). Note that within the terminal it doesn't matter where you click the middle
 
mouse button – the text will always be inserted at the current cursor position.
 
 
 
The simple way to stop a process
 
Sometimes a command or program you run in the terminal goes on too long, or is obviously doing something
 
you did not plan. If there is no obvious way (such as a menu option or button) to stop the program running,
 
try using Control and c (more commonly written as Ctrl-c). i.e. hold down the Control key and hit the c
 
key. This requests the program to stop immediately, though the program may ignore the request.
 
Note that this is the same key combination used in most graphical applications for copying text. Remember
 
that highlighting text in a Linux terminal automatically copies it into the buffer – you don't need to press
 
Ctrl-c before pasting with the middle button.
 
 
 
Putting a command to one side
 
Sometimes, you are in the middle of typing a long command, and you suddenly realise you need to do
 
something else in the terminal, like list the current directory contents or check the manpage, before you run
 
the command. Z-shell provides a handy shortcut for this: Alt-q. When you press Alt-q the current
 
command disappears and you have a new empty prompt, but the unfinished command has been remembered
 
and will reappear with the next prompt ready for you to edit and run it.
 
An alternative is to hit Ctrl-c. Within the shell, Ctrl-c does not cause the shell to exit but it does cause the
 
current command to be abandoned and a fresh prompt to appear. Unlike with Alt-q the unfinished command
 
will still be visible in the terminal display so you can select it and paste it back in with the middle button if
 
you decide you want it after all. (Try it!)
 
 
 
Logging out of a session
 
To logout, you can press the Power Icon on the far right of the top taskbar (Figure 2) and choose the Log
 
Out option.
 
To shut down the machine, you can choose the Shut Down option on the same menu. If you are working on
 
the console of a machine with users apart from you, then please check with your system administrator before
 
powering down the machine. Other people might want to log in remotely.
 
 
 
Clearing your terminal of text
 
Your terminal windows can fill up with lots of text, and it can become difficult to see the information you
 
want because of all the clutter. You can clear the terminal window by typing
 
clear
 
 
 
31
 
 
 
�Accessing a running program or working with others interactively
 
If you just run a job and then close down the terminal you ran it from, normally the job will be terminated. It
 
would be nice to be able to leave a long job running and be able to log out and then log back in again to see
 
how it is progressing. This is especially true if you log in remotely via SSH and experience network
 
disruptions, or if you run programs that can take quite a long time, but ask you for input periodically.
 
Luckily, there is a tool that makes it possible to leave programs running with no danger of them terminating
 
if you log off or your terminal is closed. In addition, when you log back into your system, either locally or
 
remotely, you can “re-attach” to your earlier session so it feels like you are picking up where you left off, in
 
the same window you were running your program from.
 
The utility that allows you to do this is called screen. It must be run before you start running other programs
 
in your window. Screen can also allow two people on different machines to work in the same session – i.e.
 
Real time collaborative editing is possible with screen.
 
Unfortunately, how to work with screen is beyond the scope of this course. However, the link below provides
 
a useful beginners tutorial about screen and multi-user sessions:
 
https://www.linode.com/docs/networking/ssh/using-gnu-screen-to-manage-persistent-terminalsessions#screen-basics
 
An extensive list of command options can be found in the screen manpage (ie. type man screen).
 
There are many useful commands available on Linux and we cannot begin to cover them in this course. We
 
recommend that you consider buying a book to help you learn how to use Linux efficiently.
 
 
 
Accessing your machine including a full graphical desktop - remotely
 
Bio-Linux is set up for secure remote access. We can't demonstrate this on the Live system but it is well
 
worth knowing that if you have an installed Bio-Linux system you can connect to it securely over the
 
network, so long as your account is enabled in the ssh group and you have network access to the machine (ie.
 
not blocked by a site firewall)
 
You can connect to your (installed) Bio-Linux system remotely using X2Go software. If you download an
 
X2Go client to another Windows, Linux or Mac system, you can connect to an installed Bio-Linux system
 
and run a full, graphical, desktop session remotely. Further details on how to do this can be found on the
 
website at:
 
http://environmentalomics.org/bio-linux-remote-access
 
Note that due to limitations of the remote protocol, X2Go will use a fallback desktop “MATE” session which
 
is slightly different to the default “Unity” desktop environment described in this tutorial.
 
 
 
32
 
 
 
�Part Two: Introduction to Bioinformatics on Bio-Linux
 
This section of the tutorial introduces you to running bioinformatics software on Bio-Linux, including how
 
to find out what is available for particular types of bioinformatics tasks, some options you have for running
 
programs on the system, and where to find documentation about the software on the system. This course
 
does not cover the detailed use or understanding of any particular piece of software.
 
You should read through the general information in the next few pages, then look at which specific programs
 
are of most interest to you.
 
The main points we hope you take away after completing this section of the tutorial are:
 
a) You can discover and run bioinformatics tools even if you have not explicitly been taught
 
how to use them.
 
b) If you have repetitive tasks to carry out, chances are there are ways of fully or partially
 
automating them.
 
c) Web interfaces are easy, and have certain benefits, but a competence with the command line
 
gives you access to more possibilities and sometimes these will suit your needs better.
 
 
 
Documentation and Help for Bioinformatics Software on Bio-Linux
 
There are a number of sources of information about the bioinformatics software on Bio-Linux, including
 
 
 
 
Bio-Linux bioinformatics documentation
 
 
 
 
 
 
local copies of software documentation – look in /usr/share/doc
 
 
 
 
 
 
options under the help menus in some graphical programs
 
 
 
 
 
 
web pages
 
 
 
 
 
 
journal articles.
 
 
 
Bio-Linux Bioinformatics Documentation
 
Categorised information about bioinformatics software on the Bio-Linux system can be accessed via the
 
Bioinformatics Docs icon on the left hand side of your desktop. Software can be listed by name or by
 
functional category.
 
The information for each program includes an overview of what it does, with links to local documentation
 
when available, as well as links to information on the internet.
 
 
 
An apology – the Bioinformatics Docs are currently (in 2014) out-of-date and in severe need of
 
attention. The plan is to integrate this catalogue with the ELIXIR tools registry but this work will
 
take many months to complete.
 
This notwithstanding, we highly recommend that you read the documentation for any programs
 
you intend to run.
 
This is especially important for programs that use heuristic algorithms (methods involving some
 
level of approximation, such as BLAST), and those that output numerical results.
 
33
 
 
 
�Exercise 2-1
 
 
 
 
Click on the Bio-Linux Documentation icon on the desktop, then on Bioinformatics Docs
 
 
 
 
 
 
Select a category under the Browse by Category section.
 
 
 
Click on the names of any of the programs that might interest you and view the information
 
in the resulting web page.
 
 
 
 
Return to the search form and click on the link to List all categories. This shows a view of
 
all the documented software according to the functional category (or categories) they are listed
 
in.
 
 
 
 
Please refer to the bioinformatics documentation throughout this tutorial to find out more about the
 
programs introduced, or look on-line. Most current software will have web pages and online resources
 
for users. For example QIIME has a very active user community.
 
If you know of a good information resource for a program on Bio-Linux that is not mentioned in our
 
bioinformatics documentation system, or you have any problems with the system, please let us know by
 
emailing us at helpdesk@nebc.nerc.ac.uk.
 
 
 
Help Functions within the Programs
 
Documentation is available from within many programs. For example, many graphical programs have a Help
 
menu or button; many command line programs provide help if you type the name of the program followed
 
by –h, –help or --help. Some programs even have their own manual pages that can be accessed by typing
 
man followed by the program name.
 
 
 
Example data for this tutorial
 
The sequences referred to in this tutorial can be unpacked from the file
 
/usr/local/bioinf/documentation/bio-linux/intro_course/bioinf_files.tar.gz.
 
If you have just done the associated Introduction to Linux tutorial, you will already have these files – please
 
move on to the next section of the tutorial.
 
If you have joined the tutorial at this point, please refer to Exercise 1-1, parts b, c and d to download and
 
unpack the necessary sample data files.
 
For some parts you will also need qiime_tutorial_data.tar.gz, mothur_tutorial_data.tar.gz and
 
assembly_taster.tar.xz which are available in the same directory.
 
 
 
34
 
 
 
�Interface choices
 
Software can be run on the command line, via graphical programs on your computer, via web interfaces, via
 
web services and/or via scripts. Bioinformatics programs can often be run using more than one of these
 
options. Each type of interface has pros and cons. We have summarised some of these for reference below.
 
Interface
 
Command line
 
 
 
Pros
 
 
 
Cons
 
 
 
Fast to run once you know the program
 
 
 
Have to learn the syntax
 
 
 
Very flexible; usually many options
 
 
 
Have to find out what options are available
 
 
 
Type out the command Repetitive tasks are easy to run or automate
 
and press enter
 
Easy to log in remotely and carry out tasks
 
Easy to run; don't have to remember the
 
Prompted command command line syntax
 
line
 
Easy to log in remotely and carry out tasks
 
 
 
Easy to forget the diversity of options for a
 
program because of the temptation to just
 
reply to prompts provided
 
 
 
Type out the command
 
and respond to
 
prompts on screen
 
 
 
Slower to get running than “pure” command
 
line
 
 
 
Graphical interface
 
 
 
Often more intuitive and visually pleasing
 
than the command line
 
 
 
Can be slower to use than the command line,
 
especially for repetitive tasks
 
 
 
Extensive help is often available via a menu
 
option or button
 
 
 
For some programs, the command line
 
version provides more functionality.
 
 
 
Some programs (not all!) can be run by
 
Start the program and clicking an icon in the Applications |
 
Bioinformatics menu on your system.
 
interact via menus
 
 
 
You may need your system admin to set up
 
programs so that you can run graphical
 
programs when logging in remotely
 
 
 
Appropriate for visual tasks such as
 
alignment editing, detailed annotation
 
checking, etc.
 
Usually intuitive
 
 
 
Web interface
 
 
 
Can provide functionality not available via
 
locally-run programs such as access to
 
important data resources or results presented
 
in useful formats, e.g. including links to
 
related data resources, graphics, etc.
 
 
 
Run via a web browser
 
window, usually at a Some websites allow a certain degree of
 
“pipelining”, where the outputs of one
 
remote site
 
program can intuitively be supplied as input
 
to another.
 
 
 
Can be slow to use relative to the command
 
line, especially for repetitive tasks
 
You are subject to the rules and restrictions
 
of the site you are working on (e.g. data
 
volume, number of tasks, options available,
 
etc.)
 
You may not want to send private data over
 
the internet (e.g. if you are applying for a
 
patent?)
 
You can be subject to the whims of network
 
connectivity
 
 
 
35
 
 
 
�Web services
 
Runs tasks over the
 
internet from a
 
program, usually
 
locally installed or run
 
via java webstart.
 
 
 
Can bring together the ease of a locally run
 
program with the data and computing
 
resources of a remote site
 
 
 
You are dependent on the consistency of the
 
remote server where the functions you need
 
Can be used via graphical programs or scripts are running
 
You are dependent on the functionality the
 
remote site offers; this may not be as
 
extensive as the functionality you get locally
 
for some programs.
 
Very flexible
 
 
 
Scripts
 
 
 
You are dependent on network connectivity
 
 
 
Great for automating tasks
 
 
 
Using a small
 
Great for carrying out customised tasks
 
program that runs a
 
program or programs Straightforward to learn enough to alter
 
existing scripts to do exactly the task you
 
for you
 
want.
 
 
 
You have to write the script or find a script
 
that does the job. This means learning a
 
programming language (or asking someone
 
who knows one to help you)
 
 
 
For repetitive tasks, we highly recommend the use of the command line, workflow software and/or scripting.
 
 
 
General points about working with bioinformatics programs
 
Sequence formats
 
A simple thing that often trips people up is sequence formats. There are many different sequence formats;
 
the reasons for this are both historical and functional.
 
Historically, when people first started writing analysis programs for molecular data, they designed a format
 
that they felt suited their needs. As time went on, numerous formats came into existence. We live with the
 
legacy of this. We must know what format our data is in, and whether the program we want to run can use
 
data in that format.
 
Functionally, a program may require information that can be included with data held in certain formats, but
 
not others. For example, EMBL format files can, in addition to the sequence data itself, contain descriptive
 
information about a sequence, such as its features. In contrast, plain format contains nothing inside the file
 
except the sequence data, while FASTA format allows a small amount of information about a sequence to be
 
given in a header line and FASTQ adds read quality information alongside the sequence. Clustal and msf
 
formats handle multiple aligned sequences, while phylip and nexus format files contain aligned sequences as
 
well as information relevant to phylogenetic analysis programs.
 
 
 
To analyse data, it must be presented to the analysis program in a format the progam
 
understands.
 
This seems obvious, but frequent errors (or worse, misleading results) occur when the data entered into
 
a program is not appropriate.
 
 
 
36
 
 
 
�Converting files to different sequence formats used to be a frequent, and often time consuming, task in
 
bioinformatics. Luckily there are file conversion programs that take care of this easily for many formats. In
 
addition, many program understand more than one format.
 
Some common bioinformatics sequence formats, along with common filename conventions used for those
 
formats, are listed in the table that follows the next section.
 
We recommend the following page for more information and examples of common bioinformatics file
 
formats:
 
http://www.molecularevolution.org/resources/fileformats
 
 
 
File naming conventions in bioinformatics
 
The suffix, (the part of the filename after the final dot), is often used to denote to you, and other people, what
 
the format of the data inside the file is.
 
For example, the common suffix for clustal formatted alignments is aln. .A bioinformatics file that ends in
 
.aln is usually assumed to be a clustal formatted alignment file.
 
Another multiple sequence alignment format is phylip. A common suffix used on files containing sequences
 
in phylip format is phy.
 
Common suffices used for files containing data in particular formats are listed in the table following this
 
section. We highly recommend that you follow conventions when naming your data files.
 
Benefits to following the convention for filename endings include:
 
 
 
 
You will know your data format just by looking at the name of the file.
 
 
 
Following standard conventions, (rather than making up your own naming system), makes it
 
easier for other people looking at your files, (e.g. collaborators, or people helping you); they will
 
know the data format just by looking at the name.
 
 
 
 
Some graphical programs have filters set so that only files with particular suffices will be
 
listed in the file browser window when you try to load some data. If you use conventional
 
filename endings, this is less likely to cause problems for you.
 
 
 
 
Certain programs use information in the filename to interpret aspects of the data, (not just the data format).
 
Such programs have strict naming conventions for the whole filename. For example, some sequence
 
assembly programs either require, or are benefited by, defined naming schemes for sequence traces. The
 
filename will inform them about which sequences are read pairs, what direction sequence reads are in, and
 
other information relevant to assembly or visualisation. You will need to read the program documentation to
 
find out what is required in such instances.
 
 
 
You are not restricted to naming your files in any particular way but we highly recommend that you
 
follow the convention for the type of file you are generating/saving.
 
Following file naming conventions from the beginning will save you, and your collaborators,
 
a lot of time!
 
 
 
37
 
 
 
�Common bioinformatics file formats
 
Format
 
Embl or
 
swissprot
 
 
 
Some common
 
filename endings
 
.dat
 
.embl
 
.sprot
 
.swiss
 
 
 
Comments
 
Usually these files, along with genbank files, contain feature information
 
as well as sequence.
 
Embl and Swisprot (or Uniprot) format are the same. Embl files contains
 
nucleotide sequences and Uniprot files contain peptide sequences.
 
Files downloaded from EMBL or Uniprot websites use the suffix .dat.
 
Often these are compressed with gzip, and so end in .dat.gz
 
Files generated by individuals in embl format will tend to end in .embl.
 
 
 
Genbank
 
 
 
.seq
 
.gb
 
.genbank
 
 
 
These files, along with embl and swissprot files, usually contain feature
 
information as well as sequence.
 
Individuals using this format, usually use the .gb or .genbank suffix. The
 
NCBI usually uses .seq for genbank sections.
 
 
 
FASTA
 
 
 
FASTQ
 
 
 
Plain
 
 
 
.fasta
 
.fsa
 
.fa
 
 
 
Possibly the most common sequence format.
 
 
 
.fastq
 
.fq
 
 
 
Very common for NextGen reads. Like FASTA with extra quality info
 
per sequence.
 
Alternative extensions may indicate the type of sequencing technology
 
- .fastqsanger, .fastqsolexa, etc.
 
 
 
.pln
 
.staden
 
.sdn
 
 
 
Not commonly used, as the file contents contain nothing but the sequence
 
itself; the only identifier of the sequence is in the filename.
 
 
 
It may contain nucleotide or peptide sequence(s) and a single-line header
 
per sequence.
 
 
 
Staden programs use the plain format, accounting for the last two of the
 
file suffices given.
 
Clustal
 
 
 
.aln
 
 
 
Multiple sequence alignment format
 
Originally from the clustalw program, but now recognised by many
 
programs that accept or output multiple sequence alignments.
 
 
 
Phylip
 
 
 
.phy
 
.phylip
 
 
 
Multiple sequence alignment format
 
Used by the Phylip suite of programs and many others, especially those
 
associated with phylogenetic analysis.
 
 
 
Msf
 
 
 
.msf
 
 
 
Multiple sequence alignment format
 
This was the standard output format from some of the suite of programs
 
called GCG. The format is still sometimes used.
 
Other multiple alignment formats are more generally used and thus are
 
often a better option to choose if you have a choice.
 
 
 
Nexus
 
 
 
.nxs
 
.nex
 
 
 
Multiple sequence alignment format
 
Used by a number of phylogenetics programs.
 
 
 
GFF
 
 
 
38
 
 
 
.gff
 
 
 
A format for describing genes and other features associated with DNA,
 
RNA and Protein sequences. Not generally used as input for analyses.
 
 
 
�Naming files and the danger of over-writing previous results
 
Many programs will suggest a name for your results file. Sometimes this name is generated by taking the
 
beginning of the name of your input file, and adding a new suffix. However, sometimes it is just a generic
 
name like prettyplot.ps or clustalw.aln. We encourage you to change generic names to something
 
meaningful.
 
Apart from the fact that filenames like prettyplot.ps give you little idea what is in the file, if you do not
 
change the name, the next time a file of the same name is generated, you will overwrite previous results.
 
 
 
A common problem: what is a text file and what is not
 
If you didn't work through the section on text files in part 1 we suggest you do so now. This part reiterates
 
the key points.
 
Sequence data are usually stored in text or binary files. Text files contain data you can look at in a text editor.
 
Binary files are not human readable. The file formats referred to in the table above are all text formats.
 
Examples of binary formats include ABI sequences and SFF sequence files.
 
Word documents may look like text, but they aren’t. The letters you see on the page of a Word document
 
(or OpenOffice Write, or other word processing programs) are stored along with layout data in a binary
 
format.
 
Most sequence analysis programs expect text. Plain old, nothing fancy, text.
 
It is an unusual situation to need to use sequence data that has been stored as a Word document (if it is not
 
unusual to you, you are probably doing things the hard way!). To get a text document when using Word,
 
save it as text only.
 
 
 
Rule of thumb
 
If you are using Word or any other word processing program at any stage your work with sequences, then it is
 
very likely that your life could be made a lot easier.
 
Please seek advice about other ways to handle your data. You will almost certainly save yourself time and
 
frustration. Honest.
 
 
 
39
 
 
 
�Exercise 2-2a
 
A useful Linux command to find out what type of file you are dealing with is file. This does not
 
look at the filename but interrogates the file contents directly.
 
In your bioinf_files directory is the file example.xls. Move into your bioinf_files directory
 
if you are not already there and try running the command
 
 
  
file example.xls
+
= Examples of running bioinformatics programs on Bio-Linux =
 
  
In the bioinf_files directory is a file called testseq1.embl. Try running the command
+
== Analysing sequences with QIIME ==
file testseq1.embl
 
  
GZipped files in bioinformatics
+
QIIME (pronounced ‘chime’) is a pipeline for performing microbial community analysis that integrates many third party tools which have become standard in the field. QIIME can run on a laptop, a supercomputer, and systems in between such as multicore desktops. QIIME is now included in the standard Bio-Linux distribution.
gzip is a simple compression program, which you met right at the start of this course when you unpacked a
 
.tar.gz file. Any file can be compressed with gzip and .fastq.gz is now particularly popular as it saves a lot of
 
disk space. Some programs deal with .fastq.gz files directly, but for others you have to gunzip them first.
 
You can unpack the file on disk or use pipe syntax to feed it directly to your application. The zcat command
 
prints out the uncompressed contents of a gzipped file, so something like
 
zcat some_file.fastq.gz | some_app will work in many situations. Remember that the "–" by convention tells the application to process the data
 
received via the pipe. This way you never have to store the big uncompressed file on disk.
 
bzip2 and xz are similar compression programs. The tools bunzip2/bzcat and unxz/xzcat are provided to
 
unpack these files from the command line, but if in doubt just click on the file in the File Browser. The
 
graphical File Roller application will know how to unpack these and more file types.
 
  
40
+
As an example, we will use data from a study of the response of mouse gut microbial communities to fasting (Crawford et al., 2009). To make this tutorial run quickly on a personal computer, we will use a subset of the data generated from 5 animals kept on the control ad libitum fed diet, and 4 animals fasted for 24 hours before sacrifice. At the end of our tutorial, we will be able to compare the community structure of control vs. fasted animals. In particular, we will be able to compare taxonomic profiles for each sample type, differences in diversity metrics within the samples and between the groups, and perform comparative clustering analysis to look for overall differences in the samples.
  
�Examples of running bioinformatics programs on Bio-Linux
 
Analysing sequences with QIIME
 
QIIME (pronounced ‘chime’) is a pipeline for performing microbial community analysis that
 
integrates many third party tools which have become standard in the field. QIIME can run on a
 
laptop, a supercomputer, and systems in between such as multicore desktops. QIIME is now
 
included in the standard Bio-Linux distribution.
 
As an example, we will use data from a study of the response of mouse gut microbial communities
 
to fasting (Crawford et al., 2009). To make this tutorial run quickly on a personal computer, we will
 
use a subset of the data generated from 5 animals kept on the control ad libitum fed diet, and 4
 
animals fasted for 24 hours before sacrifice. At the end of our tutorial, we will be able to compare
 
the community structure of control vs. fasted animals. In particular, we will be able to compare
 
taxonomic profiles for each sample type, differences in diversity metrics within the samples and
 
between the groups, and perform comparative clustering analysis to look for overall differences in
 
the samples.
 
 
To process our data, we will perform the following steps, each of which is described in more detail
 
To process our data, we will perform the following steps, each of which is described in more detail
 
in the Data Analysis Steps:
 
in the Data Analysis Steps:
 
 
 
 
 
 
 
 
 
Filter the sequence reads for quality and assign multiplexed reads to starting samples by
 
Filter the sequence reads for quality and assign multiplexed reads to starting samples by
 
nucleotide barcode.
 
nucleotide barcode.
Line 2,226: Line 1,537:
 
41
 
41
  
�Note: Commands to type are shown in grey boxes like this. Some commands in QIIME are too
+
== Note: Commands to type are shown in grey boxes like this. Some commands in QIIME are too ==
 
long to print on one line, so where you see ... , you need to continue typing the command on the
 
long to print on one line, so where you see ... , you need to continue typing the command on the
 
same line.
 
same line.
  
Preparation
+
= Preparation =
 
First, we must copy the tutorial data to your home directory and extract it:
 
First, we must copy the tutorial data to your home directory and extract it:
 
cd
 
cd
Line 2,257: Line 1,568:
 
below will only be recognised within the special QIIME shell.
 
below will only be recognised within the special QIIME shell.
  
Assign Samples to Multiplex Reads
+
= Assign Samples to Multiplex Reads =
 
The first task is to assign the multiplex reads to samples based on their nucleotide barcode. Also,
 
The first task is to assign the multiplex reads to samples based on their nucleotide barcode. Also,
 
this step performs quality filtering based on the characteristics of each sequence, removing any low
 
this step performs quality filtering based on the characteristics of each sequence, removing any low
Line 2,265: Line 1,576:
 
42
 
42
  
�cd qiime_tutorial_data
+
== cd qiime_tutorial_data ==
 
pwd
 
pwd
 
mkdir out
 
mkdir out
Line 2,286: Line 1,597:
 
information on any barcode errors that were corrected.
 
information on any barcode errors that were corrected.
  
Processing sequences into OTUs
+
= Processing sequences into OTUs =
 
There are several steps to go through to produce the annotated OTUs from the input sequences,
 
There are several steps to go through to produce the annotated OTUs from the input sequences,
 
however the following 5 steps can be called using the ‘pick_de_novo_otus’ command found at the
 
however the following 5 steps can be called using the ‘pick_de_novo_otus’ command found at the
Line 2,314: Line 1,625:
 
43
 
43
  
�4. Make OTU table
+
== 4. Make OTU table ==
 
Tabulates the number of times an OTU is found in each sample, and adds the taxonomic predictions
 
Tabulates the number of times an OTU is found in each sample, and adds the taxonomic predictions
 
for each OTU in the last column if a taxonomy file is supplied.
 
for each OTU in the last column if a taxonomy file is supplied.
Line 2,350: Line 1,661:
 
# Make sure you change the path in the custom_parameters.txt file before running this command
 
# Make sure you change the path in the custom_parameters.txt file before running this command
  
Data to information
+
= Data to information =
 
QIIME has many different ways to visualize and interrogate the data. Here we will explore just a
 
QIIME has many different ways to visualize and interrogate the data. Here we will explore just a
 
few.
 
few.
Line 2,358: Line 1,669:
 
44
 
44
  
�Heatmap
+
== Heatmap ==
 
The QIIME pipeline includes a very useful utility to generate images of the OTU table. You can
 
The QIIME pipeline includes a very useful utility to generate images of the OTU table. You can
 
open this file with any web browser, and will be prompted to enter a value for “Filter by Counts per
 
open this file with any web browser, and will be prompted to enter a value for “Filter by Counts per
Line 2,366: Line 1,677:
 
make_otu_heatmap_html.py -i out/otu_table.biom -o out/otu_heatmap
 
make_otu_heatmap_html.py -i out/otu_table.biom -o out/otu_heatmap
  
Taxonomy Summary Charts
+
= Taxonomy Summary Charts =
 
The taxa of the samples can be visualised at each taxonomic level (see the –L flag).
 
The taxa of the samples can be visualised at each taxonomic level (see the –L flag).
 
Here, summarize_taxa.py produces a text file at the Phylum level (Level 2=Domain, 3=Phylum,
 
Here, summarize_taxa.py produces a text file at the Phylum level (Level 2=Domain, 3=Phylum,
Line 2,373: Line 1,684:
 
plot_taxa_summary.py -i out/taxa_summary/otu_table_L3.txt -l Phylum -o out/taxa_charts -k white
 
plot_taxa_summary.py -i out/taxa_summary/otu_table_L3.txt -l Phylum -o out/taxa_charts -k white
  
Diversity
+
= Diversity =
 
Community ecologists typically describe the microbial diversity within their study. This diversity
 
Community ecologists typically describe the microbial diversity within their study. This diversity
 
can be assessed within a sample (alpha diversity) or between a collection of samples (beta
 
can be assessed within a sample (alpha diversity) or between a collection of samples (beta
Line 2,383: Line 1,694:
 
alpha_rarefaction.py -i out/otu_table.biom -m Fasting_Map.txt -o out/arare -p custom_parameters.txt -t out/rep_set.tre
 
alpha_rarefaction.py -i out/otu_table.biom -m Fasting_Map.txt -o out/arare -p custom_parameters.txt -t out/rep_set.tre
  
Beta
+
= Beta =
 
Beta diversity can be represented in many different ways, shown below. By rarefying the samples to
 
Beta diversity can be represented in many different ways, shown below. By rarefying the samples to
 
the smallest set (in this example dataset, 146 sequences) sample heterogeneity can be removed.
 
the smallest set (in this example dataset, 146 sequences) sample heterogeneity can be removed.
Line 2,399: Line 1,710:
 
45
 
45
  
�make_2d_plots.py -i out/bdiv_even146/unweighted_unifrac_pc.txt -o out/bdiv_even146/unweighted_unifrac_2d
+
== make_2d_plots.py -i out/bdiv_even146/unweighted_unifrac_pc.txt -o out/bdiv_even146/unweighted_unifrac_2d ==
  
 
...
 
...
Line 2,450: Line 1,761:
 
46
 
46
  
�Analysing sequences with MOTHUR
+
== Analysing sequences with MOTHUR ==
 
MOTHUR is another popular pipeline for performing microbial community analysis that integrates
 
MOTHUR is another popular pipeline for performing microbial community analysis that integrates
 
many third party tools which have become standard in the field. MOTHUR is included in the
 
many third party tools which have become standard in the field. MOTHUR is included in the
Line 2,465: Line 1,776:
 
2009. 75(23):7537-41
 
2009. 75(23):7537-41
  
Preparation
+
= Preparation =
 
First, we must copy the tutorial data to your home directory and extract it:
 
First, we must copy the tutorial data to your home directory and extract it:
 
cd
 
cd
Line 2,488: Line 1,799:
 
47
 
47
  
�mothur
+
== mothur ==
  
 
Assign Samples to Multiplex Reads and Quality Filtering
 
Assign Samples to Multiplex Reads and Quality Filtering
Line 2,515: Line 1,826:
 
from.
 
from.
  
Generating Alignment & Distance Matrix
+
= Generating Alignment & Distance Matrix =
 
The first thing we want to do is to simplify the dataset by working with only the unique sequences.
 
The first thing we want to do is to simplify the dataset by working with only the unique sequences.
 
We are not chucking anything here, we are just making the life of your CPU and RAM a bit easier.
 
We are not chucking anything here, we are just making the life of your CPU and RAM a bit easier.
Line 2,532: Line 1,843:
 
48
 
48
  
�filter.seqs(fasta=Fasting_Example.trim.unique.align)
+
== filter.seqs(fasta=Fasting_Example.trim.unique.align) ==
  
 
Next, we want to calculate the column-formatted distance matrix, but we are only interested in
 
Next, we want to calculate the column-formatted distance matrix, but we are only interested in
Line 2,538: Line 1,849:
 
dist.seqs(fasta=Fasting_Example.trim.unique.filter.fasta, cutoff=0.15)
 
dist.seqs(fasta=Fasting_Example.trim.unique.filter.fasta, cutoff=0.15)
  
Classify Sequences
+
= Classify Sequences =
 
We then need to classify our sequences using the MOTHUR version of the “Bayesian” classifier.
 
We then need to classify our sequences using the MOTHUR version of the “Bayesian” classifier.
 
We do this with classify.seqs command using the SILVA-compatible reference file and taxonomy
 
We do this with classify.seqs command using the SILVA-compatible reference file and taxonomy
Line 2,545: Line 1,856:
 
template=data/silva.bacteria.fasta, taxonomy=data/silva.bacteria.silva.tax)
 
template=data/silva.bacteria.fasta, taxonomy=data/silva.bacteria.silva.tax)
  
Renaming Files
+
= Renaming Files =
 
This step is done only to make our life easier by making copies of some files and giving it nice and
 
This step is done only to make our life easier by making copies of some files and giving it nice and
 
short names. The command system() allows you to run programs outside of MOTHUR without
 
short names. The command system() allows you to run programs outside of MOTHUR without
Line 2,555: Line 1,866:
 
system(cp Fasting_Example.trim.unique.filter.silva.wang..taxonomy final.taxonomy)
 
system(cp Fasting_Example.trim.unique.filter.silva.wang..taxonomy final.taxonomy)
  
Clustering Sequences
+
= Clustering Sequences =
 
Now we want to assign these sequences to OTUs for every possible distance up to and including a
 
Now we want to assign these sequences to OTUs for every possible distance up to and including a
 
distance of 0.15. By default, this method uses the average neighbour algorithm.
 
distance of 0.15. By default, this method uses the average neighbour algorithm.
 
cluster(column=final.dist, name=final.names, cutoff=0.15)
 
cluster(column=final.dist, name=final.names, cutoff=0.15)
  
Generating OTU Table and Normalisation
+
= Generating OTU Table and Normalisation =
 
Now that we have a list file, we need to create a table that indicates the number of times an OTU
 
Now that we have a list file, we need to create a table that indicates the number of times an OTU
 
shows up in each sample. This is called a shared file and can be created using the make.shared
 
shows up in each sample. This is called a shared file and can be created using the make.shared
Line 2,572: Line 1,883:
 
49
 
49
  
�count.groups()
+
== count.groups() ==
  
 
From the output we see that the sample with the fewest sequences had 146 sequences in it, so we
 
From the output we see that the sample with the fewest sequences had 146 sequences in it, so we
Line 2,578: Line 1,889:
 
sub.sample(shared=final.an.shared, size=146)
 
sub.sample(shared=final.an.shared, size=146)
  
Classifying OTU
+
= Classifying OTU =
 
The last thing we'd like to do is to get the taxonomy information for each of our OTUs. To do this
 
The last thing we'd like to do is to get the taxonomy information for each of our OTUs. To do this
 
we will use the classify.otu command to give us the majority consensus taxonomy.
 
we will use the classify.otu command to give us the majority consensus taxonomy.
 
classify.otu(list=final.an.list, name=final.names, taxonomy=final.taxonomy)
 
classify.otu(list=final.an.list, name=final.names, taxonomy=final.taxonomy)
  
Converting the shared file to BIOM-format
+
= Converting the shared file to BIOM-format =
 
The make.biom command allows you to convert your shared file to a biom file. Please refer to
 
The make.biom command allows you to convert your shared file to a biom file. Please refer to
 
http://biom-format.org/documentation/biom_format.html for detail.
 
http://biom-format.org/documentation/biom_format.html for detail.
 
make.biom(shared=final.an.shared, contaxonomy=final.an.unique.cons.taxonomy)
 
make.biom(shared=final.an.shared, contaxonomy=final.an.unique.cons.taxonomy)
  
Data to information
+
= Data to information =
 
MOTHUR has many different ways to visualise and interrogate the data. Here we explore just a few.
 
MOTHUR has many different ways to visualise and interrogate the data. Here we explore just a few.
  
Heatmap
+
= Heatmap =
 
Now we'd like to compare the membership and structure of the various samples using an OTUbased approach. Let's start by generating a heatmap of the relative abundance of each OTU across
 
Now we'd like to compare the membership and structure of the various samples using an OTUbased approach. Let's start by generating a heatmap of the relative abundance of each OTU across
 
the 24 samples using the heatmap.bin command.
 
the 24 samples using the heatmap.bin command.
Line 2,611: Line 1,922:
 
50
 
50
  
�venn(shared=final.an.shared, groups=PC.354-PC.355-PC.356-PC.481)
+
== venn(shared=final.an.shared, groups=PC.354-PC.355-PC.356-PC.481) ==
  
Finding and running useful scripts
+
= Finding and running useful scripts =
 
Scripts are small programs written in a scripting language such as Perl or Python or even by compiling
 
Scripts are small programs written in a scripting language such as Perl or Python or even by compiling
 
commands you'd run directly in the shell into a shell script file. Unlike normal binary applications, the
 
commands you'd run directly in the shell into a shell script file. Unlike normal binary applications, the
Line 2,658: Line 1,969:
 
A practical exercise using fastagrep is included in the next section.
 
A practical exercise using fastagrep is included in the next section.
  
Aligning sequences using MUSCLE
+
= Aligning sequences using MUSCLE =
 
Aligning multiple sequences is a very common task, as it is the first step to comparing related sequences.
 
Aligning multiple sequences is a very common task, as it is the first step to comparing related sequences.
 
There are many algorithms for performing gapped global alignments over a set of sequences, most of which
 
There are many algorithms for performing gapped global alignments over a set of sequences, most of which
Line 2,670: Line 1,981:
 
51
 
51
  
�•
+
== • ==
  
 
You want to fine tune the options passed to the aligner
 
You want to fine tune the options passed to the aligner
Line 2,686: Line 1,997:
 
You want to run several alignments at once using a loop or a short script
 
You want to run several alignments at once using a loop or a short script
  
Exercise
+
= Exercise =
 
Plants contain many closely related genes in the cellulose synthase family. Previous studies have examined
 
Plants contain many closely related genes in the cellulose synthase family. Previous studies have examined
 
these in some model organisms, eg maize[ref below]. It might be useful to compare the cellulose synthase
 
these in some model organisms, eg maize[ref below]. It might be useful to compare the cellulose synthase
Line 2,716: Line 2,027:
 
52
 
52
  
�BLAST
+
== BLAST ==
 
The Basic Local Alignment Search Tool (BLAST) searches for regions of local similarity between
 
The Basic Local Alignment Search Tool (BLAST) searches for regions of local similarity between
 
sequences. The program compares nucleotide or protein sequences or patterns to sequence, or sequencerelated, databases and calculates the statistical significance of matches.
 
sequences. The program compares nucleotide or protein sequences or patterns to sequence, or sequencerelated, databases and calculates the statistical significance of matches.
Line 2,726: Line 2,037:
  
 
A few examples of ways to run BLAST, on Bio-Linux or otherwise
 
A few examples of ways to run BLAST, on Bio-Linux or otherwise
+
* Locally installed command line against locally installed BLAST databases
 
 
Locally installed command line against locally installed BLAST databases
 
 
 
 
 
 
Locally installed command line against remote databases
 
 
 
 
  
Locally through options in graphical programs (e.g. under the Run menu in Artemis)
+
* Locally installed command line against remote databases
  
+
* Locally through options in graphical programs (e.g. under the Run menu in Artemis)
  
Remotely through ssh tunnelling or the remote BLAST options in Artemis.
+
* Remotely through ssh tunnelling or the remote BLAST options in Artemis.
  
+
* Remotely on websites such as those available at the NCBI and EBI
  
Remotely on websites such as those available at the NCBI and EBI
+
* Remotely using webservices, either through programs such as Taverna, or through scripting
 
 
 
 
 
Remotely using webservices, either through programs such as Taverna, or through scripting
 
  
 
For this course, we assume that you are familiar with running BLAST searches using at least one web-based
 
For this course, we assume that you are familiar with running BLAST searches using at least one web-based
Line 2,762: Line 2,061:
 
carry out different types of searches, as well as to create databases, reformat reports, etc.
 
carry out different types of searches, as well as to create databases, reformat reports, etc.
  
What this course covers
+
= What this course covers =
 
This course covers how to run BLAST+ programs via the command line and a few simple steps you can take
 
This course covers how to run BLAST+ programs via the command line and a few simple steps you can take
 
to work with more than one sequence at a time. We also cover how to install your own BLAST databases in
 
to work with more than one sequence at a time. We also cover how to install your own BLAST databases in
Line 2,768: Line 2,067:
 
results.
 
results.
  
Why use BLAST on the command line?
+
= Why use BLAST on the command line? =
 
The web resources available for BLAST are highly developed, usually stable, and have access to a much
 
The web resources available for BLAST are highly developed, usually stable, and have access to a much
 
greater set of data than most people will have available locally. They also often provide lovely graphics and
 
greater set of data than most people will have available locally. They also often provide lovely graphics and
Line 2,780: Line 2,079:
 
53
 
53
  
�searches. (This last point has been made easier than ever in the newer BLAST+ programs, where you can, to
+
== searches. (This last point has been made easier than ever in the newer BLAST+ programs, where you can, to ==
 
a certain extent, specify which information to return in a tab delimited format 1.)
 
a certain extent, specify which information to return in a tab delimited format 1.)
  
Line 2,788: Line 2,087:
 
about blast programs.
 
about blast programs.
  
General considerations for database searching
+
= General considerations for database searching =
 
Database searching should be approached like an experiment. In particular: define your aims before your
 
Database searching should be approached like an experiment. In particular: define your aims before your
 
start. This will save you an enormous amount of time, both in terms of time taken doing searches and time
 
start. This will save you an enormous amount of time, both in terms of time taken doing searches and time
Line 2,803: Line 2,102:
 
● Where and how will I store a record of the parameters I've used and the database version I've searched
 
● Where and how will I store a record of the parameters I've used and the database version I've searched
 
with?
 
with?
+
* A very, very brief introduction to BLAST+
 
 
A very, very brief introduction to BLAST+
 
 
BLAST+ includes programs to perform searches with different types of input against databases holding
 
BLAST+ includes programs to perform searches with different types of input against databases holding
 
different types of data. Each search combination is referred to by a particular name and has its own
 
different types of data. Each search combination is referred to by a particular name and has its own
 
command. A table of the basic BLAST “flavours” and what they do is given below.
 
command. A table of the basic BLAST “flavours” and what they do is given below.
  
Blastall flavour
+
= Blastall flavour =
 
blastn
 
blastn
 
blastp
 
blastp
 
blastx
 
blastx
  
Input sequence type
+
= Input sequence type =
 
nucleotide
 
nucleotide
 
peptide
 
peptide
Line 2,830: Line 2,127:
 
translation is created during run)
 
translation is created during run)
  
Database sequence type
+
= Database sequence type =
 
nucleotide
 
nucleotide
 
peptide
 
peptide
Line 2,846: Line 2,143:
 
54
 
54
  
�There are many other programs available as part of the BLAST+ release apart from the ones above. These
+
== There are many other programs available as part of the BLAST+ release apart from the ones above. These ==
 
include blastdbcmd, dustmasker, psiblast, rpsblast+, segmasker and srsearch.. These programs are not
 
include blastdbcmd, dustmasker, psiblast, rpsblast+, segmasker and srsearch.. These programs are not
 
covered here, but are worth learning about for your own work.
 
covered here, but are worth learning about for your own work.
  
How a BLAST database looks on the file system
+
= How a BLAST database looks on the file system =
 
A typical BLAST database consists of three files names with extensions .pin .phr and .psq for protein
 
A typical BLAST database consists of three files names with extensions .pin .phr and .psq for protein
 
databases or .nin .nhr and .nsq for nucleotide databases. These files represent a specially indexed version
 
databases or .nin .nhr and .nsq for nucleotide databases. These files represent a specially indexed version
Line 2,857: Line 2,154:
 
without any extension (see examples). BLAST will know to find and read the three files.
 
without any extension (see examples). BLAST will know to find and read the three files.
  
A simple blastp search
+
= A simple blastp search =
 
The following is a basic blastp command – you can run it from within the course folder.
 
The following is a basic blastp command – you can run it from within the course folder.
 
blastp -db blastdb/sprot –query cd4_cerae.fasta –evalue 0.0001 > cd4_cerae.blastp
 
blastp -db blastdb/sprot –query cd4_cerae.fasta –evalue 0.0001 > cd4_cerae.blastp
Line 2,898: Line 2,195:
 
55
 
55
  
�For the purposes of this tutorial, we will give each BLAST command the explicit location of the BLAST
+
== For the purposes of this tutorial, we will give each BLAST command the explicit location of the BLAST ==
 
database to search.
 
database to search.
  
Exercise
+
= Exercise =
+
* Move into the bioinf_files directory if you are not already there.
 
 
Move into the bioinf_files directory if you are not already there.
 
  
 
List the files in the blastdb subdirectory. The files called sprot.p* are the files that BLAST uses when
 
List the files in the blastdb subdirectory. The files called sprot.p* are the files that BLAST uses when
 
it searches.
 
it searches.
+
* * From within the bioinf_files directory, run the example command given previously, ie:
 
 
 
 
 
From within the bioinf_files directory, run the example command given previously, ie:
 
 
blastp -db blastdb/sprot –query cd4_cerae.fasta –evalue 0.0001 > cd4_cerae.blastp
 
blastp -db blastdb/sprot –query cd4_cerae.fasta –evalue 0.0001 > cd4_cerae.blastp
  
+
* Look at the results file that has been created.
 
 
Look at the results file that has been created.
 
  
 
Try a blastx search on the file unknown.fasta. This time set the evalue to 1 and save the results in
 
Try a blastx search on the file unknown.fasta. This time set the evalue to 1 and save the results in
 
unknown.blastx. The command you use will start like this:
 
unknown.blastx. The command you use will start like this:
+
* blastx -db blastdb/sprot -query unknown.fasta ...???...
 
 
blastx -db blastdb/sprot -query unknown.fasta ...???...
 
 
Recall that a blastx search translates a nucleotide sequence in six frames and searches a peptide database.
 
Recall that a blastx search translates a nucleotide sequence in six frames and searches a peptide database.
+
* Look at the results file.
 
 
Look at the results file.
 
  
 
blastp expects a peptide query file, and blastx expects nucleotides. What would you expect to happen
 
blastp expects a peptide query file, and blastx expects nucleotides. What would you expect to happen
 
if you use an inappropriate BLAST flavour? Try it and see.
 
if you use an inappropriate BLAST flavour? Try it and see.
+
* Formatting BLAST output
 
 
Formatting BLAST output
 
 
You have now seen the default report format for BLAST searches. There are many options available using
 
You have now seen the default report format for BLAST searches. There are many options available using
 
the -outfmt option with a numerical argument between 0 and 11. The default is -outfmt 0.
 
the -outfmt option with a numerical argument between 0 and 11. The default is -outfmt 0.
Line 2,940: Line 2,223:
 
blastx -help | less
 
blastx -help | less
  
Exercise
+
= Exercise =
 
Run either of the above BLAST searches again, this time adding the parameter -outfmt 6 to the
 
Run either of the above BLAST searches again, this time adding the parameter -outfmt 6 to the
 
command. Make sure you change the name of the output file as well, or else just let the results get printed
 
command. Make sure you change the name of the output file as well, or else just let the results get printed
 
to the screen.
 
to the screen.
+
* Look at the results from this search and compare it to what was returned using default formatting. Is it
 
 
Look at the results from this search and compare it to what was returned using default formatting. Is it
 
 
easier or harder to read? Is there information present in one report that is not in the other?
 
easier or harder to read? Is there information present in one report that is not in the other?
+
* Note: BLAST+ programs offer finer control over the format and contents of results returned – see the help
 
 
Note: BLAST+ programs offer finer control over the format and contents of results returned – see the help
 
 
page as mentioned above.
 
page as mentioned above.
 
56
 
56
  
�Handling multiple sequences
+
== Handling multiple sequences ==
 
BLAST makes it easy to deal with a medium-sized number of sequences at once – say up to a few hundred.
 
BLAST makes it easy to deal with a medium-sized number of sequences at once – say up to a few hundred.
 
For thousands of sequences, you will probably want to use the ideas introduced here, in conjunction with
 
For thousands of sequences, you will probably want to use the ideas introduced here, in conjunction with
Line 2,965: Line 2,244:
 
BLAST searching using fasta files containing more than one sequence
 
BLAST searching using fasta files containing more than one sequence
  
Exercise
+
= Exercise =
 
Look at the contents of the file multiseqs.fasta in your bioinf_files directory. How many sequences
 
Look at the contents of the file multiseqs.fasta in your bioinf_files directory. How many sequences
 
are in this file?
 
are in this file?
+
* * Run a blastx search using multiseqs.fasta as the input file.
 
 
 
 
 
Run a blastx search using multiseqs.fasta as the input file.
 
 
blastx -db blastdb/sprot -query multiseqs.fasta -evalue 0.4 > multiseqs_1.blastx
 
blastx -db blastdb/sprot -query multiseqs.fasta -evalue 0.4 > multiseqs_1.blastx
  
 
Look at the results file to see how the results have been reported. How easy would this be to read and
 
Look at the results file to see how the results have been reported. How easy would this be to read and
 
understand? Could you load the results into other software tools?
 
understand? Could you load the results into other software tools?
+
* * Try the above query again, but with the -outfmt 6 flag.
 
 
 
 
 
Try the above query again, but with the -outfmt 6 flag.
 
  
 
Read about the -num_descriptions, -num_alignments and -max_target_seqs flags in the BLAST+
 
Read about the -num_descriptions, -num_alignments and -max_target_seqs flags in the BLAST+
 
documentation. For very small studies, where you might read through the BLAST reports yourself rather
 
documentation. For very small studies, where you might read through the BLAST reports yourself rather
 
than doing further processing on them using the computer, these flags may help you otherwise.
 
than doing further processing on them using the computer, these flags may help you otherwise.
+
* Processing multiple files using a foreach loop
 
 
Processing multiple files using a foreach loop
 
 
This section introduces a powerful shell feature that allows you to quickly automate repetitive tasks. In this
 
This section introduces a powerful shell feature that allows you to quickly automate repetitive tasks. In this
 
case we'll use BLAST to illustrate the use of the loop, so you'll need to look at the previous exercise before
 
case we'll use BLAST to illustrate the use of the loop, so you'll need to look at the previous exercise before
Line 3,004: Line 2,273:
 
57
 
57
  
�The foreach loop explained step by step
+
== The foreach loop explained step by step ==
 
Please note that the syntax used this section assumes that you are in the default Z­shell. If the
 
Please note that the syntax used this section assumes that you are in the default Z­shell. If the
 
commands fails for you and you are sure that you have typed them in correctly, please check your shell.
 
commands fails for you and you are sure that you have typed them in correctly, please check your shell.
Line 3,028: Line 2,297:
 
explanation follows.
 
explanation follows.
 
Explanation of the first line of a foreach loop:
 
Explanation of the first line of a foreach loop:
+
* we have used the command “foreach”. It's not the only way to write a loop but it is the most used.
 
 
we have used the command “foreach”. It's not the only way to write a loop but it is the most used.
 
  
 
the “j” is a name we choose to refer to “each thing” – more specifically, for each thing we get to in the
 
the “j” is a name we choose to refer to “each thing” – more specifically, for each thing we get to in the
 
list, let's refer to it by the name j. This is an arbitrary name. You can use whatever you want. So the
 
list, let's refer to it by the name j. This is an arbitrary name. You can use whatever you want. So the
 
following are equally correct to the line given above:
 
following are equally correct to the line given above:
+
* foreach myThing in testseq[1-5].fasta
 
 
foreach myThing in testseq[1-5].fasta
 
  
 
calls each list item in turn “myThing”
 
calls each list item in turn “myThing”
Line 3,055: Line 2,320:
 
58
 
58
  
�The keyword in is followed by a list of things to loop over. In this case the list is being generated as the
+
== The keyword in is followed by a list of things to loop over. In this case the list is being generated as the ==
 
result of a single glob pattern expansion, but this need not be the case. You can list items explicitly, use
 
result of a single glob pattern expansion, but this need not be the case. You can list items explicitly, use
 
multiple patterns, or even generate a list on-the-fly using backtick substitution (not covered in this tutorial).
 
multiple patterns, or even generate a list on-the-fly using backtick substitution (not covered in this tutorial).
+
* The semicolon serves to terminate the list of items to be processed, and do primes the shell to accept
 
 
The semicolon serves to terminate the list of items to be processed, and do primes the shell to accept
 
 
one or more commands to be run within the loop. The single command done terminates this list.
 
one or more commands to be run within the loop. The single command done terminates this list.
+
* So the overall effect of that one line is: “foreach thing that matches the pattern testseq[1-5].fasta, do
 
 
So the overall effect of that one line is: “foreach thing that matches the pattern testseq[1-5].fasta, do
 
 
the following:”, and after that you just supply a regular command to run. Note how we can reference $j as
 
the following:”, and after that you just supply a regular command to run. Note how we can reference $j as
 
the input sequence and also use $j.blastx to generate a filename for the results – ie. the original name
 
the input sequence and also use $j.blastx to generate a filename for the results – ie. the original name
 
with .blastx appended.
 
with .blastx appended.
+
* Hint: It is usually a good idea to check that the command or pattern used to create a list does actually
 
 
Hint: It is usually a good idea to check that the command or pattern used to create a list does actually
 
 
generate the list you expect before including it within a foreach loop. Once common trick is to add echo
 
generate the list you expect before including it within a foreach loop. Once common trick is to add echo
 
on the start of the command within the loop, so the commands are printed to the screen but not run.
 
on the start of the command within the loop, so the commands are printed to the screen but not run.
  
Exercise
+
= Exercise =
 
Set up a foreach loop to run blastx searches using the five testseq*.fasta sequences with the Swissprot
 
Set up a foreach loop to run blastx searches using the five testseq*.fasta sequences with the Swissprot
 
database:
 
database:
+
* Type this command to begin the foreach loop as described above:
 
 
Type this command to begin the foreach loop as described above:
 
 
foreach j in testseq[1-5].fasta ; do
 
foreach j in testseq[1-5].fasta ; do
  
+
* You will now be seeing something like:
 
 
You will now be seeing something like:
 
 
live@machine[bioinf_files] foreach j in testseq[1-5].fasta ; do
 
live@machine[bioinf_files] foreach j in testseq[1-5].fasta ; do
 
foreach>
 
foreach>
Line 3,090: Line 2,345:
 
The foreach> is a prompt, much like the regular prompt – it is here we tell the computer what we
 
The foreach> is a prompt, much like the regular prompt – it is here we tell the computer what we
 
want it to do with each item in the list. To do this, type:
 
want it to do with each item in the list. To do this, type:
+
* blastx –db blastdb/sprot -query $j -evalue 0.01 -out $j.blastx
 
 
blastx –db blastdb/sprot -query $j -evalue 0.01 -out $j.blastx
 
 
Recall that we defined each thing that we want to work on by the letter j in the first line of the
 
Recall that we defined each thing that we want to work on by the letter j in the first line of the
 
foreach loop. In each subsequent line of the foreach loop, we refer to each thing by prefacing the j
 
foreach loop. In each subsequent line of the foreach loop, we refer to each thing by prefacing the j
Line 3,099: Line 2,352:
 
So here, the blastall command is executed with each filename in turn, and output files are named
 
So here, the blastall command is executed with each filename in turn, and output files are named
 
using the sequence filename with .blastx appended.
 
using the sequence filename with .blastx appended.
+
* You will now see another foreach> prompt, inviting a second command, but you are done so type
 
 
You will now see another foreach> prompt, inviting a second command, but you are done so type
 
 
done
 
done
 
This indicates that there are no more processing steps to include in this foreach loop.
 
This indicates that there are no more processing steps to include in this foreach loop.
  
+
* After running the foreach loop successfully, type the command
 
 
After running the foreach loop successfully, type the command
 
 
ls -l *blastx
 
ls -l *blastx
  
 
59
 
59
  
�You should now see that you have five blastx results files. Imagine you had 100 sequences to blast – you
+
== You should now see that you have five blastx results files. Imagine you had 100 sequences to blast – you ==
 
could set up a foreach loop and go get a coffee. (Of course, you still need to figure out how you're going to
 
could set up a foreach loop and go get a coffee. (Of course, you still need to figure out how you're going to
 
use or analyse the results files if you're working with large numbers of sequences.)
 
use or analyse the results files if you're working with large numbers of sequences.)
Line 3,123: Line 2,372:
 
your names do not match up, but will just substitute blank spaces into the command.
 
your names do not match up, but will just substitute blank spaces into the command.
  
Exercise
+
= Exercise =
+
* Look through all the files called testseq*.blastx by using the command less:
 
 
Look through all the files called testseq*.blastx by using the command less:
 
 
less testseq*.blastx
 
less testseq*.blastx
  
+
* To go to the next document, you need to type the two-character command :n
  
To go to the next document, you need to type the two-character command :n
+
* To quit, press q
 
 
 
 
 
To quit, press q
 
  
 
Why go to all this trouble when we could just create a multiple fasta file and run a BLAST search in one go?
 
Why go to all this trouble when we could just create a multiple fasta file and run a BLAST search in one go?
Line 3,145: Line 2,388:
 
60
 
60
  
�Exercise – advanced looping
+
== Exercise – advanced looping ==
 
If you have time, you can run the following foreach loop. Try to figure out what it does before running it.
 
If you have time, you can run the following foreach loop. Try to figure out what it does before running it.
 
You may need to read the man pages for basename and cut to understand all the steps being taken. Note,
 
You may need to read the man pages for basename and cut to understand all the steps being taken. Note,
Line 3,168: Line 2,411:
 
-x. To return to normal output type set +x.
 
-x. To return to normal output type set +x.
  
Working with lots of BLAST results
+
= Working with lots of BLAST results =
 
Reading a few BLAST reports is fine, but when you have thousands, you presumably won't be reading them
 
Reading a few BLAST reports is fine, but when you have thousands, you presumably won't be reading them
 
one by one yourself.
 
one by one yourself.
Line 3,182: Line 2,425:
 
61
 
61
  
�EMBOSS Programs
+
== EMBOSS Programs ==
 
EMBOSS is an extensive package of programs that cover areas of bioinformatics analysis including:
 
EMBOSS is an extensive package of programs that cover areas of bioinformatics analysis including:
+
* Sequence alignment
 
 
Sequence alignment
 
 
 
 
 
 
Rapid database searching with sequence patterns
 
 
 
 
  
Protein motif identification, including domain analysis
+
* Rapid database searching with sequence patterns
  
+
* Protein motif identification, including domain analysis
  
Nucleotide sequence pattern analysis---for example to identify CpG islands or repeats
+
* Nucleotide sequence pattern analysis---for example to identify CpG islands or repeats
  
+
* Codon usage analysis for small genomes
  
Codon usage analysis for small genomes
+
* Rapid identification of sequence patterns in large scale sequence sets
  
+
* Presentation tools for publication
 
 
Rapid identification of sequence patterns in large scale sequence sets
 
 
 
 
 
 
Presentation tools for publication
 
  
 
We recommend that you refer to the official EMBOSS overview at
 
We recommend that you refer to the official EMBOSS overview at
Line 3,218: Line 2,447:
 
own EMBOSS tools.
 
own EMBOSS tools.
  
Ways to run EMBOSS programs:
+
= Ways to run EMBOSS programs: =
+
* Locally installed, via the jemboss graphical interface on your Bio-Linux machine*
 
 
Locally installed, via the jemboss graphical interface on your Bio-Linux machine*
 
  
 
Locall installed via graphical interfaces available under the Applications | Bioinformatics | Emboss
 
Locall installed via graphical interfaces available under the Applications | Bioinformatics | Emboss
 
menu
 
menu
 
 
+
* Locally installed, via the command line on your Bio-Linux machine*
 
 
Locally installed, via the command line on your Bio-Linux machine*
 
 
 
 
 
 
Remotely on websites such as Mobyl: http://mobyle.pasteur.fr
 
  
+
* Remotely on websites such as Mobyl: http://mobyle.pasteur.fr
  
Remotely using webservices
+
* Remotely using webservices
  
Biological databases and EMBOSS on Bio-Linux
+
= Biological databases and EMBOSS on Bio-Linux =
 
Certain EMBOSS programs can talk to local or remote biological databases. The version of EMBOSS
 
Certain EMBOSS programs can talk to local or remote biological databases. The version of EMBOSS
 
installed on Bio-Linux machines is pre-configured to access data from embl, emblcds, uniprot (including
 
installed on Bio-Linux machines is pre-configured to access data from embl, emblcds, uniprot (including
Line 3,251: Line 2,472:
 
62
 
62
  
�A comparison of the Jemboss and command line interfaces for EMBOSS programs
+
== A comparison of the Jemboss and command line interfaces for EMBOSS programs ==
 
Interface
 
Interface
 
Jemboss
 
Jemboss
Line 3,257: Line 2,478:
 
Interface
 
Interface
  
Pros
+
== Pros ==
  
Cons
+
== Cons ==
  
 
Easy to see the programs available and what Much slower to set programs running than
 
Easy to see the programs available and what Much slower to set programs running than
Line 3,304: Line 2,525:
 
Documentation still easy to access
 
Documentation still easy to access
  
Working with EMBOSS programs
+
= Working with EMBOSS programs =
 
We will run a simple 3 stage task twice – once using Jemboss and once using the command line so that you
 
We will run a simple 3 stage task twice – once using Jemboss and once using the command line so that you
 
can experience ,and get a feeling for the differences between, the two interfaces. The task is to fetch a
 
can experience ,and get a feeling for the differences between, the two interfaces. The task is to fetch a
Line 3,312: Line 2,533:
 
63
 
63
  
�Exercise – using Jemboss
+
== Exercise – using Jemboss ==
+
* Start Jemboss on Bio-Linux by typing jemboss on the command line. It can also be started by clicking
 
 
Start Jemboss on Bio-Linux by typing jemboss on the command line. It can also be started by clicking
 
 
on the icon under the Applications | Bioinformatics menu.
 
on the icon under the Applications | Bioinformatics menu.
  
+
* Click on each of the categories (e.g. Alignment, Display, etc) to see what programs are listed.
  
Click on each of the categories (e.g. Alignment, Display, etc) to see what programs are listed.
+
* When you're finished exploring, click on the Data Retrieval category and choose coderet which is
 
 
 
 
 
When you're finished exploring, click on the Data Retrieval category and choose coderet which is
 
 
under Sequence Data.
 
under Sequence Data.
  
+
* Scroll to the bottom of the window and click on the
 
 
Scroll to the bottom of the window and click on the
 
 
Read about what coderet does.
 
Read about what coderet does.
  
Line 3,342: Line 2,555:
 
64
 
64
  
�Exercise continued
+
== Exercise continued ==
 
Scroll back to the top of the coderet form in the Jemboss window, and fill in a Sequence Filename. In
 
Scroll back to the top of the coderet form in the Jemboss window, and fill in a Sequence Filename. In
 
fact, we want to pull a sequence directly from embl at the EBI. The sequence we want is from a plasmid
 
fact, we want to pull a sequence directly from embl at the EBI. The sequence we want is from a plasmid
 
and has the accession number U80928. To fetch it from the EBI, you need to type:
 
and has the accession number U80928. To fetch it from the EBI, you need to type:
+
* embl:U80928
 
 
embl:U80928
 
 
into the Sequence Filename box.
 
into the Sequence Filename box.
 
Enter a filename into the outfile file name box. For example, to distinguish from your later
 
Enter a filename into the outfile file name box. For example, to distinguish from your later
 
work, you could use the name: jemboss_bx.coderet.
 
work, you could use the name: jemboss_bx.coderet.
+
* * Scroll to the bottom of the window and hit the GO button.
 
 
 
 
 
Scroll to the bottom of the window and hit the GO button.
 
  
 
● When the program has finished, a new window called Saved Results should appear. (Don't be
 
● When the program has finished, a new window called Saved Results should appear. (Don't be
Line 3,363: Line 2,570:
 
jemboss_bx.coderet) The others will likely be called things like u80928.cds, u80928.noncoding,
 
jemboss_bx.coderet) The others will likely be called things like u80928.cds, u80928.noncoding,
 
etc.
 
etc.
+
* Take a look at the type of information in each tab. In particular, take note that:
 
 
Take a look at the type of information in each tab. In particular, take note that:
 
 
each of the tabs that contains sequence information contains multiple sequences
 
each of the tabs that contains sequence information contains multiple sequences
 
the command line you would use to run this program identically to how you just ran it via
 
the command line you would use to run this program identically to how you just ran it via
Line 3,377: Line 2,582:
 
from later work -e.g. jemboss_bx.cds. Do not close the Saved Results window as we want to
 
from later work -e.g. jemboss_bx.cds. Do not close the Saved Results window as we want to
 
refer to the information under the cmd tab later.
 
refer to the information under the cmd tab later.
+
* Go back to the main Jemboss window, go to the Nucleic | Repeats section and choose
 
 
Go back to the main Jemboss window, go to the Nucleic | Repeats section and choose
 
 
palindrome from the list of programs.
 
palindrome from the list of programs.
+
* Browse for the file you just saved using the Browse files... button next to the box under
 
 
Browse for the file you just saved using the Browse files... button next to the box under
 
 
Sequence Filename near the top of the page. Note that you'll have to set the Files of Type: option
 
Sequence Filename near the top of the page. Note that you'll have to set the Files of Type: option
 
to All Files to find your saved file because it has a .cds suffix.
 
to All Files to find your saved file because it has a .cds suffix.
+
* Check that you're happy with all the required options, and give a filename in the outfile file
 
 
Check that you're happy with all the required options, and give a filename in the outfile file
 
 
name box. For example, jemboss_palin.txt. Then press the GO button.
 
name box. For example, jemboss_palin.txt. Then press the GO button.
+
* * Scan through the results to see what has been returned to you.
 
 
 
 
 
Scan through the results to see what has been returned to you.
 
  
 
You can also view listings of the files on your system using the Jemboss file manager functionality. Click on
 
You can also view listings of the files on your system using the Jemboss file manager functionality. Click on
Line 3,402: Line 2,597:
 
better way to keep track of what files have been created or deleted.
 
better way to keep track of what files have been created or deleted.
  
Using the EMBOSS command line
+
= Using the EMBOSS command line =
 
All EMBOSS commands follow a similar pattern:
 
All EMBOSS commands follow a similar pattern:
+
* If you just type the command name, then you are prompted for required information.
 
 
If you just type the command name, then you are prompted for required information.
 
 
65
 
65
  
�If you type the command name followed by -opt then you are prompted for optional
+
== If you type the command name followed by -opt then you are prompted for optional ==
 
information as well as required information.
 
information as well as required information.
+
* If you type the command name, followed by a minimum amount of information, and -auto, the
 
 
If you type the command name, followed by a minimum amount of information, and -auto, the
 
 
program runs and uses defaults for anything you have not specified in the command.
 
program runs and uses defaults for anything you have not specified in the command.
+
* The full command (i.e. the command and all relevant options and values) can be specified by
 
 
The full command (i.e. the command and all relevant options and values) can be specified by
 
 
including parameters and arguments on the command line.
 
including parameters and arguments on the command line.
+
* The command name followed by -h or -help brings up information about the main options for
 
 
The command name followed by -h or -help brings up information about the main options for
 
 
the program.
 
the program.
+
* * The command name followed by -h -v brings up information about all options for the program
 
 
 
  
The command name followed by -h -v brings up information about all options for the program
+
* Typing tfm followed by the command name brings up the full documentation for the program.
 
 
 
 
 
Typing tfm followed by the command name brings up the full documentation for the program.
 
  
 
So, using the EMBOSS program seqret as an example, we could run:
 
So, using the EMBOSS program seqret as an example, we could run:
Line 3,455: Line 2,636:
 
http://emboss.sourceforge.net/developers/acd/commandline.html
 
http://emboss.sourceforge.net/developers/acd/commandline.html
  
Exercise – using EMBOSS command line
+
= Exercise – using EMBOSS command line =
+
* Look at the cmd tab in your jemboss results window for coderet. You should see the following:
 
 
Look at the cmd tab in your jemboss results window for coderet. You should see the following:
 
 
coderet -seqall embl:U80928 -outfile jemboss_bx.coderet -auto
 
coderet -seqall embl:U80928 -outfile jemboss_bx.coderet -auto
  
Line 3,464: Line 2,643:
 
indicates that you do not want to be prompted for further information. This results in default values being
 
indicates that you do not want to be prompted for further information. This results in default values being
 
used for all options you have not specified on the command line.
 
used for all options you have not specified on the command line.
+
* Read about coderet by bringing up the information via the command line:
 
 
Read about coderet by bringing up the information via the command line:
 
 
coderet -h or coderet -help
 
coderet -h or coderet -help
 
coderet -h -v
 
coderet -h -v
Line 3,477: Line 2,654:
 
brings up the full documentation
 
brings up the full documentation
  
(EMBOSS commands exercise continued)
+
== (EMBOSS commands exercise continued) ==
 
To make things simple, we will edit the command line in the coderet cmd tab of the Saved Results
 
To make things simple, we will edit the command line in the coderet cmd tab of the Saved Results
 
window in Jemboss, and then copy and paste our final command line into a terminal to run the program.
 
window in Jemboss, and then copy and paste our final command line into a terminal to run the program.
+
* Go to the coderet cmd tab of the Saved Results window in Jemboss, and edit the command to give a
 
 
Go to the coderet cmd tab of the Saved Results window in Jemboss, and edit the command to give a
 
 
new output filename. e.g.
 
new output filename. e.g.
 
coderet -seqall embl:U80928 -outfile cl_bx.coderet -auto
 
coderet -seqall embl:U80928 -outfile cl_bx.coderet -auto
 
Open a new terminal window and cd to your bioinf_files directory. Make a new directory to store your
 
Open a new terminal window and cd to your bioinf_files directory. Make a new directory to store your
 
result files (as it will make it easier to see what files the program generates by default).
 
result files (as it will make it easier to see what files the program generates by default).
+
* mkdir cl_dir
 
 
mkdir cl_dir
 
 
Change directory into your new directory, copy and paste the coderet command line above into the
 
Change directory into your new directory, copy and paste the coderet command line above into the
 
terminal and press the return key. (Recall that we covered highlighting and pasting text using mouse
 
terminal and press the return key. (Recall that we covered highlighting and pasting text using mouse
 
buttons near the end of the first half of this tutorial.) ie:
 
buttons near the end of the first half of this tutorial.) ie:
+
* cd cl_dir
 
 
cd cl_dir
 
 
coderet -seqall embl:U80928 -outfile cl_bx.coderet -auto
 
coderet -seqall embl:U80928 -outfile cl_bx.coderet -auto
 
When the program finishes, list the files in your directory. What has coderet produced? How does this
 
When the program finishes, list the files in your directory. What has coderet produced? How does this
 
compare with the tabs presented to you when you ran coderet via Jemboss?
 
compare with the tabs presented to you when you ran coderet via Jemboss?
+
* You may notice that we have generated a lot of files we don't need. We could have specified to coderet that
 
 
You may notice that we have generated a lot of files we don't need. We could have specified to coderet that
 
 
we only wanted the mRNA sections from the embl entry BX255937. To find out how, you'll need to refer
 
we only wanted the mRNA sections from the embl entry BX255937. To find out how, you'll need to refer
 
to the coderet documentation (the lists of options won't tell you enough).
 
to the coderet documentation (the lists of options won't tell you enough).
Line 3,507: Line 2,676:
 
command in the Jemboss Saved Results window for palindrome, or you can type palindrome on the
 
command in the Jemboss Saved Results window for palindrome, or you can type palindrome on the
 
command line and answer the prompts. Please run palindrome now, doing one of these.
 
command line and answer the prompts. Please run palindrome now, doing one of these.
+
* Once you get to know it, the command line is much faster to get running than programs via Jemboss.
 
 
Once you get to know it, the command line is much faster to get running than programs via Jemboss.
 
 
However, the power of using the EMBOSS command line is much greater if you need to process groups of
 
However, the power of using the EMBOSS command line is much greater if you need to process groups of
 
files, or do things repetitively.
 
files, or do things repetitively.
Line 3,521: Line 2,688:
 
67
 
67
  
�Exercise
+
== Exercise ==
 
Fetching a list of sequences using seqret.
 
Fetching a list of sequences using seqret.
  
 
Look at the contents of the file hexaseqs.list in your bioinf_files directory. e.g. using the
 
Look at the contents of the file hexaseqs.list in your bioinf_files directory. e.g. using the
 
command less. You will see a list of sequence ids and the database those sequences are in.
 
command less. You will see a list of sequence ids and the database those sequences are in.
+
* * Quit less. (hit q)
 
 
 
 
 
Quit less. (hit q)
 
  
 
We need to tell EMBOSS programs when they are going to work on a list of files rather than
 
We need to tell EMBOSS programs when they are going to work on a list of files rather than
 
just a single file. To do this, we preface the filename with the @ symbol. So, to fetch the list of
 
just a single file. To do this, we preface the filename with the @ symbol. So, to fetch the list of
 
sequences in the hexaseqs.list file, we can use the command:
 
sequences in the hexaseqs.list file, we can use the command:
+
* seqret -sequence @hexaseqs.list
 
 
seqret -sequence @hexaseqs.list
 
  
 
The default behaviour of seqret is to fetch sequences in fasta format, with all sequences in a
 
The default behaviour of seqret is to fetch sequences in fasta format, with all sequences in a
 
single file with a filename that uses the id of the first sequence. By now you should know
 
single file with a filename that uses the id of the first sequence. By now you should know
 
how to go about finding out how to alter aspects of the program behaviour like these.
 
how to go about finding out how to alter aspects of the program behaviour like these.
+
* Take a look at the sequence file you have generated.
 
 
Take a look at the sequence file you have generated.
 
  
 
You can use this same “list of sequences” syntax with Jemboss. e.g. you could run seqret via
 
You can use this same “list of sequences” syntax with Jemboss. e.g. you could run seqret via
Line 3,555: Line 2,714:
 
68
 
68
  
�A very basic sequence assembly
+
== A very basic sequence assembly ==
 
This demonstration takes you through a very simple assembly of some reads from a mitochondrial genome.
 
This demonstration takes you through a very simple assembly of some reads from a mitochondrial genome.
 
This is in no way supposed to be a tutorial on genome assembly, but rather a way to see various tools in
 
This is in no way supposed to be a tutorial on genome assembly, but rather a way to see various tools in
Line 3,591: Line 2,750:
 
$ mkdir results
 
$ mkdir results
  
Quality Checking
+
= Quality Checking =
 
Firstly, in receiving a set of sequence data it is paramount to assess the quality of the dataset. A useful tool is
 
Firstly, in receiving a set of sequence data it is paramount to assess the quality of the dataset. A useful tool is
 
FastQC which gives a quick graphical overview of the dataset.
 
FastQC which gives a quick graphical overview of the dataset.
Line 3,610: Line 2,769:
 
69
 
69
  
�•
+
== • ==
  
 
Use fastx splitter splits mt_reads.fastq by barcode.
 
Use fastx splitter splits mt_reads.fastq by barcode.
Line 3,622: Line 2,781:
 
results/mt1.fastq.
 
results/mt1.fastq.
  
Clean Up
+
= Clean Up =
 
To remove artefacts and improve the assembly we will do two steps:
 
To remove artefacts and improve the assembly we will do two steps:
 
1) Trim barcodes
 
1) Trim barcodes
Line 3,645: Line 2,804:
 
70
 
70
  
�Assembly With Velvet
+
== Assembly With Velvet ==
 
Velvet (https://www.ebi.ac.uk/~zerbino/velvet/) is a highly popular short-read assembler which is available
 
Velvet (https://www.ebi.ac.uk/~zerbino/velvet/) is a highly popular short-read assembler which is available
 
on Bio-Linux. There are countless parameters and combinations to achieve the best assembly, but we will
 
on Bio-Linux. There are countless parameters and combinations to achieve the best assembly, but we will
Line 3,666: Line 2,825:
 
$ tablet velvet_k21/velvet_asm.afg &
 
$ tablet velvet_k21/velvet_asm.afg &
  
Quick ‘cheat’
+
= Quick ‘cheat’ =
 
VelvetOptimiser is a script which automatically tries multiple parameter combinations and returns the best
 
VelvetOptimiser is a script which automatically tries multiple parameter combinations and returns the best
 
assembly it can find. It can be helpful in pointing you in the right direction.
 
assembly it can find. It can be helpful in pointing you in the right direction.
Line 3,676: Line 2,835:
 
$ tablet auto_data_31/velvet_asm.afg &
 
$ tablet auto_data_31/velvet_asm.afg &
  
Assembly With Abyss
+
= Assembly With Abyss =
 
Abyss (http://www.bcgsc.ca/platform/bioinfo/software/abyss) is another popular assembler which we will
 
Abyss (http://www.bcgsc.ca/platform/bioinfo/software/abyss) is another popular assembler which we will
 
run to give a comparison. Again, multitudes of parameters are available, but here we will run mostly with
 
run to give a comparison. Again, multitudes of parameters are available, but here we will run mostly with
Line 3,685: Line 2,844:
 
71
 
71
  
�•
+
== • ==
  
 
Run abyss in single-end mode with k=21
 
Run abyss in single-end mode with k=21
Line 3,700: Line 2,859:
 
# produce output for each permutation.
 
# produce output for each permutation.
  
Assessing The Assemblies
+
= Assessing The Assemblies =
 
We used tablet to view the output from Velvet assemblies. This isn’t possible with the Abyss output as the
 
We used tablet to view the output from Velvet assemblies. This isn’t possible with the Abyss output as the
 
program does not provide a full assembly, just the consensus contigs. We can obtain some simple statistics
 
program does not provide a full assembly, just the consensus contigs. We can obtain some simple statistics
Line 3,708: Line 2,867:
 
 
  
Compare assemblies with gnx-tools
+
= Compare assemblies with gnx-tools =
  
 
$ for f in velvet_k21/contigs.fa auto_data_31/contigs.fa abyss_contigs.fa
 
$ for f in velvet_k21/contigs.fa auto_data_31/contigs.fa abyss_contigs.fa
 
for> gnx-tools $f
 
for> gnx-tools $f
  
Adding Some Annotation
+
= Adding Some Annotation =
 
If sequence assembly is a tricky process to master then sequence annotation is a bona fide black art. There
 
If sequence assembly is a tricky process to master then sequence annotation is a bona fide black art. There
 
are various approaches that one can use and several pipelines available that will help. But in this case, we
 
are various approaches that one can use and several pipelines available that will help. But in this case, we
Line 3,737: Line 2,896:
 
72
 
72
  
�Artemis
+
== Artemis ==
 
Artemis is a DNA sequence viewer and annotation tool, allowing visualisation of sequence features and the
 
Artemis is a DNA sequence viewer and annotation tool, allowing visualisation of sequence features and the
 
results of analyses within the context of the sequence and its six-frame translation. Artemis can read embl or
 
results of analyses within the context of the sequence and its six-frame translation. Artemis can read embl or
 
genbank format files. Sequences can be loaded from local files or via the network from the EBI.
 
genbank format files. Sequences can be loaded from local files or via the network from the EBI.
  
Ways to run Artemis:
+
= Ways to run Artemis: =
 
from a locally installed version on your Bio-Linux machine*
 
from a locally installed version on your Bio-Linux machine*
 
via Java Web Start from the Sanger Centre
 
via Java Web Start from the Sanger Centre
 
(http://www.sanger.ac.uk/resources/software/artemis/java/artemis.jnlp)
 
(http://www.sanger.ac.uk/resources/software/artemis/java/artemis.jnlp)
 
 
+
* Figure 16: Artemis Entry window after hsy14768.embl is loaded.
 
 
Figure 16: Artemis Entry window after hsy14768.embl is loaded.
 
  
 
73
 
73
  
�Exercise
+
== Exercise ==
 
Start Artemis on Bio-Linux by typing artemis on the command line or by choosing the
 
Start Artemis on Bio-Linux by typing artemis on the command line or by choosing the
 
option Artemis from under the Bioinformatics Applications graphical menu.
 
option Artemis from under the Bioinformatics Applications graphical menu.
+
* Now choose the option Open... from under the Artemis File menu, and select the
 
 
Now choose the option Open... from under the Artemis File menu, and select the
 
 
file hsy14768.embl from within the bioinf_files directory.
 
file hsy14768.embl from within the bioinf_files directory.
+
* This should open up a large window, as shown in Figure 14, where this sequence is displayed
 
 
This should open up a large window, as shown in Figure 14, where this sequence is displayed
 
 
graphically .
 
graphically .
  
 
Open a terminal window and view the text of the embl entry using the command
 
Open a terminal window and view the text of the embl entry using the command
 
less hsy14768.embl
 
less hsy14768.embl
+
* Notice how Artemis is providing a graphical representation of what is in the text file.
 
 
Notice how Artemis is providing a graphical representation of what is in the text file.
 
  
 
Try choosing Mark Open Reading Frames from under the Create menu of
 
Try choosing Mark Open Reading Frames from under the Create menu of
 
Artemis.
 
Artemis.
+
* * Choose to mark open reading frames with a minimum size of 200.
 
 
 
 
 
Choose to mark open reading frames with a minimum size of 200.
 
  
 
You should now see two boxes near the top in the Entry section, the first called hsy14768.embl
 
You should now see two boxes near the top in the Entry section, the first called hsy14768.embl
Line 3,783: Line 2,930:
 
Uncheck the box next to hsy14768.embl. You should now be able to scroll along the
 
Uncheck the box next to hsy14768.embl. You should now be able to scroll along the
 
window horizontally and easily see the open reading frames you marked.
 
window horizontally and easily see the open reading frames you marked.
+
* Check the box next to hsy14768.embl again. Look at the information in the bottom
 
 
Check the box next to hsy14768.embl again. Look at the information in the bottom
 
 
frame of the window. Notice how it is related to the images in the frames above.
 
frame of the window. Notice how it is related to the images in the frames above.
+
* Try clicking on some of the lines in the bottom frame and seeing what happens in the
 
 
Try clicking on some of the lines in the bottom frame and seeing what happens in the
 
 
images in the other two frames.
 
images in the other two frames.
+
* Explore the options available to you. (Not all options will be functional by default. See the
 
 
Explore the options available to you. (Not all options will be functional by default. See the
 
 
information about the Run menu below)
 
information about the Run menu below)
+
* * Close the Artemis Entry Editing window using File | Close.
 
 
 
 
 
Close the Artemis Entry Editing window using File | Close.
 
  
 
You can also load up files direct from the EBI. If you want to try this, then choose File |
 
You can also load up files direct from the EBI. If you want to try this, then choose File |
 
Open from the EBI – Dbfetch... option in the original small Artemis window and enter the
 
Open from the EBI – Dbfetch... option in the original small Artemis window and enter the
 
accession number BX255937.
 
accession number BX255937.
+
* When you are done, close Artemis by choosing File | Close in the sequence entry
 
 
When you are done, close Artemis by choosing File | Close in the sequence entry
 
 
window and then choosing File | Quit in the main (small) Artemis window.
 
window and then choosing File | Quit in the main (small) Artemis window.
+
* You can run various programs on your sequence, or parts of your sequence, from under the Run menu in
 
 
You can run various programs on your sequence, or parts of your sequence, from under the Run menu in
 
 
Artemis. Some of the options in this menu need to be configured to be appropriate for your site. There is
 
Artemis. Some of the options in this menu need to be configured to be appropriate for your site. There is
 
information on how to do this on our website at:
 
information on how to do this on our website at:
Line 3,822: Line 2,955:
 
BLAST betrween two or more sequences.
 
BLAST betrween two or more sequences.
  
�Appendix A – BLAST references and documentation
+
== Appendix A – BLAST references and documentation ==
 
Web pages
 
Web pages
 
The blastall and blast+ page in your Bio-Linux Bioinformatics Docs provides links to local web pages with
 
The blastall and blast+ page in your Bio-Linux Bioinformatics Docs provides links to local web pages with
Line 3,840: Line 2,973:
 
http://nebc.nerc.ac.uk/support/training/course-notes/past-notes/nebc-introduction-to-bioinformaticsmsc.-biology-2007
 
http://nebc.nerc.ac.uk/support/training/course-notes/past-notes/nebc-introduction-to-bioinformaticsmsc.-biology-2007
  
References
+
= References =
 
The book by Ian Korf is a good place to start in learning about what BLAST can do, how it does it and what BLAST output means. It
 
The book by Ian Korf is a good place to start in learning about what BLAST can do, how it does it and what BLAST output means. It
 
is now out of date however, and should be read in conjunction with the new blast+ documentation. Also note that wu-blast is now
 
is now out of date however, and should be read in conjunction with the new blast+ documentation. Also note that wu-blast is now
Line 3,867: Line 3,000:
 
75
 
75
  
�Appendix B – Creating local BLAST databases
+
== Appendix B – Creating local BLAST databases ==
 
Obtaining local BLAST databases
 
Obtaining local BLAST databases
 
To get the most from BLAST, you should search against a relevant database, which may mean using the
 
To get the most from BLAST, you should search against a relevant database, which may mean using the
Line 3,953: Line 3,086:
 
76
 
76
  
�One thing to note in the table above is that uniprot divisions are provided in embl format. However, BLAST
+
== One thing to note in the table above is that uniprot divisions are provided in embl format. However, BLAST ==
 
indices are created from fasta format files. Unfortunately, the EMBOSS program seqret, which you saw
 
indices are created from fasta format files. Unfortunately, the EMBOSS program seqret, which you saw
 
earlier, does not handle entire database divisions well. Instead, you can use a simple script to do the
 
earlier, does not handle entire database divisions well. Instead, you can use a simple script to do the
Line 3,961: Line 3,094:
 
can be slightly different than the database that results from downloading and formatting your own.
 
can be slightly different than the database that results from downloading and formatting your own.
  
Understand your databases
+
= Understand your databases =
 
It is important to read the documentation about the databases you choose to work with.
 
It is important to read the documentation about the databases you choose to work with.
 
For example, uniprot and nr are not the same. nt is not a non-redundant database; nr is.
 
For example, uniprot and nr are not the same. nt is not a non-redundant database; nr is.
Line 3,970: Line 3,103:
 
http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz?-page+top
 
http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz?-page+top
  
Building BLAST indices from local sequence files
+
= Building BLAST indices from local sequence files =
 
We will use the uniprot swissprot virus division as an example here. As this is distributed in embl format,
 
We will use the uniprot swissprot virus division as an example here. As this is distributed in embl format,
 
and we need it in fasta format, we include a format conversion step in the instructions below.
 
and we need it in fasta format, we include a format conversion step in the instructions below.
Line 3,988: Line 3,121:
 
working in. To set this variable for every session, you can add the line to your ~/.zshrc file.
 
working in. To set this variable for every session, you can add the line to your ~/.zshrc file.
 
export BLASTDB=”$HOME/blastdb”
 
export BLASTDB=”$HOME/blastdb”
+
* Download the database section of interest. Here we will work with the uniprot swissprot virus division:
 
 
Download the database section of interest. Here we will work with the uniprot swissprot virus division:
 
  
 
wget
 
wget
Line 3,997: Line 3,128:
 
77
 
77
  
�If you don't already have a sequence conversion tool, download the emblToFastaAndPreProcess.pl
+
== If you don't already have a sequence conversion tool, download the emblToFastaAndPreProcess.pl ==
 
script from the NEBC site.
 
script from the NEBC site.
+
* wget http://nebc.nerc.ac.uk/downloads/scripts/bioinf/emblToFastaAndPreProcess.pl
 
 
wget http://nebc.nerc.ac.uk/downloads/scripts/bioinf/emblToFastaAndPreProcess.pl
 
  
 
This script converts embl sequence to fasta sequence. Due to issues that sometimes appear because of the
 
This script converts embl sequence to fasta sequence. Due to issues that sometimes appear because of the
Line 4,007: Line 3,136:
 
conversion. A version of the script that does not pre-edit the feature lines is also available:
 
conversion. A version of the script that does not pre-edit the feature lines is also available:
 
http://nebc.nerc.ac.uk/downloads/scripts/bioinf/emblToFasta.pl
 
http://nebc.nerc.ac.uk/downloads/scripts/bioinf/emblToFasta.pl
+
* Make this script executable.
 
 
Make this script executable.
 
 
chmod u+x emblToFastaAndPreProcess.pl
 
chmod u+x emblToFastaAndPreProcess.pl
  
 
This script can handle compressed files, so you can create a fasta formatted copy of the
 
This script can handle compressed files, so you can create a fasta formatted copy of the
 
uniprot_sprot_viruses division by running the command:
 
uniprot_sprot_viruses division by running the command:
+
* ./emblToFastaAndPreProcess.pl uniprot_sprot_viruses.dat.gz
 
 
./emblToFastaAndPreProcess.pl uniprot_sprot_viruses.dat.gz
 
 
Notice the ./ at the start of the line. You need this if you are running the script from the directory you are in.
 
Notice the ./ at the start of the line. You need this if you are running the script from the directory you are in.
 
There are better ways to do this if you plan to keep this script for use again, but they are not covered here.
 
There are better ways to do this if you plan to keep this script for use again, but they are not covered here.
 
When the script is finished, you should find a file called uniprot_sprot_viruses.fasta in your directory.
 
When the script is finished, you should find a file called uniprot_sprot_viruses.fasta in your directory.
 
This is the file we build the BLAST database from.
 
This is the file we build the BLAST database from.
+
* makeblastdb -dbtype prot -in uniprot_sprot_viruses.fasta -out sprot_virus
 
 
makeblastdb -dbtype prot -in uniprot_sprot_viruses.fasta -out sprot_virus
 
 
You should now have four new files in your directory: sprot_virus.psq, sprot_virus.pin, sprot_virus.phr
 
You should now have four new files in your directory: sprot_virus.psq, sprot_virus.pin, sprot_virus.phr
 
and formatdb.log. The last of these lets you know how the BLAST formatting went.
 
and formatdb.log. The last of these lets you know how the BLAST formatting went.
+
* The sprot_virus.p* files are your BLAST indices. You search against them by specifying the BLAST
 
 
The sprot_virus.p* files are your BLAST indices. You search against them by specifying the BLAST
 
 
database name sprot_virus.
 
database name sprot_virus.
 
Note:
 
Note:
Line 4,042: Line 3,163:
 
78
 
78
  
�Appendix C - Cheat sheet of basic Linux commands
+
== Appendix C - Cheat sheet of basic Linux commands ==
 
bg
 
bg
  
Line 4,168: Line 3,289:
 
Info about who has logged onto the machine recently
 
Info about who has logged onto the machine recently
  
79
+
less
 
 
�80
 
 
 
less
 
  
 
Type a file to the screen one page at a time (press q to quit,
 
Type a file to the screen one page at a time (press q to quit,
Line 4,310: Line 3,427:
  
 
Suspend a process, see also jobs, fg and bg
 
Suspend a process, see also jobs, fg and bg
 
�81
 
 
 

Latest revision as of 23:09, 14 April 2017

Contents

Starting with an empty open terminal

Open a new terminal by clicking the icon in the dash ---> Type the following at the command prompt and press the enter key :

tar -xz -f bioinf_files.tar.gz

This command uncompresses and unpacks the contents of the tar file into your current working directory, which in this case is your home folder. You should then see a new prompt, just like this:

If you see an error, try typing the command again, making sure it is exactly as shown above including spaces, hyphens, underscores, etc. If the error says "No such file or directory " then check you really did copy the file in step (b) above. You can confirm the extraction worked by looking in the file browser or using the ls command.

Re-visiting the command above

Press the up arrow key while in the terminal. The previous command should re-appear for you to edit. You can move the cursor left and right using the keyboard but don't try to move it with the mouse – that won't work.

  • Edit the command by adding an extra 'v' righ after '-xz' so that the full command reads:
tar -xzv -f bioinf_files.tar.gz

Hit the enter key to run it. You don't need to scroll the cursor back the end before you do this. What is the result this time?

The letters after the hyphens are parameters of the tar command: x means “unpack/extract”, the z means “the file should be uncompressed with gzip”, the f indicates the file to unpack, and the v you just added means "be verbose". Therefore on this occasion you should have seen a list of the files being unpacked. This is a common behavior for many Linux commands. If the command runs successfully without errors it says nothing and just goes right back to the prompt. If you want the command to tell you what it is doing, adding -v makes it verbose, otherwise you may assume that "no news is good news". The use of the cursor keys to re-visit commands is a major time-saver in the terminal and you must get in the habit of doing this. The other major time-saver is Tab completion which we will come to soon.

Removing the compressed tarball

The unpacked files that you will be working with in this tutorial are now in a directory called bioinf_files. You can remove the compressed tar file now if you wish. Again, this can be done via the command line or using the graphical file browser but we'll stick with the command line version. More details about how to remove files from the system are covered in the Removing Files and Directories part of this tutorial.

  • Open a terminal window if you don't have one already.
  • Type the following into the terminal, then press Enter:
rm bioinf_files.tar.gz

Enter “y” to agree when you are asked if you wish to delete the file.

Finding your way on the system

In Linux/Unix systems, documents are usually referred to as files, and file folders are referred to as directories.

Your Bio-Linux file system can be thought of as a huge file folder (directory), inside of which are many other file folders (directories). Inside these there are more nested file folders (directories), and so on. As in the real world, where file folders can contain documents and other file folders, in Linux directories can contain files and other directories. The hierarchy of folders is called the directory tree.

Your personal Home folder is one directory within the tree of directories that make up your Bio-Linux machine. In your account, you can create other directories, store data, run programs, etc. A graphical view of your home directory is available by clicking on the file cabinet Files icon in the Dash toolbar (Figure 5). This opens up a window that shows the files and directories in your Home. The full name of this folder on the system is /home/live, ie. a directory named after the login account, live, within the top-level directory named /home, but the graphical file browser just shows it as Home.

Linux enforces file permissions depending on the login account. By default on Bio-Linux, your account has the right to create, delete and edit files in your own Home folder, but not in other people’s accounts or in system directories. You can be given permission (or give yourself permission, if it's your system) to work on files in such areas, and some information on setting file permissions is given later in this course. Your system administrator or local IT support should be able to help you with sharing files if they are on a shared server.

You can use the graphical file browser to explore directory areas on the machine, and to move around in your own files. It allows you to accomplish most typical file operation, including opening files and copying, moving or deleting files using drag and drop or copy/cut/paste. To view areas of the system outside your Home directory, click on Computer under Devices in the left hand pane to see the root directory of the system.

Exercise 1-2

  • If you have not done so already, click on the filing cabinet Files icon near the top of the Dash
  • Double-click on the bioinf_files directory that you unpacked in Exercise 1-1, to view the contents

Investigate the options under the file browser menus. These appear on the bar at the very top of the screen.

  • Click on the Computer icon in the left panel. This allows you to see the root directory – the base of the whole filesystem hierarchy.
  • Find the folder called home and double click on it.

You should see a single folder called live listed. Select this to get back to your Home folder. If you are not working on a live-booted system you should see a folder with your username, and other user folders may also listed. A lock symbol on a folder would inform you that you do not have permission to view the contents of that folder.

  • The Root Folder

The name of the base directory of the whole system, the one within which every file on the system is contained, is the root directory. It is referred to by a single forward slash “ / ”. When you work in the graphical file browser it shows your location relative to your Home folder, unless you are looking at files outside your Home in which case it shows the location relative to the root. You should have seen how the location changed as you browsed folders in exercise 1-2. Figure 6: Location path for Templates folder in File Browser view.

Your personal home folder (actually called live but labeled as Home), sits within the directory called home (with a small h), that contains homes for all users. This directory home is under the root directory, represented by a tiny picture of a disk in the graphical view or a single forward slash in the terminal. In other words, this information tells you where you are in system.

The location of a file or directory within the system is its path. If you are asked for the full path or absolute path to a file, you need to provide a complete listing of all the directories traversed on the system to get to that file. That is, you need to give the full path from the root directory to that file. The path is written by starting with a forward slash “/” then listing the names of the directories you need to traverse in the system to find that file, with each directory name separated with another forward slash.

To see the full path in the conventional format most command-line programs would expect you to provide, press Ctrl-L while viewing a File Browser window. You should see something like this:

Figure 7: Location in graphical file browser given in text; this is the the full path to the Templates folder in the home directory of the live user account.

To summarize the syntax provided in Figures 9 and 10: /home /home/live

home is a directory located within the root directory live is a directory within the directory home which is within the root directory. This special directory will sometimes be shown as

Home, with a capital H, because it is the home folder for the live user.

As another example: the full path to the file capsall.fasta, in the bioinf_files directory within the home directory of the live user:

/home/live/bioinf_files/capsall.fasta

Often you can provide just the route from where you are on the system to where your file is; this is referred to as a relative path. For example, if you are working in your home directory, the relative path to the file mentioned above would be bioinf_files/capsall.fasta.

Keeping things organised

Everyone knows it, but it's worth restating: if you start by creating a folder structure with meaningfully named subfolders, name your files so that the names indicate the contents (or follow some defined naming convention), and store your files in the right place, your life will be much, much easier!

Using the command shell

The real power of Linux/Unix systems is the command line.

A list of common Linux commands is provided in Appendix D of this document for reference.

Many programs and facilities are available through graphical options on Linux, but all programs and facilities can be accessed by the command line, also known as the shell. Some tasks are easier, or more appropriately done using graphical interfaces. Equally though, other things are easier or more appropriately done using the command line. Obvious examples include when you need to work with large numbers of files or want to automate processes. First steps on the command line can be hard but the rewards are worth it (we promise!)

Access to the command line is done through a terminal window. You can open a new terminal by:

  • clicking the middle button on the terminal icon on the Dash toolbar

or, going into an already open terminal and typing a command to open a second terminal:

gnome-terminal &

Anatomy of a Command

Linux/Unix commands usually take the form shown in Figure 11. You've already seen a good example in Exercise 1-1 part c. command what I want to do eg: tar

parameters how I want to do it -xvz -f

arguments on what do I want to do it bioinf_files.tar.gz

Figure 8: The Linux/Unix command line structure. Each part of a command is separated by one or more spaces.

The first word you supply on the command line is interpreted by the system as a command; that is – something the system should do or a program to be run. Items that appear after that on on the same line are separated by spaces. The additional input on the command line indicates to the system how the command should work. For example, what file you want the command to work on, or the format for the information that should be returned to you. Most commands have options available that will alter the way the command functions. You make use of these options by providing the command with parameters, some of which will take arguments. Examples in the following sections should make it clear how this works. With some commands you don't need to issue any parameters or arguments. Occasionally this is because there are none available, but usually this is because the command will use default settings if nothing is specified. If a command runs successfully, it will usually not report anything back to you, unless reporting to you was the purpose of the command (eg. ls). If the command does not execute properly, you will see an error message returned. Some of these messages are hard to decipher until you have a bit of Linux experience but ultimately they should tell you what has gone wrong. Note: Items supplied on the command line separated by spaces are interpreted as individual pieces of information for the system. For this reason, a filename with a space in it will be interpreted as two filenames by default. How to get around this is is addressed in more detail later in the course. Note 2: The use of the ampersand in the previous example, gnome-terminal &, is explained in a few pages time. You would not put an ampersand on the end of most shell commands.

Listing files in a directory

The command ls lists files in a directory. By default, the command will list the filenames of the files in your current working directory. When you first open a shell this is your home directory.

If you add a space followed by a –l (that is, a hyphen and a small letter L), after the ls command, it alters the behavior of the command: it will now list the files in your current directory, but with details about them including who owns them, what the size is, and what kind of file it is. Information about this is shown in

Date and time modified

twilliams hybInfo.txt targets_v1.txt targets_v2.txt

Filename

Figure 9: The detailed output of the command ls when run with the -l flag

Exercise

a) Try browsing files in both the terminal and the graphical file browser:

  • Open a new terminal by clicking the terminal icon

In the terminal, type the command ls. Compare what you see listed with what you see in the graphical representation of your Home directory.

  • Type the command ls –l and note the kind of information being provided and how it compares to the

graphical representation of your files.

  • In the graphical File Browser, click on the List option under the View menu, and compare this

information to that provided using the ls –l command.

  • In the console, type ls –l bioinf_files and also click on the bioinf_files folder in the graphical file

browser and compare what you are seeing.

  • You can also use glob patterns to identify file names by pattern.

? []

an asterisk means any string of characters a question mark means a single character square brackets can be used to designate a group of characters

More details about this are given in the Linux shorthand and shortcuts section below.

b) Try these commands that use wildcards to match multiple files:

  • List all the files in the directory bioinf_files. that start with the letters tes

ls bioinf_files/tes*

  • List all the files in your directory that start with tes, and end in 1.embl, 2.embl or 3.embl
ls bioinf_files/tes*[123].embl

Learning about Linux commands

Most Linux commands have a manual page that provides information about the command and options that can alter its behaviour. Many tasks can be made easier by using command options. A good rule of thumb is to ask yourself whether what you want to do is something many others may have wanted to do. If the answer is yes, then there may well be commands and options available to do that task. Linux manual pages are referred to as man pages. To open the man page for a particular command, you just need to type man followed by the name of the command you are interested in. To browse through a man page, use the cursor keys (↓ and ↑). To close the man page simply hit the q key on your keyboard. If you do not know the name of a command to use for a particular job, you can search using man –k followed by the type of thing you are trying to do. An example of this is in exercise 1-3, part c).

c)

  • Look up the manual information for the ls command by typing the following in a terminal:

man ls

Skim through the man page. You can scroll forward using the up and down arrow keys on your keyboard. You can go forward a page by using the space bar, and move backwards a page by using the b key.

  • * What does the -h option do? What about the -a option? What would running ls -lrt do?
  • Press the q key when you want to quit reading the man page.
  • Try running ls using some of the options mentioned above.
  • Look up some programs with man pages with the keywords “list directory”

man –k “list directory”

Basic Linux tips for filenames

  • Linux does not deal well with spaces in filenames!

Or to be more precise, Linux itself deals perfectly well with spaces and all manner of special characters in filenames but many programs you'll want to run on Linux do not, and if you're talking about those files in the terminal you'll need to remember to quote them as described below. If you stick with letters, numbers, hyphens, underscores and full stops, you will be fine.

Filenames with spaces in them are a common problem when transferring files to Linux from computers running Windows, or Mac operating systems. Normally the simplest thing is to rename the files before you work with them.

If you want to reference filenames with spaces in them, you will need to enclose the entire filename in quotation marks so that Linux understands that the space is part of one single name. Alternatively, you can “escape” the space using a backslash. For example, if I have a file called my document Linux will see this as two words, “my” and “document”.

But you could write either of the following to make it understand you mean a single file:

"my document"
my\ document

To avoid worrying about this, a common practice is to replace the space with an underscore. For example:

mv “my document” my_document
  • Everything is case sensitive

Linux systems consider capital letters different from lower case letters. The filename myFile is not the same as the filename Myfile or myfile. You could have all three of these in the same folder. There are some common naming conventions in place for biological data that you should try to follow. More is said on this in the second part of this tutorial.

Getting the prompt back when running graphical applications from the terminal

On an earlier page the command gnome-terminal & was suggested as a way to start a new terminal, but the ampersand symbol was not explained. By default, when you run a command the shell expects that the command will want to display text in the terminal window so it gets out fo the way until the command is finished. Ending a command with & tells the shell to go immediately back to the prompt, not waiting for the command to complete. This makes most sense when you expect the command to open up a new graphical window. It is also possible, though more fiddly, to change your mind and get the prompt back while the command is running. Confusingly, some graphical programs will always signal the shell to keep going even if you omit the & from the command. To demonstrate the default behavior we can use a very simple program called xcalc. The following exercise will hopefully help you understand how all this works.

Exercise – understanding the function of "&":

1. In a terminal, type the command xcalc 1. A basic calculator should appear. Try it out. 2. Try to type another command (eg. pwd) back in your terminal window. 3. Close the xcalc window and now see what happens back in the terminal. 2. Run xcalc again and leave it running. Now we're going to get the terminal prompt back... 1. Back at the terminal, type Ctrl-z (ie. hold down Ctrl and tap z). 2. What message do you see? Hopefully you can run commands again. 3. Try using the calculator. 4. In the terminal, give the command bg and try using the calculator again. 3. Run xcalc once again with an ampersand after the command – xcalc &

Linux shorthand and shortcuts

Understanding Linux commands can seem daunting at first. This is in part due to particular characters (full stops, question marks, etc.) having special meaning in commands. Once you learn the basics, these shorthand characters are extremely useful and time saving. The following incomplete list covers the symbols you will see most often today and describes their meanings as you will most likely encounter them in this course.

  • "*", matches any character appearing 0 or more times, also known as a wildcard
ls mydir/*
ls cat*
ls cat*hat
  • "?"
 list all the files under the directory mydir
 list all files starting with the letters cat
 list all files starting with the letters cat and ending in hat

matches a single character ls cat??hat

list all files starting with the letters cat followed by any 2 letters, and then hat

.

the directory you are currently in – ie. the last one you moved to using cd

..

the directory one level above the one you are currently in, aka. the parent directory

  • "~" shorthand for your home directory, eg. /home/live
  • $var dollar sign indicates a variable substitution, even within double quotes

– see the section on environment variables

  • "!" used for history substitution – not covered in this course
  • "-" often seen preceding a parameter (eg. ls -l)

also, the command cd - is a special case meaning “cd to previous directory”

  • ";" a semicolon can be used to separate two commands on the same line;

it is also used when writing loops – see p59

Changing directories

The command used to change directories is cd

If you think of your directory structure, (i.e. this set of nested file folders you are in), as a tree structure, then the simplest directory change you can do is move into a directory directly above or below the one you are in.

To change to a directory one below you are in, just use the cd command followed by the subdirectory name: cd subdir_name To change directory to the one above your are in, use the shorthand for “the directory above” .. cd .. If you need to change directory without worrying where you are now, you could explicitly state the full path: cd /usr/local/bin If you wish to return to your home directory at any time, just type cd by itself. cd And finally, you can type cd – This returns you to the last directory you were working in before this one. If you get lost and want to confirm where you are in the directory structure , use the pwd command (print working directory). This will return the full path of the directory you are currently in. Also by default in BioLinux, you see the name of the current directory you are working in as part of your prompt. For example, when you first opened the terminal in a live session you should see the prompt: live@biolinux[live] This means you are logged in as the user live on the machine named biolinux, and you are in a directory called live. (Recall that the full path of your home directory is /home/live.) If you move into the bioinf_files directory cd bioinf_files you would see the prompt: live@biolinux[bioinf_files]

14

Exercise 1-4

Ensure you start in your home directory by using the cd command on its own. Change directory from your home directory to the directory bioinf_files by typing

  • cd bioinf_files
  • Find the full path to where you are by typing

pwd

  • Type cd bioinf_files a second time. Why doesn't this work?
  • Change directory into the /usr/bin directory by typing

cd /usr/bin

  • List the files in this directory.

This is the main directory of runnable programs on the system. Some bioinformatics software can be found in here. Others are in /usr/local/bin

How can you get back to the bioinf_files folder from here? Can you work out how to do it with a single command?

  • Tab completion

Tab completion is an incredibly useful facility for working on the command line. The main thing tab completion does is complete the filename or program name you have started typing, saving you typing time and reducing spelling errors. For example, from your home directory, you could type: cd bio and hit the tab key. If there is only one directory with a name starting with the letters “bio”, the rest of the name will be completed for you. Here this would give you: cd bioinf_files The terminal environment on Bio-Linux is set up such that if there is more than one file with that combination of letters, all the files will be shown to you. You can choose the one you want by typing more of the filename, or by continuing to hit the tab key multiple times.

15

Exercise 1-5

  • Return to your home directory if you are not already there by typing cd
  • Type cd bio and use tab completion for the rest of the command. Only then press the return key.
  • You will now be in the bioinf_files directory.
  • Type ls testseq and use tab completion. This will show you a list of files that start with testseq.

You now have the option of completing the filename yourself, or “tabbing” through the filenames available.

  • Press the tab key a number of times to see what happens.
  • Type ls c and press tab once to view the files available.
  • Type a further a such that you now have ls ca on the command line.
  • Now press the tab key again.

As you get faster with this, it will save you a lot of typing effort. Also, tab completion knows how to escape spaces and other non-standard characters in file names for you.

Exercise 1-6

In the previous exercise tab completion was finding files in the working directory, but it can also help you find command and program names because the system knows that the first word you type is going to be a command name.

  • Type a on the command line and then press the tab key.
  • Add rte to the a so that you now have arte on the command line. Press the tab key again.
  • You will see that there is only one command that starts with these letters: artemis

For programs that might contain case sensitive names, tab completion can be especially useful.

  • Type bl on the command line and press the tab key. You will see a number of program names listed.
  • Keep pressing the tab key to see how the filenames will cycle through on the command line.

16

Command history

Previous commands you have used are stored in your history. You can save a lot of typing by using your command history effectively.

If you use the up arrow key when you are at the prompt in your terminal, you can see previous commands you have run. This is particularly useful if you have mistyped something and want to edit the command without writing the whole command out again. You can also view past commands using the command history.

By default, history will return a list of the last 15 commands run. You can add a number as a parameter to the command to ask for longer or shorter lists. For example, to return the last 30 commands run, you would type:

history -30

It is possible to "speed search" previously-executed commands by pressing the key combination:

Ctrl-r (ie. hold down Ctrl and tap the R key)

Then start to type. The command history will be scanned and the last matching command will be displayed on the console. Type Ctrl-r repeatedly to cycle through the entire list of matching commands.

Exercise 1-7

  • Type history -n 10 on the command line.
  • Type Ctrl-r, then start typing ist.

Making a directory

To make a new directory, use the command mkdir (make directory). For example:

mkdir newdir

would create a new directory called newdir.

Exercise 1-8

  • Start in your bioinf_files directory.
  • Make a new directory called testdir

The graphical view of your account should immediately update to show this new directory.

  • Move into the new directory testdir

Move straight back into the bioinf_files directory using a single command. (see the shorthand and shortcuts section above for a hint)

Office software

Leaving the command line for a short while... There are a number of word processors and spreadsheet programs available for your system. In this course we will look at the LibreOffice suite of programs, previously known as OpenOffice. This is an open source alternative to Microsoft Office and can be run on both Linux and Windows. The programs within LibreOffice can be run graphically from the icons in the Dash toolbar.

Word processor

Spreadsheet

Presentation editor Figure 10: LibreOffice Applications in the dash toolbar

Exercise 1-9

  • Click on the LibreOffice Calc Spreadsheet icon.
  • Under the File menu, click on Open.
  • Look inside the bioinf_files directory.
  • Open the file called example.xls.
  • Make a few changes and save the file using the Save or Save As… options under the File menu.
  • Close LibreOffice Calc by choosing Exit from under the File menu.

Text files, Word Processors and Bioinformatics

Documents written using a word processor such as Microsoft Word or LibreOffice Write are not plain text documents. If your filename has an extension such as .doc or .odt, it is unlikely to be a plain text document. (Try opening a Word document in notepad on Windows if you want proof of this.)

Word processors are very useful for preparing printed documents, but we recommend you do not use them when working with bioinformatics data files.

There is a handy command called simply file that will inspect a file and tell you what it looks like. If you run this on a FASTA file it will say "ASCII text" because FASTA is a plain text format. If it says "binary data" or "HTML" or "OpenDocument Text" or whatever then this is not actually a FASTA file, even if it resembles one when viewed in soem applications.

Using text editors

Plain text files are important, both as input to bioinformatics programs and as input or configuration files for system programs. We highly recommend that you learn to use a text editor to prepare and edit plain text files.

There are a number of different text editors available on Bio-Linux. These range in ease of use, and each has its pros and cons. In this practical we will briefly look at two editors, nano and gedit.

Nano

  • Pros

very simple – for example, most command options are visible at the bottom of the window can be used right in the terminal without graphical support fast to start up and use supports syntax highlighting 

  • Cons

 due to simplicity, lacks some advanced features – eg. line numbering, search by pattern  it is not completely intuitive for people who are used to graphical word processors

Gedit

  • Pros

very easy to start using supports syntax highlighting looks similar to a word processor, but is in fact a powerful text editor. has many useful plugins that you can easily install

Cons:  it is a graphical program and cannot be run from a text-only environment  it is slightly slower to start up than nongraphical editors  for real power users, it's not a match for Vim or Emacs


Exercise: Editing a file with Gedit

To start up Gedit, you can use the command line, or find it in the Dash menu. Choose one of the two methods to open gedit: Command line Type gedit & Graphical menu Click the Dash Home at the top left of the screen, then type edit and click the Text Editor icon.

  • Type three or four lines of text into the gedit window.
  • Save your file using the save option under the File menu (note, you have to move your mouse right to

the top of the screen to see this) or simply click the Save button on the Toolbar. Save it as myfirstfile.txt in your testdir directory.

To save a file under the testdir directory, you may have to click on the drop down arrow to Browse for other folders. This will expand this section into a File Browser like the one you've seen in past exercises. Simply browse through to the location testdir is in and click the Save button. Add a new line to your file and save the file again using the Save As… option under the File menu. Save this file as mysecondfile.txt in the testdir directory.

  • Add more functionality to gedit by choosing the menu options; Edit → Preferences. A pop-up box

will appear with 4 tabs:

  • View

Editor

Font & Colours

Plugins

Seeing the line numbers in a file helps to keep track of your position in that file. We will enable line numbers here.

  • On the View tab enable Display line numbers. Now you can see the line numbers on the left.

Next, click on the Plugins tab and enable the Change Case and the Document Statistics plugins. Browse around the other plugins and see what functionality they provide.

  • Under the Tools menu, click on Document Statistics.

Try out the other newly added plugin, by selecting a piece of text from the document you are editing with the mouse and click on the Edit menu. Hover the mouse over the Change Case menu and choose one of the options you are presented with.

  • Change part of one of the lines in this file and save it again using the Save As… option under the File

menu. This time save it as mythirdfile.txt in the testdir directory.

  • * Quit gedit by choosing the option Quit under the File menu.

Reading text files

There are many commands available for reading text files on Linux/Unix. These are useful when you want to look at the contents of a file, but not edit it. Among the most common of these commands are cat, more, and less.

cat simply prints out a whole file in the terminal, which is often a very useful thing to do. However, cat streams the entire contents of a file to your terminal at once and is thus not that useful for reading long files as the text streams past too quickly to read. (Note – cat is short for concatenate because if you give it multiple files it will string them together in order before printing them.) more and less are commands that show the contents of a file one screenful at a time. less has more functionality than more; specifically it can scroll backwards, hence the name. With both more and less, you can use the space bar to scroll down the page, and typing the letter q causes the program to quit – returning you to your command line prompt. Once you are reading a document with more or less, typing a forward slash / will start a prompt at the bottom of the page, and you can then type in text that is searched for below the point in the document you were at. Typing in a ? also searches for a text string you enter, but it searches in the document above the point you were at. Hitting the n key during a search looks for the next instance of that text in the file.

With less (but not more), you can use the arrow keys to scroll up and down the page, and the b key to move back up the document if you wish to.

Exercise 1-11a

  • Move into the bioinf_files directory.
  • Read the file hsy14768.embl using the commands cat, more and less.

Don’t forget that tab completion can save you typing effort.

cat hsy14768.embl
more hsy14768.embl
less hsy14768.embl

Use the spacebar to scroll down Press q to quit. Use the spacebar to scroll down, b to go up a page, and the up and down arrow keys to move up and down the file line by line. Press the / key and search for the letters sequen in the file. Press the ? key and search for the letters gene in the file. Press the n key to search for other instances of gene in the file.

In almost all cases, if you want to look at a file in the terminal you want to use less. The cat command is more usually used in conjunction with other commands or when you actually want to concatenate files. The more command does nothing that less can't do.

Remember the man pages There are many command line options available for each of the above commands, as well as functionality we do not cover here. To read more about them, consult the manual pages: man cat man less

As you'll see, the manual pages are actually displayed for you using less.

An important note on line endings – CR and LF

There is one major gotcha when working with text files, and it stems from a decision made way back in the olden days of line printers. To print a text file on such a device, you would send the raw text file directly down the serial line to the printer and at the end of each line you sent two control codes, one to advance the paper (line feed) and the other to move the print carriage back to the start (carriage return).

In MS-DOS, later Windows, both these codes were embedded in standard text files at the end of every line.

In UNIX, and later Linux, a single LF character is used to indicate a newline. On old Macs it was a single LF. New Macs use the UNIX convention, so text files with single LF newlines are rare.

Many programs on Linux are written to deal with all these conventions – they just helpfully regard any combination of CR and LF as meaning "next line". Others are not, and will either complain the file is invalid or worse will try to process the extra characters as meaningful data and produce nonsense results. You don't need this hassle so, much like we recommended removing spaces from filenames above, we also recommend ensuring all your text files are in order before attempting any bioinformatics on them. The next exercise shows how you might do this.

Exercise 1-11b

In Gedit, open the file hexaseqs.list which is provided in bioinf_files. Without editing the file, save it as a new file named hexaseqs_crlf.list but on the Save As dialog switch the Line Ending option to Windows. ● Try these commands in order: ○ file hexaseqs.list hexaseqs_crlf.list ○ ls -l hexaseqs.list hexaseqs_crlf.list Note the difference in file sizes in the fourth column ●

cat hexaseqs.list cat hexaseqs_crlf.list

○ ○

cat -A hexaseqs.list cat -A hexaseqs_crlf.list

Now run these. Remember that the * in a filename is a shorthand to match multiple files at once. Don't worry about the specific meaning of the sed command but do ensure you type it exactly like as shown.

sed -i "s/\r//" hexaseqs*.list file hexaseqs*.list

In summary

○ The line endings problem is a historical annoyance that won't go away. ○ The file and cat -A commands are the quickest ways to detect troublesome CRLF line endings. ○ Using Gedit and saving with the Unix/Linux mode is the simplest and safest way to remove them. ○ The command shown above using sed (sed is a handy tool but we don't really have time to cover it in this course) can quickly strip all the CR characters from multiple files in one go. It's safe to run this on any regular text file, but if you run it on, say, and Excel file or an image or a .zip or .tar.gz file then the file will effectively be destroyed.

Copying files

The basic command used to copy files using the command line is cp. At a minimum, you must specify two arguments: the name of the file to be copied, and where you wish to copy the file to. The main things to know about using the cp command are:

  • if you provide the name of an existing directory as the second argument, the file named in the first

argument will be copied into that directory. otherwise, it will be assumed that the second argument is the new name to be used for the copy you are making, whether the name corresponds to an existing file or not if you provide more than two arguments to cp, the final argument needs to be the name of a directory that already exists and all the preceding arguments need to be files that will be copied to the directory

Examples (try these in the bioinf_files folder if you like, or go straight on to 1-12): cp unknown.fasta my_new_file.fasta - clones unknown.fasta with the new name my_new_file.fasta

cp unknown.fasta my_new_directory - probably not what you wanted! It just makes another file.

mkdir an_actual_directory cp unknown.fasta an_actual_directory - copy unknown.fasta into an_actual_directory you just made cp *.embl an_actual_directory - copy all the .embl files into the new directory in one go To copy whole directories, with all the subfiles and subdirectories, use the –R option, (meaning recursive). cp –R an_actual_directory foo - copy directory and its contents as a new directory, foo The Linux shorthand for “this directory right here” (a dot . ) and "the parent directory" ( .. ) comes in handy when copying: cd foo cp –R ../blastdb .

copy blastdb from the directory above and put the copy here in foo

Make sure you leave a space between the directory name and the final dot.

Also useful is the shorthand for someone’s home account. e.g. instead of having to know and type the location of their account, you can use ~username In the case of your own account, you use just the ~ symbol, followed by a / if you want to specify any subdirectories in your account. (note the next two examples don't work on the demo system as the files are not in place)

cp ~user2/somefile .

copy the file somefile from user2’s home directory to my current working directory. Note that you need the appropriate permissions to do this!

cp ~/Documents/mytext .

copy the file or directory called mytext from within my Documents directory to my current working directory.

Exercise 1-12

  • Move into your directory testdir from exercise 1-8.
  • List the files in this directory.
  • Make a copy of myfirstfile.txt called test.txt
  • Make a copy of mythirdfile.txt called myfourthfile.txt.
  • Make a directory called subdir.
  • Copy mysecondfile.txt into subdir
  • Copy all the files that have the letters fil in the name into the subdir directory.
  • Move back into the bioinf_files directory
  • Copy all the files that start with the letters tes and end in .embl into the directory subdir.

Linking to files

Sometimes you want to access a file or directory at a different location but you don't actually want to copy it. For example if you have a data file in a system folder or network drive that you want to be able to access quickly from your desktop, but you don't actually want the entire file to be copied to your desktop folder:

ln -s /usr/local/bioinf/sampledata/nucleotide_seqs/multiple_seqs.fasta ~/Desktop/multiple.fasta

If you now try to open multiple.fasta in any application (eg. Gedit), you will see the data from the linked file as if you accessed it directly. If you write to the link you will be writing data straight to the original file (but in this case you will not have permission to do so).

You can examine links using the long output mode of ls.

ls -l ~/Desktop/multiple.fasta
lrwxrwxrwx 1 live live 35 2011-05-12 11:46
/home/live/Desktop/multiple.fasta ->
/usr/local/bioinf/sampledata/nucleotide_seqs/file1.fasta

The initial letter 'l' shows we are dealing with a link. Links do not have their own permission settings so ls shows them all as enabled, but links do have an owner depending on who created them. The target of the link is shown last. The target can be any file, directory or even another link. Note that Linux will not stop you from making a link where the target is non-existent or inaccessible, but ls will help you to spot these "dangling links" by colouring them in red.

Removing files and directories

The key difference between deleting something from the command line and using the graphical file browser is that in the first case the file vanishes immediately, but in the second it will be stored for a while in the Rubbish Bin and can be retrieved.

Option 1: Using the command line (effect: deletes files from the system) To remove a file or files, use the rm command followed by the name of the file(s) you wish to delete.

rm file1
rm file2 file3 file4
rm foo/*

remove all files in foo but not the directory itself

To remove an empty directory, you can use the rmdir command:

rmdir thisdir

If that directory contains any files, you will not able to delete the directory using rmdir until you have deleted all the files within it. To delete a directory and all the files in it at the same time, use the rm command with the option -r (for recursive) rm –r fulldir If you use the above command on Bio-Linux, you will be prompted to confirm that you wish to delete each file. While sometimes useful, this can be tedious. If you are certain that you want to delete all the files in that directory, as well as the directory itself, then you can combine the recursive flag with the force (-f) flag rm -rf anydir So if you are 100% confident that you will never make a mistake, you can use rm -rf for all deletions, but for mere mortals it is good practice to use the more specific commands, as this can mitigate mistakes. Option 2: Using the File Browser (effect: moves files into the Rubbish Bin) If you are in the graphical file browser, just find the file you wish to remove, right click on it and choose the Move to Rubbish Bin option or else press the Delete key on the keyboard. Note that this file will not be removed from your system, only hidden, and can be retrieved via the Rubbish Bin icon in the bottom right of the screen.

If you were deleting the file to make space, you now have to empty it from the Rubbish Bin to actually get the disk space back. You can remove the file permanently in one go by holding down the Shift key on your keyboard and while keeping this key depressed, pressing the Delete key. A message box will pop up asking you to confirm that you really wish to permanently delete your file.

Exercise 1-13

  • Move into the testdir directory.
  • Delete mythirdfile.txt using the command line
  • Delete myfourthfile.txt using the graphical file browser. Is the files now sitting in the Rubbish Bin?
  • Back on the command line, move back into your Home directory.
  • Then delete myfirstfile.txt from testdir without moving back to the testdir directory.

Delete the entire testdir/subdir directory without being prompted about the deletion of each file individually.

Notes on Reading, Copying and Removing Files and Directories

On Bio-Linux the commands cp, mv and rm have been aliased to cp –i , mv –i and rm –i respectively. This means the system will ask you if you really mean to overwrite files should the situation arise with cp or mv, or delete the file you have just asked to delete when using rm. You must respond with a y or Y if you do wish to proceed. Hitting any other key will cause the action you requested to be ignored. You cannot assume that any other Linux/Unix systems you work on will be configured this way, but you can always set these settings yourself.

Redirecting output to files You have seen how the cat command can take the contents of a file and put it straight into the terminal, but we can also do what is essentially the opposite and capture output that would normally go to the terminal and put it in a file. This is done by the redirection operator >. For example: ls > file_list.txt In this case the output of ls will not appear on the screen but you will see a new file called file_list.txt. If you cat this file or open it in gedit you'll see the file list. Note that the result is no longer coloured, as there is no way to represent colour information in a plain text file, and has been formatted into a single column list, but otherwise is identical.

25

Piping output between applications

A remarkably powerful facility on the Linux command line is the ability to take the output of one command and use it directly as the input to another command. This is referred to as piping the output of one command into another command. The vertical bar symbol used for this is called a pipe and looks like:

|

Standard UK PC keyboards have the pipe symbol on the same key as the backslash symbol, at the bottom, left hand side of the keyboard. So pressing the Shift key and the backslash key together will give you the pipe symbol. On some keyboards, the pipe symbol is at the top left hand side, on the same key as the backtick. To type a pipe symbol on such keyboards, hold down the key Alt Gr and hit the back tick ( ` ) key (left of the number 1 key). An example of when you want to use a pipe would be if you wanted to list all the files in a directory, but there are too many to fit on a single page. You probably saw this when you listed the contents of /usr/bin back in Ex. 1-4. You can pipe the output of the ls command (a list of files) into the less command, which will allow you to view the list page by page. To list the files in /usr/bin and view them page by page, the command would be: ls /usr/bin | less Another useful command to use with pipes is the wc command, which stands for wordcount. By default, wc returns the number of newlines, words and bytes in a file. Or you can tell wc to return just the number of lines by using the -l parameter (see the manpage for wc). For example, you could find out how many files you had in a directory by typing: ls | wc -l

Diff, Grep and Sort

In this section, we look briefly at three very useful commands: diff, grep and sort. As with all the commands covered today, we recommend that you read the manual page for more information about how these work and what options are available.

Diff

diff compares files line by line and reports the differences between the files. In fact, diff can be used for more involved tasks as well, like comparing the contents of directories. This can be very useful when you are looking for changes that you or someone else has made.

Exercise 1-14

  • Move into the testdir directory.
  • Type diff test.txt mysecondfile.txt to see what diff reports to you.
  • Type cat mysecondfile.txt | diff - test.txt

In the above command the hyphen (-) refers to the information being given to diff through the pipe. That is, the information resulting from the command cat mysecondfile.txt is put directly into the diff command. Obviously, in this instance it would be easier just to give the name of the file, mysecondfile.txt, but there are many instances where being able to use – to mean “what I am sending in via the pipe” can be useful.

Grep

grep stands for global regular expression print; you use this command to search for text patterns in a file (or any stream of text). Eg try this. grep "adge" /usr/share/dict/words You can also use flexible search terms, known as regular expressions, in your grep searches. You have already used glob pattern expressions in this practical, but regular expressions are somewhat different and more powerful. For example, when you listed all files with the pattern tes*embl* you were using a glob pattern comprising explicit characters (e.g. tes) and special symbols (* meaning any character or characters). The equivalent in grep would be “tes.*embl.*” where the period signifies any single character and the * signifies any number of repeats. Therefore to convert from a shell glob pattern to a regular expression replace each * with .* and each ? with . . You also need to enclose the expression in quotes to tell the shell not to try and interpret it as a glob. Unmodified glob patterns fed to grep but will not work as intended. For example the pattern tes* in grep means te followed by any number of s characters in sequence (te, tes, tess, tesss, ...). The question mark now signifies optionality – so tes? means te followed by zero or one s character (te, tes). Regular expressions are found in several places other than grep, most notably in the Perl scripting language. The full syntax is extensive and powerful but is beyond the scope of this course, so back to the grep command itself... grep requires a regular expression pattern as a parameter, and prints all the lines in a file containing that pattern. grep is especially useful in combination with pipes as you can filter the results of other commands. For example, perhaps you only want to see only the information in an EMBL file relating to the origin of the sequence, that is, the DE line. You do not need to search the file in an editor, you can just grep for lines beginning in DE, as in the next exercise. 27

Exercise 1-15

  • While in the bioinf_files directory, type the command: grep "DE" hsy14768.embl

What is this command doing?

Can you see why the above command results in the output you see? An explanation of this command can be found below this exercise box.

  • Try the commands: grep "^DE" hsy14768.embl and grep -x "DE.*" hsy14768.embl

What are the ^ symbol and the -x parameter in these commands doing? Check the manpage for grep to be sure.

  • Try the command: cat hsy14768.embl | grep "^DE". Does that do what you expected?
  • Move to your home directory and type ls –lR

Read the manual page for ls if it is not clear what this command returns.

  • Use the above command with a pipe and a grep command to search for files created or

modified today.

  • List the files in the bioinf_files directory and use the grep command to look for those containing the

characters d4. The first command in the previous exercise searches all the text in the hsy14768.embl file and returns the lines in which it finds the letter D followed by the letter E. The second command in the exercise also returns lines in the file that have a letter D followed by a letter E, but only where DE is found at the beginning of a line. This is because the ^ symbol means “match at the beginning of a line”. The $ symbol can be used similarly to mean “at the end of a line”. These are known as anchors. Passing the -x flag to grep tells it to automatically anchor both ends of the search pattern. What this anchoring does in the example above is return to you just the organism information in the embl file. This is because none of the other lines returned in the previous command started with DE, they just contained DE somewhere in them. This is an example where knowing how information is stored in an given file, along with a few basic Linux commands, allows you to retrieve information quickly. Another common example is counting how many sequences are in a set of multi-fasta files. We can do this with pipes between the commands cat, grep and the ever-handy wc, which here we use to count lines found by grep. cat *seqs.fasta | grep "^>" | wc -l Each sequence in a fasta file starts with a header line that begins with a > . The above command streams the contents of all files matching the glob pattern *seqs.fasta through a search with grep looking for lines that start with the symbol > . The quotes around the pattern ^> are necessary, as otherwise it is interpreted as a request for redirection of output to a file, rather than as a character to look for. As before, the ^ symbol means “match only at the beginning of the line”. The output of this grep search is sent to the wc command, with the -l indicating that you want to know the number of lines – ie. the number of headers and by implication the number of sequences. So a synopsis of the command above is: Read through all files with names ending seqs.fasta and look for all the header lines in the combined output, then count up those lines that matched and return the number to screen. We cover sequence formats later on in part 2 of the tutorial. 28

Environment Variables

We have seen that the way commands run can be modified by the options passed on the command line. Some commands also read values called environment variables which affect their behaviour. Environmental variables are set within the shell via the export command and are passed to any processes you run. This is useful when you want to set some parameter that is common to all invocations of a command, or applies across several commands. For example, your favourite text editor may be, say, Gedit, or Nano, or Vim, or Emacs. In the shell you can say: export EDITOR=vim Now any command that wants to run a text editor knows what your preferred editor is. Within the shell you can get at the current value of en environment variable by prefixing it with a $ sign, eg. echo $EDITOR

prints the current value of the EDITOR environment variable to the screen

The printenv command dumps all environment variables. Note that environment variables are only set in the current shell and are not saved by default, so if you run a command in another terminal or close and restart the terminal any values you set will be lost. For information on making the settings permanent by editing your .zshrc file see the user guide under Supported Shells.

Exercise 1-16

  • Give the command: export VAR1=hello (with no spaces around the = sign) then:

◦ echo $VAR1 ◦ echo $ VAR1 ◦ echo "$VAR1" ◦ echo '$VAR1'

  • Start a new terminal window by typing: gnome-terminal &

◦ Within this new terminal: echo $VAR1

  • Start a second new terminal by right-clicking the icon in the Dash and selecting New Terminal

◦ Within this new shell: echo $VAR1

  • Go back to the original shell window
unset VAR1
echo $VAR1

Has this affected either of the other two shells you started? Check them:

echo $VAR1

Environment variables are inherited when one process starts another, much like genetic material is inherited when a cell divides. Hopefully this explains the behaviour you see in the exercise above. When you start a terminal from en existing shell it inherits the environment from that shell. When you start one from the system menu it inherits just the base system environment. Furthermore, once a program is running no external program can modify its environment variables.

Changing permissions on files and directories

Every file on the system has a set of permissions on it that dictate who on the system can read, change or delete, or execute the file. By default, all the files you create in your account are readable, changeable or executable by you. However, you can grant other users permissions to access parts of your account if you wish. Below is some basic information about file permissions. Since there is only one user on the live system this isn't really relevant to your current setup. If you are working on a shared system and want to set up access to your files for other people on the system, please get advice from your system administrator. The command to change permissions is chmod. You have to specify who you are modifying the permissions of, what the new permissions are, and what file or directory to act on. The format of the chmod command is: chmod who ± permissions filename(s) who can be:

  • "u" means user and refers to the owner of the file
  • "g" means group, and refers to the group the file belongs to
  • "o" means others, everyone on your systems apart from those above
  • "a" means all three, i.e. user, group and others

permissions can be:

  • "r" means read permission
  • "w" means write permission
  • "x" means execute permission

Each user has a default group and possibly extra group memberships. Use the id command to view your group memberships. When you create a new file it will be owned by you and by your default group. If you are a member of additional groups, you can switch the file to any of those groups using the chgrp command.

(Please refer to the manual pages for the commands chown, chgrp and chmod for more on this topic.)

For simplicity, let us assume that you and a co-worker have both been put in the default group labusers and wish to share your data files found in ~/bioinf_files.

chmod a+x ~
chmod g+rx ~/bioinf_files

chmod g+r ~/bioinf_files/* directory

give permission to anyone to execute, in this case, so that they can move through, your home directory. give permission to people in the group to access files in the bioinf_files directory under your home directory, including listing the files with ls give permission to people in the group to read the files in the

The first command could have been “chmod g+x ~”. This would unlock your home directory only to users in the labusers group. However, enabling access for anyone is generally safe, as long as permissions on the files and subfolders prevent anyone from actually accessing them, and unless you set a+w in addition to a+x nobody but you will be able to list the files in your home directory.

Some other useful information

Copying and pasting text Most Linux applications, including the shell terminal windows, have Copy and Paste options in the Edit menu or available in the pop-up menu when you click the right mouse button. You can copy text within the application or between different applications. There is also a quick way to copy text within the terminal by highlighting text to select it, and using the middle mouse button to paste the text. The exact way to select, copy and paste text from within a terminal windows depends on how your mouse has been set up. Normally you would highlight text by dragging the mouse across it with your left mouse button depressed to copy the text, and paste by clicking the middle mouse button (or the two outer mouse buttons pressed simultaneously). Note that within the terminal it doesn't matter where you click the middle mouse button – the text will always be inserted at the current cursor position.

The simple way to stop a process

Sometimes a command or program you run in the terminal goes on too long, or is obviously doing something you did not plan. If there is no obvious way (such as a menu option or button) to stop the program running, try using Control and c (more commonly written as Ctrl-c). i.e. hold down the Control key and hit the c key. This requests the program to stop immediately, though the program may ignore the request. Note that this is the same key combination used in most graphical applications for copying text. Remember that highlighting text in a Linux terminal automatically copies it into the buffer – you don't need to press Ctrl-c before pasting with the middle button.

Putting a command to one side

Sometimes, you are in the middle of typing a long command, and you suddenly realise you need to do something else in the terminal, like list the current directory contents or check the manpage, before you run the command. Z-shell provides a handy shortcut for this: Alt-q. When you press Alt-q the current command disappears and you have a new empty prompt, but the unfinished command has been remembered and will reappear with the next prompt ready for you to edit and run it. An alternative is to hit Ctrl-c. Within the shell, Ctrl-c does not cause the shell to exit but it does cause the current command to be abandoned and a fresh prompt to appear. Unlike with Alt-q the unfinished command will still be visible in the terminal display so you can select it and paste it back in with the middle button if you decide you want it after all. (Try it!)

Logging out of a session

To logout, you can press the Power Icon on the far right of the top taskbar (Figure 2) and choose the Log Out option. To shut down the machine, you can choose the Shut Down option on the same menu. If you are working on the console of a machine with users apart from you, then please check with your system administrator before powering down the machine. Other people might want to log in remotely.

Clearing your terminal of text

Your terminal windows can fill up with lots of text, and it can become difficult to see the information you want because of all the clutter. You can clear the terminal window by typing clear

Accessing a running program or working with others interactively

If you just run a job and then close down the terminal you ran it from, normally the job will be terminated. It would be nice to be able to leave a long job running and be able to log out and then log back in again to see how it is progressing. This is especially true if you log in remotely via SSH and experience network disruptions, or if you run programs that can take quite a long time, but ask you for input periodically.

Luckily, there is a tool that makes it possible to leave programs running with no danger of them terminating if you log off or your terminal is closed. In addition, when you log back into your system, either locally or remotely, you can “re-attach” to your earlier session so it feels like you are picking up where you left off, in the same window you were running your program from.

The utility that allows you to do this is called screen. It must be run before you start running other programs in your window. Screen can also allow two people on different machines to work in the same session – i.e. Real time collaborative editing is possible with screen.

Unfortunately, how to work with screen is beyond the scope of this course. However, the link below provides a useful beginners tutorial about screen and multi-user sessions:

https://www.linode.com/docs/networking/ssh/using-gnu-screen-to-manage-persistent-terminalsessions#screen-basics

An extensive list of command options can be found in the screen manpage (ie. type man screen). There are many useful commands available on Linux and we cannot begin to cover them in this course. We recommend that you consider buying a book to help you learn how to use Linux efficiently.

Accessing your machine – including a full graphical desktop - remotely Bio-Linux is set up for secure remote access. We can't demonstrate this on the Live system but it is well worth knowing that if you have an installed Bio-Linux system you can connect to it securely over the network, so long as your account is enabled in the ssh group and you have network access to the machine (ie. not blocked by a site firewall) You can connect to your (installed) Bio-Linux system remotely using X2Go software. If you download an X2Go client to another Windows, Linux or Mac system, you can connect to an installed Bio-Linux system and run a full, graphical, desktop session remotely. Further details on how to do this can be found on the website at: http://environmentalomics.org/bio-linux-remote-access Note that due to limitations of the remote protocol, X2Go will use a fallback desktop “MATE” session which is slightly different to the default “Unity” desktop environment described in this tutorial.

Part Two: Introduction to Bioinformatics on Bio-Linux

This section of the tutorial introduces you to running bioinformatics software on Bio-Linux, including how to find out what is available for particular types of bioinformatics tasks, some options you have for running programs on the system, and where to find documentation about the software on the system. This course does not cover the detailed use or understanding of any particular piece of software. You should read through the general information in the next few pages, then look at which specific programs are of most interest to you.

The main points we hope you take away after completing this section of the tutorial are: a) You can discover and run bioinformatics tools even if you have not explicitly been taught how to use them. b) If you have repetitive tasks to carry out, chances are there are ways of fully or partially automating them. c) Web interfaces are easy, and have certain benefits, but a competence with the command line gives you access to more possibilities and sometimes these will suit your needs better.

  • local copies of software documentation – look in /usr/share/doc
  • options under the help menus in some graphical programs
  • web pages
  • journal articles.

Help Functions within the Programs

Documentation is available from within many programs. For example, many graphical programs have a Help menu or button; many command line programs provide help if you type the name of the program followed by –h, –help or --help. Some programs even have their own manual pages that can be accessed by typing man followed by the program name.

Example data for this tutorial

The sequences referred to in this tutorial can be unpacked from the file

/usr/local/bioinf/documentation/bio-linux/intro_course/bioinf_files.tar.gz.

If you have just done the associated Introduction to Linux tutorial, you will already have these files – please move on to the next section of the tutorial. If you have joined the tutorial at this point, please refer to Exercise 1-1, parts b, c and d to download and unpack the necessary sample data files. For some parts you will also need qiime_tutorial_data.tar.gz, mothur_tutorial_data.tar.gz and assembly_taster.tar.xz which are available in the same directory.

34

Interface choices

Software can be run on the command line, via graphical programs on your computer, via web interfaces, via web services and/or via scripts. Bioinformatics programs can often be run using more than one of these options. Each type of interface has pros and cons. We have summarised some of these for reference below. Interface Command line

Pros

Cons

Fast to run once you know the program

Have to learn the syntax

Very flexible; usually many options

Have to find out what options are available

Type out the command Repetitive tasks are easy to run or automate and press enter Easy to log in remotely and carry out tasks Easy to run; don't have to remember the Prompted command command line syntax line Easy to log in remotely and carry out tasks

Easy to forget the diversity of options for a program because of the temptation to just reply to prompts provided

Type out the command and respond to prompts on screen

Slower to get running than “pure” command line

Graphical interface

Often more intuitive and visually pleasing

than the command line

Can be slower to use than the command line, especially for repetitive tasks

Extensive help is often available via a menu option or button

For some programs, the command line version provides more functionality.

Some programs (not all!) can be run by Start the program and clicking an icon in the Applications | Bioinformatics menu on your system. interact via menus

You may need your system admin to set up programs so that you can run graphical programs when logging in remotely

Appropriate for visual tasks such as alignment editing, detailed annotation checking, etc. Usually intuitive

Web interface

Can provide functionality not available via locally-run programs such as access to important data resources or results presented in useful formats, e.g. including links to related data resources, graphics, etc.

Run via a web browser

window, usually at a Some websites allow a certain degree of “pipelining”, where the outputs of one remote site program can intuitively be supplied as input to another.

Can be slow to use relative to the command line, especially for repetitive tasks You are subject to the rules and restrictions of the site you are working on (e.g. data volume, number of tasks, options available, etc.) You may not want to send private data over the internet (e.g. if you are applying for a patent?) You can be subject to the whims of network connectivity

Web services

Runs tasks over the internet from a program, usually locally installed or run via java webstart.

Can bring together the ease of a locally run program with the data and computing resources of a remote site

You are dependent on the consistency of the remote server where the functions you need Can be used via graphical programs or scripts are running You are dependent on the functionality the remote site offers; this may not be as extensive as the functionality you get locally for some programs. Very flexible

Scripts

You are dependent on network connectivity

Great for automating tasks

Using a small Great for carrying out customised tasks program that runs a program or programs Straightforward to learn enough to alter existing scripts to do exactly the task you for you want.

You have to write the script or find a script that does the job. This means learning a programming language (or asking someone who knows one to help you)

For repetitive tasks, we highly recommend the use of the command line, workflow software and/or scripting.

General points about working with bioinformatics programs Sequence formats A simple thing that often trips people up is sequence formats. There are many different sequence formats; the reasons for this are both historical and functional. Historically, when people first started writing analysis programs for molecular data, they designed a format that they felt suited their needs. As time went on, numerous formats came into existence. We live with the legacy of this. We must know what format our data is in, and whether the program we want to run can use data in that format. Functionally, a program may require information that can be included with data held in certain formats, but not others. For example, EMBL format files can, in addition to the sequence data itself, contain descriptive information about a sequence, such as its features. In contrast, plain format contains nothing inside the file except the sequence data, while FASTA format allows a small amount of information about a sequence to be given in a header line and FASTQ adds read quality information alongside the sequence. Clustal and msf formats handle multiple aligned sequences, while phylip and nexus format files contain aligned sequences as well as information relevant to phylogenetic analysis programs.

To analyse data, it must be presented to the analysis program in a format the progam understands. This seems obvious, but frequent errors (or worse, misleading results) occur when the data entered into a program is not appropriate.

36

Converting files to different sequence formats used to be a frequent, and often time consuming, task in

bioinformatics. Luckily there are file conversion programs that take care of this easily for many formats. In addition, many program understand more than one format. Some common bioinformatics sequence formats, along with common filename conventions used for those formats, are listed in the table that follows the next section. We recommend the following page for more information and examples of common bioinformatics file formats: http://www.molecularevolution.org/resources/fileformats

File naming conventions in bioinformatics

The suffix, (the part of the filename after the final dot), is often used to denote to you, and other people, what the format of the data inside the file is. For example, the common suffix for clustal formatted alignments is aln. .A bioinformatics file that ends in .aln is usually assumed to be a clustal formatted alignment file. Another multiple sequence alignment format is phylip. A common suffix used on files containing sequences in phylip format is phy. Common suffices used for files containing data in particular formats are listed in the table following this section. We highly recommend that you follow conventions when naming your data files. Benefits to following the convention for filename endings include:

  • You will know your data format just by looking at the name of the file.

Following standard conventions, (rather than making up your own naming system), makes it easier for other people looking at your files, (e.g. collaborators, or people helping you); they will know the data format just by looking at the name.

  • Some graphical programs have filters set so that only files with particular suffices will be

listed in the file browser window when you try to load some data. If you use conventional filename endings, this is less likely to cause problems for you.

  • Certain programs use information in the filename to interpret aspects of the data, (not just the data format).

Such programs have strict naming conventions for the whole filename. For example, some sequence assembly programs either require, or are benefited by, defined naming schemes for sequence traces. The filename will inform them about which sequences are read pairs, what direction sequence reads are in, and other information relevant to assembly or visualisation. You will need to read the program documentation to find out what is required in such instances.

You are not restricted to naming your files in any particular way but we highly recommend that you follow the convention for the type of file you are generating/saving. Following file naming conventions from the beginning will save you, and your collaborators, a lot of time!

37

Common bioinformatics file formats

Format Embl or swissprot

Some common

filename endings .dat .embl .sprot .swiss

Comments

Usually these files, along with genbank files, contain feature information as well as sequence. Embl and Swisprot (or Uniprot) format are the same. Embl files contains nucleotide sequences and Uniprot files contain peptide sequences. Files downloaded from EMBL or Uniprot websites use the suffix .dat. Often these are compressed with gzip, and so end in .dat.gz Files generated by individuals in embl format will tend to end in .embl.

Genbank

.seq .gb .genbank

These files, along with embl and swissprot files, usually contain feature information as well as sequence. Individuals using this format, usually use the .gb or .genbank suffix. The NCBI usually uses .seq for genbank sections.

FASTA

FASTQ

Plain

.fasta .fsa .fa

Possibly the most common sequence format.

.fastq .fq

Very common for NextGen reads. Like FASTA with extra quality info per sequence. Alternative extensions may indicate the type of sequencing technology - .fastqsanger, .fastqsolexa, etc.

.pln .staden .sdn

Not commonly used, as the file contents contain nothing but the sequence itself; the only identifier of the sequence is in the filename.

It may contain nucleotide or peptide sequence(s) and a single-line header per sequence.

Staden programs use the plain format, accounting for the last two of the file suffices given. Clustal

.aln

Multiple sequence alignment format

Originally from the clustalw program, but now recognised by many programs that accept or output multiple sequence alignments.

Phylip

.phy .phylip

Multiple sequence alignment format

Used by the Phylip suite of programs and many others, especially those associated with phylogenetic analysis.

Msf

.msf

Multiple sequence alignment format This was the standard output format from some of the suite of programs called GCG. The format is still sometimes used. Other multiple alignment formats are more generally used and thus are often a better option to choose if you have a choice.

Nexus

.nxs .nex

Multiple sequence alignment format Used by a number of phylogenetics programs.

GFF

38

.gff

A format for describing genes and other features associated with DNA, RNA and Protein sequences. Not generally used as input for analyses.

Naming files and the danger of over-writing previous results

Many programs will suggest a name for your results file. Sometimes this name is generated by taking the beginning of the name of your input file, and adding a new suffix. However, sometimes it is just a generic name like prettyplot.ps or clustalw.aln. We encourage you to change generic names to something meaningful. Apart from the fact that filenames like prettyplot.ps give you little idea what is in the file, if you do not change the name, the next time a file of the same name is generated, you will overwrite previous results.

A common problem: what is a text file and what is not If you didn't work through the section on text files in part 1 we suggest you do so now. This part reiterates the key points. Sequence data are usually stored in text or binary files. Text files contain data you can look at in a text editor. Binary files are not human readable. The file formats referred to in the table above are all text formats. Examples of binary formats include ABI sequences and SFF sequence files. Word documents may look like text, but they aren’t. The letters you see on the page of a Word document (or OpenOffice Write, or other word processing programs) are stored along with layout data in a binary format. Most sequence analysis programs expect text. Plain old, nothing fancy, text. It is an unusual situation to need to use sequence data that has been stored as a Word document (if it is not unusual to you, you are probably doing things the hard way!). To get a text document when using Word, save it as text only.

Rule of thumb

If you are using Word or any other word processing program at any stage your work with sequences, then it is very likely that your life could be made a lot easier. Please seek advice about other ways to handle your data. You will almost certainly save yourself time and frustration. Honest.

Exercise 2-2a

A useful Linux command to find out what type of file you are dealing with is file. This does not look at the filename but interrogates the file contents directly. In your bioinf_files directory is the file example.xls. Move into your bioinf_files directory if you are not already there and try running the command

  • file example.xls
  • In the bioinf_files directory is a file called testseq1.embl. Try running the command

file testseq1.embl

GZipped files in bioinformatics

gzip is a simple compression program, which you met right at the start of this course when you unpacked a .tar.gz file. Any file can be compressed with gzip and .fastq.gz is now particularly popular as it saves a lot of disk space. Some programs deal with .fastq.gz files directly, but for others you have to gunzip them first. You can unpack the file on disk or use pipe syntax to feed it directly to your application. The zcat command prints out the uncompressed contents of a gzipped file, so something like zcat some_file.fastq.gz | some_app will work in many situations. Remember that the "–" by convention tells the application to process the data received via the pipe. This way you never have to store the big uncompressed file on disk. bzip2 and xz are similar compression programs. The tools bunzip2/bzcat and unxz/xzcat are provided to unpack these files from the command line, but if in doubt just click on the file in the File Browser. The graphical File Roller application will know how to unpack these and more file types.

Examples of running bioinformatics programs on Bio-Linux

Analysing sequences with QIIME

QIIME (pronounced ‘chime’) is a pipeline for performing microbial community analysis that integrates many third party tools which have become standard in the field. QIIME can run on a laptop, a supercomputer, and systems in between such as multicore desktops. QIIME is now included in the standard Bio-Linux distribution.

As an example, we will use data from a study of the response of mouse gut microbial communities to fasting (Crawford et al., 2009). To make this tutorial run quickly on a personal computer, we will use a subset of the data generated from 5 animals kept on the control ad libitum fed diet, and 4 animals fasted for 24 hours before sacrifice. At the end of our tutorial, we will be able to compare the community structure of control vs. fasted animals. In particular, we will be able to compare taxonomic profiles for each sample type, differences in diversity metrics within the samples and between the groups, and perform comparative clustering analysis to look for overall differences in the samples.

To process our data, we will perform the following steps, each of which is described in more detail in the Data Analysis Steps:  Filter the sequence reads for quality and assign multiplexed reads to starting samples by nucleotide barcode. Pick Operational Taxonomic Units (OTUs) based on sequence similarity within the reads, and pick a representative sequence from each OTU. Assign the OTU to a taxonomic identity using reference databases. Align the OTU sequences and create a phylogenetic tree. Calculate diversity metrics for each sample and compare the types of communities, using the taxonomic and phylogenetic assignments. Generate UPGMA and PCoA plots to visually depict the differences between the samples, and dynamically work with these graphs to generate publication quality figures.

What follows is a streamlined version of the exemplary tutorial provided by QIIME (which can be found at http://qiime.sourceforge.net/tutorials/tutorial.html). Further details and parameters on the below commands and many more can be found at this site. The material was compiled and adapted by Daniel Pass, School of Biosciences, University of Cardiff, for Bio-Linux courses June 2011. Editorialised for QIIME 1.6 by Tim Booth, NEBC. QIIME allows analysis of high-throughput community sequencing data J Gregory Caporaso, Justin Kuczynski, Jesse Stombaugh, Kyle Bittinger, Frederic D Bushman, Elizabeth K Costello, Noah Fierer, Antonio Gonzalez Pena, Julia K Goodrich, Jeffrey I Gordon, Gavin A Huttley, Scott T Kelley, Dan Knights, Jeremy E Koenig, Ruth E Ley, Catherine A Lozupone, Daniel McDonald, Brian D Muegge, Meg Pirrung, Jens Reeder, Joel R Sevinsky, Peter J Turnbaugh, William A Walters, Jeremy Widmann, Tanya Yatsunenko, Jesse Zaneveld and Rob Knight; Nature Methods, 2010; doi:10.1038/nmeth.f.303

41

Note: Commands to type are shown in grey boxes like this. Some commands in QIIME are too

long to print on one line, so where you see ... , you need to continue typing the command on the same line.

Preparation

First, we must copy the tutorial data to your home directory and extract it: cd tar -xvzf /usr/local/bioinf/documentation/bio-linux/intro_course/qiime_tutorial_data.tar.gz Entering the directory (cd qiime_tutorial_data) and listing the files (ls) will show what was extracted: Sequences (.fna) This is the 454-machine generated FASTA file. Quality Scores (.qual) This is the 454-machine generated quality score file, which contains a score for each base in each sequence included in the FASTA file. Mapping File (Tab-delimited .txt) The mapping file is generated by the user. This file contains all of the information about the samples necessary to perform the data analysis. At a minimum, the mapping file should contain the name of each sample, the barcode sequence used for each sample, the linker/primer sequence used to amplify the sample, and a Description column. custom_parameters.txt Structured file which can be customised to easily tune each analysis. qiime_tutorial_commands_serial.sh This is a script which will run all of the commands that we are about to see without user input. Data This directory contains the reference files required for alignment of the OTUs. To begin working with QIIME, you must enter the QIIME shell by typing ‘qiime’ in your working directory. This has been successful if the prompt changes to end in ‘qiime >’. The commands below will only be recognised within the special QIIME shell.

Assign Samples to Multiplex Reads

The first task is to assign the multiplex reads to samples based on their nucleotide barcode. Also, this step performs quality filtering based on the characteristics of each sequence, removing any low quality or ambiguous reads. The script for this step is split_libraries.py, but before running it we make a directory for all the output:

42

cd qiime_tutorial_data

pwd mkdir out

  1. This should show we are in qiime_tutorial_data
  2. This makes a directory for the results to go in

split_libraries.py -m Fasting_Map.txt -f Fasting_Example.fna -q Fasting_Example.qual -o split_library

This invocation will create three files in the new directory split_library/: split_library_log.txt This file contains the summary of splitting, including the number of reads detected for each sample and a brief summary of any reads that were removed due to quality considerations. histograms.txt This tab delimited file shows the number of reads at regular size intervals before and after splitting the library. seqs.fna This is a fasta formatted file where each sequence is renamed according to the sample it came from. The header line also contains the name of the read in the input fasta file and information on any barcode errors that were corrected.

Processing sequences into OTUs

There are several steps to go through to produce the annotated OTUs from the input sequences, however the following 5 steps can be called using the ‘pick_de_novo_otus’ command found at the end of this section. 1. Pick OTUs Using the seqs.fna file generated from split_libraries.py, the sequences are clustered into Operational Taxonomic Units (OTUs) based on their sequence similarity. This basic command uses the default parameters: uclust matching, 0.97 sequence similarity, no reverse strand matching. pick_otus.py -i split_library/seqs.fna -o out/uclust_picked_otus

2. Pick representative Since each OTU may be made up of many sequences, we will pick a representative sequence for that OTU for downstream analysis. This representative sequence will be used for taxonomic identification of the OTU and phylogenetic alignment. (options: random, longest, most_abundant, first) mkdir out/rep_set

  1. This makes a subdirectory to store the representative set

pick_rep_set.py -i out/uclust_picked_otus/seqs_otus.txt -f split_library/seqs.fna ... -o out/rep_set/seqs_rep_set.fasta --rep_set_picking_method most_abundant

3. Assign taxonomy You can compare your OTUs against a reference database of your choosing. For our example, we will use the default RDP classification system assignment method which comes ready with QIIME, however BLAST is also an option. assign_taxonomy.py -i out/rep_set/seqs_rep_set.fasta -o out/rdp_assigned_taxonomy

43

4. Make OTU table

Tabulates the number of times an OTU is found in each sample, and adds the taxonomic predictions for each OTU in the last column if a taxonomy file is supplied. make_otu_table.py -i out/uclust_picked_otus/seqs_otus.txt

...

-t out/rdp_assigned_taxonomy/seqs_rep_set_tax_assignments.txt -o out/otu_table.biom

5. Align sequences Alignments can either be generated de novo using programs such as MUSCLE, or through assignment to an existing alignment with tools like PyNAST. For small studies such as this tutorial, either method is possible. However, for studies involving many sequences (roughly, more than 1000), the de novo aligners are very slow and assignment with PyNAST is preferred. align_seqs.py -i out/rep_set/seqs_rep_set.fasta -o out/pynast_aligned_seqs

...

--alignment_method pynast -t data/core_set_aligned.imputed.fasta

6. Filter alignment command Before building the tree, the alignment must be filtered to remove columns comprised only of gaps. filter_alignment.py -i out/pynast_aligned_seqs/seqs_rep_set_aligned.fasta ... -o out/pynast_aligned_seqs --lane_mask_fp data/lanemask_in_1s_and_0s

7. Build phylogenetic tree command Produces a newick formatted tree file (.tre) which can be viewed using most tree visualization tools. Method options: clearcut, clustalw, raxml, fasttree_v1, fasttree(default), muscle make_phylogeny.py -i out/pynast_aligned_seqs/seqs_rep_set_aligned_pfiltered.fasta -o out/rep_set.tre

The above commands are integral to QIIME and further downstream analysis. Once their function and process is understood, the parameters can be set in the custom_parameters.txt file and run sequentially using the workflow script: pick_de_novo_otus.py -i split_library/seqs.fna -p custom_parameters.txt -o out

  1. Make sure you change the path in the custom_parameters.txt file before running this command

Data to information

QIIME has many different ways to visualize and interrogate the data. Here we will explore just a few. Note: To open a HTML file type: firefox filename

44

Heatmap

The QIIME pipeline includes a very useful utility to generate images of the OTU table. You can open this file with any web browser, and will be prompted to enter a value for “Filter by Counts per OTU”. Only OTUs with total counts at or above this threshold will be displayed. The OTU heatmap displays raw OTU counts per sample, where the counts are coloured based on the contribution of each OTU to the total OTU count present in that sample. make_otu_heatmap_html.py -i out/otu_table.biom -o out/otu_heatmap

Taxonomy Summary Charts

The taxa of the samples can be visualised at each taxonomic level (see the –L flag). Here, summarize_taxa.py produces a text file at the Phylum level (Level 2=Domain, 3=Phylum, 4=Class, 5=Order, 6=Family, 7=Genus) and plot_taxa_summary.py produces the html output. summarize_taxa.py -i out/otu_table.biom -o out/taxa_summary -L 3 plot_taxa_summary.py -i out/taxa_summary/otu_table_L3.txt -l Phylum -o out/taxa_charts -k white

Diversity

Community ecologists typically describe the microbial diversity within their study. This diversity can be assessed within a sample (alpha diversity) or between a collection of samples (beta diversity). Alpha Alpha diversity will be calculated and displayed though using this workflow. The full list of metrics available can be found at http://qiime.sourceforge.net/scripts/alpha_diversity_metrics.html. The html visualisation file can be found at ‘out/arare/alpha_rarefaction_plots/rarefaction_plots.html’ alpha_rarefaction.py -i out/otu_table.biom -m Fasting_Map.txt -o out/arare -p custom_parameters.txt -t out/rep_set.tre

Beta

Beta diversity can be represented in many different ways, shown below. By rarefying the samples to the smallest set (in this example dataset, 146 sequences) sample heterogeneity can be removed. Firstly, 3d plots are generated using unifrac. beta_diversity_through_plots.py -i out/otu_table.biom -o out/bdiv_even146 -p custom_parameters.txt

...

-m Fasting_Map.txt -t out/rep_set.tre -e 146

To view a 3d plot, navigate to the jar directory within the metric you wish to view (weighted/unweighted, continuous/discrete) and enter ‘java -jar jar/king.jar */*.kin’ where you can then view the output. The more traditional 2d plots are also generated by unifrac:

45

make_2d_plots.py -i out/bdiv_even146/unweighted_unifrac_pc.txt -o out/bdiv_even146/unweighted_unifrac_2d

...

-m Fasting_Map.txt -k white -p out/bdiv_even146/prefs.txt

These are easiest viewed through the html page: ‘out/bdiv_even146/unweighted_unifrac_2d/unweighted_unifrac_pc_2D_PCoA_plots.html’ Inter-Sample Distance Distance Histograms are a way to compare different categories and see which tend to have larger/smaller distances than others. make_distance_histograms.py -d out/bdiv_even146/unweighted_unifrac_dm.txt

...

-m Fasting_Map.txt -o out/bdiv_even146/distance_histograms -p out/bdiv_even146/prefs.txt

The html is found at: ‘out/bdiv_even146/distance_histograms/unweighted_unifrac_dm_distance_histograms.html’ Jackknifing & UPGMA To measure robustness of the sequencing effort, we perform a jackknifing analysis, wherein a small number of sequences are chosen at random from each sample, and the resulting UPGMA tree from this subset of data is compared with the tree representing the entire available data set. This produces jackknifed weighted and unweighted 2d and 3d plots like above, and also jackknifed trees found in the out/jack/ directory. jackknifed_beta_diversity.py -i out/otu_table.biom -o out/jack -p custom_parameters.txt

...

-e 110 -t out/rep_set.tre -m Fasting_Map.txt make_bootstrapped_tree.py -m out/jack/unweighted_unifrac/upgma_cmp/master_tree.tre -s out/jack/unweighted_unifrac/upgma_cmp/jackknife_support.txt -o ... out/jack/unweighted_unifrac/upgma_cmp/jackknife_named_nodes.pdf

...

evince out/jack/unweighted_unifrac/upgma_cmp/jackknife_named_nodes.pdf

A key feature of the QIIME interface is the ability to list the steps which you wish to run and have them sequentially performed by running them as a standard shell script. In the file qiime_tutorial_commands_serial.sh in your working qiime directory, you will find the commands which we have just gone through. This can be called directly from the QIIME shell prompt and will produce the same output as we have achieved, with no user input. This can be edited, along with custom_parameters.txt to tune the analyses to your specific requirements. What is described above is a brief introduction to the type of analyses which QIIME can perform. Extensive details of the commands, parameters and metrics used can be found at http://www.qiime.org/scripts or through typing a QIIME command followed by ‘-help’ into the qiime shell prompt.

46

Analysing sequences with MOTHUR

MOTHUR is another popular pipeline for performing microbial community analysis that integrates many third party tools which have become standard in the field. MOTHUR is included in the standard Bio-Linux distribution. As an example, we will use the same data used in the previous QIIME tutorial. Please refer to the previous QIIME tutorial for the description of the experiment and the data.

What follows is an adapted version of the exemplary tutorial provided by MOTHUR (which can be found at http://www.mothur.org/wiki/Sogin_data_analysis). Further details and parameters on the below commands and many more can be found at this site. The material was compiled and adapted by Soon Gweon, NBAF. Introducing mothur: Open-source, platform-independent, community-supported software for describing and comparing microbial communities. Schloss, P.D., et al., Appl Environ Microbiol, 2009. 75(23):7537-41

Preparation

First, we must copy the tutorial data to your home directory and extract it: cd tar -xvzf /usr/local/bioinf/documentation/bio-linux/intro_course/mothur_tutorial_data.tar.gz cd mothur_tutorial_data Entering the directory (cd mothur_tutorial_data) and listing the files (ls) will show what was extracted: Fasting_Example.fna This is the 454-machine generated FASTA file. Fasting_Example.qual This is the 454-machine generated quality score file, which contains a score for each base in each sequence included in the FASTA file. Fasting_Example.oligos This is generated by the user. This file is used to provide barcodes and primers to MOTHUR. data This directory contains the reference files required for alignment of the OTUs. To begin working with MOTHUR, you must enter the MOTHUR shell by typing ‘mothur’ in your working directory. This has been successful if the prompt changes to end in ‘mothur >’. The commands below will only be recognised within the special MOTHUR shell.

47

mothur

Assign Samples to Multiplex Reads and Quality Filtering First, we need to separate each sequence according to the barcode and primer combination. The first task is to assign the multiplex reads to samples based on their nucleotide barcode using the information from oligos file. Also, this step screens sequences based on the quality file, truncating reads at where the quality score falls below the threshold. The script for this step is trim.seqs: trim.seqs(fasta=Fasting_Example.fna, oligos=Fasting_Example.oligos, qfile=Fasting_Example.qual, qaverage=25, minlength=200, maxlength=1000)

This creates five files in the current directory: Fasting_Example.trim.fasta This is the processed fasta file. Fasting_Example.trim.qual This is the precessed quality file. Fasting_Example.scrap.fasta This file contains sequences which fell below the thresholds (below quality score of 25, shorter than 200 bps or longer than 1000 bps) Fasting_Example.scrap.qual This is the quality file for the scrapped sequences. Fasting_Example.groups This is a two-column list with the first column indicating the sequence names of those sequences in the Fasting_Example.trim.fasta file and the second column the group that it came from.

Generating Alignment & Distance Matrix

The first thing we want to do is to simplify the dataset by working with only the unique sequences. We are not chucking anything here, we are just making the life of your CPU and RAM a bit easier. We do this with the command: unique.seqs unique.seqs(fasta=Fasting_Example.trim.fasta)

We then need to generate an alignment of our data using the align.seqs command by aligning it to SILVA-compatible alignment database reference alignment. Please note that this step can take awhile to complete. align.seqs(fasta=Fasting_Example.trim.unique.fasta, reference=data/silva.bacteria.fasta, flip=T)

Next, we need to filter our alignment so that all of our sequences only overlap in the same region and remove any columns in the alignment that don't contain data. We do this by running the filter.seqs command.

48

filter.seqs(fasta=Fasting_Example.trim.unique.align)

Next, we want to calculate the column-formatted distance matrix, but we are only interested in distances smaller than 0.15 at this stage. We will do this using dist.seqs command. dist.seqs(fasta=Fasting_Example.trim.unique.filter.fasta, cutoff=0.15)

Classify Sequences

We then need to classify our sequences using the MOTHUR version of the “Bayesian” classifier. We do this with classify.seqs command using the SILVA-compatible reference file and taxonomy file (http://www.mothur.org/wiki/Silva_reference_alignment) classify.seqs(fasta=Fasting_Example.trim.unique.filter.fasta, name=Fasting_Example.trim.names, template=data/silva.bacteria.fasta, taxonomy=data/silva.bacteria.silva.tax)

Renaming Files

This step is done only to make our life easier by making copies of some files and giving it nice and short names. The command system() allows you to run programs outside of MOTHUR without leaving the MOTHUR shell. system(cp Fasting_Example.trim.unique.filter.fasta final.fasta) system(cp Fasting_Example.trim.names final.names) system(cp Fasting_Example.groups final.groups) system(cp Fasting_Example.trim.unique.filter.dist final.dist) system(cp Fasting_Example.trim.unique.filter.silva.wang..taxonomy final.taxonomy)

Clustering Sequences

Now we want to assign these sequences to OTUs for every possible distance up to and including a distance of 0.15. By default, this method uses the average neighbour algorithm. cluster(column=final.dist, name=final.names, cutoff=0.15)

Generating OTU Table and Normalisation

Now that we have a list file, we need to create a table that indicates the number of times an OTU shows up in each sample. This is called a shared file and can be created using the make.shared command. We are only interested in the distance of 0.03 from the list file, so we give 0.03 to “label” parameter. make.shared(list=final.an.list, group=final.groups, label=0.03)

We then normalise the number of sequences in each sample. In order to do this, we need to know how many sequences are in each step. You can do this with the count.groups command.

49

count.groups()

From the output we see that the sample with the fewest sequences had 146 sequences in it, so we normalise all the samples to this number of sequences. sub.sample(shared=final.an.shared, size=146)

Classifying OTU

The last thing we'd like to do is to get the taxonomy information for each of our OTUs. To do this we will use the classify.otu command to give us the majority consensus taxonomy. classify.otu(list=final.an.list, name=final.names, taxonomy=final.taxonomy)

Converting the shared file to BIOM-format

The make.biom command allows you to convert your shared file to a biom file. Please refer to http://biom-format.org/documentation/biom_format.html for detail. make.biom(shared=final.an.shared, contaxonomy=final.an.unique.cons.taxonomy)

Data to information

MOTHUR has many different ways to visualise and interrogate the data. Here we explore just a few.

Heatmap

Now we'd like to compare the membership and structure of the various samples using an OTUbased approach. Let's start by generating a heatmap of the relative abundance of each OTU across the 24 samples using the heatmap.bin command. heatmap.bin(shared=final.an.shared)

The output will be in a SVG-formatted file called final.an.0.03.heatmap.bin.svg. In this heatmap, the red colors indicate communities that are more similar than those with black colors. Venn Diagram MOTHUR allows you to generate a Venn diagram with venn command. Let's take a look at the Venn diagram for PC.354 and PC.355. venn(shared=final.an.shared, groups=PC.354-PC.355)

This generates a file called final.an.0.03.sharedsobs.PC.354-PC.355.svg. To view the file, type the following in another terminal: eog final.an.0.03.sharedsobs.PC.354-PC.355.svg

When generating Venn diagrams we are limited by the number of samples that we can analyze simultaneously. MOTHUR can generate up to 4-way Venn diagram: 50

venn(shared=final.an.shared, groups=PC.354-PC.355-PC.356-PC.481)

Finding and running useful scripts

Scripts are small programs written in a scripting language such as Perl or Python or even by compiling commands you'd run directly in the shell into a shell script file. Unlike normal binary applications, the program files can be examined and edited directly using a text editor. However, Linux is able to run these text files as if they were compiled programs by automatically invoking the appropriate interpreter named on the first line of the script – for example if the first line of a script says:

  1. !/usr/bin/perl

Then the script will be run using the Perl interpreter. Writing scripts is beyond the scope of this course, but it is useful to be able to run scripts that others have written. Exercise http://nebc.nerc.ac.uk/tools/code-corner/scripts • • • •

Visit the above link, then find the “fastagrep” script located under “Sequence Formatting and Other Text Manipulation”. (If you don't have a net connection there is also a copy in bioinf_files) Make a folder called “scripts” in your home directory and save the file there. In a terminal run the command chmod a+x scripts/fastagrep to tell Linux that this file is an executable script. Type ~/scripts/fastagrep to actually run the script. In this case you will see basic help.

Fastagrep is a script to help extracting sequences of interest form a multi-FASTA file by matching text in the header lines. It is a FASTA-aware version of the standard Linux 'grep' command introduced in part 1. An example invocation of fastagrep in the case where the FASTA file has Uniprot-style headers would be:

~/scripts/fastagrep -F 'OS=Zea mays' uniprot_sprot.fasta •

Here, the -F flag specifies an exact text match and the 'OS=...' syntax is specific to the headers used by Uniprot.

Tip: • •

If you get a “permission denied” error when running the script, it normally means that you missed out the chmod a+x ... part. If you get a “bad interpreter” error it means that the interpreter named on the first line of the file cannot be found on the system. You can always run the interpreter explicitly – eg. by typing perl scripts/fastagrep.

A practical exercise using fastagrep is included in the next section.

Aligning sequences using MUSCLE

Aligning multiple sequences is a very common task, as it is the first step to comparing related sequences. There are many algorithms for performing gapped global alignments over a set of sequences, most of which can be used on either nucleotide or peptide input. Many web based tools offer to align sequences, for example http://uniprot.org can align sequences retrieved from a search on the reference database, and additional sequences can also be uploaded and added to the alignment. GUI applications like ClustalX and Jalview can call alignment applications like Clustal, MUSCLE, and MAFFT for you and display the results graphically. Sometimes you may want to run the alignment directly from the command line – reasons for this include:

51

You want to fine tune the options passed to the aligner

You want to use an aligner program that is not supported by the GUI or website you are using

You want to run the alignment remotely – for example on a powerful departmental server

You want to run several alignments at once using a loop or a short script

Exercise

Plants contain many closely related genes in the cellulose synthase family. Previous studies have examined these in some model organisms, eg maize[ref below]. It might be useful to compare the cellulose synthase genes in another plant of interest, or to align bacterial homologues against the plant genes.

For use in this exercise, the file all_cellulose_synthase.fasta in the example files directory contains all the reference cellulose synthase genes from Uniprot (selected with the query “name:cellulose synthase”). 1. Ensure that you have the fastagrep script available from the previous exercise. 2. Use fastagrep to extract all the sequences that come from oilseed rape (Brassica napus). 3. Modify your command so that instead of printing the matching sequences to the terminal the results are saved as a file. • Hint – this involves using the > operator 4. Now invoke MUSCLE with the default parameters to perform the alignment. Use the following command but replace the ??? with the appropriate filename: muscle -in ??? -out seqs.aln 5. Run the Jalview application from the bioinformatics menu. Close the default project windows that appear, and select “Input Alignment -> from File”. Now load seqs.aln, enable colouring in the Colour menu and bring up the overview window from the view menu.

Jalview has many options for viewing and editing the alignment, drawing trees, etc. For comparing alignments, you may want to add the “-stable” flag to the muscle command in order to maintain the sequences in the same order as the input FASTA file. [ref for paper mentioned above] Holland et al. 2000. A comparative analysis of the plant cellulose synthase (CesA) gene family. http://www.ncbi.nlm.nih.gov/sites/entrez?db=pubmed&cmd=search&term=10938350

52

BLAST

The Basic Local Alignment Search Tool (BLAST) searches for regions of local similarity between sequences. The program compares nucleotide or protein sequences or patterns to sequence, or sequencerelated, databases and calculates the statistical significance of matches. The documentation here covers only the most commonly used BLAST implementation, BLAST+ from NCBI. There are several other BLAST varients that essentially do the same thing. Some are commercial, for example AB-BLAST from Advanced Biocomputing LLC, formerly known as WU-BLAST. There are also many other programs that search sequence databases and perform local alignments. Before relying on BLAST as your search tool you should consider whether one of these might better suit your analysis needs.

A few examples of ways to run BLAST, on Bio-Linux or otherwise

  • Locally installed command line against locally installed BLAST databases
  • Locally installed command line against remote databases
  • Locally through options in graphical programs (e.g. under the Run menu in Artemis)
  • Remotely through ssh tunnelling or the remote BLAST options in Artemis.
  • Remotely on websites such as those available at the NCBI and EBI
  • Remotely using webservices, either through programs such as Taverna, or through scripting

For this course, we assume that you are familiar with running BLAST searches using at least one web-based interface. If you are not, then this is a good time to look at the facilities offered through one of these sites, and to try BLASTing some of the example sequences in the coruse folder: NCBI: http://blast.ncbi.nlm.nih.gov/Blast.cgi EBI: http://www.ebi.ac.uk/Tools/sss/ Bio-Linux includes both the BLAST+ package and the older NCBI “blastall” implementation. Information and links in the Bio-Linux Bionformatics Documentation System (icon on your Desktop) provide information on both packages. The ncbi-blast+ package contains a number of programs allowing you to carry out different types of searches, as well as to create databases, reformat reports, etc.

What this course covers

This course covers how to run BLAST+ programs via the command line and a few simple steps you can take to work with more than one sequence at a time. We also cover how to install your own BLAST databases in Appendix C. We do not cover the internals of BLAST searching in any detail or how to interpret BLAST results.

Why use BLAST on the command line?

The web resources available for BLAST are highly developed, usually stable, and have access to a much greater set of data than most people will have available locally. They also often provide lovely graphics and links out to other data resources or analysis programs. So why use the command line at all? For small volumes of data, where you wish to search a commonly available database or subset of data available through a website, then web access is a very good option. Web-based utilities are also good for experimenting with parameters when determining useful settings for your investigation. The command line comes into its own for setting up searches quickly, for processing large volumes of data, for automating your searches, and for giving you the ability to get just the information you want returned from the BLAST

53

searches. (This last point has been made easier than ever in the newer BLAST+ programs, where you can, to

a certain extent, specify which information to return in a tab delimited format 1.)

We HIGHLY recommend you invest time learning about what BLAST does in detail, including how it works and what the statistics is produces mean. The “take the top hit” method will rarely serve your research well. We provide a list of references and helpful web pages in Appendix C that we hope will help you learn more about blast programs.

General considerations for database searching

Database searching should be approached like an experiment. In particular: define your aims before your start. This will save you an enormous amount of time, both in terms of time taken doing searches and time taken bringing together and reporting your findings later. Before you start searching with a sequence, it is useful to outline your answers to questions like: What am I trying to find out/what do I want to do with the results? ● What kind of database do I want to search with my sequence? E.g. nucleotide, protein, pattern, profile? ● Which database(s) in particular do I want to search? Why? ● Are there are any subsets of the database that I could or should restrict my search to? ● Do I want to take into account potential frameshifts in my coding sequences? ● What format is my sequence in? ● Do I want to filter my sequence for repeats and low complexity regions before searching? ● Is the scoring system I’ve chosen appropriate? ● Where and how will I store a record of the parameters I've used and the database version I've searched with?

  • A very, very brief introduction to BLAST+

BLAST+ includes programs to perform searches with different types of input against databases holding different types of data. Each search combination is referred to by a particular name and has its own command. A table of the basic BLAST “flavours” and what they do is given below.

Blastall flavour

blastn blastp blastx

Input sequence type

nucleotide peptide nucleotide (6 frame conceptual translation is created during run)

tblastn

peptide

tblastx

nucleotide (6 frame conceptual translation is created during run)

Database sequence type

nucleotide peptide peptide nucleotide (6 frame conceptual translation is created during run) nucleotide (6 frame conceptual translation is created during run)

1 You can return most information you want using the tab delimited output options in BLAST+. However, a key thing missing is the Description field – usually the most interesting field for a biologist! To get this field, along with others, out of a BLAST report, it is still necessary to consider custom scripting – or grabbing someone else's script that does the job!

54

There are many other programs available as part of the BLAST+ release apart from the ones above. These

include blastdbcmd, dustmasker, psiblast, rpsblast+, segmasker and srsearch.. These programs are not covered here, but are worth learning about for your own work.

How a BLAST database looks on the file system

A typical BLAST database consists of three files names with extensions .pin .phr and .psq for protein databases or .nin .nhr and .nsq for nucleotide databases. These files represent a specially indexed version of a multi-fasta source file. Do not try to examine the files in a regular text editor (they appear as garbage), and do not try to split the files apart. When invoking BLAST commands, just give the path to the database without any extension (see examples). BLAST will know to find and read the three files.

A simple blastp search

The following is a basic blastp command – you can run it from within the course folder. blastp -db blastdb/sprot –query cd4_cerae.fasta –evalue 0.0001 > cd4_cerae.blastp The command is easy to understand when you break it down. It means: ➔ ➔ ➔ ➔ ➔

run blastp, i.e. a peptide sequence will be used to search a peptide database. The database (-db) to be searched is called sprot and can be found in the blastdb directory. The input sequence (-query) is cd4_cerae.fasta. Only report results of sequences with e-values (-evalue) better than (i.e. lower than) 0.0001. Put the results of this search in the file cd4_cerae.blastp, using standard shell redirection (>).

You can fine tune BLAST easily using additional command line options. We highly recommend that you read about BLAST and determine appropriate settings for your research questions. This will ultimately save you a huge amount of time and energy. A copy of the Swissprot part of Uniprot, formatted for BLAST searches, is located in the directory blastdb, under your bioinf_files directory. We do not fully cover the use of makeblastdb in this course, but some more info is shown in Appendix C. For completeness, the steps we took, including the command we used to create the BLAST formatted Swissprot database, are as follows: We downloaded the fasta formatted swissprot file from ftp://ftp.ebi.ac.uk/pub/databases/fastafiles/uniprot/swissprot.gz into the blastdb directory under bioinf_files. We then used the makeblastdb command in a one-liner run within the blastdb/ directory. gunzip -c swissprot.gz | makeblastdb -title Swissprot -out sprot -dbtype prot -in Note the use of a hyphen “-” in place of a filename tells the command to get the input via the pipe “|”. This does not work in all cases but is a common convention in command line tools. Reference databases for BLASTing would normally be stored in a shared location You can either give the full or relative PATH to your blast databases within the blast command, or you can store your blast databases in a location that is supplied as the value for the BLASTDB environmental variable and just provide the database name in the blast command line. When loading reference BLAST databases onto Bio-Linux 6 you can can put them in the default BLASTDB location /home/db/blastdb OR change the environmental variable BLASTDB to a location appropriate for your work. If you do not have sudo access you will need to talk to the system administrator of the machine about this. Note that the default location for blast databases may be different on different machines, and may change on Bio-Linux in the future. 55

For the purposes of this tutorial, we will give each BLAST command the explicit location of the BLAST

database to search.

Exercise

  • Move into the bioinf_files directory if you are not already there.

List the files in the blastdb subdirectory. The files called sprot.p* are the files that BLAST uses when it searches.

  • * From within the bioinf_files directory, run the example command given previously, ie:

blastp -db blastdb/sprot –query cd4_cerae.fasta –evalue 0.0001 > cd4_cerae.blastp

  • Look at the results file that has been created.

Try a blastx search on the file unknown.fasta. This time set the evalue to 1 and save the results in unknown.blastx. The command you use will start like this:

  • blastx -db blastdb/sprot -query unknown.fasta ...???...

Recall that a blastx search translates a nucleotide sequence in six frames and searches a peptide database.

  • Look at the results file.

blastp expects a peptide query file, and blastx expects nucleotides. What would you expect to happen if you use an inappropriate BLAST flavour? Try it and see.

  • Formatting BLAST output

You have now seen the default report format for BLAST searches. There are many options available using the -outfmt option with a numerical argument between 0 and 11. The default is -outfmt 0. The BLAST+ commands don't (currently) have man pages, but to see a list of all the -outfmt options you can use the builtin help function: blastx -help | less

Exercise

Run either of the above BLAST searches again, this time adding the parameter -outfmt 6 to the command. Make sure you change the name of the output file as well, or else just let the results get printed to the screen.

  • Look at the results from this search and compare it to what was returned using default formatting. Is it

easier or harder to read? Is there information present in one report that is not in the other?

  • Note: BLAST+ programs offer finer control over the format and contents of results returned – see the help

page as mentioned above. 56

Handling multiple sequences

BLAST makes it easy to deal with a medium-sized number of sequences at once – say up to a few hundred. For thousands of sequences, you will probably want to use the ideas introduced here, in conjunction with running your searches on a compute cluster and using scripts to pull out information of relevance from the result files. The general principle of needing more sophisticated techniques as the data volume increases applies to pretty much any bioinformatics task. First we'll look at BLASTing a file containing more than one sequence In the next section we'll process multiple sequences as input using a “foreach” loop BLAST searching using fasta files containing more than one sequence

Exercise

Look at the contents of the file multiseqs.fasta in your bioinf_files directory. How many sequences are in this file?

  • * Run a blastx search using multiseqs.fasta as the input file.

blastx -db blastdb/sprot -query multiseqs.fasta -evalue 0.4 > multiseqs_1.blastx

Look at the results file to see how the results have been reported. How easy would this be to read and understand? Could you load the results into other software tools?

  • * Try the above query again, but with the -outfmt 6 flag.

Read about the -num_descriptions, -num_alignments and -max_target_seqs flags in the BLAST+ documentation. For very small studies, where you might read through the BLAST reports yourself rather than doing further processing on them using the computer, these flags may help you otherwise.

  • Processing multiple files using a foreach loop

This section introduces a powerful shell feature that allows you to quickly automate repetitive tasks. In this case we'll use BLAST to illustrate the use of the loop, so you'll need to look at the previous exercise before attempting this one. A foreach loops say to the computer: “For each thing in this list, do the following:” So, when running multiple BLAST searches, you might want to do something like: “For each sequence in my list, run a blastx search against my Swissprot database.” You can also create nested foreach loops. For example, if you had a list of sequences and a list of databases, you could use a nested foreach loop to get the computer to do something like this: “For each sequence in my sequence list, run a blastx search against each database in my database list” You can run a foreach loop on arbitrarily long lists. However, for the exercises below, we will use just five sequences: testseq1.fasta, testseq2.fasta, testseq3.fasta, testseq4.fasta and testseq5.fasta. 57

The foreach loop explained step by step

Please note that the syntax used this section assumes that you are in the default Z­shell. If the commands fails for you and you are sure that you have typed them in correctly, please check your shell. You can identify your current shell by typing the command echo $0. If you are not in the z­shell (zsh) already, just type zsh in your terminal window. Other shells provide the same functionality as the foreach loop demonstrated here, but the syntax is different. You need to tell the computer the list of files to work on. Here, we will use a glob pattern match to indicate the list of sequences we want to work with. Recall that echo simply prints its arguments and so can be used to show glob expansions: echo testseq*.fasta or, if we wanted to be more specific: echo testseq[1-5].fasta We bind each file in the list to a loop variable within the first line of the foreach loop. So the following says: “take each file in this list in turn and refer to it as j”: foreach j in testseq[1-5].fasta When we finish, our complete foreach loop will state: foreach j in testseq[1-5].fasta ; do blastx –db blastdb/sprot -query $j -evalue 0.01 -out $j.blastx done This means: for each sequence in the list in the first line, run the command in the second line. When all the sequences in the list have been dealt with, then finish. Loops are very powerful and useful, so it is worth understanding exactly how they work. A more detailed explanation follows. Explanation of the first line of a foreach loop:

  • we have used the command “foreach”. It's not the only way to write a loop but it is the most used.

the “j” is a name we choose to refer to “each thing” – more specifically, for each thing we get to in the list, let's refer to it by the name j. This is an arbitrary name. You can use whatever you want. So the following are equally correct to the line given above:

  • foreach myThing in testseq[1-5].fasta

calls each list item in turn “myThing”

foreach x in testseq[1-5].fasta

calls each list item in turn “x”

foreach seq in testseq[1-5].fasta

calls each list item in turn “seq”

Once you have chosen a name for each thing in your list, you must use that name with a dollar symbol “$” to refer to the list item in any commands that follow within the foreach loop. Recall how the $ construct also lets you access the contents of environment variables, like $BLASTDB.

58

The keyword in is followed by a list of things to loop over. In this case the list is being generated as the

result of a single glob pattern expansion, but this need not be the case. You can list items explicitly, use multiple patterns, or even generate a list on-the-fly using backtick substitution (not covered in this tutorial).

  • The semicolon serves to terminate the list of items to be processed, and do primes the shell to accept

one or more commands to be run within the loop. The single command done terminates this list.

  • So the overall effect of that one line is: “foreach thing that matches the pattern testseq[1-5].fasta, do

the following:”, and after that you just supply a regular command to run. Note how we can reference $j as the input sequence and also use $j.blastx to generate a filename for the results – ie. the original name with .blastx appended.

  • Hint: It is usually a good idea to check that the command or pattern used to create a list does actually

generate the list you expect before including it within a foreach loop. Once common trick is to add echo on the start of the command within the loop, so the commands are printed to the screen but not run.

Exercise

Set up a foreach loop to run blastx searches using the five testseq*.fasta sequences with the Swissprot database:

  • Type this command to begin the foreach loop as described above:

foreach j in testseq[1-5].fasta ; do

  • You will now be seeing something like:

live@machine[bioinf_files] foreach j in testseq[1-5].fasta ; do foreach>

The foreach> is a prompt, much like the regular prompt – it is here we tell the computer what we want it to do with each item in the list. To do this, type:

  • blastx –db blastdb/sprot -query $j -evalue 0.01 -out $j.blastx

Recall that we defined each thing that we want to work on by the letter j in the first line of the foreach loop. In each subsequent line of the foreach loop, we refer to each thing by prefacing the j with a $ sign. Each $j in that command will be replaced by the name of a file from the list. So here, the blastall command is executed with each filename in turn, and output files are named using the sequence filename with .blastx appended.

  • You will now see another foreach> prompt, inviting a second command, but you are done so type

done This indicates that there are no more processing steps to include in this foreach loop.

  • After running the foreach loop successfully, type the command

ls -l *blastx

59

You should now see that you have five blastx results files. Imagine you had 100 sequences to blast – you

could set up a foreach loop and go get a coffee. (Of course, you still need to figure out how you're going to use or analyse the results files if you're working with large numbers of sequences.) We mentioned above that the j in the foreach loop was an arbitrary name. As an example, if we had used seq instead of j, the foreach loop would have been written: foreach seq in testseq[1-5].fasta ; do blastx –db blastdb/sprot -query $seq -evalue 0.01 -out $seq.blastx done Notice that we have just replaced each instance of $j with $seq. Be careful, as the shell will not notice if your names do not match up, but will just substitute blank spaces into the command.

Exercise

  • Look through all the files called testseq*.blastx by using the command less:

less testseq*.blastx

  • To go to the next document, you need to type the two-character command :n
  • To quit, press q

Why go to all this trouble when we could just create a multiple fasta file and run a BLAST search in one go? Well, there is often more than one way to do a task, but foreach loops can be used with any programs – not just BLAST – and not all programs will take multiple inputs, so this method is widely applicable. Multiple tasks, and even inner loops can be carried out in a single foreach loop, as the following example shows.

60

Exercise – advanced looping

If you have time, you can run the following foreach loop. Try to figure out what it does before running it. You may need to read the man pages for basename and cut to understand all the steps being taken. Note, the text has been indented for clarity but you need not type it like this. Also note the special quotes in the second line are backticks obtained with the key at the top left of the keyboard, next to number 1. These serve to capture the output of the basename command into the newname variable, and later to drive an inner loop from a list contained in a file. (Earlier, we said these wouldn't be covered in the course, but here's a little taster. Backticks are a powerful feature for any aspiring command-line guru to master!) foreach seq in testseq[1-3].fasta ; do newname=`basename $seq .fasta` mkdir $newname pushd $newname blastx -db ../blastdb/sprot -query ../$seq -evalue 0.01 -outfmt 6 -out $newname.blastx cat $newname.blastx | cut -f2 > top5.list for hit in `cat top5.list` ; do wget -q "http://www.uniprot.org/uniprot/$hit.txt" done popd done You can get the Z-shell to report what it is doing within loops and functions by running the command set -x. To return to normal output type set +x.

Working with lots of BLAST results

Reading a few BLAST reports is fine, but when you have thousands, you presumably won't be reading them one by one yourself. A common way to handle large volumes of BLAST results is to get the computer to process the report files, pulling out key information. You can try using the various -outfmt options, which give you a great deal of fine tuned control over what to report in tab delimited format. Alternatively, you can use a customised script. You might choose to load such extracted information into a database, or for small scale studies, into a spreadsheet. This topic is not covered further in this course, but we recommend BioPerl modules for parsing BLAST report files. Example BioPerl scripts for BLAST parsing can be found on your Bio-Linux machine under the following directory: /usr/share/doc/bioperl/examples/searchio

61

EMBOSS Programs

EMBOSS is an extensive package of programs that cover areas of bioinformatics analysis including:

  • Sequence alignment
  • Rapid database searching with sequence patterns
  • Protein motif identification, including domain analysis
  • Nucleotide sequence pattern analysis---for example to identify CpG islands or repeats
  • Codon usage analysis for small genomes
  • Rapid identification of sequence patterns in large scale sequence sets
  • Presentation tools for publication

We recommend that you refer to the official EMBOSS overview at http://emboss.sourceforge.net/what/#Overview to find out more about the extensive functionality available via EMBOSS programs. EMBOSS also consists of an underlying programming library, in case you are interested in building your own EMBOSS tools.

Ways to run EMBOSS programs:

  • Locally installed, via the jemboss graphical interface on your Bio-Linux machine*

Locall installed via graphical interfaces available under the Applications | Bioinformatics | Emboss menu ●

  • Locally installed, via the command line on your Bio-Linux machine*
  • Remotely using webservices

Biological databases and EMBOSS on Bio-Linux

Certain EMBOSS programs can talk to local or remote biological databases. The version of EMBOSS installed on Bio-Linux machines is pre-configured to access data from embl, emblcds, uniprot (including swissprot and trembl) and Refseq from the EBI. Information about how to change this configuration can be found at http://nebc.nerc.ac.uk/tools/bioinformatics-docs/other-bioinf/emboss-applications-and-databases Sequence formats and EMBOSS EMBOSS programs accept most common sequence formats. EMBOSS also includes a versatile tool called seqret that can be used to convert between sequence formats should you need to do this for other bioinformatics programs.

62

A comparison of the Jemboss and command line interfaces for EMBOSS programs

Interface Jemboss Graphical Interface

Pros

Cons

Easy to see the programs available and what Much slower to set programs running than type of analysis they do on the command line Easy to run Many programs accept input files with multiple sequences, either directly or using lists of sequence or filenames. Documentation is easy to access

Not always obvious how to save and where to save output Additional programs with EMBOSS interfaces are not available via this interface. e.g. there are emboss interfaces for phylip and hmmer programs, among others, which are useful when creating pipelines and automating tasks. Programs that are interfaces to others (e.g. emma is an EMBOSS interface to clustalw) may not always work smoothly via Jemboss, even though they are fine via the command line.

Command Line

Prompted command line makes programs easy to run

Prompted command line makes it easy to overlook many of the options available

Programs accept input files with multiple sequences either directly or using lists of sequence or filenames.

You have to read the documentation to find out about the options available

Easy to automate tasks and create pipelines of tasks Documentation still easy to access

Working with EMBOSS programs

We will run a simple 3 stage task twice – once using Jemboss and once using the command line so that you can experience ,and get a feeling for the differences between, the two interfaces. The task is to fetch a sequence file from the EMBL database, extract all the mRNA sequences from the feature table and search for palindromes in those mRNA sequences.

63

Exercise – using Jemboss

  • Start Jemboss on Bio-Linux by typing jemboss on the command line. It can also be started by clicking

on the icon under the Applications | Bioinformatics menu.

  • Click on each of the categories (e.g. Alignment, Display, etc) to see what programs are listed.
  • When you're finished exploring, click on the Data Retrieval category and choose coderet which is

under Sequence Data.

  • Scroll to the bottom of the window and click on the

Read about what coderet does.

button to bring up a documentation window.

Figure 1: The Jemboss graphical interface to EMBOSS programs

Figure 2: The GO button is pressed when you are ready to run the program. The i button pops up a window with documentation. Some, but not all programs, will also have an Advanced Options button that will bring up, often very useful, optional fields.

64

Exercise continued

Scroll back to the top of the coderet form in the Jemboss window, and fill in a Sequence Filename. In fact, we want to pull a sequence directly from embl at the EBI. The sequence we want is from a plasmid and has the accession number U80928. To fetch it from the EBI, you need to type:

  • embl:U80928

into the Sequence Filename box. Enter a filename into the outfile file name box. For example, to distinguish from your later work, you could use the name: jemboss_bx.coderet.

  • * Scroll to the bottom of the window and hit the GO button.

● When the program has finished, a new window called Saved Results should appear. (Don't be fooled – your results haven't been saved yet!) There should be a number of tabs in that window. One will be called the name you entered into the the outfile file name box (e.g. jemboss_bx.coderet) The others will likely be called things like u80928.cds, u80928.noncoding, etc.

  • Take a look at the type of information in each tab. In particular, take note that:

each of the tabs that contains sequence information contains multiple sequences the command line you would use to run this program identically to how you just ran it via Jemboss is provided to you under the cmd tab. This will be useful later. ➢ ➢

To work with any of this data further, you have to save it to a local file. Click on the tab with the name ending in .cds. Choose the File | Save to Local File... option and save this to a location you can find again (e.g. under your bioinf_files directory). Give it a name that will distinguish it from later work -e.g. jemboss_bx.cds. Do not close the Saved Results window as we want to refer to the information under the cmd tab later.

  • Go back to the main Jemboss window, go to the Nucleic | Repeats section and choose

palindrome from the list of programs.

  • Browse for the file you just saved using the Browse files... button next to the box under

Sequence Filename near the top of the page. Note that you'll have to set the Files of Type: option to All Files to find your saved file because it has a .cds suffix.

  • Check that you're happy with all the required options, and give a filename in the outfile file

name box. For example, jemboss_palin.txt. Then press the GO button.

  • * Scan through the results to see what has been returned to you.

You can also view listings of the files on your system using the Jemboss file manager functionality. Click on the symbol at the bottom right side of the Jemboss window. If you double click on the name of a file that contains text, it will pop up in another window for you to view or edit. Note: the file listings in the Jemboss window are not updated unless you refresh them manually - the regular file browser or the ls command are a better way to keep track of what files have been created or deleted.

Using the EMBOSS command line

All EMBOSS commands follow a similar pattern:

  • If you just type the command name, then you are prompted for required information.

65

If you type the command name followed by -opt then you are prompted for optional

information as well as required information.

  • If you type the command name, followed by a minimum amount of information, and -auto, the

program runs and uses defaults for anything you have not specified in the command.

  • The full command (i.e. the command and all relevant options and values) can be specified by

including parameters and arguments on the command line.

  • The command name followed by -h or -help brings up information about the main options for

the program.

  • * The command name followed by -h -v brings up information about all options for the program
  • Typing tfm followed by the command name brings up the full documentation for the program.

So, using the EMBOSS program seqret as an example, we could run: seqret seqret -opt seqret -sequence embl:X03487 information. seqret -sequence embl:XO3487 -auto options. seqret -help seqret -h -v tfm seqret

Run seqret and prompt for required information. Run seqret and prompt for required and optional information. Run seqret, specifying the sequence. Prompts for additional Run seqret, specifying the sequence. Defaults are used for all other Show information about the main options for seqret Show information about all options for seqret Show full documentation for seqret

Much more information about the EMBOSS command line syntax is available at: http://emboss.sourceforge.net/developers/acd/commandline.html

Exercise – using EMBOSS command line

  • Look at the cmd tab in your jemboss results window for coderet. You should see the following:

coderet -seqall embl:U80928 -outfile jemboss_bx.coderet -auto

This command runs coderet, specifies the sequence to use and sets the output file name. The -auto option indicates that you do not want to be prompted for further information. This results in default values being used for all options you have not specified on the command line.

  • Read about coderet by bringing up the information via the command line:

coderet -h or coderet -help coderet -h -v tfm coderet

66

brings up a list of main options brings up a list of all available options brings up the full documentation

(EMBOSS commands exercise continued)

To make things simple, we will edit the command line in the coderet cmd tab of the Saved Results window in Jemboss, and then copy and paste our final command line into a terminal to run the program.

  • Go to the coderet cmd tab of the Saved Results window in Jemboss, and edit the command to give a

new output filename. e.g. coderet -seqall embl:U80928 -outfile cl_bx.coderet -auto Open a new terminal window and cd to your bioinf_files directory. Make a new directory to store your result files (as it will make it easier to see what files the program generates by default).

  • mkdir cl_dir

Change directory into your new directory, copy and paste the coderet command line above into the terminal and press the return key. (Recall that we covered highlighting and pasting text using mouse buttons near the end of the first half of this tutorial.) ie:

  • cd cl_dir

coderet -seqall embl:U80928 -outfile cl_bx.coderet -auto When the program finishes, list the files in your directory. What has coderet produced? How does this compare with the tabs presented to you when you ran coderet via Jemboss?

  • You may notice that we have generated a lot of files we don't need. We could have specified to coderet that

we only wanted the mRNA sections from the embl entry BX255937. To find out how, you'll need to refer to the coderet documentation (the lists of options won't tell you enough). Now run palindrome on the mRNA sequence. To do this, you could edit, copy and paste the the command in the Jemboss Saved Results window for palindrome, or you can type palindrome on the command line and answer the prompts. Please run palindrome now, doing one of these.

  • Once you get to know it, the command line is much faster to get running than programs via Jemboss.

However, the power of using the EMBOSS command line is much greater if you need to process groups of files, or do things repetitively. Below we'll go through an example of running an emboss program on a batch of files using a single command. If you want to run a job like this repetitively, you can save the commands in a text file and then set things up to get those command executed whenever you want (either by you directly, or by your computer at a time you schedule). We do not cover this in these course notes, but please ask the demonstrator if you would like to know more about this.

67

Exercise

Fetching a list of sequences using seqret.

Look at the contents of the file hexaseqs.list in your bioinf_files directory. e.g. using the command less. You will see a list of sequence ids and the database those sequences are in.

  • * Quit less. (hit q)

We need to tell EMBOSS programs when they are going to work on a list of files rather than just a single file. To do this, we preface the filename with the @ symbol. So, to fetch the list of sequences in the hexaseqs.list file, we can use the command:

  • seqret -sequence @hexaseqs.list

The default behaviour of seqret is to fetch sequences in fasta format, with all sequences in a single file with a filename that uses the id of the first sequence. By now you should know how to go about finding out how to alter aspects of the program behaviour like these.

  • Take a look at the sequence file you have generated.

You can use this same “list of sequences” syntax with Jemboss. e.g. you could run seqret via Jemboss and specify the sequence name as @hexaseqs.list. General things to keep in mind If you suspect there may be a more efficient way to do what you are doing, there probably is! If you find yourself doing anything repetitively, there is probably an easier way to do it. Please read documentation and seek advice. It will save you a lot of time in the end!

68

A very basic sequence assembly

This demonstration takes you through a very simple assembly of some reads from a mitochondrial genome. This is in no way supposed to be a tutorial on genome assembly, but rather a way to see various tools in action on a small dataset. This section of the course was originally written as a separate tutorial by Dan Pass. Note that, in all the commands given in this tutorial, $ represents your terminal prompt. This is a common convention, even though the real prompt will be something like “live@biolinux[live]”. Lines beginning with # are comments and not to be typed. Setup •

Open up the Bio-Linux Documentation icon in the Dash menu, then the Introductory Tutorial folder. You should see several tar files. Select assembly_taster.tar.xz and right click it. Select Extract To... from the pop-up menu. Extract to your home directory, which on the Live USB system is listed as live in the list on the left.

Open a terminal, then change into the new directory and list the files:

$ cd assembly_taster

  1. -lh options to ls show human-readable file size

$ ls -lh •

To get a quick look at the input data, you can view it in the less text file viewer:

$ less mt_reads.fastq

  1. as usual, press q to return to the terminal.

Make a new directory to store your results:

$ mkdir results

Quality Checking

Firstly, in receiving a set of sequence data it is paramount to assess the quality of the dataset. A useful tool is FastQC which gives a quick graphical overview of the dataset. • Run FastQC on the dataset $ fastqc -o results mt_reads.fastq

Open the HTML report file.

  1. The ampersand (&) will put the process in the background so you can still use the terminal

$ firefox results/mt_reads_fastqc/fastqc_report.html & •

Split Barcodes The sequencing data may be barcoded, depending on the experimental set up. Here, two mitochondria have been sequenced together, with differing 10bp barcodes at the 5’ end. This allows us to split the data into two sets whilst only performing one sequencing run. Here we use a standard script from the fastx toolkit (http://hannonlab.cshl.edu/fastx_toolkit/index.html)

69

Use fastx splitter splits mt_reads.fastq by barcode.

  1. --bol indicates that the barcodes are at the 5’ end.
  2. Note the following command should be typed on a single line:

$ fastx_barcode_splitter.pl <mt_reads.fastq --bcfile mt_barcodes.txt --bol --suffix .fastq --prefix results/

There are now two .fastq files in the results directory; one for each barcode. There is also an unmatched.fasta file which should be empty. We will be focusing on the first mitochondrion, ie. the one now in results/mt1.fastq.

Clean Up

To remove artefacts and improve the assembly we will do two steps: 1) Trim barcodes This removes the barcode sequences from the beginning of each read. The -Q33 is required due to differences in sanger and illumina encoding. $ cd results $ fastx_trimmer -i mt1.fastq -f 8 -o trimmed_mt1.fastq -Q33 2) Quality Filter Removing low quality sequences increases the accuracy of the assembly. Here we remove any sequences which do not have >25 phred quality score (-q) at 80% of bases (-p). (n.b. https://en.wikipedia.org/wiki/Phred_quality_score). • Run the quality filter

  1. -v instructs the script to give ‘verbose’ output and it is common to find in similar scripts.

$ fastq_quality_filter -i trimmed_mt1.fastq -q 25 -p 80 -o qual_trim_mt1.fastq -Q33 -v

Note that you could have run both the previous commands in one shot, combined as a pipeline. $ fastx_trimmer -i mt2.fastq -f 8 -Q33 | fastq_quality_filter -q 25 -p 80 -Q33 -o qual_trim_mt2.fastq

70

Assembly With Velvet

Velvet (https://www.ebi.ac.uk/~zerbino/velvet/) is a highly popular short-read assembler which is available on Bio-Linux. There are countless parameters and combinations to achieve the best assembly, but we will run close to default here. We will assess the quality of the assemblies in the next step. Run velvet in single-end mode with k=21 ‘k’ signifies the Kmer length i.e. the length of sub sequences that the data is being broken up into, and is one of the most important parameters to manipulate. Full parameters can be seen by typing either command with no flags.

  1. You should still be in the results directory at this point
  2. velveth is a ‘hash program’ which breaks down your data into Kmer sized sequences

$ velveth velvet_k21 21 -short -fastq qual_trim_mt1.fastq

  1. velvetg performs de Bruijn graph construction, error removal and repeat resolution

$ velvetg velvet_k21 -read_trkg yes -amos_file yes •

Inspect the results in the Tablet graphical viewer (not ideal - we have 139 contigs): $ tablet velvet_k21/velvet_asm.afg &

Quick ‘cheat’

VelvetOptimiser is a script which automatically tries multiple parameter combinations and returns the best assembly it can find. It can be helpful in pointing you in the right direction. •

Try using velvetoptimiser

$ velvetoptimiser -s 27 -e 31 -f '-short -fastq qual_trim_mt1.fastq' -a 1 $ tablet auto_data_31/velvet_asm.afg &

Assembly With Abyss

Abyss (http://www.bcgsc.ca/platform/bioinfo/software/abyss) is another popular assembler which we will run to give a comparison. Again, multitudes of parameters are available, but here we will run mostly with default settings, just optimising the K-mer length. A major benefit of working in a command-line environment is the ability to loop easily through multiple values. Without an existing ‘optimiser’ type program, a shell loop can be used to try many values.

71

Run abyss in single-end mode with k=21

$ abyss -k21 qual_trim_mt1.fastq -o abyss_contigs.fa •

Try abyss with multiple kmer values

  1. Type the first line and press return. The prompt will change to “for>”

$ for k in {15..20} for> abyss -k$k qual_trim_mt1.fastq -o abyss_k$k.fa

  1. This will run abyss for all values of k between 15 and 20, and
  2. produce output for each permutation.

Assessing The Assemblies

We used tablet to view the output from Velvet assemblies. This isn’t possible with the Abyss output as the program does not provide a full assembly, just the consensus contigs. We can obtain some simple statistics on all the assembly results on the command line. For example, the gnx-tools command will output basic statistics on the multi-fasta file produced by the assembler. •

Compare assemblies with gnx-tools

$ for f in velvet_k21/contigs.fa auto_data_31/contigs.fa abyss_contigs.fa for> gnx-tools $f

Adding Some Annotation

If sequence assembly is a tricky process to master then sequence annotation is a bona fide black art. There are various approaches that one can use and several pipelines available that will help. But in this case, we just want to get something to look at in Artemis. We’ll quickly scan the assembled genome for likely open reading frames. We’ll use the Abyss output as this has (hopefully!) produced a single contig. Glimmer3 (http://ccb.jhu.edu/software/glimmer/index.shtml) is an application for predicting open reading frames in prokaryotic genomes. As with the assemblers above, it should generally be tuned for the specific organism that you are working with and also provided with an appropriate training data set. But in this case we will just run it quickly with the default options (don't do this if you want actual meaningful results). A Perl script is provided to convert the output from Glimmer into something that Artemis can view. You don’t need to be a Perl programmer to re-use useful scripts like this. $ g3-from-scratch abyss_contigs.fa glimmer $ perl ../glimmer_to_gbk.perl <glimmer.predict >glimmer.gbk $ artemis abyss_contigs.fa &

You should now be looking at a view of the contig in Artemis. From the File menu select Read An Entry… and choose the file glimmer.gbk. To conclude this section, load the file human_mitochondrial.gbk into Artemis for comparison. This is not exectly the same as the mitochondrial data you’ve just assembled (which is from Lumbricus rubellus) but it is fully annotated. Annotation will have been achieved using a combination of automated tools and manual editing in Artemis. You can find more on Artemis, and on how to identify genes using BLAST, in the next section.

72

Artemis

Artemis is a DNA sequence viewer and annotation tool, allowing visualisation of sequence features and the results of analyses within the context of the sequence and its six-frame translation. Artemis can read embl or genbank format files. Sequences can be loaded from local files or via the network from the EBI.

Ways to run Artemis:

from a locally installed version on your Bio-Linux machine* via Java Web Start from the Sanger Centre (http://www.sanger.ac.uk/resources/software/artemis/java/artemis.jnlp) ●

  • Figure 16: Artemis Entry window after hsy14768.embl is loaded.

73

Exercise

Start Artemis on Bio-Linux by typing artemis on the command line or by choosing the option Artemis from under the Bioinformatics Applications graphical menu.

  • Now choose the option Open... from under the Artemis File menu, and select the

file hsy14768.embl from within the bioinf_files directory.

  • This should open up a large window, as shown in Figure 14, where this sequence is displayed

graphically .

Open a terminal window and view the text of the embl entry using the command less hsy14768.embl

  • Notice how Artemis is providing a graphical representation of what is in the text file.

Try choosing Mark Open Reading Frames from under the Create menu of Artemis.

  • * Choose to mark open reading frames with a minimum size of 200.

You should now see two boxes near the top in the Entry section, the first called hsy14768.embl and the other called ORFS_200+. Uncheck the box next to hsy14768.embl. You should now be able to scroll along the window horizontally and easily see the open reading frames you marked.

  • Check the box next to hsy14768.embl again. Look at the information in the bottom

frame of the window. Notice how it is related to the images in the frames above.

  • Try clicking on some of the lines in the bottom frame and seeing what happens in the

images in the other two frames.

  • Explore the options available to you. (Not all options will be functional by default. See the

information about the Run menu below)

  • * Close the Artemis Entry Editing window using File | Close.

You can also load up files direct from the EBI. If you want to try this, then choose File | Open from the EBI – Dbfetch... option in the original small Artemis window and enter the accession number BX255937.

  • When you are done, close Artemis by choosing File | Close in the sequence entry

window and then choosing File | Quit in the main (small) Artemis window.

  • You can run various programs on your sequence, or parts of your sequence, from under the Run menu in

Artemis. Some of the options in this menu need to be configured to be appropriate for your site. There is information on how to do this on our website at: http://nebc.nerc.ac.uk/tools/bioinformatics-docs/faq#blast_art If you are not the system administrator of your Bio-Linux machine, then you will probably need to liaise with the person who is to get this set up properly.

74

We also highly recommend Artemis’ sister program Act, which can be used to graphically view a pairwise BLAST betrween two or more sequences.

Appendix A – BLAST references and documentation

Web pages The blastall and blast+ page in your Bio-Linux Bioinformatics Docs provides links to local web pages with information about NCBI BLAST programs. You can also access this remotely at the URL: http://nebc.nerc.ac.uk/bioinformatics/docs/blastall.html http://nebc.nerc.ac.uk/bioinformatics/docs/blast+.html NCBI BLAST Manual pages http://www.ncbi.nlm.nih.gov/books/NBK1763/ http://www.ncbi.nlm.nih.gov/blast/blast_help.shtml NCBI BLAST Web Interface paper http://nar.oxfordjournals.org/cgi/content/full/36/suppl_2/W5 Sequence similarity statistics http://www.ncbi.nlm.nih.gov/BLAST/tutorial/Altschul-1.html NEBC BLAST Frequently asked questions http://nebc.nerc.ac.uk/tools/bioinformatics-docs/other-bioinf/blastfaq NEBC November 2007 Masters Bioinformatics Course (covers older blastall, rather than BLAST+) http://nebc.nerc.ac.uk/support/training/course-notes/past-notes/nebc-introduction-to-bioinformaticsmsc.-biology-2007

References

The book by Ian Korf is a good place to start in learning about what BLAST can do, how it does it and what BLAST output means. It is now out of date however, and should be read in conjunction with the new blast+ documentation. Also note that wu-blast is now AB-blast, which is licensed software from Advanced Biocomputing LLC. S. F. Altschul, T. L. Madden, A. A. Schaffer, J. Zhang, Z. Zhang, W. Miller, and D. J. Lipman. Gapped blast and psi-blast: a new generation of protein database search programs. Nucleic Acids Res, 25(17):3389–402, 1997. Lm05110/lm/nlm Journal Article Research Support, U.S. Gov’t, P.H.S. Review England. S. F. Altschul, J. C. Wootton, E. M. Gertz, R. Agarwala, A. Morgulis, A. A. Schaffer, and Y. K. Yu. Protein database searches using compositionally adjusted substitution matrices. Febs J, 272(20):5101–9, 2005. Z01 lm000072-10/lm/nlm Journal Article Review England. C. Camacho, G. Coulouris, V. Avagyan, M.N. Papadopoulos, K. Bealer and T.L. Madden. Blast+: architecture and applciations. BMC Bioinformatics, 10: 421, 2009 S. R. Eddy. Where did the blosum62 alignment score matrix come from? Nat Biotechnol, 22(8):1035–6, 2004. Evaluation Studies Journal Article Review United States. Ian Korf, Mark Yandell, Joseph Bedell, and Stephen Altschul. BLAST. [“An essential guide to the Basic Local Alignment Search Tool”. Includes bibliographical references and index.] O’Reilly, Sebastopol, Calif. ; Farnham, 2003. GB A3-Y7706 ill. ; 24 cm. A. A. Schaffer, L. Aravind, T. L. Madden, S. Shavirin, J. L. Spouge, Y. I. Wolf, E. V. Koonin, and S. F. Altschul. Improving the accuracy of psi-blast protein database searches with composition-based statistics and other refinements. Nucleic Acids Res, 29(14):2994–3005, 2001. Journal Article Review England. Y. K. Yu, E. M. Gertz, R. Agarwala, A. A. Schaffer, and S. F. Altschul. Retrieval accuracy, statistical significance and compositional similarity in protein sequence database searches. Nucleic Acids Res, 34(20):5966–73, 2006. Evaluation Studies Journal Article Research Support, N.I.H., Intramural England.

75

Appendix B – Creating local BLAST databases

Obtaining local BLAST databases To get the most from BLAST, you should search against a relevant database, which may mean using the relevant parts of a larger database. In general, BLAST searching against the whole of nr or the whole of embl is not a particularly good idea. It takes up your time and computer resources, returns BLAST results with less useful statistics and often less meaningful results. For example, if you are studying marine viruses, do you really care about all the mouse sequence in nr or embl? Web resources often offer different data subsets you can search against. For example, using the NCBI BLAST pages, you can choose from a certain number of database sections, or you can fine tune the sequence set you blast against using Entrez queries: http://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=FAQ#entrez http://www.ncbi.nlm.nih.gov/bookshelf/br.fcgi?book=helpentrez&part=EntrezHelp Using the EBI BLAST services, you can choose from a number of data subsets, as well as having a choice of WU-blast or NCBI blastall. http://www.ebi.ac.uk/Tools/blast/ To run BLAST locally, you need to index your collection of sequences; it is these indices that BLAST reads when searching. For some databases or database divisions, you can download prepared BLAST indices from sites such as the NCBI. These are convenient, but do restrict you to searching against particular sets of sequences. It is often useful to create a set of sequences chosen for the types of searches you wish to carry out (e.g. organism or tissue specific) and format them into a database you can search using BLAST. Any set of fasta sequences can be indexed for BLAST searching. Creating useful sets of sequences is beyond the scope of this course, but two resources to consider are SRS (http://srs.ebi.ac.uk) and Entrez (http://www.ncbi.nlm.nih.gov/books/bookres.fcgi/helpentrez/EntrezHelp.pdf). For NCBI blastall, the formatdb command is run on fasta formatted files to create BLAST indices. For BLAST+, the program used is called makeblastdb, and this is the you want to use, though BLAST+ will happily search databases made with formatdb. Some data resources useful for local BLAST URL

Database File format

Contents

ftp://ftp.ebi.ac.uk/pub/databases/fastafiles/uniprot/

uniprot

fasta

Uniprot, swissprot and trembl

ftp://ftp.ebi.ac.uk/pub/databases/uniprot/current_rele uniprot ase/knowledgebase/taxonomic_divisions/

embl

Uniprot divisions

ftp://ftp.ebi.ac.uk/pub/databases/fastafiles/emblreleas embl e/

fasta

Individual embl divisions

ftp://ftp.ebi.ac.uk/pub/databases/embl/release/

embl

embl

Individual embl divisions

ftp://ftp.ncbi.nlm.nih.gov/blast/db/ ftp://ftp.ebi.ac.uk/pub/blast/db/

various

blast

nr, nt, env and a few other BLAST formatted databases or database sections.

ftp://ftp.ncbi.nlm.nih.gov/genbank

genbank

genbank

Individual genbank divisions

76

One thing to note in the table above is that uniprot divisions are provided in embl format. However, BLAST

indices are created from fasta format files. Unfortunately, the EMBOSS program seqret, which you saw earlier, does not handle entire database divisions well. Instead, you can use a simple script to do the conversion. Instructions on this are below. If you choose to use pre-formatted BLAST databases, make sure you read the notes about them (usually available as a file called something like REAMDE on the FTP site you get the BLAST files from) as they can be slightly different than the database that results from downloading and formatting your own.

Understand your databases

It is important to read the documentation about the databases you choose to work with. For example, uniprot and nr are not the same. nt is not a non-redundant database; nr is. Knowing what is in a database you work with is vital in understanding your results. Nucleic Acids Research publishes a database issue in January of each year. This is an excellent resource for finding out more about available database resources. Another useful resource is the information available via the links on the Library page of SRS at the EBI: http://srs.ebi.ac.uk/srsbin/cgi-bin/wgetz?-page+top

Building BLAST indices from local sequence files

We will use the uniprot swissprot virus division as an example here. As this is distributed in embl format, and we need it in fasta format, we include a format conversion step in the instructions below. Bio-Linux machines by default have the BLASTDB environmental variable set to a central location. To find out where it is set to on your machine, you can use the command: echo $BLASTDB If you are logged in as an administrative user, then you will be able to download and work in any area on the machine using your sudo privileges. If you are on a multi-user system and are not an administrative user, the default location for BLAST databases may not be writable by you. In this case, you should talk to your system administrator: either to ask them to give you privileges in the central BLAST database folder, or warn them that you are about to use lots of space in your account for BLAST databases. These instructions assume that you are working from the directory where you will be storing your BLAST database files. This is not normally the case. Usually, if you download BLAST databases into your account, it is easiest to set the BLASTDB environmental variable to the location of these BLAST databases, and then work from a convenient folder where you plan to store your results. You can set the BLASTDB environmental variable for a single session by typing a line of the form below in the terminal you are working in. To set this variable for every session, you can add the line to your ~/.zshrc file. export BLASTDB=”$HOME/blastdb”

  • Download the database section of interest. Here we will work with the uniprot swissprot virus division:

wget ftp://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/knowledgebase/taxonomic_divisions/uniprot_sprot_viruses.dat.gz

77

If you don't already have a sequence conversion tool, download the emblToFastaAndPreProcess.pl

script from the NEBC site.

This script converts embl sequence to fasta sequence. Due to issues that sometimes appear because of the formatting of information in the feature table, it does so by removing the feature lines from the entry before conversion. A version of the script that does not pre-edit the feature lines is also available: http://nebc.nerc.ac.uk/downloads/scripts/bioinf/emblToFasta.pl

  • Make this script executable.

chmod u+x emblToFastaAndPreProcess.pl

This script can handle compressed files, so you can create a fasta formatted copy of the uniprot_sprot_viruses division by running the command:

  • ./emblToFastaAndPreProcess.pl uniprot_sprot_viruses.dat.gz

Notice the ./ at the start of the line. You need this if you are running the script from the directory you are in. There are better ways to do this if you plan to keep this script for use again, but they are not covered here. When the script is finished, you should find a file called uniprot_sprot_viruses.fasta in your directory. This is the file we build the BLAST database from.

  • makeblastdb -dbtype prot -in uniprot_sprot_viruses.fasta -out sprot_virus

You should now have four new files in your directory: sprot_virus.psq, sprot_virus.pin, sprot_virus.phr and formatdb.log. The last of these lets you know how the BLAST formatting went.

  • The sprot_virus.p* files are your BLAST indices. You search against them by specifying the BLAST

database name sprot_virus. Note: If you were interested in the swissprot virus division, you would probably be interested in the trembl virus division also. You could download and format that division as described above, and then search the swissprot and trembl virus divisions separately, or as a single, virtual database. Alternatively, you could create a single BLAST formatted database from the two fasta files using cat and makeblastdb: cat uniprot_sprot_viruses.fasta uniprot_trembl_viruses.fasta | makeblastdb -in - -out uniprot_viruses -dbtype prot -title "combined sprot and trembl virus divisions"

What is the best division to search against depends on what you need to accomplish.

78

Appendix C - Cheat sheet of basic Linux commands

bg

To send a suspended job to the background

cat fileName1

Output a file to the screen (see also more and less)

cat file1 file2 file3 > newfile

Append three files together and put the result in newfile

cat -nA file1

Output a file to screen, numbering all lines and revealing nonprinting characters

cd dirName

Change to directory dirName. Use cd .. to go up one dir or just cd to go home.

chmod

To change the permissions or protection on a file, to allow everyone to read a file (chmod a+r somefile)

clear

clear the terminal screen

cp fileName1 fileName2

create a copy of the file called fileName1 and call the copy fileName2

cp fileName directoryName

copy the file fileName into a directory called directoryName

cp –R dirName1 dirName2

copy a whole directory called dirName1 and its contents into another directory called dirName2.

date

Print the current date and time

df –h

File system information including space usage

diff file1 file2

Summarise differences between two similar text files file1 and file 2. See also the graphical tool, meld

echo $NAME

Print the value of an environment variable called $NAME

emacs

A text editor, more powerful than gedit, but more complex.

evince

A command for viewing postscript or PDF formatted files

exit

Exit the current terminal

export NAME=value

Set the environment variable $NAME to “value”

fg

Brings a suspended or background job to the foreground

file fileName

Tries to determine what fileName is by looking at the contents

find -name “test*”

Scans for filenames matching a given glob pattern in the current folder and subfolders. This command is tricky to use. To scan the whole system for files, try locate.

gedit

The standard text editor

grep

Search for the occurrence of a pattern

groups or id

Show what groups a user is in.

head fileName

Show just the first few lines of fileName

history

List log of previous commands you have entered

jobs

Lists any suspended or background processes that you have running. See also ps and pgrep

kill pid

Kill a process that is running where pid is the process id number (see ps). Also consider pgrep and pkill.

last

Info about who has logged onto the machine recently

less

Type a file to the screen one page at a time (press q to quit, spacebar for next page, b to go back a page)

ls

List the files in your directory

ls –l

List the files in your directory but with “longer” information. (Add -h for more readable file sizes)

man command

For help about UNIX command “command”

man -k keyword

Lists all UNIX commands that mention the word “keyword”

mkdir dirName

Make a directory

more fileName

Type a file to the screen a page at a time (press q to quit, spacebar for next page).

mv file1 dirName

Assuming dirName is an existing directory, move a file called file1 into a directory called dirName

mv file1 file2

Rename file1 and call it file2

nano

A basic text editor that runs in the terminal

passwd

Change your password

pgrep pattern

Find process names that contain the pattern. See also ps

pkill processname

Kill a running process using the process name. Be careful with this! See also ps, pgrep and kill

pwd

Print the full path of your current directory

ps –u

List your current processes

ps –aux

List all processes on the machine. See also top

rm fileName

Delete a file

rm –rf dirName

Delete a directory and all its contents

rmdir

Delete an empty directory

screen

Run the screen manager (read the man page first!)

stat fileName

Show detailed info on fileName, similar to ls -l

tail

Show just the last few lines of a file. See also head.

tar -xvz -f fileName.tar.gz

Unpack a tarball from the file fileName.tar.gz

someCommand | tee fileName

Save output of someCommand to fileName and also print to screen. Use instead of >fileName if you want to redirect but still see the output.

top

List the processes running that are using the most CPU

touch fileName

Create an empty file (also updates file timestamps)

wc -l fileName

Count lines in fileName

which commandName

Reveal what will really be run when you give a command

w or who

List users currently logged on

yes

A very useful command ;-)

Ctrl-c

Stop (interrupt) a process

Ctrl-r

Interactively search in command log. See history

Ctrl-z

Suspend a process, see also jobs, fg and bg