Difference between revisions of "Quality Control and Preprocessing Talk"

From wiki
Jump to: navigation, search
Line 45: Line 45:
 
= GFF3 file example =
 
= GFF3 file example =
  
−
##gff-version 3
+
[[File:gff.png]]
−
##sequence-region ctg123 1 1497228
 
−
ctg123 . gene 1000 9000 . + . ID=gene00001;Name=EDEN
 
−
ctg123 . TF_binding_site 1000 1012 . + . Parent=gene00001
 
−
ctg123 . mRNA 1050 9000 . + . ID=mRNA00001;Parent=gene00001
 
−
ctg123 . mRNA 1050 9000 . + . ID=mRNA00002;Parent=gene00001
 
−
ctg123 . mRNA 1300 9000 . + . ID=mRNA00003;Parent=gene00001
 
−
ctg123 . exon 1300 1500 . + . Parent=mRNA00003
 
−
ctg123 . exon 1050 1500 . + . Parent=mRNA00001,mRNA00002
 
−
ctg123 . exon 3000 3902 . + . Parent=mRNA00001,mRNA00003
 
−
ctg123 . exon 5000 5500 . + . Parent=mRNA00001,mRNA00002,mRNA00003
 
−
ctg123 . exon 7000 9000 . + . Parent=mRNA00001,mRNA00002,mRNA00003
 
−
ctg123 . CDS 1201 1500 . + 0 ID=cds00001;Parent=mRNA00001
 
−
ctg123 . CDS 3000 3902 . + 0 ID=cds00001;Parent=mRNA00001
 
−
ctg123 . CDS 5000 5500 . + 0 ID=cds00001;Parent=mRNA00001
 
−
ctg123 . CDS 7000 7600 . + 0 ID=cds00001;Parent=mRNA00001
 
−
ctg123 . CDS 1201 1500 . + 0 ID=cds00002;Parent=mRNA00002
 
−
ctg123 . CDS 5000 5500 . + 0 ID=cds00002;Parent=mRNA00002
 
−
ctg123 . CDS 7000 7600 . + 0 ID=cds00002;Parent=mRNA00002
 
−
ctg123 . CDS 3301 3902 . + 0 ID=cds00003;Parent=mRNA00003
 
−
ctg123 . CDS 5000 5500 . + 2 ID=cds00003;Parent=mRNA00003
 
−
ctg123 . CDS 7000 7600 . + 2 ID=cds00003;Parent=mRNA00003
 
−
ctg123 . CDS 3391 3902 . + 0 ID=cds00004;Parent=mRNA00003
 
−
ctg123 . CDS 5000 5500 . + 2 ID=cds00004;Parent=mRNA00003
 
−
ctg123 . CDS 7000 7600 . + 2 ID=cds00004;Parent=mRNA00003
 
−
 
 
−
{|style="width:90%"
 
−
| Col1
 
−
| Col2
 
−
| Col3
 
−
| Col4
 
−
| Col5
 
−
| Col6
 
−
| Col7
 
−
| Col8
 
−
| Col9
 
−
|-
 
−
| "seqid"
 
−
| "source"
 
−
| "type"
 
−
| "start"
 
−
| "end"
 
−
| "score"
 
−
| "strand"
 
−
| "phase"
 
−
| "attributes"
 
−
|}
 
  
 
= GFF graphicalGFF =
 
= GFF graphicalGFF =
−
representation
+
* representation
−
structure
+
* structure
  
 
[[File:sascha.png]]
 
[[File:sascha.png]]
Line 170: Line 124:
 
Per sequence quality scores
 
Per sequence quality scores
  
−
[[File:psqs.png]]
+
[[File:bimods.png]]
−
{|style="width:80%"
 
−
| Good data = most reads
 
−
are high-quality sequences
 
−
|-
 
−
| Bad data = Distribution
 
−
with bi-modalities
 
−
|}
 
  
 
= Per tile sequence quality 1 =
 
= Per tile sequence quality 1 =
−
[[File:ptsq.png]]
+
[[File:ptsq3.png]]
−
 
 
−
{|style="width:90%"
 
−
| Good data = Blue all over
 
−
|-
 
−
| Bad data = Presence of hot colours
 
−
|}
 
  
 
= Per tile sequence quality 2 =
 
= Per tile sequence quality 2 =
−
[[File:ptsq2.png]]
+
[[File:ptsq4.png]]
−
 
 
−
{|style="width:90%"
 
−
| Problems in some tiles
 
−
|-
 
−
| Filtering of reads with Q30 on 90% of the read
 
−
|}
 
  
 
= Per base sequence content =
 
= Per base sequence content =
  
−
[[File:pbsc.png]]
+
[[File:pbsc3.png]]
−
 
 
−
{|style="width:90%"
 
−
| Good data = smooth over the read
 
−
|-
 
−
| Bad data = Sequence position bias and adapter contamination
 
−
|}
 
  
 
= Per base GC content =
 
= Per base GC content =
  
−
[[File:pbgcc.png]]
+
[[File:pbgcc3.png]]
−
 
 
−
{|style="width:90%"
 
−
| Good data = smooth over the read
 
−
|-
 
−
| Bad data = Sequence position bias and adapter contamination
 
−
|}
 
  
 
= Per sequence GC content =
 
= Per sequence GC content =
  
−
[[File:psgcc.png]]
+
[[File:psgcc3.png]]
−
 
 
−
{|style="width:90%"
 
−
| Good data = Normal distribution, Distribution fits with expected, Organism dependent
 
−
|-
 
−
| Bad data = Distribution doesn’t fit with expected.  Possibility of contamination
 
−
|}
 
  
 
= Per sequence N content =
 
= Per sequence N content =
  
−
[[File:psnc.png]]
+
[[File:psnc3.png]]
−
 
 
−
{|style="width:90%"
 
−
| Good data
 
−
|-
 
−
| Bad data = There are peaks of Ns per base position.
 
−
|}
 
  
 
= Sequence duplication levels =
 
= Sequence duplication levels =
  
−
[[File:sdl.png]]
+
[[File:sdl3.png]]
−
 
 
−
{|style="width:90%"
 
−
| Good data
 
−
|-
 
−
| Bad data = High number of duplicates. Indicates some kind of enrichment bias.
 
−
|}
 
−
* Note:
 
−
:- Only few sequences are used to make this judgment.
 
−
:- For RNASeq, higher number of duplicated sequences are expected.
 
  
 
= Overrepresented sequences and k-mer content =
 
= Overrepresented sequences and k-mer content =
  
−
[[File:ovrep.png]]
+
[[File:ovrep3.png]]
−
* Exact same sequences too many times
 
−
* PCR primers, Adapters, etc.
 
−
 
 
−
* Note:
 
−
:- Sometimes this is expected
 
  
 
= Sequence Filtering 1 =
 
= Sequence Filtering 1 =
  
 
* It is important to remove bad quality data as our confidence on downstream analysis will be improved.
 
* It is important to remove bad quality data as our confidence on downstream analysis will be improved.
−
[[File:sf.png]]
+
[[File:sf1.png]]
  
 
= Sequence Filtering 2 =
 
= Sequence Filtering 2 =
−
[[File:sf2.png]]
+
[[File:sf3.png]]
−
* Mean quality
 
−
* Read length after trimming
 
−
* Percentage of bases above a quality threshold
 
−
* Adapter trimming
 
  
 
= Sequence filtering tools =
 
= Sequence filtering tools =

Revision as of 23:02, 8 May 2017

Quality control and data pre-processing

Contents

  • Data formats
– Fasta and Fastq formats
– Sequence quality encoding
  • Quality Control (QC)
– Evaluation of sequence quality
– Quality control tools
– Addressing QC with FastQC
– Typical artifacts and sequence filtering

Data formats

  • Text-based formats
  • If not compressed, it can be huge
  • Almost every programming language has a parser

Fformat.png

Data formats

Fastq Format: Sequence quality encoding

Ascii.png

Sequence Data Format

Raw sequence data format (Flat/Binary files)

Processed sequence data format (Flat files)

  • Column separated files containing genomic features

and their chromosomal coordinates.

  • Different files
  • GFF and GTF
  • BED

GFF

  • Column separated file format contains features located at chromosomal locations
  • Not a compact format
  • Several versions

– GFF 3 most currently used – GFF 2.5 is also called GTF (used at Ensembl for describing gene features)

GFF3 file example

Gff.png

GFF graphicalGFF

  • representation
  • structure

Sascha.png

GFF3 can describes the representation of a protein-coding gene (From Sascha Steinbiss' genome-tools suite of programs: http://genometools.org/)

BED

  • Created by UCSC Genome team
  • Contains similar information to the GFF, but optimized for viewing in the UCSC genome browser
- Essentially about features and ranges.
  • BIG BED, optimized for next gen data – essentially a binary version
– It can be displayed at UCSC Web browser (even several Gbs !!)

Quality Control

Evaluation of sequence quality

  • Primary tool to assess sequencing
  • Evaluating sequences in depth is a valuable approach to assess how reliable our results will be.
  • QC determines posterior filtering
- Any filtering decision will affect downstream analysis.
  • QC must be run after every critical step.

Quality control tools 1

Fastx.png

Ngsqc.png

Quality control tools

Fastqc.png

Multiple Sample Quality control

- uses FastQC output

Multiqc.png

Addressing QC with FastQC

  • various screen devoted to plots of the following:
- Basic stats
- Per base sequence quality
- Per read sequence quality
- Per base sequence content
- Per base GC content
- Per sequence GC content
- Per base N content
- Sequence length distribution
- Duplicate sequences
- Overrepresented sequences
- Overrepresented k-mers

Examples on web:

- Good quality:

http://www.bioinformatics.babraham.ac.uk/projects/fastqc/good_sequence_short_fastqc.html

- Bad quality:

http://www.bioinformatics.babraham.ac.uk/projects/fastqc/bad_sequence_fastqc.html

Per base sequence quality, good

Pbsqg.png Good data = Consistent high quality along the read

Per base sequence quality, bad

Pbsqb4.png

Bad data = Quality decreases towards the end of the read and High variance

Addressing QC with FastQC

Per sequence quality scores

Bimods.png

Per tile sequence quality 1

Ptsq3.png

Per tile sequence quality 2

Ptsq4.png

Per base sequence content

Pbsc3.png

Per base GC content

Pbgcc3.png

Per sequence GC content

Psgcc3.png

Per sequence N content

Psnc3.png

Sequence duplication levels

Sdl3.png

Overrepresented sequences and k-mer content

Ovrep3.png

Sequence Filtering 1

  • It is important to remove bad quality data as our confidence on downstream analysis will be improved.

Sf1.png

Sequence Filtering 2

Sf3.png

Sequence filtering tools

  • Fastq-mcf
- https://code.google.com/p/ea-utils/wiki/FastqMcf
  • Cutadapt
- https://code.google.com/p/cutadapt/
  • SeqTK
- https://github.com/lh3/seqtk
  • Trimmomatic
- http://www.usadellab.org/cms/?page=trimmomatic)

Next

Practical sequence filtering session

Retrieved from "http://stab.st-andrews.ac.uk/wiki/index.php?title=Quality_Control_and_Preprocessing_Talk&oldid=1554"