Journal:Bioinformatics workflow for clinical whole genome sequencing at Partners HealthCare Personalized Medicine
|Full article title||Bioinformatics workflow for clinical whole genome sequencing at Partners HealthCare Personalized Medicine|
|Journal||Journal of Personalized Medicine|
|Author(s)||Tsai, E.A.; Shakbatyan, R.; Evan, J.; Rossetti, P.; Graham, C.; Sharma, H.; Lin, C.-F., Lebo, M.S.|
|Author affiliation(s)||Partners HealthCare, Brigham and Women’s Hospital, Harvard Medical School|
|Primary contact||Tel.: +1-617-768-8292|
|Editors||Weiss, S.T.; Liggett, S.B.|
|Volume and issue||6(1)|
|Distribution license||Creative Commons Attribution 4.0 International|
- 1 Abstract
- 2 Introduction
- 3 Results
- 4 Discussion
- 5 Methods and materials
- 6 Conclusions
- 7 Supplementary materials
- 8 Acknowledgments
- 9 Author contributions
- 10 Conflicts of interest
- 11 References
- 12 Notes
Effective implementation of precision medicine will be enhanced by a thorough understanding of each patient’s genetic composition to better treat his or her presenting symptoms or mitigate the onset of disease. This ideally includes the sequence information of a complete genome for each individual. At Partners HealthCare Personalized Medicine, we have developed a clinical process for whole genome sequencing (WGS) with application in both healthy individuals and those with disease. In this manuscript, we will describe our bioinformatics strategy to efficiently process and deliver genomic data to geneticists for clinical interpretation. We describe the handling of data from FASTQ to the final variant list for clinical review for the final report. We will also discuss our methodology for validating this workflow and the cost implications of running WGS.
Keywords: clinical sequencing, WGS, NGS, next generation sequencing, bioinformatics, validation, precision medicine
Precision medicine is becoming an increasing focus in medical research. To achieve the resolution necessary to personalize clinical care, greater attention has been drawn towards higher resolution of the patient genome. Next generation sequencing (NGS) provided a cost-effective method for targeted sequencing of known disease genes at base pair resolution. Moreover, the advent of exome sequencing enabled rapid discovery of genes causing Mendelian disorders. While gene panels and exome sequencing have proved fast and cost-effective for delivering genomic results back to the patient, these technologies are limited by our current knowledge of the exome, which changes over time. Additionally, the use of targeted capture may introduce biases to the data, including PCR duplicates, depth of coverage disparities, and failures at difficult to amplify target regions.
Practical considerations such as sequencing costs, data processing and maintenance, and data analysis complexities are important considerations when a laboratory is considering a new NGS program. These issues are amplified in whole genome sequencing (WGS) due to the volume of the data and have long been barriers to entry for clinical laboratories looking to adopt WGS. Despite the ability of WGS to interrogate the entirety of the genome, clinical interpretation still often focuses on only 3% of the genome (i.e., exome data, pharmacogenomics risk variants, and single nucleotide variants associated with complex disease risk). Therefore, WGS services may be overlooked for clinical applications as they trend towards increased costs and longer turnaround times due to a heavier computational load, increased number of variants for analysis, and larger data archives. However, the steadily decreasing cost of sequencing and storage now allow laboratories to consider genome sequencing. WGS, and more specifically PCR-free WGS, also decreases the need to re-sequence each time the coding sequences of targeted regions change, novel genes are discovered, or a new reference genome is released. The balance between cost, turnaround time, accuracy, and completeness has to be addressed when launching a WGS program. Here, we describe the workflow we adopted and the challenges we met supporting the bioinformatics of WGS in a clinical setting.
Our pipeline performs robustly as different entry points and different runs of the same sample returned exactly the same variants. Using HapMap sample NA12878 and previous Sanger confirmed regions, we identified thresholds for variants of Quality by Depth (QD) ≥ 4 and Fisher Strand Bias (FS) ≤ 30 as providing the optimal balance of sensitivity and specificity. This sample has been well characterized by our laboratory and also by other groups. Of 425 total confirmed variants, all 425 variants were detected by genome sequencing for a sensitivity of 100% (95% CI: 99.1%–100% for SNVs and 79.6%–100% for indels; Table 1a). In addition, four likely reference sequence errors, positions where only the alternative allele has ever been identified, were correctly genotyped by genome sequencing as homozygous for the alternative allele, three of which had incorrect genotype calls with previous orthogonal assays. In addition to these true positive variants, calls were also made for 21 false positive (FP) variants, including 20 substitutions and one indel (Table 1b). After filtration of variants based on optimal QD and FS thresholds discussed, only one FP remained.
Our genome-wide comparison of the variants detected through this pipeline compared to those detected at a similar coverage through the 1000 Genomes Project for NA12878 confirmed similar results (Table 1c). Out of 2.7 million SNVs and 285,000 indels detected through our pipeline, we found that 98.8% of these SNVs were also called in the 1000 Genomes dataset. In addition, 97.6% of the total variants called were concordant for genotype. Annotation and variant filtration were also performed on NA12878. Greater than 50 variants were randomly selected for manual inspection to ensure that they were properly annotated and filtered.
Known regions of poor coverage
All of the genomes delivered to us from Illumina’s CLIA-laboratory have at least 30X coverage, with an average coverage of 43X. However, this coverage varies across the coding sequences of the genome and affects both clinically relevant and clinically unknown regions of the genome. A gene-level list of the percentage of callable bases is provided in Supplementary Table S1. In total, of the 1381 genes with at least five asserted Pathogenic or Likely pathogenic variants by clinical laboratories in ClinVar, 94 had <90% coverage across their coding region. The 20 genes with the poorest coverage metrics included many with high prevalence and clinical relevance (Table 2). In our clinical process, for indication gene-list driven analyses, we report back the coverage of analyzed genes, highlighting those with coverage issues. This provides a useful guide to determining potential false negative findings, including those due to regions that are difficult to sequence with NGS technologies.
Cost analysis and scalability
The majority of our bioinformatics costs of WGS lies in data storage and not in computational processing. As part of our clinical workflow, the FASTQ, BAM, and VCF files are periodically archived on a replicated, secondary storage site. We also store MD5 checksums to ensure data consistency during transfer and storage. These disks are primarily used to store data, and any access to these files must be done after they are copied back onto a faster storage location (e.g., primary storage). Based on the calculations of CPU time used and the lifespan of our cluster as well as the hourly costs of a bioinformatics analyst, we estimated that it costs $77 for the computational time and $128 for the hands-on time necessary to process a genome. However, approximately 300 GB per genome of storage is required to keep files containing unaligned and aligned reads as well as the final variant calls (respective FASTQ, BAM, and VCF files).
In Table 3, we estimate the annual cost of storage per genome to be around $40–$55, depending on whether the genome is stored on a primary storage device or a secondary (or archival) disk; the bioinformatics process and storage of a genome for one year for our laboratory is approximately $245. Disk storage of sequencing data incurs significant cost in addition to the sequencing and interpretive components, and depending upon laboratory needs and policies, the availability of the data on either active or deep storage may differ. This pipeline is built on a high performance computing (HPC) cluster, so the ability to process many genomes at a given time is heavily dependent on the size of the cluster. With our current infrastructure, we do not run more than 10 genomes simultaneously. The pipeline is scalable with respect to the cluster size, but we have not explored scaling this process to simultaneously handle hundreds or thousands of genomes.
The validated bioinformatics workflow we described above has been used to process hundreds of genomes. We have been updating its functionality and re-validating the components of this process multiple times since initially implementing the pipeline. The modularity of our process allows us to repeat only the annotation, upload, and filtration processes when new or updated annotation resources are available. Similarly, filtrations can be easily re-queried from the persistent variant data in the Oracle database.
We have also compared the results of our NA12878 with the Genome in a Bottle Consortium (GIAB) dataset driven by the National Institute of Standards and Technology (NIST). A similar tool known as the Genetic Testing Reference Materials Coordination Program (GeT-RM) allows users to query across different alignments of the same dataset to see if the variant of interest found by your pipeline also is found by other pipelines. This dataset includes variant data generated by our laboratory. Both of these resources are tremendously valuable in validating clinical genome sequencing.
The limitation of genome sequencing is an important consideration. Although it is often times termed “whole genome,” it has many regions that cannot be accurately determined by current technologies. These regions are inaccessible to current, standard clinical WGS analysis and include regions of high homology for which reads cannot be uniquely mapped, regions where the reference genome contains errors, regions with multiple reference haplotypes, and tandem repeats that extend beyond the sequenced read length. These regions can overlap with clinically relevant areas and can potentially lead to false negative results if not dealt with carefully. In Table 2, we have indicated a subset of these that consistently have the lowest coverage in our WGS assay. Current techniques for managing these regions include repeat-primed PCR for repeat expansion disorders, long-range PCR for genes with homology issues, and allele-based barcoding techniques to anchor reads to unique neighboring regions. Aligning to the GRCh38 reference genome may help for regions with multiple reference alleles, but it is still unclear how those regions will be implemented in a clinical workflow. Eventually the use of longer read technologies, and possibly de novo assembly, will enable more accurate aligning and detection of variants in these difficult regions.
As genome sequencing becomes more routine, additional enhancements will continuously be made to the process. In our laboratory, these enhancements focus on a user interface for variant filtration to enable live filtering for our clinical staff, identification of copy number and structural variation from the WGS data, and upgrading the pipeline to align to the GRCh38 reference genome. This work requires a team of bioinformaticians with a broad skillset that can interact directly with both the clinical staff and the information technology teams. Finding this dynamic can prove to be one of the most challenging aspects of creating a clinical genome sequencing program.
Methods and materials
The bioinformatics pipeline processes data from the hard drives delivered by Illumina Clinical Services Laboratory (San Diego, CA, USA) to the variant report files generated for clinical interpretation in a fairly automated fashion. As these hard drives are received, they are accessioned in the laboratory and uploaded to our high performance computing (HPC) environment. The BAM files delivered on the hard drives contain reads aligned by the pipeline developed at Illumina using CASAVA (or ISAAC in recent data deliveries). These files contained both aligned and unaligned reads, so the conversion from BAM back to FASTQ format to decouple mapping information from the read sequence does not incur loss of sequencing information.
Data processing from the FASTQ file to the VCF file was performed primarily using the parameters recommended by each respective software package. Paired-end alignment of the sequencing reads to the hg19 reference genome was performed using bwa 0.6.1-r104. The aligned reads were sorted and PCR duplicates removed using samtools 0.1.18. Local indel realignment, base quality recalibration, variant calling by UnifiedGenotyper, and variant recalibration were performed using Genome Analysis ToolKit (GATK) 2.2.5 and the recommendations in the Best Practices Workflow by the GATK development team at the Broad Institute (Cambridge, MA, USA). The entire workflow is summarized in Figure 1. For more efficient computing, our validated process includes parallelization by dividing files into smaller pieces for even distribution across the cluster. This method was described in our application of this pipeline to the MedSeq Project, a randomized clinical trial assessing the impact of genome sequencing in clinical practice and to the discovery of a putative locus causing a novel recessive syndrome presenting with skeletal malformation and malignant lymphoproliferative disease. Similar pipelines have been used successfully in other clinical labs using NGS previously.
Successive variant annotation is accomplished through a set of scripts that independently annotate the dataset with a wide collection of information, including: (1) transcript and gene annotations from Alamut (Interactive Biosoftware, Rouen, France); (2) gene annotations from variant effect predictor (VEP); (3) variant annotations from 1000 Genomes Project, ClinVar, and Exome Sequencing Project (ESP); and (4) clinical interpretation from our laboratory maintained in GeneInsight (Cambridge, MA, USA). As a clinical laboratory, we are focused on genomic regions that are known or likely to be associated with disease. Thus, we have limited the annotation of variants called on the genome to only those that lie in our target regions. The target region contains the RefSeq coding sequence for all genes mapped to hg19, pharmacogenomic variants, and association or risk variants taken from the NHGRI GWAS catalog. These regions are buffered by 50 base pairs (bp). The VCF file is limited to only this target region prior to variant annotation. Additionally, during filtration and coverage analysis, we focus in on the regions most likely to contain clinically reportable disease causing variants. Thus, we created clinical region of interest (ROI) files containing all coding regions of all exons for each gene ±15 bp or ±2 bp for filtration and coverage, respectively.
The sample data and variant annotations are uploaded to an Oracle SQL database. The variant filtration process was designed to query this database and return the variants of interest in an excel spreadsheet for careful clinical review. Variant filtration has been an evolving process, and filtration specifications vary depending on the prevalence, penetrance, and predicted mode of inheritance of the genetic disorder. Table 4 lists some of the common filtrations used to generate the final report. Often, the final variant report is generated using Boolean logic to combine more than one of these filters. The overview of the entire bioinformatics process, from alignment and variant calling to final report generation is outlined in Figure 2.
The WGS dataset of NA12878 retrieved from Coriell Cell Repositories (Camden, NJ, USA) and sequenced at Illumina was used to validate the efficacy of this pipeline. The pipeline was run twice on the exact same dataset to test the robustness of the pipeline, proving it can consistently deliver the same results for a given dataset. For this test, the pipeline was triggered from two different upstream entry points, Illumina-aligned BAM files and FASTQ files, and the final variant output files of this test were compared. Another source of validation comes from understanding the sensitivity and specificity of the variant calling pipeline. These metrics were estimated using different standard datasets that have assessed the genome level variation on NA12878. The variant calls were first compared to internally generated sequencing data. NA12878 has been the validation standard of many laboratories, and we have sequenced about 700 kb of sequence across 195 genes from our cardiomyopathy, hearing loss, respiratory, Noonan syndrome, and Marfan syndrome tests using a combination of orthogonal technologies including Sanger sequencing, array-based sequencing, and targeted next-generation sequencing. Consequently, we estimated sensitivity and specificity of our WGS workflow using only this specific region of interest. As recommended for earlier versions of GATK, we varied the QD and FS filters for variants called to find the optimal threshold. We additionally checked the concordance rate of our genome-level data against the variants in NA12878 reported by the 1000 Genomes Project dataset of high coverage WGS (>30× average coverage). The variants were assessed by their presence in each VCF file (variant concordance) as well as the concordance of the genotype called for each variant (genotype concordance).
Characterization of poorly-covered regions
We selected 15 genomes sequenced at Illumina Clinical Services Laboratory and processed through our bioinformatics pipeline in FY2014 to generate a coverage list for the coding sequence of RefSeq genes plus 2 bp of buffer to encapsulate splice sites. We customized GATK 2.2.5 to run CallableLoci to determine the callability of every coding base pair, where “callable” is defined as a base having at least 8X of coverage from quality reads and less than 10% of the total reads in that region have poor mapping quality. The average of the callable base pairs for each gene across 15 genomes was calculated for this coverage list. Our pipeline did not perform variant calling on chrMT, so mitochondrial genes were excluded from this list.
Next, we wanted to assess the clinical relevance of the poorly covered regions identified in the coverage list. To do so, we selected the ClinVar dataset of reported variants, a public archive of variants and their clinical significance as contributed by the medical genetics community. We selected “Pathogenic” and “Likely pathogenic” variants interpreted by the clinical significance element in the XML release of the ClinVar dataset (August 2015 release). ClinVar has several categorizations of variants, and we define a unique variant as one having a unique MeasureSet ID. To identify clinically relevant genes that are currently assayed in clinical sequencing laboratories, we focused on variants submitted by laboratories and working groups. To do this, we removed variant classifications from large, research-grade databases, including OMIM. MeasureSet IDs with submissions of conflicting clinical significance, such as being reported as pathogenic by one laboratory but benign in a different laboratory, were also excluded. However, an assertion of pathogenic in one laboratory but uncertain significance in another laboratory would be consistent and therefore not counted as a conflicted variant.
Cost analysis and scalability
WGS is more computationally intensive and demanding of storage than other current sequencing solutions. It incurs additional costs as clinical sequencing requires the data to be preserved for a set amount of time. The computational costs were estimated by the price of purchasing the compute nodes adjusted for the amount of CPU-hours necessary for a genome run and the lifespan of the hardware, which is guaranteed for five years at Partners HealthCare. The hands-on time was estimated by the price we charge per hour for bioinformatics analysis multiplied by the time necessary to successfully trigger the pipeline, deliver the variants to the geneticists, and archive the data. We also calculated the cost of storage based on current internal prices offered. The cluster resources used to run the genome sequencing pipeline is shared across many bioinformatics processes. At present, we have 10 compute nodes, each with 128 GB RAM and 16-core Intel Xeon 2.60 GHz processors running CentOS 6.5 and IBM LSF job scheduler.
We have described the implementation of clinical WGS in our laboratory at Partners HealthCare Personalized Medicine. Our implementation includes specialized scripts to perform: (1) alignment and variant calling; (2) variant annotation; (3) importing data to an SQL database; and (4) variant filtration. Each of these components has been clinically validated and can be updated independently of each other. WGS has been shown to reliably identify genetic variants, but the main challenge remains in clinical interpretation of those variants. This is particularly true for the vast majority of non-protein-coding variants, which are largely omitted in whole exome sequencing. With a larger public knowledgebase on non-coding regions, such as ENCODE data and GWAS signals, we can begin to create filtrations that include larger portions of the non-coding regions as they start to have utility in the clinical setting, ultimately enabling the full utility of complete genome sequencing.
Supplementary materials can be accessed at: http://www.mdpi.com/2075-4426/6/1/12/s1. (.zip file)
We would like to thank Sivakumar Gowrisankar, Ignaty Leshchiner, Netsanet Gebremedhin, Eugene Clark, and Heidi Rehm for their contributions setting up the initial pipeline. We also thank all of our colleagues past and present at the LMM and as part of the MedSeq project for their contributions to our genome sequencing workflow. This work was supported in part by National Institutes of Health grant HG006500.
All authors were actively involved in implementing, maintaining, and enhancing the bioinformatics workflow described. C.G., J.E., P.R., and E.A.T. performed the cost analysis. C.-F.L. and M.S.L. determined the poorly covered clinically relevant gene regions. R.S. and H.S. contributed to summarizing different portions of the pipeline, including examples and strategies for variant filtration and average genome coverage analysis, respectively. E.A.T. and M.S.L. drafted the manuscript. All authors reviewed and provided edits to the final version.
Conflicts of interest
All authors work at the nonprofit Partners HealthCare Personalized Medicine, which includes the Laboratory for Molecular Medicine that offers fee-for-service genetic testing.
- Collins, F.S.; Varmus, H. (2015). "A new initiative on precision medicine". New England Journal of Medicine 372 (9): 793–5. doi:10.1056/NEJMp1500523. PMID 25635347.
- Alfares, A.A.; Kelly, M.A.; McDermott, G. et al. (2015). "Results of clinical genetic testing of 2,912 probands with hypertrophic cardiomyopathy: Expanded panels offer limited additional sensitivity". Genetics in Medicine 17 (11): 880–8. doi:10.1038/gim.2014.205. PMID 25611685.
- Harismendy, O.; Ng, P.C.; Strausberg, R.L. et al. (2009). "Evaluation of next generation sequencing platforms for population targeted sequencing studies". Genome Biology 10 (3): R32. doi:10.1186/gb-2009-10-3-r32. PMC PMC2691003. PMID 19327155. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2691003.
- Biesecker, L.G.; Mullikin, J.C.; Facio, F.M. et al. (2009). "The ClinSeq Project: Piloting large-scale genome sequencing for research in genomic medicine". Genome Research 19 (9): 1665-74. doi:10.1101/gr.092841.109. PMC PMC2752125. PMID 19602640. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2752125.
- Dewey, F.E.; Grove, M.E.; Pan, C. et al. (2014). "Clinical interpretation and implications of whole-genome sequencing". JAMA 311 (10): 1035-45. doi:10.1001/jama.2014.1717. PMC PMC4119063. PMID 24618965. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4119063.
- Jiang, Y.H.; Yuen, R.K.; Jin, X. et al. (2013). "Detection of clinically relevant genetic variants in autism spectrum disorder by whole-genome sequencing". American Journal of Human Genetics 93 (2): 249-63. doi:10.1016/j.ajhg.2013.06.012. PMC PMC3738824. PMID 23849776. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3738824.
- Lupski, J.R.; Reid, J.G.; Gonzaga-Jauregui, C. et al. (2010). "Whole-genome sequencing in a patient with Charcot-Marie-Tooth neuropathy". New England Journal of Medicine 362 (13): 1181-91. doi:10.1056/NEJMoa0908094. PMC PMC4036802. PMID 20220177. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4036802.
- MacArthur, D.G.; Balasubramanian, S.; Frankish, A. et al. (2012). "A systematic survey of loss-of-function variants in human protein-coding genes". Science 335 (6070): 823-8. doi:10.1126/science.1215040. PMC PMC3299548. PMID 22344438. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3299548.
- Zook, J.M., Chapman, B.; Wang, J. et al. (2014). "Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls". Nature Biotechnology 32 (3): 246–51. doi:10.1038/nbt.2835. PMID 24531798.
- Li, H.; Broad Institute. "Picard". GitHub. http://broadinstitute.github.io/picard/. Retrieved 22 October 2015.
- Li, H.; Durbin, R. (2009). "Fast and accurate short read alignment with Burrows-Wheeler transform". Bioinformatics 25 (14): 1754-60. doi:10.1093/bioinformatics/btp324. PMC PMC2705234. PMID 19451168. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234.
- Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). "The Sequence Alignment/Map format and SAMtools". Bioinformatics 25 (16): 2078-9. doi:10.1093/bioinformatics/btp352. PMC PMC2723002. PMID 19505943. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002.
- Van der Auwera, G.A.; Carneiro, M.O.; Hartl, C. et al. (2013). "From FastQ data to high confidence variant calls: The Genome Analysis Toolkit best practices pipeline". Current Protocols in Bioinformatics 43: 11.10.1-11.10.33. doi:10.1002/0471250953.bi1110s43. PMC PMC4243306. PMID 25431634. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4243306.
- McLaughlin, H.M.; Ceyhan-Birsoy, O.; Christensen, K.D. et al. (2014). "A systematic approach to the reporting of medically relevant findings from whole genome sequencing". BMC Medical Genetics 15: 134. doi:10.1186/s12881-014-0134-1. PMC PMC4342199. PMID 25714468. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4342199.
- Chopra, S.S.; Leshchiner, I.; Duzkale, H. et al. (2015). "Inherited CHST11/MIR3922 deletion is associated with a novel recessive syndrome presenting with skeletal malformation and malignant lymphoproliferative disease". Molecular Genetics & Genomic Medicine 3 (5): 413–23. doi:10.1002/mgg3.152. PMC PMC4585449. PMID 26436107. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4585449.
- Choudhary, A.; Mambo, E.; Sanford, T. et al. (2014). "Evaluation of an integrated clinical workflow for targeted next-generation sequencing of low-quality tumor DNA using a 51-gene enrichment panel". BMC Medical Genomics 7: 62. doi:10.1186/s12920-014-0062-0. PMC PMC4241214. PMID 25395014. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4241214.
- Miller, N.A.; Farrow, E.G.; Gibson, M. et al. (2015). "A 26-hour system of highly sensitive whole genome sequencing for emergency management of genetic diseases". Genome Medicine 7: 100. doi:10.1186/s13073-015-0221-8. PMC PMC4588251. PMID 26419432. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4588251.
- Reid, J.G.; Carroll, A.; Veeraraghavan, N. et al. (2014). "Launching genomics into the cloud: deployment of Mercury, a next generation sequence analysis pipeline". BMC Bioinformatics 15: 30. doi:10.1186/1471-2105-15-30. PMC PMC3922167. PMID 24475911. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3922167.
- 1000 Genomes Project Consortium; Abecasis, G.R.; Auton, A. et al. (2012). "An integrated map of genetic variation from 1,092 human genomes". BMC Bioinformatics 491 (7422): 56–65. doi:10.1038/nature11632. PMC PMC3498066. PMID 23128226. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3498066.
- Aronson, S.J.; Clark, E.H.; Babb, L.J. et al. (2011). "The GeneInsight Suite: a platform to support laboratory and provider use of DNA-based genetic testing". Human Mutation 32 (5): 532–6. doi:10.1002/humu.21470. PMC PMC3082613. PMID 21432942. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3082613.
- McLaren, W.; Pritchard, B.; Rios, D. et al. (2010). "Deriving the consequences of genomic variants with the Ensembl API and SNP Effect Predictor". Bioinformatics 26 (16): 2069-70. doi:10.1093/bioinformatics/btq330. PMC PMC2916720. PMID 20562413. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC2916720.
- NHLBI Exome Sequencing Project. "Exome Variant Server". University of Washington. http://evs.gs.washington.edu/EVS/. Retrieved 22 October 2015.
- Welter, D.; MacArthur, J.; Morales, J. et al. (2014). "The NHGRI GWAS Catalog, a curated resource of SNP-trait associations". Nucleic Acids Research 42 (D1): D1001-D1006. doi:10.1093/nar/gkt1229. PMC PMC3965119. PMID 24316577. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3965119.
- Pugh, T.J.; Kelly, M.A.; Gowrisankar, S. et al. (2014). "The landscape of genetic variation in dilated cardiomyopathy as surveyed by clinical DNA sequencing". Genetics in Medicine 16 (8): 601-8. doi:10.1038/gim.2013.204. PMID 24503780.
- Zimmerman, R.S.; Cox, S.; Lakdawala, N.K. et al. (2010). "A novel custom resequencing array for dilated cardiomyopathy". Genetics in Medicine 12 (5): 268-78. doi:10.1097/GIM.0b013e3181d6f7c0. PMC PMC3018746. PMID 20474083. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3018746.
- Landrum, M.J.; Lee, J.M.; Riley, G.R. et al. (2014). "ClinVar: Public archive of relationships among sequence variation and human phenotype". Nucleic Acids Research 42 (D1): D980-5. doi:10.1093/nar/gkt1113. PMC PMC3965032. PMID 24234437. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3965032.
This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. In one case, a project URL had changed from Sourceforge to GitHub, and the updated URL was used.