Journal:4273π: Bioinformatics education on low cost ARM hardware

From LIMSWiki
Revision as of 20:50, 28 December 2015 by Shawndouglas (talk | contribs) (→‎Notes: Added cat.)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search
Full article title 4273π: Bioinformatics education on low cost ARM hardware
Journal BMC Bioinformatics
Author(s) Barker, Daniel; Ferrier, David E.K.; Holland, Peter W.H.; Mitchell, John B.O.; Plaisier, Heleen; Ritchie, Michael G.; Smart, Steven D.
Author affiliation(s) University of St. Andrews, University of Oxford
Primary contact Email: db60@st-andrews.ac.uk
Year published 2013
Volume and issue 14
Page(s) 243
DOI 10.1186/1471-2105-14-243
ISSN 1471-2105
Distribution license Creative Commons Attribution 2.0 Generic
Website http://www.biomedcentral.com/1471-2105/14/243
Download http://www.biomedcentral.com/content/pdf/1471-2105-14-243.pdf (PDF)

Abstract

Background: Teaching bioinformatics at universities is complicated by typical computer classroom settings. As well as running software locally and online, students should gain experience of systems administration. For a future career in biology or bioinformatics, the installation of software is a useful skill. We propose that this may be taught by running the course on GNU/Linux running on inexpensive Raspberry Pi computer hardware, for which students may be granted full administrator access.

Results: We release 4273π, an operating system image for Raspberry Pi based on Raspbian Linux. This includes minor customisations for classroom use and includes our Open Access bioinformatics course, 4273π Bioinformatics for Biologists. This is based on the final-year undergraduate module BL4273, run on Raspberry Pi computers at the University of St Andrews, Semester 1, academic year 2012–2013.

Conclusions: 4273π is a means to teach bioinformatics, including systems administration tasks, to undergraduates at low cost.

Keywords: Bioinformatics education; Teaching material; Raspberry Pi; Linux

Background

Bioinformatics is increasingly included in the undergraduate curriculum for biology students. Teaching bioinformatics is made difficult, however, by the constraints of a typical university computer classroom. Some areas of basic bioinformatics may be taught using such classrooms, where all that is required is an Internet connection and Web browser (e.g. BLAST[1] searches at the NCBI[2]). More in-depth teaching requires the re-creation of a bioinformatics research environment, consisting of a Linux or UNIX operating system, standard GNU utilities[3], specialist bioinformatics software, and sequence databases.

Undergraduate modules ought, ideally, to prepare students for research in an academic research group. Students who do pursue a research career will often find that institutional computer support is targeted to generic computer use (e.g. Microsoft software) rather than installing and maintaining systems suitable for bioinformatics. Particularly outside of bioinformatics research (but also occasionally within it), the principal investigator of the research group may never have used Linux, may have a limited idea of the procedures, and may expect group members to ‘pick things up’ and deal with problems themselves. This requires researchers to have a high level of proficiency with Linux, including the ability to install both standard Linux packages and software for which no standard package may be available. A single taught module cannot prepare a student for all eventualities, but ought to leave the student with the basic skills and confidence to be able to discover solutions, and implement them, as required. Hence, a certain amount of system administration should appear in an undergraduate bioinformatics module for biologists.

Traditionally, the environment required for an undergraduate bioinformatics module has been created in one of four ways. Firstly, one may set up a central GNU/Linux server on the campus and allow students to connect by Secure Shell, ssh (‘the server approach’). The server will typically run either a standard Linux distribution or a specialist bioinformatics distribution such as NEBC Bio-Linux.[4] The server approach allows the instructor to have full control over the server, and allows students to connect from existing computing classrooms with little or no adjustment to the classroom software. For students to connect to the server via the intranet, classroom computers only require an ssh client, the X Window System (X11), and a means of file transfer such as secure copy (scp). Students may also connect to the server from home (typically requiring them to install virtual private network software in addition to ssh, X11 and an scp client) or elsewhere on campus. Secondly, one may provide students with a virtual machine, consisting of an environment similar to that which they might experience on the Linux server but running on a classroom computer, either with a standard Linux distribution or a specialist bioinformatics distribution such as DNA Linux Virtual Desktop Edition[5] (‘the VM approach’). This has the advantage that students may be given administrator access to their virtual machine. Thirdly, one may provide students with a Linux system on removable media (‘the USB stick approach’, for example[6]; where files and settings do not have to be saved, a DVD may be used instead[7]). So long as students have the media to hand, this allows them to boot into ‘their own’ Linux. As with the VM approach, students may be given administrator access. The additional advantage is that the media may be portable between computer classrooms and home computers, without requiring students to move virtual machine image files. Fourthly, students may be loaned or required to buy laptops of a specific kind, with a suitable operating system, data and software installed (‘the laptop approach’). This avoids hardware incompatibilities that the USB stick approach may, in practice, experience.[6]

Because administrator access cannot be allowed, the server approach fails to give students experience of the standard mechanism of software installation. It also involves competition for resources such as CPU time, especially if the class is large or the server is also shared with research colleagues. The VM approach solves both these problems but is less portable. Although, in theory, students may transfer a VM from one computer to another (assuming the destination has the necessary virtualisation software installed), the task is non-trivial, and more time consuming than a simple transfer of data or documents. The USB stick approach reduces the portability problem, since it is trivial to move a USB stick from one computer to another. However, smooth operation on all hardware is not guaranteed and requires ongoing efforts from the developers of the Linux distribution as new hardware is released. The laptop approach avoids all these problems by providing a portable computer holding everything required for the course. However, it is expensive.

As a fifth approach, we propose loaning a Raspberry Pi computer[8] and associated peripherals to students for the duration of the course (‘the Raspberry Pi approach’). This includes a customised version of Linux, appropriate software and data. This allows students full administrator access to a suitable operating system, without the difficulties of the VM or USB stick approaches. Should the student accidentally damage critical files, the system can be re-written from a master image.

The Raspberry Pi Model B — with 256 MB (now 512 MB) RAM, an ARM11 CPU running at 700 MHz before over clocking and a Video core IV GPU — was released for public sale in 2012[9] and costs £28.07a or £31.20.[10][11] Though additional items are required to turn it into a functioning, general-purpose computer (case, charger, SD card, mouse, keyboard, monitor and cable; and an entirely separate computer for initialising the SD card), it is still relatively low-cost (Additional file 1: Table S1). The existence of the Raspberry Pi is partly a celebration of the early days of popular computing in the 1980s, and an attempt to recreate that excitement among young people today.[12] It is also a symptom of the rapidly decreasing costs and increasing performance of computer hardware. The Raspberry Pi uses an ARM CPU.[13] Because of their high performance-per-watt, ARM CPUs are frequently found in small electronic appliances such as mobile phones and tablets. With CPU innovation increasingly driven by such applications, as opposed to more traditional areas such as desktop, laptop and server computers, the prevalence and utility of ARM-based computer hardware is likely to increase. Indeed, ARM-based servers are starting to appear in data centres, due to their modest requirements for power.[14]

Additional file 1. Table S1
Example prices of Raspberry Pi peripherals we found to work well in practice. These are presented without any endorsement. A case for the Raspberry Pi (various models and suppliers; ~£5-£10), the Raspberry Pi itself (see main text) and a monitor are not shown. Standard consumer prices, including UK tax but excluding any delivery charge, were obtained from the Insight UK Web site or via the Amazon UK Web site on 7 April 2013.

Format: PDF; Size: 110KB Download file

Though far slower than current desktop and laptop computers, the Raspberry Pi is notably faster than the Cray 1 supercomputer[15], a marvel of computer speed in its day. The valid question arises as to how much computer power is actually required to teach undergraduates bioinformatics? We propose that the answer is, by current standards, ‘not much’. The Raspberry Pi is more than adequate for the task. The Raspberry Pi approach includes all the benefits of the laptop approach, above, but at lower cost. In addition, the Raspberry Pi is a new and exciting computer system, which in itself can add interest to the course.

A variety of operating systems is available for the Raspberry Pi.[16] These include Raspbian[17], which is based on Debian GNU/Linux.[18] Over 35,000 Debian software packages are available pre-compiled for Raspbian, including Web browsers, text editors, word processors, and a wide range of bioinformatics packages.[19] Other software will usually compile and run without problems. Some features of recent CPUs (e.g. 64-bit addressing or vector operations) are absent, but we have not found these to be at all necessary for our proposed use of the Raspberry Pi. The most serious limitation, affecting structure visualisation software in particular, is limited graphics performance. Even so, some structural visualisation software does work on the Raspberry Pi.[20] New system software, improving graphics performance by making better use of the Raspberry Pi’s GPU, is under development.[21]

We provide 4273π, a customised version of Linux for Raspberry Pi computer hardware. 4273π includes an Open Access bioinformatics course, 4273π Bioinformatics for Biologists.

Implementation

4273π Bioinformatics for Biologists is based on the module BL4273 Bioinformatics for Biologists at the University of St Andrews[22], an optional module of 15 SCOTCAT credits, equivalent to 7.5 ECTS credits or ~4 US credits. BL4273 is intended for final-year undergraduate students on BSc(Hons) Biology, BSc(Hons) Biochemistry and other degree courses taught by the School of Biology. BL4273 was taught on Raspberry Pi hardware in Semester 1 of academic year 2012–2013. During this time, students were loaned a Raspberry Pi, SD card and USB stick for backup. Teaching was carried out in a small computer room in which students connected to monitors more generally used with Windows desktop PCs. Students were allowed to take all equipment home on loan (including keyboard, mouse and cables), with the exception of the monitors; several copies of the main textbook[23] were also available. Course material was released week-by-week using an rsync server (running on a Raspberry Pi) on the university intranet. Following the conclusion of the module, material was edited to remove closed-access material (e.g. images in lectures), typographical errors were corrected, and the SD card image was re-created using a recent version of Raspbian.[17]

Preparation of a release of 4273π begins with the latest Raspbian SD card image.[24] This is then customised to produce a ‘master’ SD card for the release, partly by a series of scripts which alter the configuration and use Raspbian’s port of the APT mechanism of Debian[18] to install specialist bioinformatics packages[19] and more general packages, and partly by a series of commands entered manually (e.g. to install BLAST databases and 4273π Bioinformatics for Biologists in the ~/4273pi/ directory). The master SD card image is stored on a separate computer and uploaded to the 4273π Web site. A ‘work instruction’ detailing the steps performed to convert Raspbian into 4273π, and all scripts used, are distributed with 4273π.

For the permanent record, the teaching material included in the current release – excluding Linux, software and BLAST databases – is available as Additional file 2. The latest version may be downloaded from the 4273π Web site.[25]

Additional file 2. 4273π Bioinformatics for Biologists teaching material, Version 1.01
The module handbook, lectures and practicals are included. The latest version, including Linux, software and BLAST databases, is available at the 4273π Web site[25]

Format: ZIP; Size: 7MB Download file

As the Raspbian operating system, Raspberry Pi firmware and hardware and 4273π Bioinformatics for Biologists teaching material develop, further releases of 4273π will be made available. It is anticipated that there will be a minimum of two releases per year during the next four years.

Results and discussion

4273π provides an attractive, general-purpose computing environment, within which the course 4273π Bioinformatics for Biologists is embedded (Figure 1; Table 1). Student feedback questionnaires for the module BL4273, upon which it is based, are too few to draw strong conclusions (five returned from a class of six students; Additional file 1: Table S2). However, they were sufficiently positive to continue the module without modification.

Fig1 Barker BMCBio2013 14.jpg

Figure 1: 4273π screenshot. Practical, Week 7. Predicted gene structures of the Or98b gene in the Drosophila sechellia genome are being compared. Different predictions are made de novo by SNAP[26], run in a terminal on the Raspberry Pi; GeneWise[27], run online at the European Bioinformatics Institute[28], with the D. melanogaster protein being aligned against D. sechellia genomic DNA; and using TBLASTN[1], run online at FlyBase.[29] FlyBase also displays the annotation of this gene from its database. This case study arose during research for a comparative study of the chemoreceptor superfamily in Drosophila.[30] The 4273π background desktop image includes hand-written versions of material from 4273π Bioinformatics for Biologists and from the module BL3320 Practical Statistics for Biologists, which most students at the University of St Andrews would have taken earlier. It also includes a copy of one of the first published phylogenetic trees, from Vestiges of the Natural History of Creation, written in St Andrews.[31]

Table 1. Timetable for 4273π Bioinformatics for Biologists
Week Lecture Practical class
1 Genomes, sequences and bioinformatics data Linux and Perl
2 - Linux, Perl and protein BLAST
3 - Linux, Perl and delimiting gene/protein families
4 Multiple alignment and phylogeny Multiple alignment and phylogeny
5 Gene family evolution Gene family evolution
6 BLAST; DNA sequence analysis DNA sequence analysis
7 Looking at species differences Detecting positive selection
8 Function and evolution of enzymes Function and evolution of enzymes
9 Seminar: student presentations -
Lectures last slightly less than an hour. Practical classes all use the Raspberry Pi, and last two hours. In addition, in their own time, students engage in self-study and complete a coursework ‘practical project’.

Conclusions

(Administrator) power to the people

Rather than present systems administration as a complex task to be delegated to a technical support unit — which in students’ future careers may not be available — 4273π Bioinformatics for Biologists introduces software installation on Linux through standard package management (APT), through compilation (GNU ‘make’ for SNAP[26] and PAML[32]) and through command-line launch of a JAR (Modelgenerator[33] and Mesquite[34]). Other administrative tasks covered include upgrading Linux and installing a MySQL server. Although these matters of system administration are incidental to an intellectual understanding of bioinformatics, they do not take much time to teach, and we believe they leave students well-prepared for a bioinformatics research career – in many cases, better prepared than any other members of the research group within which they are working.

The virtue of patience

No serious compromises in content were required to teach bioinformatics on the Raspberry Pi. A BLAST search of the GenPept database (‘nr’) is, in practice, too slow. However a BLAST search of the SwissProt database takes only a few minutes. Delimitation of protein families across two prokaryotic genomes using BLAST and OrthoMCL is entirely feasible, and is central to the coursework component of 4273π Bioinformatics for Biologists. Java (Modelgenerator, Mesquite) programs run slowly, but not unbearably so. Speed will likely improve once Oracle Java becomes available for the platform. Floating-point-intensive tasks (PhyML[35] and PAML) are also slow, but feasible. Waiting an hour for an analysis is, in fact, realistic training for bioinformatics research. Research will tend to use a far faster computer or cluster, but with far larger input. Teaching bioinformatics on rather slow hardware was not universally popular among students (Additional file 1: Table S2). However, it is a valuable lesson in the transferrable virtue of patience, and the value of checking the configuration of analyses before launching them.

Low-cost teaching and learning

When the undergraduate module BL4273 at the University of St Andrews ran previously, in Semester 1 of academic year 2008–2009, it used the server approach. Students used desktop computers running Windows in a computer classroom to connect to a 64-bit IBM System x3755 8877 server with 4 AMD CPU cores and 16 GB RAM, running Debian Linux. When purchased in 2008, this server cost £19,537 after educational discount, including 20 TB external storage (14 TB usable as RAID5), uninterruptable power supply (UPS), tape library for backup, and 3 years’ hardware support. It was housed in a secure air-conditioned server room, where it required the University’s IT Services department for installation and maintenance (and for setting up the tape library in a different building) and the University’s Estates department to set up a new 15A electrical connection for the UPS. Although this did not happen, there was always the fear of down-time at some crucial stage in teaching, which is more problematic for a single server being used by all students than for (say) one desktop computer in a classroom. As well as teaching, the server was used for research purposes, leading to worries about conflicting resource use between students and researchers.

To use 4273π, the investment in hardware per set of equipment, including the Raspberry Pi but excluding the monitor, before any quantity or educational discount is ~ £147-£159, depending on what exactly is bought and from where (Additional file 1: Table S1; Background). This is far cheaper than the server used in 2008–2009, but is not extremely cheap. However, these are mostly one-off expenses, since the equipment may be re-used; some components are likely already present in an educational establishment; the maximum cost of any one part is no more than £31.20, allowing cheap repairs compared to ‘the laptop approach’; and the cost of some parts of the equipment (e.g. SD card) continues to fall noticeably. Among its other advantages, the Raspberry Pi approach is a low-cost method for bioinformatics teaching and learning.

Open learning

By including an explicit Open Access licence, and removing or replacing material incompatible with this from 4273π Bioinformatics for Biologists, we have been able to share it with anyone interested, the world over, in such a way that they can — with minimal care — re-use and adapt it without accusation of plagiarism or copyright violation. This approach is broadly in common with the pioneering EcoEd Digital Library[36] and related portals[37], but is in contrast to most of the teaching material that can be found by an online search, for which the licence is unclear. We expect our approach will lead to mutual benefits, for example the contribution of corrections or teaching material by others. As Open Access publication is becoming more standard for research, we predict that Open Access will become more standard for teaching material.

Availability and requirements

Project name: 4273π

Project home page: http://eggg.st-andrews.ac.uk/4273pi

Operating Systems: Linux

Other requirements: Raspberry Pi computer hardware

Licence: 4273π Bioinformatics for Biologists has a Creative Commons Attribution licence (http://creativecommons.org/licenses/by/2.0)

Any restrictions to use by non-academics: no

Endnotes

aPrices in British Pounds (GBP), including UK tax but excluding delivery charges, obtained on 26 June 2013. £1 converts to approximately $1.54 US Dollars, €1.18 Euro, or R93.04 Indian Rupees.[38]

Abbreviations

AMD: Advanced Micro Devices

APT: A package tool

BSc(Hons): Bachelor of Science with Honours

CPU: Central processing unit

ECTS: European credit transfer and accumulation system

GB: Gigabyte

GNU: GNU’s not UNIX!

GPU: Graphics processing unit

IBM: International Business Machines

IT: Information technology

JAR: Java archive

MHz: Megahertz

NCBI: National Center for Biotechnology Information

NEBC: NERC Environmental Bioinformatics Centre

NERC: Natural Environment Research Council

PC: Personal computer

RAID5: Redundant array of independent disks, level 5

RAM: Random-access memory

SCOTCAT: Scottish Credit Accumulation and Transfer

scp: Secure copy

SD: Secure digital

ssh: secure shell

TB: Terabyte

USB: Universal serial bus

UPS: Uninterruptable power supply

US: United States

VM: Virtual machine

X11: The X Window System

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

DB conceived the study, and wrote the manuscript with contributions from other authors. DB, DEKF, PWHH, JBOM and MGR designed and wrote teaching material, which was edited by DB and HP. DB configured Raspbian and prepared the 4273π release, with contributions from SDS and HP. All authors read and approved the final manuscript.

Acknowledgements

Losia Lagisz contributed to the lecture material on Looking at Species Differences. Ian Grieve and Lianne Baker assisted with the task of obtaining sufficient Raspberry Pi computers, during summer and autumn 2012 when they were scarce. All students of BL4273 Bioinformatics for Biologists at the University of St Andrews are thanked for testing teaching material, particularly those in academic year 2012–2013, who used the Raspberry Pi. Their contribution has been invaluable. We thank the developers of Raspbian Linux, upon which 4273π is based, for their rapid and professional work. Clare Peddie’s enthusiasm (as Director of Teaching for the School of Biology, University of St Andrews) allowed the necessary equipment purchases to go ahead. JBOM is grateful to the Scottish Universities Life Sciences Alliance (SULSA) for financial support and thanks the BBSRC for funding his research through grant BB/I00596X/1. The University of St Andrews provided funding for the Open Access charge.

References

  1. 1.0 1.1 Altschul, S.F.; Madden, T.L.; Schäffer, A.A.; Zhang, J.; Zhang, Z.; Miller, W.; Lipman, D.J. (1997). "Gapped BLAST and PSI-BLAST: a new generation of protein database search programs". Nucleic Acids Research 25 (17): 3389-3402. doi:10.1093/nar/25.17.3389. PMC PMC146917. PMID 9254694. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC146917. 
  2. "BLAST: Basic Local Alignment Search Tool". National Center for Biotechnology Information. http://blast.ncbi.nlm.nih.gov/Blast.cgi. 
  3. "The GNU Operating Systems and the Free Software Movement". Free Software Movement, Inc. http://www.gnu.org/. 
  4. Field, D.; Tiwari, B. Booth, T.; Houten, S.; Swan, D.; Bertrand, N.; Thurston, M. (2006). "Open software for biologists: from famine to feast". Nature Biotechnology 24 (7): 801–803. doi:10.1038/nbt0706-801. PMID 16841067. 
  5. Bassi, S.; Gonzalez, V.C. (2007). "DNALinux Virtual Desktop Edition". Nature Precedings. doi:10.1038/npre.2007.670.1. 
  6. 6.0 6.1 "Bio-Linux 7". Environmental Omics Network. http://nebc.nerc.ac.uk/tools/bio-linux/live-usbkey. 
  7. Yu, G.; Wang, L.G.; Meng, X.H.; He, Q.Y. (2012). "LXtoo: an integrated live Linux distribution for the bioinformatics community". BMC Research Notes 5: 360. doi:10.1186/1756-0500-5-360. PMC PMC3461469. PMID 22813356. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3461469. 
  8. "Raspberry Pi". Raspberry Pi Foundation. https://www.raspberrypi.org/. 
  9. "The Raspberry Pi computer goes on general sale". BBC. 29 February 2012. http://www.bbc.com/news/technology-17190918. 
  10. "Raspberry Pi". Element14 Community. Premier Farnell plc. http://www.element14.com/community/community/raspberry-pi. 
  11. "Raspberry Pi". RS Components Ltd. http://uk.rs-online.com/web/generalDisplay.html?id=raspberrypi. 
  12. Roberts, Jonathan (1 May 2012). "Is the Raspberry Pi the future of computing?". TechRadar Pro. Future Publishing Limited. http://www.techradar.com/news/computing/pc/is-the-raspberry-pi-the-future-of-computing-1078276. 
  13. "ARM: The Architecture for the Digital World". ARM Ltd. http://www.arm.com/. 
  14. Latif, Lawrence (5 April 2013). "ARM sees its 32-bit chips being deployed in future servers: Not everything needs 64-bit addressing". The Inquirer. Incisive Business Media Limited. http://www.theinquirer.net/inquirer/news/2259386/arm-sees-its-32bit-chips-being-deployed-in-future-servers. 
  15. Nic (19 November 2012). "A Cray for $35". 2000 Nickels. http://2000nickels.com/blog/2012/11/19/a-cray-for-35-dollars/. 
  16. "Raspberry Pi - Operating system distributions". Raspberry Pi Foundation. https://www.raspberrypi.org/forums/viewforum.php?f=18. 
  17. 17.0 17.1 "Raspbian". Mike Thompson and the Raspbian community. http://www.raspbian.org/. 
  18. 18.0 18.1 "Debian". Software in the Public Interest, Inc. http://www.debian.org/. 
  19. 19.0 19.1 Möller, S.; Krabbenhöft, H.N.; Tille, A.; Paleino, D.; Williams, A.; Wolstencroft, K.; Goble, C.; Holland, R.; Belhachemi, D.; Plessy, C. (2010). "Community-driven computational biology with Debian Linux". BMC Bioinformatics 11 (Suppl 12): S5. doi:10.1186/1471-2105-11-S12-S5. PMC PMC3040531. PMID 21210984. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3040531. 
  20. O'Boyle, Noel (19 January 2013). "Chemistrify your Raspberry Pi Part III". Noel O'Blog. http://baoilleach.blogspot.co.uk/2013/01/chemistrify-your-raspberry-pi-part-iii_19.html. 
  21. Upton, Eben (24 May 2013). "Wayland project". Raspberry Pi Blog. http://www.raspberrypi.org/archives/4053. 
  22. "Undergraduate Course Catalogue 2012-2013". University of St. Andrews. 11 December 2012. http://www.st-andrews.ac.uk/coursecatalogue/ug/2012-2013/. 
  23. Bradnam, Keith; Korf, Ian (2012). UNIX and Perl to the Rescue! - A Field Guide for the Life Sciences (and Other Data-rich Pursuits). Cambridge, England: Cambridge University Press. pp. 428. ISBN 9780521169820. http://www.cambridge.org/us/academic/subjects/life-sciences/genomics-bioinformatics-and-systems-biology/unix-and-perl-rescue-field-guide-life-sciences-and-other-data-rich-pursuits. 
  24. "Downloads". Raspberry Pi Foundation. https://www.raspberrypi.org/downloads/. 
  25. 25.0 25.1 "4273π". University of St. Andrews. http://4273pi.org/. 
  26. 26.0 26.1 Korf, I. (2004). "Gene finding in novel genomes". BMC Bioinformatics 14 (5): 59. doi:10.1186/1471-2105-5-59. PMC PMC421630. PMID 15144565. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC421630. 
  27. Birney, E.; Clamp, M.; Durbin, R. (2004). "GeneWise and Genomewise". Genome Research 14 (5): 988-995. doi:10.1101/gr.1865504. PMC PMC479130. PMID 15123596. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC479130. 
  28. "GeneWise". European Bioinformatics Institute. http://www.ebi.ac.uk/Tools/psa/genewise/. 
  29. Marygold, S.J.; Leyland, P.C.; Seal, R.L.; Goodman, J.L.; Thurmond, J.; Strelets, V.B.; Wilson, R.J.; FlyBase Consortium (2013). "FlyBase: improvements to the bibliography". Nucleic Acids Research 41 (Database issue): D751-7. doi:10.1093/nar/gks1024. PMC PMC3531214. PMID 23125371. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3531214. 
  30. Gardiner, A.; Barker, D.; Butlin, R.K.; Jordan, W.C.; Ritchie, M.G. (2008). "Drosophila chemoreceptor gene evolution: selection, specialization and genome size". Molecular Ecology 17 (7): 1648-57. doi:10.1111/j.1365-294X.2008.03713.x. PMID 18371013. 
  31. Chambers, R. (1844). Vestiges of the natural history of creation. London, England: John Churchill. 
  32. Yang, Z. (2007). "PAML 4: Phylogenetic analysis by maximum likelihood". Molecular Biology and Evolution 24 (8): 1586-91. doi:10.1093/molbev/msm088. PMID 17483113. 
  33. Keane, T.M.; Creevey, C.J.; Pentony, M.P.; Naughton, T.J.; Mclnerney, J.O. (2006). "Assessment of methods for amino acid matrix selection and their use on empirical data shows that ad hoc assumptions for choice of matrix are not justified". BMC Evolutionary Biology 6: 29. doi:10.1186/1471-2148-6-29. PMC PMC1435933. PMID 16563161. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1435933. 
  34. Maddison, W.P.; Maddison, D.R. (2011). "Mesquite". Tangient, LLC. http://mesquiteproject.org/. 
  35. Guindon, S.; Dufayard, J.F.; Lefort, V.; Anisimova, M.; Hordijk, W.; Gascuel, O. (2010). "New algorithms and methods to estimate maximum-likelihood phylogenies: assessing the performance of PhyML 3.0". Systematic Biology 59 (3): 307-21. doi:10.1093/sysbio/syq010. PMID 20525638. 
  36. "EcoEd Digital Library". Ecological Society of America. http://ecoed.esa.org/. 
  37. "EvoEd Digital Library". Society for the Study of Evolution. http://evoed.evolutionsociety.org/index.php?P=DRD_People. 
  38. "XE Currency Converter". XE.com, Inc. http://www.xe.com/currencyconverter/. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.