Journal:Ten simple rules for developing usable software in computational biology
|Full article title||Ten simple rules for developing usable software in computational biology|
|Journal||PLOS Computational Biology|
|Author(s)||List, Markus; Ebert, Peter; Albrecht, Felipe|
|Author affiliation(s)||Max Planck Institute for Informatics, Saarland Informatics Campus|
|Primary contact||Email: pebert at mpi-inf dot mpg dot de|
|Volume and issue||13(1)|
|Distribution license||Creative Commons Attribution 4.0 International|
- 1 Introduction
- 2 Rule 1: Identify the missing pieces
- 3 Rule 2: Collect feedback from prospective users
- 4 Rule 3: Be ready for data growth
- 5 Rule 4: Use standard data formats for input and output
- 6 Rule 5: Expose only mandatory parameters
- 7 Rule 6: Expect users to make mistakes
- 8 Rule 7: Provide logging information
- 9 Rule 8: Get users started quickly
- 10 Rule 9: Offer tutorial material
- 11 Rule 10: Consider the future of your tool
- 12 Conclusions
- 13 Further reading
- 14 Acknowledgments
- 15 Competing interests
- 16 References
- 17 Notes
The rise of high-throughput technologies in molecular biology has led to a massive amount of publicly available data. While computational method development has been a cornerstone of biomedical research for decades, the rapid technological progress in the wet laboratory makes it difficult for software development to keep pace. Wet lab scientists rely heavily on computational methods, especially since more research is now performed in silico. However, suitable tools do not always exist, and not everyone has the skills to write complex software. Computational biologists are required to close this gap, but they often lack formal training in software engineering. To alleviate this, several related challenges have been previously addressed in the Ten Simple Rules series, including reproducibility, effectiveness, and open-source development of software.
Here, we want to shed light on issues concerning software usability. Usability is commonly defined as "a measure of interface quality that refers to the effectiveness, efficiency, and satisfaction with which users can perform tasks with a tool." Considering the subjective nature of this topic, a broad consensus may be hard to achieve. Nevertheless, good usability is imperative for achieving wide acceptance of a software tool in the community. In many cases, academic software starts out as a prototype that solves one specific task and is not geared for a larger user group. As soon as the developer realizes that the complexity of the problems solved by the software could make it widely applicable, the software will grow to meet the new demands. At least by this point, if not sooner, usability should become a priority. Unfortunately, efforts in scientific software development are constrained by limited funding, time, and rapid turnover of group members. As a result, scientific software is often poorly documented, non-intuitive, non-robust with regards to input data and parameters, and hard to install. For many use cases, there is a plethora of tools that appear very similar and make it difficult for the user to select the one that best fits their needs. Not surprisingly, a substantial fraction of these tools are probably abandonware; i.e., these are no longer actively developed or supported in spite of their potential value to the scientific community.
To our knowledge, software development as part of scientific research is usually carried out by individuals or small teams with no more than two or three members. Hence, the responsibility of designing, implementing, testing, and documenting the code rests on few shoulders. Additionally, there is pressure to produce publishable results or, at least, to contribute analysis work to ongoing projects. Consequently, academic software is typically released as a prototype. We acknowledge that such a tool cannot adhere to and should not be judged by the standards that we take for granted for production grade software. However, widespread use of a tool is typically in the interest of a researcher. To this end, we propose 10 simple rules that, in our experience, have a considerable impact on improving usability of scientific software.
Rule 1: Identify the missing pieces
Unless you are a pioneer, and few of us are, the problem you are working on is likely addressed by existing tools. As a professional, you are aware of this software but may consider it cumbersome, non-functional, or otherwise unacceptable for your demands. Make sure that your judgment is shared by a substantial fraction of the prospective users before you start developing a new tool. Usable software should offer the features needed and behave as expected by the community. Moreover, a new tool needs to provide substantial novelty over existing solutions. For this purpose, list the requirements on the software and create a comparison table to set the new tool against existing solutions. This allows you to carve out the selling points of your tool in a systematic fashion.
Rule 2: Collect feedback from prospective users
Software can be regarded as providing the interface between wet lab science and data analysis. A lack of communication between both sides will lead to misunderstandings that need to be rectified by substantially changing the code base in a late phase of the project. Avoid this pitfall by exposing potential users to a prototype. Discussions on data formats or on the design of the user interface will reveal unforeseen challenges and help to determine if a tool is sufficiently intuitive. To plan your progress, keep a record of suggested improvements and existing issues.
Rule 3: Be ready for data growth
First estimate the expected data growth in your field and then design your software accordingly. To this end, consider parallelization and make sure your tool can be integrated seamlessly in workflow management systems (e.g., GALAXY and Taverna), pipeline frameworks (e.g., Ruffus and SnakeMake), or a cluster framework (e.g., Hadoop, http://hadoop.apache.org/). Moreover, make sure that the user interface can scale to growing data volumes. For example, consider that the visualizations should still be comprehensible for larger datasets, e.g., by displaying only parts of the data or through aggregation of results.
Rule 4: Use standard data formats for input and output
As an expert in your research domain, you know the established data standards and related programming libraries for reading and writing commonly used data formats. Make sure that your tool’s output follows standard specifications to the letter, but be as lenient as possible when users provide non-standard input. Tools that follow this rule are more likely to become successful. If you are working in an emerging field with no prevalent model for data exchange, provide data in a structured text file (e.g., tab-separated tables, XML/XSD, or JSON) and aim for self-documenting output by including header lines and data type descriptions. In this case, document how users can derive suitable input data for your tool.
Rule 5: Expose only mandatory parameters
Exposing all (possible) parameters to a user can be confusing and carries the risk of nonsensical parameters settings. When possible, users will thus rely on default parameters. The same applies to benchmark studies comparing your tool against the state-of-the-art competitors. This has three important implications: (i) expose only a small set of parameters by default whose effects on results can be easily understood by any user, (ii) offer advanced parameters only in an expert section and describe them thoroughly in the documentation, and (iii) choose conservatively (and if possible, justify) the default values for parameters such that the tool can operate in a wide range of scenarios and within reasonable run time.
Rule 6: Expect users to make mistakes
You should never assume that your tool is self-explanatory, that requirements concerning the input data are obvious, or that the user will immediately grasp all details of the problem at hand. Ideally, your tool supports the user in using it appropriately, e.g., by checking that data remain inside required ranges or that identifiers are unique, and provides descriptive error messages in case of unexpected values. If performance penalties due to such checks are a real concern (which should be tested), make the checks optional and enabled by default. Finally, allow users to stop ongoing operations in case they realize they made a mistake.
Rule 7: Provide logging information
Two types of logs improve usability and also support the user in making their research more reproducible. Configuration logs keep track of basic information, such as the time stamp of the analysis, the version of your tool and of third-party libraries, as well as the parameter settings and input data. Archiving this information is particularly important in long-running research projects in order to trace irregularities in the results at any later point in time. Technical logs, on the other hand, contain progress messages that help users to pinpoint errors in the execution flow and allow clear communication of these issues to the developer. As much as possible, avoid exposing potentially sensitive user information in the logs.
Rule 8: Get users started quickly
Complex setup routines introduce dependency or configuration debt; i.e., the user has to spend substantial time installing software and learning about the execution parameters of a tool. These raise the bar for unhindered exploration of software features. Such issues can be solved by implementing a web application (if feasible with respect to resource demands), by providing a standalone executable, or by providing a system-specific software package. Alternatively, issues of a program’s dependence on third-party libraries can be avoided by encapsulating your tool in a virtual machine image or, e.g., a Docker container (https://docker.com). Finally, it is imperative to provide demo data that enable users to immediately interact with the software. A successful test run proves to the user that your software works as expected and will be essential if you want your tool to be published.
Rule 9: Offer tutorial material
Researchers can seldom afford the time to thoroughly read complex user manuals. They will thus appreciate a number of clearly written code examples, illustrations, or video screen casts to get started. Most importantly, documented use cases enable users to quickly assess if your tool is suited for the problem at hand and allow fast learning by doing. Keep in mind that these materials have to be updated together with your tool.
Rule 10: Consider the future of your tool
For long-term availability of your software, use suitable repositories such as GitHub (https://github.com) or Bitbucket (https://bitbucket.com) throughout the development process. Explicitly state under which software license you release your code for third parties (see https://opensource.org/licenses). Without such a license, using your software might be prohibitive for many organizations or companies. More importantly, keeping your code in a public repository will also allow you to engage with the users through issue tracking (e.g., bugs, suggestions). After releasing your tool, expect support requests and take them seriously. See them as an opportunity to continuously improve the usability of your tool.
In the above ten simple rules, we highlight that software should not only be scientifically sound but also be perceived as usable for widespread and effective application. To these ends, developers should also be the first to apply their tool, to reveal usability issues as early as possible. However, effort is required from both users and developers to further improve a tool. Even engaging with only a few users (Rule 2) is likely to have a large impact on usability, since, as Jakob Nielsen put it, "Zero users give zero insights."
Usability is an important topic in software design, and we would like to provide a few starting points for further reading:
- Baxter, S.M.; Day, S.W.; Fetrow, J.S. et al. (2006). "Scientific software development is not an oxymoron". PLOS Computational Biology 2 (9): e87. doi:10.1371/journal.pcbi.0020087. PMC PMC1560404. PMID 16965174. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC1560404.
- Borchardt, J.-C. (30 June 2011). "Usability in Free Software". jancborchardt.net. http://jancborchardt.net/usability-in-free-software.
- Macaulay, C.; Sloan, D.; Jiang, X. et al. (2009). "Usability and user-centered design in scientific software development". IEEE Software 26 (1): 96–102. doi:10.1109/MS.2009.27.
- Nichols, D.; Twidale, M. (2003). "The usability of open source software". First Monday 8 (1). doi:10.5210/fm.v8i1.1018.
- Seffah, A.; Metzker, E. (2004). "The obstacles and myths of usability and software engineering". Communications of the ACM 47 (12): 71–76. doi:10.1145/1035134.1035136.
- Sloan, D.; Macaulay, C.; Forbes, P. et al. (2009). "User research in a scientific software development project". Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology 2009: 423–429. http://dl.acm.org/citation.cfm?id=1671066.
We would like to thank Thomas Lengauer, Nico Pfeifer, and Fabian Müller for their critical reading of the manuscript and insightful comments.
FA and PE acknowledge the support of the German Federal Ministry of Education and Research grant no. 01KU1216A (DEEP project). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The authors have declared that no competing interests exist.
- Sandve, G.K.; Nekrutenko, A.; Taylor, J.; Hovig, E.. "Ten simple rules for reproducible computational research". PLOS Computational Biology 9 (10): e1003285. doi:10.1371/journal.pcbi.1003285. PMC PMC3812051. PMID 24204232. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3812051.
- Osborne, J.M.; Bernabeu, M.O.; Bruna, M. et al.. "Ten simple rules for effective computational research". PLOS Computational Biology 10 (3): e1003506. doi:10.1371/journal.pcbi.1003506. PMC PMC3967918. PMID 24675742. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3967918.
- Prlić, A.; Procter, J.B.. "Ten simple rules for the open development of scientific software". PLOS Computational Biology 8 (12): e1002802. doi:10.1371/journal.pcbi.1002802. PMC PMC3516539. PMID 23236269. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3516539.
- Perez-Riverol, Y.; Gatto, L.; Wang, R. et al.. "Ten simple rules for taking advantage of Git and GitHub". PLOS Computational Biology 12 (7): e1004947. doi:10.1371/journal.pcbi.1004947. PMC PMC4945047. PMID 27415786. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4945047.
- Dillon, A. (2001). "Human Acceptance of Information Technology". In Karwowski, W.. Encyclopedia of Human Factors and Ergonomics. Taylor & Francis. pp. 673–675. ISBN 9780748408474.
- Thielsch, M.T.; Engel, R.; Hirschfeld, G. (2015). "Expected usability is not a valid indicator of experienced usability". PeerJ Computer Science 1: e19. doi:10.7717/peerj-cs.19.
- Giardine, B.; Riemer, C.; Hardison, R.C. et al. (2005). "Galaxy: A platform for interactive large-scale genome analysis". Genome Research 15 (10): 1451–1455. doi:10.1101/gr.4086505. PMC PMC1240089. PMID 16169926. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089.
- Wolstencroft, K.; Haines, R.; Fellows, D. et al. (2013). "The Taverna workflow suite: Designing and executing workflows of Web Services on the desktop, web or in the cloud". Nucleic Acids Research 41 (W1): W557-W561. doi:10.1093/nar/gkt328. PMC PMC3692062. PMID 23640334. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3692062.
- Goodstadt, L. (2010). "Ruffus: A lightweight Python library for computational pipelines". Bioinformatics 26 (21): 2778-9. doi:10.1093/bioinformatics/btq524. PMID 20847218.
- Köster, J.; Rahmann, S. (2012). "Snakemake: A scalable bioinformatics workflow engine". Bioinformatics 28 (19): 2520-2. doi:10.1093/bioinformatics/bts480. PMID 22908215.
- Morgenthaler, J.D.; Gridney, M.; Sauciuc, R. et al. (2012). "Searching for build debt: Experiences managing technical debt at Google". Proceedings of the Third International Workshop on Managing Technical Debt 2012: 1–6. ISBN 9781467317498.
- Sculley, D.; Holt, G.; Golovin, D. et al. (2014). "Machine learning: The high interest credit card of technical debt". Proceedings of SE4ML: Software Engineering for Machine Learning 2014 2014: 1–9.
- Nielsen, J. (19 March 2000). "Why You Only Need to Test with 5 Users". Nielsen Norman Group. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/. Retrieved 30 September 2016.
This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. In the original conclusion, the authors provided suggested reading, but the original only included them as citations to an associated opening sentence. These have been moved and expanded out into a new section after the conclusion called "Further reading" and organized alphabetically by author (but now no longer appear as citations to the opening sentence).