Difference between revisions of "User:Shawndouglas/sandbox/sublevel4"

From LIMSWiki
Jump to navigationJump to search
m (Added Glossary link)
 
(433 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Infobox journal article
<div class="nonumtoc">__TOC__</div>
|name        =
{{ombox
|image        =
| type     = notice
|alt          = <!-- Alternative text for images -->
| style    = width: 960px;
|caption      =
| text     = This is sublevel4 of my sandbox, where I play with features and test MediaWiki code. If you wish to leave a comment for me, please see [[User_talk:Shawndouglas|my discussion page]] instead.<p></p>
|title_full  = Generalized Procedure for Screening Free Software and Open Source Software Applications
|journal     =
|authors      = Joyce, John
|affiliations = Arcana Informatica; Scientific Computing
|contact     = Email:  
|editors      =
|pub_year    = 2015
|vol_iss      =
|pages        =
|doi          =
|issn        =
|license      = [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International]
|website      =
|download    =
}}
}}
==Abstract==
Free Software and [[:Category:Open-source software|Open Source Software projects]] have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process we have initiated a project to generate a library of in-depth reviews of Free Software and Open Source Software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we will examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we will synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure will be executed against a selected class of applications.


==Introduction==
==Sandbox begins below==
There is much confusion regarding Free Software and Open Source Software and many people use these terms interchangeably, however, to some the connotations associated with the terms is highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with Open Source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).


When discussing Free Software, we are not explicitly discussing software for which no fee is charged, rather we are referring to free in terms of liberty. To quote the Free Software Foundation (FSF)<ref name="FSFWhat15">{{cite web |url=http://www.gnu.org/philosophy/free-sw.html |title=What is free software? |work=GNU Project |publisher=Free Software Foundation, Inc |date=2015 |accessdate=17 June 2015}}</ref>:
*Discussion and practical use of [[artificial intelligence]] (AI) in the [[laboratory]] is, perhaps to the surprise of some, not a recent phenomena. In the mid-1980s, researchers were developing computerized AI systems able "to develop automatic decision rules for follow-up analysis of &#91;[[clinical laboratory]]&#93; tests depending on prior information, thus avoiding the delays of traditional sequential testing and the costs of unnecessary parallel testing."<ref>{{Cite journal |last=Berger-Hershkowitz |first=H. |last2=Neuhauser |first2=D. |date=1987 |title=Artificial intelligence in the clinical laboratory |url=https://www.ccjm.org/content/54/3/165 |journal=Cleveland Clinic Journal of Medicine |volume=54 |issue=3 |pages=165–166 |doi=10.3949/ccjm.54.3.165 |issn=0891-1150 |pmid=3301059}}</ref> In fact, discussion of AI in general was ongoing even in the mid-1950s.<ref name="MinskyHeuristic56">{{cite book |url=https://books.google.com/books?hl=en&lr=&id=fvWNo6_IZGUC&oi=fnd&pg=PA1 |title=Heuristic Aspects of the Artificial Intelligence Problem |author=Minsky, M. |publisher=Ed Services Technical Information Agency |date=17 December 1956 |accessdate=16 February 2023}}</ref><ref>{{Cite journal |last=Minsky |first=Marvin |date=1961-01 |title=Steps toward Artificial Intelligence |url=http://ieeexplore.ieee.org/document/4066245/ |journal=Proceedings of the IRE |volume=49 |issue=1 |pages=8–30 |doi=10.1109/JRPROC.1961.287775 |issn=0096-8390}}</ref>


<blockquote>A program is free software if the program's users have the four essential freedoms:
*Hiring demand for laboratorians with AI experience (2015–18) has historically been higher in non-healthcare industries, such as manufacturing, mining, and agriculture, shedding a light on how AI adoption in the clinical setting may be lacking. According to the Brookings Institute, "Even for the relatively-skilled job postings in hospitals, which includes doctors, nurses, medical technicians, research lab workers, and managers, only approximately 1 in 1,250 job postings required AI skills." They add: "AI adoption may be slow because it is not yet useful, or because it may not end up being as useful as we hope. While our view is that AI has great potential in health care, it is still an open question."<ref name=":11">{{Cite web |last=Goldfarb, A.; Teodoridis, F. |date=09 March 2022 |title=Why is AI adoption in health care lagging? |work=Series: The Economics and Regulation of Artificial Intelligence and Emerging Technologies |url=https://www.brookings.edu/research/why-is-ai-adoption-in-health-care-lagging/ |publisher=Brookings Institute |accessdate=17 February 2023}}</ref>


* The freedom to run the program as you wish, for any purpose (freedom 0).
*Today, AI is being practically used in not only clinical diagnostic laboratories but also clinical research labs, life science labs, and research and development (R&D) labs, and more. Practical uses of AI can be found in:
* The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
* The freedom to redistribute copies so you can help your neighbor (freedom 2).
* The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.</blockquote>


This does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also 'free software' or 'freeware' that is given away at no charge, or gratis, but without the source code, this would not be considered Free Software under the FSF definition.
:clinical research labs<ref name=":0">{{Cite journal |last=Damiani |first=A. |last2=Masciocchi |first2=C. |last3=Lenkowicz |first3=J. |last4=Capocchiano |first4=N. D. |last5=Boldrini |first5=L. |last6=Tagliaferri |first6=L. |last7=Cesario |first7=A. |last8=Sergi |first8=P. |last9=Marchetti |first9=A. |last10=Luraschi |first10=A. |last11=Patarnello |first11=S. |date=2021-12-07 |title=Building an Artificial Intelligence Laboratory Based on Real World Data: The Experience of Gemelli Generator |url=https://www.frontiersin.org/articles/10.3389/fcomp.2021.768266/full |journal=Frontiers in Computer Science |volume=3 |pages=768266 |doi=10.3389/fcomp.2021.768266 |issn=2624-9898}}</ref>
:hospitals<ref name=":0" /><ref name=":1">{{Cite journal |last=University of California, San Francisco |last2=Adler-Milstein |first2=Julia |last3=Aggarwal |first3=Nakul |last4=University of Wisconsin-Madison |last5=Ahmed |first5=Mahnoor |last6=National Academy of Medicine |last7=Castner |first7=Jessica |last8=Castner Incorporated |last9=Evans |first9=Barbara J. |last10=University of Florida |last11=Gonzalez |first11=Andrew A. |date=2022-09-29 |title=Meeting the Moment: Addressing Barriers and Facilitating Clinical Adoption of Artificial Intelligence in Medical Diagnosis |url=https://nam.edu/meeting-the-moment-addressing-barriers-and-facilitating-clinical-adoption-of-artificial-intelligence-in-medical-diagnosis |journal=NAM Perspectives |volume=22 |issue=9 |doi=10.31478/202209c |pmc=PMC9875857 |pmid=36713769}}</ref>
:medical diagnostics labs<ref name=":1" /><ref name=":12">{{Cite web |last=Government Accountability Office (GAO); National Academy of Medicine (NAM) |date=September 2022 |title=Artificial Intelligence in Health Care: Benefits and Challenges of Machine Learning Technologies for Medical Diagnostics |url=https://www.gao.gov/assets/gao-22-104629.pdf |format=PDF |publisher=Government Accountability Office |accessdate=16 February 2023}}</ref><ref name=":13">{{Cite journal |last=Wen |first=Xiaoxia |last2=Leng |first2=Ping |last3=Wang |first3=Jiasi |last4=Yang |first4=Guishu |last5=Zu |first5=Ruiling |last6=Jia |first6=Xiaojiong |last7=Zhang |first7=Kaijiong |last8=Mengesha |first8=Birga Anteneh |last9=Huang |first9=Jian |last10=Wang |first10=Dongsheng |last11=Luo |first11=Huaichao |date=2022-09-24 |title=Clinlabomics: leveraging clinical laboratory data by data mining strategies |url=https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-022-04926-1 |journal=BMC Bioinformatics |language=en |volume=23 |issue=1 |pages=387 |doi=10.1186/s12859-022-04926-1 |issn=1471-2105 |pmc=PMC9509545 |pmid=36153474}}</ref><ref name=":7">{{Cite journal |last=DeYoung |first=B. |last2=Morales |first2=M. |last3=Giglio |first3=S. |date=2022-08-04 |title=Microbiology 2.0–A “behind the scenes” consideration for artificial intelligence applications for interpretive culture plate reading in routine diagnostic laboratories |url=https://www.frontiersin.org/articles/10.3389/fmicb.2022.976068/full |journal=Frontiers in Microbiology |volume=13 |pages=976068 |doi=10.3389/fmicb.2022.976068 |issn=1664-302X |pmc=PMC9386241 |pmid=35992715}}</ref><ref name=":5">{{Cite web |last=Schut, M. |date=01 December 2022 |title=Get better with bytes |url=https://www.amsterdamumc.org/en/research/news/get-better-with-bytes.htm |publisher=Amsterdam UMC |accessdate=16 February 2023}}</ref><ref name="AlbanoCal19">{{cite web |url=https://physicianslab.com/calculations-to-diagnosis-the-artificial-intelligence-shift-thats-already-happening/ |title=Calculations to Diagnosis: The Artificial Intelligence Shift That’s Already Happening |author=Albano, V.; Morris, C.; Kent, T. |work=Physicians Lab |date=06 December 2019 |accessdate=16 February 2023}}</ref>
:chromatography labs<ref name="AlbanoCal19" />
:biology and life science labs<ref name=":6">{{Cite journal |last=de Ridder |first=Dick |date=2019-01 |title=Artificial intelligence in the lab: ask not what your computer can do for you |url=https://onlinelibrary.wiley.com/doi/10.1111/1751-7915.13317 |journal=Microbial Biotechnology |language=en |volume=12 |issue=1 |pages=38–40 |doi=10.1111/1751-7915.13317 |pmc=PMC6302702 |pmid=30246499}}</ref>
:medical imaging centers<ref name="Brandao-de-ResendeAIWeb22">{{cite web |url=https://siim.org/page/22w_clinical_adoption_of_ai |title=AI Webinar: Clinical Adoption of AI Across Image Producing Specialties |author=Brandao-de-Resende, C.; Bui, M.; Daneshjou, R. et al. |publisher=Society for Imaging Informatics in Medicine |date=11 October 2022}}</ref>
:ophthalmology clinics<ref>{{Cite journal |last=He |first=Mingguang |last2=Li |first2=Zhixi |last3=Liu |first3=Chi |last4=Shi |first4=Danli |last5=Tan |first5=Zachary |date=2020-07 |title=Deployment of Artificial Intelligence in Real-World Practice: Opportunity and Challenge |url=https://journals.lww.com/10.1097/APO.0000000000000301 |journal=Asia-Pacific Journal of Ophthalmology |language=en |volume=9 |issue=4 |pages=299–307 |doi=10.1097/APO.0000000000000301 |issn=2162-0989}}</ref>
:reproduction clinics<ref name=":9">{{Cite journal |last=Trolice |first=Mark P. |last2=Curchoe |first2=Carol |last3=Quaas |first3=Alexander M |date=2021-07 |title=Artificial intelligence—the future is now |url=https://link.springer.com/10.1007/s10815-021-02272-4 |journal=Journal of Assisted Reproduction and Genetics |language=en |volume=38 |issue=7 |pages=1607–1612 |doi=10.1007/s10815-021-02272-4 |issn=1058-0468 |pmc=PMC8260235 |pmid=34231110}}</ref><ref name="ESHREArti22">{{cite web |url=https://www.focusonreproduction.eu/article/ESHRE-News-22AI |title=Annual Meeting 2022: Artificial intelligence in embryology and ART |author=European Society of Human Reproduction and Embryology |work=Focus on Reproduction |date=06 July 2022 |accessdate=16 February 2023}}</ref><ref name="HinckleyApply21">{{cite web |url=https://rscbayarea.com/blog/applying-ai-for-better-ivf-success |title=Applying AI (Artificial Intelligence) in the Lab for Better IVF Success |author=Hinckley, M. |work=Reproductive Science Center Blog |publisher=Reproductive Science Center of the Bay Area |date=17 March 2021 |accessdate=16 February 2023}}</ref>
:digital pathology labs<ref name="YousifArt21">{{cite web |url=https://clinlabint.com/artificial-intelligence-is-the-key-driver-for-digital-pathology-adoption/ |title=Artificial intelligence is the key driver for digital pathology adoption |author=Yousif, M.; McClintock, D.S.; Yao, K. |work=Clinical Laboratory Int |publisher=PanGlobal Media |date=2021 |accessdate=16 February 2023}}</ref>
:material testing labs<ref name=":2">{{Cite journal |last=MacLeod |first=B. P. |last2=Parlane |first2=F. G. L. |last3=Morrissey |first3=T. D. |last4=Häse |first4=F. |last5=Roch |first5=L. M. |last6=Dettelbach |first6=K. E. |last7=Moreira |first7=R. |last8=Yunker |first8=L. P. E. |last9=Rooney |first9=M. B. |last10=Deeth |first10=J. R. |last11=Lai |first11=V. |date=2020-05-15 |title=Self-driving laboratory for accelerated discovery of thin-film materials |url=https://www.science.org/doi/10.1126/sciadv.aaz8867 |journal=Science Advances |language=en |volume=6 |issue=20 |pages=eaaz8867 |doi=10.1126/sciadv.aaz8867 |issn=2375-2548 |pmc=PMC7220369 |pmid=32426501}}</ref><ref name=":3">{{Cite journal |last=Chibani |first=Siwar |last2=Coudert |first2=François-Xavier |date=2020-08-01 |title=Machine learning approaches for the prediction of materials properties |url=http://aip.scitation.org/doi/10.1063/5.0018384 |journal=APL Materials |language=en |volume=8 |issue=8 |pages=080701 |doi=10.1063/5.0018384 |issn=2166-532X}}</ref><ref name="MullinTheLab21">{{Cite journal |last=Mullin, R. |date=28 March 2021 |title=The lab of the future is now |url=http://cen.acs.org/business/informatics/lab-future-ai-automated-synthesis/99/i11 |journal=Chemical & Engineering News |volume=99 |issue=11 |archiveurl=https://web.archive.org/web/20220506192926/http://cen.acs.org/business/informatics/lab-future-ai-automated-synthesis/99/i11 |archivedate=06 May 2022 |accessdate=16 February 2023}}</ref>
:chemical experimentation and molecular discovery labs<ref name="MullinTheLab21" /><ref name=":4">{{Cite journal |last=Burger |first=Benjamin |last2=Maffettone |first2=Phillip M. |last3=Gusev |first3=Vladimir V. |last4=Aitchison |first4=Catherine M. |last5=Bai |first5=Yang |last6=Wang |first6=Xiaoyan |last7=Li |first7=Xiaobo |last8=Alston |first8=Ben M. |last9=Li |first9=Buyi |last10=Clowes |first10=Rob |last11=Rankin |first11=Nicola |date=2020-07-09 |title=A mobile robotic chemist |url=https://www.nature.com/articles/s41586-020-2442-2.epdf?sharing_token=HOkIS6P5VIAo2_l3nRELmdRgN0jAjWel9jnR3ZoTv0Nw4yZPDO1jBpP52iNWHbb8TakOkK906_UHcWPTvNxCmzSMpAYlNAZfh29cFr7WwODI2U6eWv38Yq2K8odHCi-qwHcEDP18OjAmH-0KgsVgL5CpoEaQTCvbmhXDSyoGs6tIMe1nuABTeP58z6Ck3uULcdCtVQ66X244FsI7uH8GnA%3D%3D&tracking_referrer=cen.acs.org |journal=Nature |language=en |volume=583 |issue=7815 |pages=237–241 |doi=10.1038/s41586-020-2442-2 |issn=0028-0836}}</ref><ref name="LemonickExplore20">{{Cite journal |last=Lemonick, S. |date=06 April 2020 |title=Exploring chemical space: Can AI take us where no human has gone before? |url=https://cen.acs.org/physical-chemistry/computational-chemistry/Exploring-chemical-space-AI-take/98/i13 |journal=Chemical & Engineering News |volume=98 |issue=13 |archiveurl=https://web.archive.org/web/20200729004137/https://cen.acs.org/physical-chemistry/computational-chemistry/Exploring-chemical-space-AI-take/98/i13 |archivedate=29 July 2020 |accessdate=16 February 2023}}</ref>
:quantum physics labs<ref name="DoctrowArti19">{{cite web |url=https://www.pnas.org/post/podcast/artificial-intelligence-laboratory |title=Artificial intelligence in the laboratory |author=Doctrow, B. |work=PNAS Science Sessions |date=16 December 2019 |accessdate=16 February 2023}}</ref>


The Open Source Initiative (OSI), originally formed to promote Free Software, which they referred to as Open Source Software (OSS) to make it sound more business friendly. The OSI defines Open Source Software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines<ref name="OSIDef15">{{cite web |url=http://opensource.org/osd |title=The Open Source Definition |publisher=Open Source Initiative |date=2015 |accessdate=17 June 2015}}</ref>:
*What's going on in these labs?


* Free redistribution
:'''Materials science''': The creation of "a modular robotic platform driven by a model-based optimization algorithm capable of autonomously optimizing the optical and electronic properties of thin-film materials by modifying the film composition and processing conditions ..."<ref name=":2" />
* Source code included
:'''Materials science''': "Most of the applications of [machine learning (ML)] in chemical and materials sciences, as we have said, feature supervised learning algorithms. The goal there is to supplement or replace traditional modeling methods, at the quantum chemical or classical level, in order to predict the properties of molecules or materials directly from their structure or their chemical composition ... Our research group was applying the same idea on a narrower range of materials, trying to confirm that for a given chemical composition, geometrical descriptors of a material’s structure could lead to accurate predictions of its mechanical features."<ref name=":3" />
* Must allow derived works
:'''Life science''': "In biological experiments, we generally cannot as easily declare victory, but we can use the systems biology approach of cycling between experimentation and modelling to see which sequences, when tested, are most likely to improve the model. In artificial intelligence, this is called active learning, and it has some similarity to the way in which we as humans learn as infants: we get some help from parents and teachers, but mainly model the world around us by exploring it and interacting with it. Ideally then, we would recreate such an environment for our machine learning algorithms in the laboratory, where we start with an initial ‘infant’ model of a certain regulatory system or protein function and let the computer decide what sequence designs to try out – a deep learning version of the ‘robot scientist’. Microbes are ideal organisms for such an approach, given the ease and speed with which they can be grown and genetically manipulated. Combined with laboratory automation, many microbial experiments can (soon) be performed with minimal human intervention, ranging from strain construction and screening, such as operated by Amyris, Gingko, Transcriptic, etc., to full-genome engineering or even the design of microbial ecologies."<ref name=":6" />
* Must preserve the integrity of the author's source code
:'''Digital pathology''': "The collaboration combines two AI solutions, VistaPath’s Sentinel, the world’s first automated tissue grossing platform, and Gestalt’s AI Requisition Engine (AIRE), a leading-edge AI algorithm for accessioning, to raise the bar in AI-driven pathology digitization. Designed to make tissue grossing faster and more accurate, VistaPath’s Sentinel uses a high-quality video system to assess specimens and create a gross report 93% faster than human technicians with 43% more accuracy. It not only improves on quality by continuously monitoring the cassette, container, and tissue to reduce mislabeling and specimen mix-up, but also increases traceability by retaining original images for downstream review."<ref>{{Cite web |last=VistaPath |date=28 July 2022 |title=VistaPath Launches New Collaboration with Gestalt Diagnostics to Further Accelerate Pathology Digitization |work=PR Newswire |url=https://www.prnewswire.com/news-releases/vistapath-launches-new-collaboration-with-gestalt-diagnostics-to-further-accelerate-pathology-digitization-301594718.html |publisher=Cision US Inc |accessdate=17 February 2023}}</ref>
* License must not discriminate against persons or groups
:'''Chemistry and molecular science''': "The benefits of combining automated experimentation with a layer of artificial intelligence (AI) have been demonstrated for flow reactors, photovoltaic films, organic synthesis, perovskites and in formulation problems. However, so far no approaches have integrated mobile robotics with AI for chemical experiments. Here, we built Bayesian optimization into a mobile robotic workflow to conduct photocatalysis experiments within a ten-dimensional space."<ref name=":4" />
* License must not discriminate against fields of endeavor
:'''Chemistry and immunology''': "Chemistry and immunology laboratories are particularly well-suited to leverage machine learning because they generate large, highly structured data sets, Schulz and others wrote in a separate review paper. Labor-intensive processes used for interpretation and quality control of electrophoresis traces and mass spectra could benefit from automation as the technology improves, they said. Clinical chemistry laboratories also generate digital images—such as urine sediment analysis—that may be highly conducive to semiautomated analyses, given advances in computer vision, the paper noted."<ref name=":8">{{Cite web |last=Blum, K. |date=01 January 2023 |title=A Status Report on AI in Laboratory Medicine |work=Clinical Laboratory News |url=https://www.aacc.org/cln/articles/2023/janfeb/a-status-report-on-ai-in-laboratory-medicine |publisher=American Association for Clinical Chemistry |accessdate=17 February 2023}}</ref>
* Distribution of licenses
:'''Clinical research''': "... retrospective analysis of existing patient data for descriptive and clustering purposes [and] automation of knowledge extraction, ranging from text mining, patient selection for trials, to generation of new research hypotheses ..."<ref name=":0" />
* License must not be specific to a production
:'''Clinical research''': "AI ... offers a further layer to the laboratory system by analyzing all experimental data collected by experiment devices, whether it be a sensor or a collaborative robot. From data collected, AI is able to produce hypotheses and predict which combination of materials or temperature is desired for the experiment. In short, this system will allow scientists to be aided by a highly intelligent system which is constantly monitoring and analyzing the experimental output. In this way, AI will help an experiment from its inception to conclusion."<ref>{{Cite web |last=Chubb, P. |date=03 November 2020 |title=How disruptive technology is helping laboratories combat COVID-19 |url=https://datafloq.com/read/disruptive-technologies-lab-help-us-prepare-future-pandemics/ |publisher=Datafloq |accessdate=16 February 2023}}</ref>
* License must not restrict other software
:'''Clinical research/medical diagnostics''': "Artificial intelligence (AI) in the laboratory is primarily used to make sense of big data, the almost impossibly large sets of data that biologists and pharmaceutical R&D teams are accustomed to working with. AI algorithms can parse large amounts of data in a short amount of time and turn that data into visualizations that viewers can easily understand. In certain data-intensive fields, such as genomic testing and virus research, AI algorithms are the best way to sort through the data and do some of the pattern recognition work."<ref>{{Cite web |last=Stewart, B. |date=18 March 2021 |title=Using LIMS for Data Visualization |work=CSols Insights |url=https://www.csolsinc.com/insights/published-articles/using-lims-for-data-visualization/ |publisher=CSols, Inc |accessdate=17 February 2023}}</ref>
* License must be technology-neutral
:'''Medical diagnostics''': Development and implementation of [[Clinical decision support system|clinical decision support systems]] <ref name=":0" /><ref name=":1" />
:'''Medical diagnostics''': "Finally, in the laboratory, AI reduces the number of unnecessary blood samples when diagnosing infection. Instead of the 'gold standard blood sample' that takes 24-72 hours, the algorithm can predict the outcome of the blood sample with almost 80% accuracy based on demographics, vital signs, medications, and laboratory and radiology results. These are all examples of how Artificial Intelligence can be used to test better and faster with information that already exists. This saves time and costs."<ref name=":5" />
:'''Medical diagnostics''': "Chang sees two overarching classes of AI models: those that tackle internal challenges in the lab, such as how to deliver more accurate results to clinicians; and those that seek to identify cohorts of patients and care processes to close quality gaps in health delivery systems. The lab, however, 'isn’t truly an island,' said Michelle Stoffel, MD, PhD, associate chief medical information officer for laboratory medicine and pathology at M Health Fairview and the University of Minnesota in Minneapolis. 'When other healthcare professionals are working with electronic health records or other applications, there could be AI-driven tools, or algorithms used by an institution’s systems that may draw on laboratory data.'"<ref name=":8" />
:'''Medical diagnostics''': AI is used for the formulation of reference ranges, improvement of quality control, and automated interpretation of results. "Continuous monitoring of specimen acceptability, collection and transport can result in the prompt identification and correction of problems, leading to improved patient care and a reduction in unnecessary redraws and delays in reporting results."<ref name=":13" />
:'''Reproduction science''': "The field of AI is the marriage of humans and computers while reproductive medicine combines clinical medicine and the scientific laboratory of embryology. The application of AI has the potential to disconnect healthcare professionals from patients through algorithms, automated communication, and clinical imaging. However, in the embryology laboratory, AI, with its focus on gametes and embryos, can avoid the same risk of distancing from the patient. Areas of application of AI in the laboratory would be to enhance and automate embryo ranking through analysis of images, the ultimate goal being to predict successful implantation. Might such a trend obviate the need for embryo morphological assessment, time-lapse imaging and preimplantation genetic testing for aneuploidy (PGT-A), including mosaicism. Additionally, AI could assist with automation through analysis of testicular sperm samples searching for viable gametes, embryo grading uniformity."<ref name=":9" />
:'''Chromatography-heavy sciences''': " A great example of this is AI in the Liquid Chromatography Mass Spectrometry (LC-MS) field. LC-MS is a great tool used to measure various compounds in the human body, including everything from hormone levels to trace metals. One of the ways AI has already integrated with LC-MS is how it cuts down on the rate limiting steps of LC-MS, which more often than not are sample prep and LC separations. One system that Physicians Lab has made use of is parallel processing using SCIEX MPX 2.0 High Throughput System. This system can couple parallel runs with one LCMS instrument, resulting in twice the speed with no loss to accuracy. It can do this by staggering two runs either using the same method, or different methods entirely. What really makes this system great is its ability to automatically detect carryover and inject solvent blanks to clean the instrument. The system will then continue its analyzing, while automatically reinjecting samples that may be affected by the carryover. It will also flag high concentration without user input, allowing for easy detection of possibly faulty samples. This allows it to operate without users from startup to shut down. Some of the other ways that it can be used to increase efficiency are by using integrated network features to work on anything from streamlining management to increased throughput."<ref name="AlbanoCal19" />
:'''Most any lab''': "Predictive analytics, for example, is one tool that the Pistoia Alliance is using to better understand laboratory instruments and how they might fail over time... With the right data management strategies and careful consideration of metadata, how to best store data so that it can be used in future AI and ML workflows is essential to the pursuit of AI in the laboratory. Utilizing technologies such as LIMS and ELN enables lab users to catalogue data, providing context and instrument parameters that can then be fed into AI or ML systems. Without the correct data or with mismatched data types, AI and ML will not be possible, or at the very least, could provide undue bias trying to compare data from disparate sources."<ref>{{Cite web |date=29 January 2021 |title=Data Analytics |work=Scientific Computing World - Building a Smart Laboratory 2020 |url=https://www.scientific-computing.com/feature/data-analytics-0 |publisher=Europa Science Ltd |accessdate=17 February 2023}}</ref>
:'''Most any lab''': "When the actionable items are automatically created by Optima, the 'engine' starts working. An extremely sophisticated algorithm is able to assign the tasks to the resources, both laboratory personnel and instruments, according to the system configuration. Optima, thanks to a large amount of time dedicated to research the best way to automate this critical process, is able to automate most of the lab resource scheduling."<ref>{{Cite web |last=Optima Team |date=15 December 2020 |title=The concept of machine learning applied to lab resources scheduling |work=Optima Blog |url=https://www.optima.life/blog/the-concept-of-machine-learning-applied-to-lab-resources-scheduling/ |publisher=Optima PLC Tracking Tools S.L |accessdate=17 February 2023}}</ref>


Open Source Software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.
*A number of challenges exist in the realm of effectively and securely implementing AI in the laboratory. This includes:


In an attempt to placate both groups when discussing the same software application, the term Free/Open Source Software (F/OSS) was developed. Since the term Free was still tending to confuse some people, the term libre, which connotes freedom, was added resulting in the term Free/Libre Open Source Software (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all Free Software fits the Open Source Software definition, while not all Open Source Software fits the Free Software definition. However, any Open Source Software that is not also Free Software is the exception, rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.
:Ethical and privacy challenges<ref name=":0" /><ref name=":8" /><ref name=":10" />
:Algorithmic limitations<ref name=":11" />
:Data access limitations, including "where to get it, how to share it, and how to know when you have enough to train a machine-learning system that will produce good results"<ref name=":11" /><ref name=":8" /><ref name=":14">{{Cite web |last=Sherwood, L. |date=10 February 2022 |title=SLAS 2022: Barriers remain to AI adoption in life sciences |work=LabPulse.com Showcasts |url=https://www.labpulse.com/showcasts/slas/2022/article/15300130/slas-2022-barriers-remain-to-ai-adoption-in-life-sciences |publisher=Science and Medicine Group |accessdate=17 February 2023}}</ref><ref name=":15">{{Cite journal |last=Bellini |first=Claudia |last2=Padoan |first2=Andrea |last3=Carobene |first3=Anna |last4=Guerranti |first4=Roberto |date=2022-11-25 |title=A survey on Artificial Intelligence and Big Data utilisation in Italian clinical laboratories |url=https://www.degruyter.com/document/doi/10.1515/cclm-2022-0680/html |journal=Clinical Chemistry and Laboratory Medicine (CCLM) |language=en |volume=60 |issue=12 |pages=2017–2026 |doi=10.1515/cclm-2022-0680 |issn=1434-6621}}</ref>
:Data integration and transformation issues<ref name=":0" /><ref name=":15" />
:Regulatory barriers<ref name=":11" /><ref name=":12" />
:Misaligned incentives<ref name=":11" />
:Lack of knowledgeable/skilled talent<ref name=":0" /><ref name=":8" /><ref name=":14" /><ref name=":15" />
:Cost of skilled talent and infrastructure for maintaining and updating AI systems<ref name=":8" />
:Legacy systems running outdated technologies<ref name=":14" />
:Lack of IT systems or specialized software systems<ref name=":15" />
:Lack of standardized, best practices-based methods of validating algorithms<ref name=":8" />
:Failure to demonstrate real-world performance<ref name=":12" />
:Failure to meet the needs of the professionals using it<ref name=":12" />


The reality is that since both groups trace their history back to the same origins, the practical differences between an application being Free Software or Open Source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in Open Source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us.<ref name="SchießleFree12">{{cite web |url=https://fsfe.org/freesoftware/basics/comparison.en.html |title=Free Software, Open Source, FOSS, FLOSS - same same but different |author=Schießle, Björn |publisher=Free Software Foundation Europe |date=12 August 2012 |accessdate=5 June 2015}}</ref> In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With in excess of 300,000 FLOSS applications currently registered in SourceForge.net<ref name="DateRepOSS12">{{cite web |url=http://events.linuxfoundation.org/images/stories/pdf/lceu2012_date.pdf |format=PDF |title=RepOSS: A Flexible OSS Assessment Repository |publisher=Northeast Asia OSS Promotion Forum WG3 |date=05 November 2012 |accessdate=05 May 2015}}</ref> and over 10 million repositories on GitHub<ref name="Doll10Mil13">{{cite web |url=https://github.com/blog/1724-10-millionrepositories |title=10 Million Repositories |author=Doll, Brian |publisher=GitHub, Inc |date=23 December 2013 |accessdate=08 August 2015}}</ref>, there are generally multiple options accessible for any class of application, be it a [[Laboratory information management system|Laboratory Information Management System]] (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an Open Source application<ref name=SarrabTheTech13">{{cite journal |title=The Technical, Non-technical Issues and the Challenges of Migration to Free and Open Source Software |journal=IJCSI International Journal of Computer Science Issues |author=Sarrab, Mohamed; Elsabir, Mahmoud; Elgamel, Laila |volume=10 |issue=2.3 |year=March 2013 |url=http://ijcsi.org/papers/IJCSI-10-2-3-464-469.pdf |format=PDF}}</ref> and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers with no indication of the vitality or independent evaluation of the project.
*Given those challenges, some considerations should be made about implementing AI-based components in the laboratory. Examples include:


What is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While once they've made a tentative selection, the organization will need to perform their own testing to confirm that the selected application meets their specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.
:'''Clinical diagnostics''': "From an industry and regulatory perspective, however, only the intended uses supported from the media manufacturer can be supported from AI applications, unless otherwise justified and substantive evidence is presented for additional claims support. This means strict adherence to specimen type and incubation conditions. Considering that the media was initially developed for human assessment using the well-trained microbiologist eye, and not an advanced imaging system with or without AI, this paradigm should shift to allow advancements in technology to challenge the status-quo of decreasing media read-times especially, as decreased read-times assist with laboratory turnaround times and thus patient management. Perhaps with an increasing body of evidence to support any proposed indications for use, either regulatory positions should be challenged, or manufacturers of media and industry AI-development specialists should work together to advance the field with new indications for use.
 
:While the use of AI in the laboratory setting can be highly beneficial there are still some issues to be addressed. The first being phenotypically distinct single organism polymorphisms that may be interpreted by AI as separate organisms, as may also be the case for a human assessment, as well as small colony variant categorization. As detailed earlier, the broader the inputs, the greater the generalization of the model, and the higher the likelihood of algorithm accuracy. In that respect, understanding and planning around these design constraints is critical for ultimate deployment of algorithms. Additionally, expecting an AI system to correctly categorize “contamination” is a difficult task as often this again seemingly innocuous decision is dependent on years of experience and understanding the specimen type and the full clinical picture with detailed clinical histories. In this respect, a fully integrated AI-LIS system where all data is available may assist, but it is currently not possible to gather this granular detail needed to make this assessment reliable."<ref name=":7" />
:[[File:Fig1 Joyce 2015.png|500px]]
:'''Clinical diagnostics and pathology''': "Well, if I’ve learned anything in my research into this topic, it’s that AI implementation needs to be a two-way street. First, any company who is active in this space must reach out to pathologists and laboratory medicine professionals to understand their daily workflows, needs, and pain points in as much detail as possible. Second, pathologists, laboratory medicine professionals, and educators must all play their important part – willingly offering their time and expertise when it is sought or proactively getting involved. And finally, it’s clear that there is an imbalanced focus on certain issues – with privacy, respect, and sustainability falling by the wayside."<ref name=":10">{{Cite web |last=Lee, G.F. |date=10 October 2022 |title=The Robot May See You Now: It’s time to stop and think about the ethics of artificial intelligence |work=The Pathologist |url=https://thepathologist.com/outside-the-lab/the-robot-may-see-you-now |accessdate=17 February 2023}}</ref>
{{clear}}
:'''Healthcare''': "While we are encouraged by the promise shown by AI in healthcare, and more broadly welcome the use of digital technologies in improving clinical outcomes and health system productivity, we also recognize that caution must be exercised when introducing any new healthcare technology. Working with colleagues across the NHS Transformation Directorate, as well as the wider AI community, we have been developing a framework to evaluate AI-enabled solutions in the health and care policy context. The aim of the framework is several-fold but is, at its core, a tool with which to highlight to healthcare commissioners, end users, patients and members of the public the considerations to be mindful when introducing AI to healthcare settings."<ref>{{Cite journal |last=Chada |first=Bharadwaj V |last2=Summers |first2=Leanne |date=2022-10-10 |title=AI in the NHS: a framework for adoption |url=https://www.rcpjournals.org/lookup/doi/10.7861/fhj.2022-0068 |journal=Future Healthcare Journal |language=en |pages=fhj.2022–0068 |doi=10.7861/fhj.2022-0068 |issn=2514-6645 |pmc=PMC9761451 |pmid=36561823}}</ref>
<blockquote>'''Illustration 1.''': This diagram, originally by Chao-Kuei and updated by several others since,<br />explains the different categories of software. It's available as a Scalable Vector Graphic and<br />as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2<br />or later, or the Creative Commons Attribution-Share Alike v2.0 or later</blockquote>
:'''Most any lab''': A code of AI ethics should address objectivity, privacy, transparency, accountability, and sustainability in any AI implementation.<ref name=":10" />
 
:'''Most any lab''': "Another approach is to implement an AI program alongside a manual process, assessing its performance along the way, as a means to ease into using the program. 'I think one of the most impactful things that laboratorians can do today is to help make sure that the lab data that they’re generating is as robust as possible, because these AI tools rely on new training sets, and their performance is really only going to be as good as the training data sets they’re given,' Stoffel said."<ref name=":8" />
The primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end it we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.
 
You can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, commercial is referring to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior quality, to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS ''is'' commercial software.<ref name="WheelerFree11">{{cite web |url=http://www.dwheeler.com/essays/commercial-floss.html |title=Free-Libre / Open Source Software (FLOSS) is Commercial Software |author=Wheeler, David A. |work=dwheeler.com |date=14 June 2011 |accessdate=28 May 2015}}</ref>
 
The final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.
 
==Results==
===Literature review===
A search of the literature returns thousands of papers related to Open Source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between Open Source projects is mentioned in a number of these papers and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large scale support in the industry. Stol and Babar have published a framework for comparing evaluation methods targeting Open Source software and include a comparison of 20 of them.<ref name="StolAComp10">{{cite book |chapter=A Comparison Framework for Open Source Software Evaluation Methods |title=Open Source Software: New Horizons |author=Stol, Klaas-Jan; Ali Babar, Muhammad |editor=Ågerfalk, P.J.; Boldyreff, C.; González-Barahona, J.M.; Madey, G.R.; Noll, J |publisher=Springer |year=2010 |pages=389–394 |isbn=9783642132445 |doi=10.1007/978-3-642-13244-5_36}}</ref> They noted that web sites that simply consisted of a suggestion list for selecting an Open Source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, and other factors involved.<ref name="FritzAClass94">{{cite book |title=A Classification And Summary Of Software Evaluation And Selection Methodologies |author=Fritz, Catherine A.; Carter, Bradley D. |publisher=Department of Computer Science, Mississippi State University |location=Mississippi State, MS |date=23 August 1994 |url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.4470}}</ref>
 
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! No.
  ! Name
  ! Year
  ! Orig
  ! Method
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Capgemini Open Source Maturity Model
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2003
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Evaluation Framework for Open Source Software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2004
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|3
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Model for Comparative Assessment of Open Source Products
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2004
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|4
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Navica Open Source Maturity Model
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2004
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|5
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Woods and Guliani's OSMM
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2005
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|6
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Business Readiness Rating (OpenBRR)<ref name="OpenBRROpen2005">{{cite web |url=http://docencia.etsit.urjc.es/moodle/file.php/125/OpenBRR_Whitepaper.pdf |format=PDF |title=OpenBRR, Business Readiness Rating for Open Source: A Proposed Open Standard to Facilitate Assessment and Adoption of Open Source Software |publisher=OpenBRR |year=2005 |accessdate=13 April 2015}}</ref><ref name="WassermanTheBus06">{{cite web |url=http://www.openbrr.org/comoworkshop/papers/WassermanPalChan_EFOSS06.pdf |archiveurl=http://web.archive.org/web/20070111113722/http://www.openbrr.org/comoworkshop/papers/WassermanPalChan_EFOSS06.pdf |format=PDF |title=The Business Readiness Rating: a Framework for Evaluating Open Source |work=Proceedings of the Workshop on Evaluation Frameworks for Open Source Software (EFOSS) at the Second International Conference on Open Source Systems |author=Wasserman, A.I.; Pal, M.; Chan, C. |location=Lake Como, Italy |pages=1–5 |date=10 June 2006 |archivedate=11 January 2007 |accessdate=15 April 2015}}</ref>
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2005
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R/I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|7
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Atos Origin Method for Qualification and Selection of Open Source Software (QSOS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2006
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|8
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Evaluation Criteria for Free/Open Source Software Products
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2006
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|9
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Quality Model for OSS Selection
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|10
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Selection Process of Open Source Software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|11
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Observatory for Innovation and Technological transfer on Open Source software (OITOS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|12
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Framework for OS Critical Systems Evaluation (FOCSE)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|13
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Balanced Scorecards for OSS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|14
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Business Quality Rating (OpenBQR)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|15
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Evaluating OSS through Prototyping
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|16
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Comprehensive Approach for Assessing Open Source Projects
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|17
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Software Quality Observatory for Open Source Software (SQO-OSS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|18
  | style="background-color:white; padding-left:10px; padding-right:10px;"|An operational approach for selecting open source components in a software development project<ref name="MajchrowskiAnOp08">{{cite book |chapter=An Operational Approach for Selecting Open Source Components in a Software Development Project |title=Software Process Improvement |author=Majchrowski, Annick; Deprez, Jean-Christophe |editor=O'Connor, R.; Baddoo, N.; Smolander, K.; Messnarz, R. |publisher=Springer |year=2008 |pages=176–188 |isbn=9783540859369 |doi=10.1007/978-3-540-85936-9_16}}</ref>
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|19
  | style="background-color:white; padding-left:10px; padding-right:10px;"|QualiPSo trustworthiness model
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|20
  | style="background-color:white; padding-left:10px; padding-right:10px;"|OpenSource Maturity Model (OMM)<ref name="PetrinjaIntro09">{{cite book |chapter=Introducing the Open Source Maturity Model |title=ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development, 2009 |author=Petrinja, E.; Nambakam, R.; Sillitti, A. |publisher=IEEE |year=2009 |pages=37–41 |doi=10.1109/FLOSS.2009.5071358 |isbn=9781424437207 }}</ref>
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2009
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
|}
|}
<blockquote>'''Table 1.''': Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar.<ref name="StolAComp10" /> The selection<br />procedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described<br /> process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)<br />or just a list of evaluation criteria (No).</blockquote>
 
Extensive comparisons between some of these methods have also been published, such as Deprez's and Alexandre's comparative assessment of the OpenBRR and QSOS techniques.<ref name="Deprezcomp08">{{cite book |chapter=Comparing Assessment Methodologies for Free/Open Source Software: OpenBRR and QSOS |title=Product-Focused Software Process Improvement |author=Deprez,Jean-Christophe; Alexandre, Simon |editor=Jedlitschka, Andreas; Salo, Outi |publisher=Springer |year=2008 |pages=189-203 |isbn=9783540695660 |doi=10.1007/978-3-540-69566-0_17}}</ref> Wasserman and Pal have also published a paper under the title of ''Evaluating Open Source Software'', which appears to be more of an updated announcement and in-depth description of the Business Readiness Rating (BRR) framework.<ref name="WassermanEval10">{{cite web |url=http://oss.sv.cmu.edu/readings/EvaluatingOSS_Wasserman.pdf |archiveurl=https://web.archive.org/web/20150218173146/http://oss.sv.cmu.edu/readings/EvaluatingOSS_Wasserman.pdf |format=PDF |title=Evaluating Open Source Software |author=Wasserman, Anthony I.; Pal, Murugan |publisher=Carnegie Mellon University - Silicon Valley |date=2010 |archivedate=18 February 2015 |accessdate=31 May 2015}}</ref> Jadhav and Sonar have also examined the issue of both evaluating and selecting software packages. They include a helpful analysis of the strengths and weaknesses of the various techniques.<ref name="JadhavEval09">{{cite journal |title=Evaluating and selecting software packages: A review |journal=Information and Software Technology |author=Jadhav, Anil S.; Sonar, Rajendra M. |volume=51 |issue=3 |year=March 2009 |pages=555–563 |doi=10.1016/j.infsof.2008.09.003}}</ref> Perhaps more importantly, they clearly point out that there is no common list of evaluation criteria. While the majority of the articles they reviewed listed the criteria used, Jadhav and Sonar indicated that these criteria frequently did not include a detailed definition, which required each evaluator to use their own, sometimes conflicting, interpretation.
 
Since the publication of Stol's and Babar's paper, additional evaluation methods have been published. Of particular interest are a series of papers by Pani, et al. describing their proposed FAME (Filter, Analyze, Measure and Evaluate) methodology.<ref name="PaniAMeth10">{{cite web |title=FAME, A Methodology for Assessing Software Maturity |work=Atti della IV Conferenza Italiana sul Software Libero |author=Pani, F.E.; Sanna, D. |date=11 June 2010 |location=Cagliari, Italy}}</ref><ref name="PaniTheFAMEApp10">{{cite book |chapter=The FAME Approach: An Assessing Methodology |title=Proceedings of the 9th WSEAS International Conference on Telecommunications and Informatics |author=Pani, F.E.; Concas, G.; Sanna, D.; Carrogu, L. |editor=Niola, V.; Quartieri, J.; Neri, F.; Caballero, A.A.; Rivas-Echeverria, F.; Mastorakis, N. |publisher=WSEAS |location=Stevens Point, WI |year=2010 |url=http://www.wseas.us/e-library/conferences/2010/Catania/TELE-INFO/TELE-INFO-10.pdf |format=PDF |isbn=9789549260021}}</ref><ref name="PaniTheFAMEtool10">{{cite journal |title=The FAMEtool: an automated supporting tool for assessing methodology |journal=WSEAS Transactions on Information Science and Applications |author=Pani, F.E.; Concas, G.; Sanna, S.; Carrogu, L. |volume=7 |issue=8 |pages=1078–1089 |year=August 2010 |url=http://www.wseas.us/e-library/transactions/information/2010/88-137.pdf |format=PDF}}</ref><ref name="PaniTrans10">{{cite book |chapter=Transferring FAME, a Methodology for Assessing Open Source Solutions, from University to SMEs |title=Management of the Interconnected World |author=Pani, F.E.; Sanna, D.; Marchesi, M.; Concas, G. |editor= D'Atri, A.; De Marco, M.; Braccini, A.M.; Cabiddu, F. |publisher=Springer |year=2010 |pages=495–502 |isbn=9783790824049 |doi=10.1007/978-3-7908-2404-9_57}}</ref> In their ''Transferring FAME'' paper, they emphasized that all of the evaluation frameworks previously described in the published literature were frequently not easy to apply to real environments, as they were developed using an analytic research approach which incorporated a multitude of factors.<ref name="PaniTrans10" />
 
Their stated design objective with FAME is to reduce the complexity of performing the application evaluation, particularly for small organizations. As specified ”The goals of FAME methodology are to aid the choice of high-quality F/OSS products, with high probability to be sustainable in the long term, and to be as simple and user friendly as possible.” They further state that “The main idea behind FAME is that the users should evaluate which solution amongst those available is more suitable to their needs by comparing technical and economical factors, and also taking into account the total cost of individual solutions and cash outflows. It is necessary to consider the investment in its totality and not in separate parts that are independent of one another.”<ref name="PaniTrans10" />
 
This paper breaks the FAME methodology into four activities:
# Identify the constraints and risks of the projects
# Identify user requirements and rank
# Identify and rank all key objectives of the project
# Generate a priority framework to allow comparison of needs and features
 
Their paper includes a formula for generating a score from the information collected. The evaluated system with the highest 'major score', ''Pjtot'', indicates the system selected. While it is a common practice to define an analysis process which condenses all of the information gathered into a single score, I highly caution against blindly accepting such a score. FAME, as well as a number or the other assessment methodologies, is designed for iterative use. The logical purpose of this is to allow the addition of factors initially overlooked into your assessment, as well as to change the weighting of existing factors as you reevaluate their importance. However, this feature means that it is also very easy to unconsciously, or consciously, skew the results of the evaluation to select any system you wish. Condensing everything down into a single value also strips out much of the information that you have worked so hard to gather. Note that you can generate the same result score using significantly different input values. While of value, selecting a system based on just the highest score could potentially leave you with a totally unworkable system.
 
Pani, et al. also describe a FAMEtool to assist in this data gathering and evaluation.<ref name="PaniTheFAMEtool10" /> However a general web search, as well as a review of their FAME papers revealed no indication of how to obtain this resource. While this paper includes additional comparisons with other FLOSS analysis methodologies and there are some hints suggesting that the FAMEtool is being provided as a web service, I have found no URL specified for it. As of now, I have received no responses from the research team via either e-mail or Skype, regarding FAME, the FAMEtool, or feedback on its use.
 
During this same time frame Soto and Ciolkowski also published papers describing the QualOSS Open Source Assessment Model and compared it to a number of the procedures in Stol's and Barbar's table.<ref name="SotoTheQual09">{{cite book |chapter=The QualOSS open source assessment model measuring the performance of open source communities |title=3rd International Symposium on Empirical Software Engineering and Measurement, 2009 |author=Soto, M.; Ciolkowski, M. |publisher=IEEE |year=2009 |pages=498-501 |doi=10.1109/ESEM.2009.5314237 |isbn=9781424448425}}</ref><ref name="SotoTheQualProc09">{{cite book |chapter=The QualOSS Process Evaluation: Initial Experiences with Assessing Open Source Processes |title=Software Process Improvement |author=Soto, M.; Ciolkowski, M. |editor=O'Connor, R.; Baddoo, N.; Cuadrado-Gallego, J.J.; Rejas Muslera, R.; Smolander, K.; Messnarz, R. |publisher=Springer |year=2009 |pages=105–116 |isbn=9783642041334 |doi=10.1007/978-3-642-04133-4_9}}</ref> Their focus was primarily on three process perspectives: product quality, process maturity, and sustainability of the development community. Due to the lack of anything more than a rudimentary process perspective examination, they felt that the following OSS project assessment models were unsatisfactory: QSOS, CapGemni OSMM, Navica OSMM, and OpenBRR. They position QualOSS as an extension of the tralatitious CMMI and SPICE process maturity models. While there are multiple items in the second paper that are worth incorporating into an in-depth evaluation process, they do not seem suitable for what is intended as a quick survey.
 
Another paper, published by Haaland and Groven also compared a number of Open Source quality models. To this paper's credit, the authors devoted a significant amount of space to discussing the different definitions of quality and how the target audience of a tool might affect which definition was used.<ref name="HaalandFree10">{{cite web |url=http://publications.nr.no/directdownload/publications.nr.no/5444/Haaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf |format=PDF |title=Free/Libre Open Source Quality Models - a comparison between two approaches |work=4th FLOSS International Workshop on Free/Libre Open Source Software |author=Haaland, Kirsten; Groven, Arne-Kristian; Glott, Ruediger; Tannenberg, Anna |location=Jena, Germany |pages=1–17 |date=01 July 2010 |accessdate=15 April 2015}}</ref> Like Stohl and Babar, they listed a number of the quality assessment models to choose from, including OSMM, QSOS, OpenBRR, and others. For their comparison, they selected OpenBRR and QualOSS. They appear to have classified OpenBRR as a first generation tool with a “User view on quality” and QualOSS as a second generation tool with a “business point of view”. An additional variation is that OpenBRR is primarily a manual tool while QualOSS is primarily an automated tool. Their analysis in this article clearly demonstrates the steps involve in using these tools and in highlighting where they are objective and subjective. While they were unable to answer their original question as to whether the first or second generation tools did a better job of evaluation, to me they answered the following even more important, but unasked question. As they proceeded through their evaluation, it became apparent as to how much the questions defined in the methods could affect the results of the evaluations. Even though the authors might have considered the questions to be objective, I could readily see how some of these questions could be interpreted in alternate ways. My takeaway is an awareness of the potential danger of using rigid tools, as they can skew the accuracy of the evaluation results depending on exactly what you want the evaluated application to do and how you plan to use it. These models can be very useful guides, but they should not be used to replace a carefully considered evaluation, as there will always be factors influencing the selection decision which did not occur to anyone when the specifications were being written.
 
Hauge, et al. have noted that despite the development of several normative methods of assessment, empirical studies have failed to show wide spread adoption of these methods.<ref name="HaugeAnEmp09">{{cite book |chapter=An empirical study on selection of Open Source Software - Preliminary results |title=ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development, 2009 |author=Hauge, O.; Osterlie, T.; Sorensen, C.-F.; Gerea, M. |publisher=IEEE |year=2009 |pages=42-47 |doi=10.1109/FLOSS.2009.5071359 |isbn=9781424437207}}</ref> From their survey of a number of Norwegian software companies, they have noticed a tendency for selectors to skip the in-depth search for what they call the 'best fit' application and fall back on what they refer to as a 'first fit'. This is an iterative procedure with the knowledge gained from the failure of one set of component tests being incorporated into the evaluation of the next one. Their recommendation is for researchers to stop attempting to develop either general evaluation schemas or normative selection methods which would be applicable to any software application and instead focus on identifying situationally sensitive factors which can be used as evaluation criteria. This is a very rational approach as all situations, even if evaluating the same set of applications, are going to be different, as each user's needs are different.
 
Ayalal, et al. have performed a study to try to more accurately determine why more people don't take advantage of the various published selection methodologies.<ref name="AyalaTowards11">{{cite book |chapter=Towards Improving OSS Products Selection – Matching Selectors and OSS Communities Perspectives |title=Open Source Systems: Grounding Research |author=Ayala, Claudia; Cruzes, Daniela S.; Franch, Xavier; Conradi, Reidar |editor=Hissam, S.; Russo, B.; de Mendonça Neto, M.G.; Kon, F. |publisher=Springer |year=2011 |pages=244–258 |isbn=9783642244186 |doi=10.1007/978-3-642-24418-6_17}}</ref> While they looked at a number of factors and identified several possible problems, one of the biggest factors was the difficulty in obtaining the needed information for the evaluation. Based on the projects they studied, many did not provide a number of the basic pieces of information required for the evaluation, or perhaps worse, required extensive examination of the project web site and documentation to retrieve the required information. From her paper, it sounded as if this issue was more of a communication breakdown than an attempt to hide any of the information, not that this had any impact on the inaccessibility of the information.
 
In addition to the low engagement rates for the various published evaluation methods, another concern is the viability of the sponsoring organizations. One of the assessment papers indicated that the published methods with the smallest footprint, or the easiest to use, appeared to be FAME and the OpenBRR. I have already mentioned my difficulty obtaining additional information regarding FAME, and OpenBRR appears to be even more problematic. BRR was first registered on SourceForge in September of 2005<ref name="OBRRSource">{{cite web |url=https://sourceforge.net/projects/openbrr/ |title=Business Readiness Rating (BRR) |author=Chan, C.; enugroho; Wasserman, T. |publisher=SourceForge |date=17 April 2013 |accessdate=21 April 2015}}</ref>, and an extensive Request For Comments from the founding members of the BRR consortium (SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel Corporation) was released.<ref name="OpenBRROpen2005" /> In 2006, in contrast to typical Open Source development groups, the OpenBRR group announced the formation of an OpenBRR Corporate Community group. Peter Galli's story indicates that "the current plan is that membership will not be open to all."<ref name="GalliOpenBRR06">{{cite web |url=http://www.eweek.com/c/a/Linux-and-Open-Source/OpenBRR-Launches-Closed-OpenSource-Group |title=OpenBRR Launches Closed Open-Source Group |author=Galli, Peter |work=eWeek |publisher=QuinStreet, Inc |date=24 April 2006 |accessdate=13 April 2015}}</ref> He quotes Murugan Pal saying "membership will be on an invitation-only basis to ensure that only trusted participants are coming into the system." However, for some reason, at least some in the group "expressed concern and unhappiness about the idea of the information discussed not being shared with the broader open-source community."<ref name="GalliOpenBRR06" />
 
While the original Business Readiness Rating web site still exists, it is currently little more than a static web page.<ref name="OBRRArch">{{cite web |url=http://www.openbrr.org/ |archiveurl=https://web.archive.org/web/20141224233009/http://www.openbrr.org/ |title=Welcome to Business Readiness Rating: A FrameWork for Evaluating OpenSource Software |publisher=OpenBRR |archivedate=24 December 2014 |accessdate=14 April 2015}}</ref> It appears that some of the original information posted on the site is still there, you just have to know what its URL is to access it, as the original links on the web site have been removed. Otherwise, you may have to turn to the Internet Archive to retrieve some of their documentation. The lack of any visible activity regarding OpenBRR prompted a blog post from one graduate student in 2012 asking "What happened to OpenBRR (Business Readiness Rating for Open Source)?"<ref name="ArjonaWhat12">{{cite web |url=https://larjona.wordpress.com/2012/01/06/what-happened-to-openbrr-business-readiness-rating-for-open-source/ |title=What happened to OpenBRR (Business Readiness Rating for Open Source)? |author=Arjona, Laura |work=The Bright Side |date=06 January 2012 |accessdate=13 April 2015}}</ref>
 
It appears that at some point, any development activity regarding OpenBRR was morphed into OSSpal.<ref name="OSSpalHome">{{cite web |url=http://osspal.org/ |title=Welcome to OSSpal |publisher=OSSpal |accessdate=18 April 2015}}</ref> However, background information on this project is sparse as well. While the site briefly mentions that OSSpal incorporates a number of lessons learned from BRR, there is very little additional information regarding the group or the methods procedures. Their 'All Projects' tab provides a list of over 30 Open Source projects, but the majority simply show 'No votes yet' under the various headings. In fact, as of now, the only projects showing any input at all are for Ubuntu and Mozilla Firefox.
 
===Initial evaluation and selection recommendations===
At this point, we'll take a step back from the evaluation methodologies papers and examine some of the more general recommendations regarding evaluating and selecting FLOSS applications. The consistency of their recommendations may provide a more useful guide for an initial survey of FLOSS applications.
 
In ''TechRepublic'', de Silva recommends 10 questions to ask when selecting a FLOSS application.<ref name="Silva10Quest09">{{cite web |url=http://www.techrepublic.com/blog/10-things/10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise/ |title=10 questions to ask when selecting open source products for your enterprise |author=Silva, Chamindra de |work=TechRepublic |publisher=CBS Interactive |date=20 December 2009 |accessdate=13 April 2015}}</ref> While he provides a brief discourse on each question in his paper to ensure you understand the point of his question, I've collected the 10 questions from his article into the following list. Once we see what overlap, if any, are amongst our general recommendations, we'll address some of the consolidated questions in more detail.
 
# Are the open source license terms compatible with my business requirements?
# What is the strength of the community?
# How well is the product adopted by users?
# Can I get a warranty or commercial support if I need it?
# What quality assurance processes exist?
# How good is the documentation?
# How easily can the system be customized to my exact requirements?
# How is this project governed and how easily can I influence the road map?
# Will the product scale to my enterprise's requirements?
# Are there regular security patches?
 
Similarly, in ''InfoWorld'' Phipps lists seven questions you should have answered before even starting to select a software package.<ref name="Phipps7Quest15">{{cite web |url=http://www.infoworld.com/article/2872094/open-source-software/seven-questions-to-ask-any-open-source-project.html |title=7 questions to ask any open source project |author=Phipps, Simon |work=InfoWorld |publisher=InfoWorld, Inc |date=21 January 2015 |accessdate=10 April 2015}}</ref> His list of questions, pulled directly from his article are:
 
# Am I granted copyright permission?
# Am I free to use my chosen business model?
# Am I unlikely to suffer patent attack?
# Am I free to compete with other community members?
# Am I free to contribute my improvements?
# Am I treated as a development peer?
# Am I inclusive of all people and skills?
 
This list of questions shows a moderately different point of view, as it is not just about someone selecting an Open Source system, but looking to be involved in its direct development. Padin, of 8th Light, Inc., takes the viewpoint of a developer who might incorporate Open Source software into their projects.<ref name="PadinHow14">{{cite web |url=https://blog.8thlight.com/sandro-padin/2014/01/03/how-i-evaluate-open-source-software.html |title=How I Evaluate Open-Source Software |author=Padin, Sandro |publisher=8th Light, Inc |date=03 January 2014 |accessdate=01 June 2015}}</ref> The list of criteria pulled directly from his blog includes:
 
# Does it do what I need it to do?
# How much more do I need it to do?
# Documentation
# Easy to review source code
# Popularity
# Tests and specs
# Licensing
# Community
 
Metcalfe of OSS Watch lists his top tips as<ref name="MetcalfeTop13">{{cite web |url=http://oss-watch.ac.uk/resources/tips |title=Top tips for selecting open source software |author=Metcalfe, Randy |work=OSSWatch |publisher=University of Oxford |date=01 February 2004 |accessdate=23 March 2015}}</ref>:
 
# Reputation
# Ongoing effort
# Standards and interoperability
# Support (Community)
# Support (Commercial)
# Version
# Version 1.0
# Documentation
# Skill setting
# Project Development Development Model
# License
 
In his LIMSexpert blog, Joel Limardo of ForwardPhase Technologies, LLC lists the following as components to check when evaluating an Open Source application<ref name="LimardoDIY13">{{cite web |url=http://www.limsexpert.com/cgi-bin/bixchange/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1363288315&title=DIY%20Evaluation%20Process |title=DIY Evaluation Process |author=Limardo, J. |work=LIMSExpert.com |publisher=ForwardPhase Technologies, LLC |date=2013 |accessdate=07 February 2015}}</ref>:
 
* Check licensing
* Check code quality
* Test setup time
* Verify extensibility
* Check for separation of concerns
* Check for last updated date
* Check for dependence on outdated toolkits/frameworks
 
Perhaps the most referenced of the general articles on selecting FLOSS applications is David Wheeler's How to Evaluate Open Source Software / Free Software (OSS/FS) Programs.<ref name="WheelerHow11">{{cite web |url=http://www.dwheeler.com/oss_fs_eval.html |title=How to Evaluate Open Source Software / Free Software (OSS/FS) Programs |author=Wheeler, David A. |work=dwheeler.com |date=05 August 2011 |accessdate=19 March 2015}}</ref> The detailed functionality to consider will vary with the types of applications being compared, but there are a number of general features that are relevant to almost any type of application. While we will cover them in more detail later, Wheeler categorizes the features to consider as the following:
 
* System functionality
* System cost – direct and in-direct
* Popularity of application, i.e. its market share for that type of application
* Varieties of product support available
* Maintenance of application, i.e, is development still taking place
* Reliability of application
* Performance of application
* Scalability of application
* Usability of application
* Security of application
* Adaptability/customizability of application
* Interoperability of application
* Licensing and other legal issues
 
While a hurried glance might suggest a lot of diversity in the features these various resources suggest, a closer look at the meaning of what they are saying show a repetitive series of concerns. The primary significant differences between the functionality lists suggested is actually due more to how wide a breadth of the analysis process the authors are considering, as well as the underlying features that they are concerned with.
 
With a few additions, the high-level screening template described in the rest of this communication is based on Wheeler's previously mentioned document describing his recommended process for evaluating open source software and free software programs. Structuring the items thus will make it easier to locate the corresponding sections in his document, which includes many useful specific recommendations, as well as a great deal of background information to help you understand the why of the topic. I highly recommend reading it and following up on some of the links he provides. I will also include evaluation suggestions from several of the previously mentioned procedures where appropriate.
 
Wheeler defines four basic steps to this evaluation process, as listed below:
 
* Identify candidate applications
* Read existing product reviews
* Compare attributes of these applications to your needs
* Analyze the applications best matching your needs in more depth
 
Wheeler categorizes this process with the acronym IRCA. In this paper we will be focusing on the IRC components of this process. To confirm the efficacy of this protocol we will later apply it to several classes of Open Source applications and examine the output of the protocol.
 
====Identify needs====
Realistically, before you can perform a survey of applications to determine which ones best match your needs, you must determine what your needs actually are. The product of determining these needs is frequently referred to as the User Requirements Specification (URS).<ref name="VOURS15">{{cite web |url=http://www.validation-online.net/user-requirements-specification.html |title=User Requirements Specification (URS) |work=validation-online.net |publisher=Validation Online |accessdate=08 August 2015}}</ref><ref name="OKeeffeHow15">{{cite web |url=http://www.askaboutgmp.com/296-how-to-create-a-bullet-proof-urs |title=How to Create a Bullet-Proof User Requirement Specification (URS) |author=O'Keefe, Graham |publisher=askaboutgmp |date=01 March 2015 |accessdate=08 August 2015}}</ref> This document can be generated in several ways, including having all of the potential users submit a list of the functions and capabilities that they feel is important. While the requirements document can be created by a single person, it is generally best to make it a group effort with multiple reviews of the draft document and including all of the users who will be working with the application. The reason for this is to ensure that an important requirement is not missed, When a requirement is missed, it is frequently due to the requirement being so basic that it never occurs to anyone that it specifically needed to be included in the requirements document. Admittedly, a detailed URS is not required at the survey level, but it is worth having if only to identify, by their implications, other features that might be significant.
 
Needs will, of course, vary with the type of application you are looking for and what you are planning to do with it. Keep in mind that the URS is a living document, subject to change through this whole process. Developing a URS is generally an iterative process, since as you explore systems, you may well see features that you hadn't considered that you find desirable. This process will also be impacted by whether the application to be selected will be used in a regulated environment. If it is, there will be existing documents that describe the minimum functionality that must be present in the system. Even if it is not to be used in a regulated environment, documents exist for many types of systems that describe the recommended functional requirements that would be expected for that type of system.
 
For a clarifying example, if you were attempting to select a Laboratory Information Management System (LIMS), you can download checklists and standards of typical system requirements from a variety of sources.<ref name="ASTMStandsLIMS">{{cite web |url=http://www.astm.org/Standards/E1578.htm |title=ASTM E1578-13, Standard Guide for Laboratory Informatics |publisher=ASTM International |location=West Conshohocken, PA |doi=10.1520/E1578 |date=2013 |accessdate=14 March 2015}}</ref><ref name="ASURCheck">{{cite web |url=http://www.autoscribeinformatics.com/services/user-requirements |title=User Requirements Checklist |publisher=Autoscribe Informatics |accessdate=10 April 2015}}</ref><ref name="LIIComplete15">{{cite web |url=http://www.limsbook.com/the-complete-guide-to-lims-laboratory-informatics-2015-edition/ |title=The Complete Guide to LIMS & Laboratory Informatics – 2015 Edition |editor=Laboratory Informatics Institute |publisher=LabLynx, Inc |date=2015 |accessdate=10 April 2015}}</ref><ref name="FDAP1115">{{cite web |url=http://www.fda.gov/regulatoryinformation/guidances/ucm125067.htm |title=Part 11, Electronic Records; Electronic Signatures — Scope and Application |publisher=U.S. Food and Drug Administration |date=26 August 2015 |accessdate=10 June 2015}}</ref> These will provide you with examples of the questions to ask, but you will have to determine which ones are important to, or required for, your particular effort.
 
Depending on the use to which this application is to be applied, you may be subject to other specific regulatory requirements as well. Which regulations may vary, since the same types of analysis performed for different industries fall under different regulatory organizations. This aspect is further complicated by the fact that you may be affected by more than one countries' regulations if your analysis are applicable to products being shipped to other countries. While some specific regulations may have changed since its publication, an excellent resource to orient you to the diverse factors that must be considered is Siri Segalstad's book ''International IT Regulations and Compliance''.<ref name="SegalstadInt08">{{cite book |title=International IT Regulations and Compliance: Quality Standards in the Pharmaceutical and Regulated Industries |author=Segalstad, Siri H. |publisher=John Wiley & Sons, Inc |year=2008 |pages=338 |isbn=9780470758823 |url=http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470758821.html}}</ref> My understanding is that an updated version of this book is currently in preparation. Keep in mind that while regulatory requirements that you must meet will vary, these regulations by and large also describe best practices, or at least the minimal allowed practices. These requirements are not put in place arbitrarily (generally) or to make things difficult for you, but to ensure the quality of the data produced. As such, any deviations should be carefully considered, whether working in a regulated environment or not. Proper due diligence would be to determine which regulations and standards would apply to your operation.
 
For a LIMS, an example of following best practices is to ensure that the application has a full and detailed [[audit trail]]. An audit trail allows you to follow the processing of items through your system, determining who did what and when. In any field where it might become important to identify the actions taken during a processing step, an audit trail should be mandatory. While your organization's operations may not fall under the FDA's [[21 CFR Part 11]] regulations, which address data access and security, including audit trails, it is still extremely prudent that the application you select complies with them. If it does not, then almost anyone could walk up to your system and modify data, either deliberately or accidentally, and you would have no idea of who made the changes or what changes they made. For that matter, you might not even be able to tell a change was made at all, which likely will raise concerns both inside and outside of your organization. This would obviously cause major problems if they became a hinge issue for any type of liability law suit.
 
For this screening procedure, you do not have to have a fully detailed URS, but it is expedient to have a list of your make-or-break issues. This list will be used later for comparing systems and determining which ones justify a more in-depth evaluation.
 
====Identify candidates====
To evaluate potential applications against your functional criteria, you must initially generate a list of potential systems. While this might sound easy, generating a comprehensive list frequently proves to be a challenge. When initiating the process, you must first determine the type of system that you are looking for, be it a LIMS, a hospital management system, a data base, et al. At this point, you should be fairly open in building your list of candidates. By that, I mean that you should be careful not to select applications based solely on the utilization label applied to them. The same piece of software can frequently be applied to solve multiple problems, so you should cast a wide net and not automatically reject a system because the label you were looking for hadn't been applied to it. While the label may give you a convenient place to start searching, it is much more important to look for the functionality that you need, not what the system is called. In any case, many times the applied labels are vague and mean very different things to different people.
 
There are a variety of ways to generate your candidate list. A good place to start is simply talking with colleagues in your field. Have they heard of or use a FLOSS application of the appropriate type that they like? Another way is to just flip through journals and trade magazines that cover your field. Any sufficiently promising applications are likely to be mentioned there. Many of the trade magazines will have a special annual issue that covers equipment and software applicable to their field. It is difficult to generate a list of all potential resources, as many of these trade publications are little known outside of their field. Also keep in mind that with the continued evolution of the World Wide Web, many of these trade publications also have associated web sites that you can scan or search. The table below includes just a minor fraction of these sites that are available. (We would welcome the suggestion of any additional resource sites that you are aware of. Please e-mail the fields covered, the resource name, and either its general URL or the URL of the specific resource section to the corresponding editor.)
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! Field
  ! Resource Name
  ! URL
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Astronomy
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Tech Support Alert
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.techsupportalert.com/best-free-astronomy-software.htm http://www.techsupportalert.com/best-free-astronomy-software.htm]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Business intelligence reporting
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Technology Innovation Management Review
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://timreview.ca/article/288 http://timreview.ca/article/288]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Community radio
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Prometheus Radio Project
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://prometheusradio.org/Free_Open_Source_Tools_C_R http://prometheusradio.org/Free_Open_Source_Tools_C_R]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Comprehensive range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Black Duck KnowledgeBase
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://www.blackducksoftware.com/products/knowledgebase https://www.blackducksoftware.com/products/knowledgebase]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Comprehensive range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|The Directory of Open Access Repositories - OpenDOAR
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.opendoar.org/ http://www.opendoar.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Data storage
  | style="background-color:white; padding-left:10px; padding-right:10px;"|InfoStor
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.infostor.com/nas/58-top-open-source-storage-project-1.html http://www.infostor.com/nas/58-top-open-source-storage-project-1.html]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Digital audio editing
  | style="background-color:white; padding-left:10px; padding-right:10px;"|25 Free Digital Audio Editors You Should Know
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.hongkiat.com/blog/25-free-digital-audio-editors/ http://www.hongkiat.com/blog/25-free-digital-audio-editors/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Digital video editing
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Best free video editing software: 20 top programs in 2015
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.techradar.com/us/news/software/applications/best-free-video-editing-software-9-top-programs-you-should-download-1136264 http://www.techradar.com/us/news/software/applications/best-free-video-editing-software-9-top-programs-you-should-download-1136264]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of Android apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia – Android applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_free_and_open-source_Android_applications https://en.wikipedia.org/wiki/List_of_free_and_open-source_Android_applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Launchpad, The Canonical Group
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://launchpad.net/ http://launchpad.net/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Microsoft CodePlex
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://www.codeplex.com/ https://www.codeplex.com/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Hub, Black Duck Software, Inc.
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://www.openhub.net/ https://www.openhub.net/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps (but site is deprecated)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Google Code
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://code.google.com/p/support/ https://code.google.com/p/support/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps (focus on infrastructure)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source Guide by Smile
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.open-source-guide.com/en http://www.open-source-guide.com/en]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps (primarily targeting Samsung products)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Samsung Open Source Release Center
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensource.samsung.com/reception.do http://opensource.samsung.com/reception.do]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of commercial open source apps and services
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia – Commercial open source applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_commercial_open-source_applications_and_services https://en.wikipedia.org/wiki/List_of_commercial_open-source_applications_and_services]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of GNU apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Savannah GNU
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://savannah.gnu.org/ http://savannah.gnu.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of non-GNU apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Savannah Non-GNU
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://savannah.nongnu.org http://savannah.nongnu.org]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of top apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Projects and Applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensource.com/resources/projects-and-applications http://opensource.com/resources/projects-and-applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Drug discovery
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Drug Discovery Today
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.sciencedirect.com/science/article/pii/S1359644605036925 http://www.sciencedirect.com/science/article/pii/S1359644605036925]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Electronic engineering
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Education Engineering (EDUCON), 2010 IEEE
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5492430 http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5492430]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Embroidery
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Machine Embroidery Portal
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.k2g2.org/portal:machine_embroidery http://www.k2g2.org/portal:machine_embroidery]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Enterprise resource planning (ERP)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Comparison of Open Source ERP Systems
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.big.tuwien.ac.at/system/theses/20/papers.pdf?1298476232 http://www.big.tuwien.ac.at/system/theses/20/papers.pdf?1298476232]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Enterprise resource planning (ERP)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source ERP Site
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.open-source-erp-site.com/list-of-open-source-erps.html http://www.open-source-erp-site.com/list-of-open-source-erps.html]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Fashion
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Ethical Fashion Forum - 9 Open-Source & Low Cost Digital Fashion Business Tools
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://source.ethicalfashionforum.com/article/9-open-source-low-cost-digital-fashion-business-tools http://source.ethicalfashionforum.com/article/9-open-source-low-cost-digital-fashion-business-tools]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geographic information systems (GIS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source GIS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensourcegis.org/ http://opensourcegis.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geophysics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geopsy project
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.geopsy.org/ http://www.geopsy.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geophysics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia - Geophysics software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/Comparison_of_free_geophysics_software https://en.wikipedia.org/wiki/Comparison_of_free_geophysics_software]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|GIS for libraries
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Library Hi Tech
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.emeraldinsight.com/doi/abs/10.1108/07378831011026742 http://www.emeraldinsight.com/doi/abs/10.1108/07378831011026742]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Help desk
  | style="background-color:white; padding-left:10px; padding-right:10px;"|CIO
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.cio.com.au/article/320110/5_open_source_help_desk_apps_watch/ http://www.cio.com.au/article/320110/5_open_source_help_desk_apps_watch/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Highly diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Code NASA
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://code.nasa.gov/#/ https://code.nasa.gov/#/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Highly diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|NASA Open Source Software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://ti.arc.nasa.gov/opensource/ http://ti.arc.nasa.gov/opensource/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Laboratory informatics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|LIMSwiki - Laboratory informatics software (open source)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.limswiki.org/index.php/Category:Laboratory_informatics_software_%28open_source%29 http://www.limswiki.org/index.php/Category:Laboratory_informatics_software_%28open_source%29]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Learning management systems (LMS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source LMS – 10 Alternatives to Moodle
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://barrysampson.com/2009/04/08/open-source-lms-10-alternatives-to-moodle/ http://barrysampson.com/2009/04/08/open-source-lms-10-alternatives-to-moodle/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Library science
  | style="background-color:white; padding-left:10px; padding-right:10px;"|FOSS4LIB
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://foss4lib.org/packages https://foss4lib.org/packages]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Limited range of iOS apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia - Open-source iOS applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_free_and_open-source_iOS_applications https://en.wikipedia.org/wiki/List_of_free_and_open-source_iOS_applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medical and laboratory informatics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Healthcare Freeware
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.healthcarefreeware.com/lim.htm http://www.healthcarefreeware.com/lim.htm]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medical office records
  | style="background-color:white; padding-left:10px; padding-right:10px;"|ZDNet
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.zdnet.com/article/free-and-open-source-healthcare-software-for-your-practice/ http://www.zdnet.com/article/free-and-open-source-healthcare-software-for-your-practice/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|'Open' Health IT Solutions for the Public & Private Sectors
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://drive.google.com/file/d/0B_TuX9zE68eWcVM1dkdDQTVCdnM/edit https://drive.google.com/file/d/0B_TuX9zE68eWcVM1dkdDQTVCdnM/edit]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|50 Successful Open Source Projects That Are Changing Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://nursingassistantguides.com/2009/50-successful-open-source-projects-that-are-changing-medicine/ http://nursingassistantguides.com/2009/50-successful-open-source-projects-that-are-changing-medicine/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Datamation
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.datamation.com/open-source/50-open-source-replacements-for-health-care-software-1.html http://www.datamation.com/open-source/50-open-source-replacements-for-health-care-software-1.html]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medfloss.org
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.medfloss.org/ http://www.medfloss.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|MIDI controllers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source MIDI Controllers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://punkmanufacturing.com/wiki/open-source-midi-controllers http://punkmanufacturing.com/wiki/open-source-midi-controllers]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Music
  | style="background-color:white; padding-left:10px; padding-right:10px;"|20 great free and open source music making programs
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.musicradar.com/tuition/tech/20-great-free-and-open-source-music-making-programs-582934 http://www.musicradar.com/tuition/tech/20-great-free-and-open-source-music-making-programs-582934]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Pattern management for textiles
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Tools for computer generated patterns
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://xxxclairewilliamsxxx.wordpress.com/tools-to-create-and-explore-digital-patterns/ https://xxxclairewilliamsxxx.wordpress.com/tools-to-create-and-explore-digital-patterns/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Personal information management
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia - Personal information managers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_personal_information_managers#Open_source_applications https://en.wikipedia.org/wiki/List_of_personal_information_managers#Open_source_applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Photography
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source Photography
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensourcephotography.org/software-list http://opensourcephotography.org/software-list]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Project management
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source Photography
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensourcephotography.org/software-list http://opensourcephotography.org/software-list]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Repository management
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Free and open-source repository software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://oad.simmons.edu/oadwiki/Free_and_open-source_repository_software http://oad.simmons.edu/oadwiki/Free_and_open-source_repository_software]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Software test management
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Software Testing Help - 15 Best Test Management Tools for Software Testers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.softwaretestinghelp.com/15-best-test-management-tools-for-software-testers/ http://www.softwaretestinghelp.com/15-best-test-management-tools-for-software-testers/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Survey and feedback
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Capterra - Top survey software products
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.capterra.com/survey-software/ http://www.capterra.com/survey-software/]
|-
|}
|}
<blockquote>'''Table 2.''': Examples of focused FLOSS resource sites available on the web</blockquote>
[[File:Fig2 Joyce 2015.png|489px|right|thumb|'''Illustration 2.''': Result page displayed on Black Duck Software's Open Hub site for the Bika Open Source LIMS 3 project.]]
I also recommend checking some of the general open source project lists, such as the ones generated by Cynthia Harvey at Datamation<ref name="HarveyData15">{{cite web |url=http://www.datamation.com/author/Cynthia-Harvey-6460.html |title=More articles by Cynthia Harvey |work=Datamation |publisher=QuinStreet, Inc |date=2015 |accessdate=12 April 2015}}</ref>, which has been covering the computer and data-processing industry since 1957. In particular, you might find their article "Open Source Software List: 2015 Ultimate List" useful. It itemizes over 1,200 open source applications, including some in categories that I didn't even know existed.<ref name="HarveyOpen15">{{cite web |url=http://www.datamation.com/open-source/open-source-software-list-2015-ultimate-list-1.html |title=Open Source Software List: 2015 Ultimate List |author=Harvey, Cynthia |work=Datamation |publisher=QuinStreet, Inc |date=05 January 2015 |accessdate=12 April 2015}}</ref>
 
It would also be prudent to search the major open source repositories, such as SourceForge<ref name="SFHome">{{cite web |url=https://sourceforge.net/ |title=SourceForge - Download, Develop and Publish Free Open Source Software |work=SourceForge |publisher=Slashdot Media |date=2015 |accessdate=14 June 2015}}</ref> and GitHub.<ref name="GHHome">{{cite web |url=https://github.com/ |title=GitHub: Where software is built |work=GitHub |publisher=GitHub, Inc |date=2015 |accessdate=14 June 2015}}</ref> ''Wikipedia'' includes a comparison of source code hosting facilities that would be worth reviewing as well.<ref name="WPSourceRepos">{{cite web |url=https://en.wikipedia.org/w/index.php?title=Comparison_of_source_code_hosting_facilities&oldid=682090863 |title=Comparison of source code hosting facilities |work=Wikipedia |publisher=Wikimedia Foundation |date=21 September 2015 |accessdate=28 September 2015}}</ref> Keep in mind that you will need to be flexible with your search terms, as the developers might be looking at the application differently than you are. While they were created for a different purpose, an examination of the books in ''The Architecture of Open Source Applications'' might prove useful as well.<ref name="AOSAHome">{{cite web |url=http://aosabook.org/en/index.html |title=The Architecture of Open Source Applications |work=AOSA |publisher=AOSA Editors |date=2015 |accessdate=08 October 2015}}</ref> Other sites where you might find interesting information regarding new Open Source applications, are the various OpenSource award sites, such as the ''InfoWorld'' Best of Open Source Software Awards, colloquially known as the Bossies.<ref name="IW5Key15">{{cite web |url=http://www.infoworld.com/article/2986769/open-source-tools/5-key-trends-in-open-source.html |title=5 key trends in open source  |author=Knorr, Eric |work=InfoWorld |publisher=InfoWorld, Inc |date=28 September 2015 |accessdate=28 September 2015}}</ref>
 
When searching the web, don't rely on just Google or Bing. Don't forget to checkout all of the journal web sites, such as SpringerLink, Wiley, ScienceDirect, PubMed, and others, as they contain a surprising amount of information on FLOSS. If you don't wish to search each of them individually, there are other search engines out there which can give you an alternate view of the research resources available. To name just two, be sure to try both Google Scholar<ref name="GSHome">{{cite web |url=https://scholar.google.com/ |title=Google Scholar |publisher=Google, Inc |date=2015 |accessdate=08 August 2015}}</ref> and Microsoft Academic Search.<ref name="MASHome">{{cite web |url=http://academic.research.microsoft.com/ |title=Microsoft Academic Search |publisher=Microsoft Corporation |date=2015 |accessdate=08 August 2015}}</ref> These tools can also be used to search for masters theses and doctoral dissertations, which likewise contain a significant amount of information regarding open source.
 
While working on creating your candidate list, be sure to pull any application reviews that you come across.<ref name="WasikeSelect10">{{cite web |url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.227.5951 |title=Selection Process of Open Source Software Component |author=Wasike, Sylvia Nasambu |date=October 2010 |accessdate=10 August 2015}}</ref> If done well, these reviews can save you a significant amount of time in screening a potential system. However, unless you're familiar with the credentials of the author, be cautious of relying on them for all of your information. While not common, people have been known to post fake reviews online, sometimes when it is not even April 1st! Another great resource, both for identifying projects and obtaining information about them, is Open HUB<ref name="OHHome">{{cite web |url=https://www.openhub.net/ |title=Open HUB, the open source network |work=Open HUB |publisher=Black Duck Software, Inc |date=2015 |accessdate=10 August 2015}}</ref>, an Open Source web site maintained by Black Duck Software, Inc. Open Hub allows you to search by Projects, People, Organizations, Forums, and Code. For example, if I searched for [[Bika LIMS]], it would currently return the results for Bika Open Source LIMS 3 along with some basic information regarding the system. If I were to click on the projects name, a much more detailed page regarding this project is displayed. Moving your mouse cursor over the graphs displays the corresponding information for that date.
 
Once a list of candidate applications has been generated, the list entries must be compared. Some of this comparison can be performed objectively, but it also requires subjective analysis of some components. As Stol and Babar have shown, there is no single recognized procedure for either the survey or detailed comparison of FLOSS applications that has shown a marked popularity above the others.
 
The importance of any one specific aspect of the evaluation will vary with the needs of the organization. General system functionality will be an important consideration, but specific aspects of the contained functionality will have different values to different groups. For instance, interoperability may be very important to some groups, while others may be using this application as their only data system and they have no interest in exchanging data files with others, so interoperability is not a concern to them. While you can develop a weighting system for different aspects of the system, this can easily skew selection resulting in a system that has a very good rating, yet is unable to perform the required function. Keep in mind that this is a high level survey, we are asking broad critical questions, not attempting to compare detailed minutia. Also keep in mind that a particular requirement might potentially fall under multiple headings. For example, compliance with 21 CFR Part 11 regulations might be included under functionality or security.
 
====Screening protocol====
While in-depth analysis of the screened systems will require a more detailed examination and comparison, for the purpose of this initial survey a much simpler assessment protocol will serve. While there is no single 'correct' evaluation protocol, something in the nature of the three leaf scoring criteria described for QSOS<ref name="Deprezcomp08" /> should be suitable. Keep in mind that for this quick assessment we are using broad criteria, so both the criteria and the scoring will both be more ambiguous than that required for an in-depth assessment. Do not be afraid to split any of these criteria up into multiple finer classes of criteria if the survey requires it. This need would be most likely to occur under System Functionality, as that is where most people's requirements greatly diverge.
 
In this process, we will assign one of three numeric values to each of the listed criteria. A score of zero indicates that the system does not meet the specified criteria. A score of one indicates that the system marginally meets the specified criteria. You can look on this as the feature is present and workable, but not to the degree you'd like it. Finally, a score of two indicates that the system fully meets or exceeds the specified requirement. In the sections below I will list some possible criteria for this table. However, you can adjust these descriptions or add a weighting factor, as many other protocols do, to adjust for the criticality of a given requirement.
 
Realistically, when it comes down to some of your potential evaluation criteria, the actuality is that for some of them, you can compensate for the missing factor in some other way. For other criteria, their presence or absence can be a drop-dead issue. That is, if the particular criteria or feature isn't present, then it doesn't matter how well any of the other criteria are ranked, that particular system is out of consideration. Deciding which, if any, criteria are drop-dead items should ideally be determined before you start your survey. This will not only be more efficient, in that it will allow you to cut off data collection at any failure point, but it will also help dampen the psychological temptation to fudge your criteria, retroactively deciding that a given criteria was not that important after all.
 
At this stage we are just wanting to reduce the number of systems for in-depth evaluation from potentially dozens, to perhaps three or four. As such, we will be refining our review criteria later, so if something isn't really a drop-dead criteria, don't mark it so. It's amazing the variety of feature tradeoffs people tend to make further down the line.
 
===In-depth evaluation===
====System functionality====
While functionality is a key aspect of selecting a system, its assessment must be used with care. Depending on how a system is designed, a key function that you are looking for might not have been implemented, but in one system it can easily be added, while in another it would take a complete redesign of the application. Also consider the possibility of whether this function must be intrinsic to the application or if you can pair the application being evaluated with another application to cover the gap.
 
In most cases, you can obtain much of the functionality information from the projects web site, or occasionally, web sites. Some projects have multiple web sites, usually with one focused on project development and another targeting general users or application support. There are two different types of functionality to be tested. The first might be termed "general functionality" that would apply to almost any system. Examples of this could include the following:
 
* User authentication
* Audit trail (I'm big on detailed audit trails, as they can make the difference between between being in regulatory compliance and having a plant shut down. Even if you aren't required to have them, they are generally a good idea, as the records they maintain may be the only way for you to identify and correct a problem.)
* Sufficient system status display that the user can understand the state of the system
* Ability to store data in a secure fashion
 
We might term the second as "application functionality." This is functionality specifically required to be able to perform your job. As the subject matter expert, ''you'' will be the one to create this list. Items might be as diverse as the following:
 
For a laboratory information management system...
* Can it print bar coded labels?
* Can it track the location of samples through the laboratory, as well as maintain the chain-of-custody for the sample?
* Can it track the certification of analysts for different types of equipment, as well as monitor the preventive maintenance on the instruments?
* Can it generate and track aliquots of the original sample?
 
For a [[geographic information system]] (GIS)<ref name="NetelerGRASS12">{{cite journal |title=GRASS GIS: A multi-purpose open source GIS |journal=Environmental Modelling & Software |author=Netelera, Markus; Bowmanb, M. Hamish; Landac, Martin; Metz, Markus |volume=31 |pages=124–130 |year=May 2012 |doi=10.1016/j.envsoft.2011.11.014}}</ref>...
* What range of map projections is supported?
* Does the system allow you to create custom parameters?
* Does it allow import of alternate system formats?
* Can it directly interface with geographic positioning devices (GPS)?
 
For a library management software system (LMSS) (or integrated library system [ILS], if you prefer, or even library information management system [LIMS]; have you ever noticed how scientists love to reuse acronyms, even within the same field?)...
* Can you identify the location of any item in the collection at a given time?
* Can you identify any items that were sent out for repair and when?
* Can it identify between multiple copies of an item and between same named items on different types of media?
* Can it handle input from RFID tags?
* Can it handle and differentiate different clients residing at the same address?
* If needed, can it correlate the clients age with the particular item they are requesting, in case you have to deal with any type of age appropriate restrictions?
** If so, can it be overridden where necessary (maintaining the appropriate records in the audit trail as to why the rule was overridden)?
 
For an archival record manager (This classification can cover a lot of ground due to all of the different ways that "archival" and "record" are interpreted.)...
* In some operations, a record can be any information about a sample, including the sample itself. By regulation, some types of information must be maintained essentially for ever. In others, you might have to keep information for five years, while you have to maintain data for another type for 50 years.
** Can the application handle tracking records for different amounts of time?
** Does the system automatically delete these records at the end of the retention period or does it ask for confirmation from a person?
** Can overrides on a particular sample be applied so that records are not allowed to be deleted, either manually of because they are past their holding date, such as those that might be related to any litigation, while again maintaining all information about the override and who applied the override in the audit trail?
* In other operations, an archival record manager may actually refer to the management of archival records, be these business plans, architectural plans, memos, art work, etc.
** Does the system keep track of the type of each record?
** Does the system support appropriate meta data on different types of records?
*** Does it just record the items location and who is responsible for it, such as a work of art?
*** If a document, does it just maintain an electronic representation of a document, such as a PDF file, or does it record the location of the original physical document, or can it do both?
*** Can it manage both permanently archived items, such as a work of art or a historically significant document, and more transitory items, where your record retention rules say to save it for five, 10, 15 years, etc., and then destroy it?
**** In the latter case, does the system require human approval before destroying any electronic documents or flagging a physical item for disposal? Does it require a single human sign-off or must a chain of people sign-off on it, just to confirm that it is something to be discarded by business rules and not an attempt to hide anything?
 
For medical records<ref name="KhanineMean15">{{cite web |url=http://www.toadworld.com/platforms/oracle/b/weblog/archive/2015/05/07/quot-meaningful-use-quot-regulations-of-medical-information-in-health-it |title="Meaningful Use" Regulations of Medical Information in Health IT |author=Khanine, Dmitri |work=Toad World - Oracle Community |publisher=Dell Software, Inc |date=07 May 2015 |accessdate=12 June 2015}}</ref><ref name="KhanineOpen15">{{cite web |url=http://www.toadworld.com/platforms/oracle/b/weblog/archive/2015/05/06/open-source-medical-record-systems-of-2015 |title=Open-Source Medical Record Systems of 2015 |author=Khanine, Dmitri |work=Toad World - Oracle Community |publisher=Dell Software, Inc |date=06 May 2015 |accessdate=28 May 2015}}</ref>...
* This is a challenging quagmire, with frequently changing regulations and requirements. Depending on how you want to break it down, this heading can be segmented into two classes: [[electronic medical record]]s (EMR) which can constitute an electronic version of the tracking of the patients health and [[electronic health record]]s (EHR) which contains extensive information on the patient, including test results, diagnostic information, and other observations by the medical staff. For those who want to get picky, you can also subdivide the heading into imaging systems, such as X-rays and CAT scans and other specialized systems.
** Can all records be accessed quickly and completely under emergency situations?
** What functionality is in place to minimize the risk of a [[Health Insurance Portability and Accountability Act]] (HIPAA) violation?
** What functionality exists for automated data transfer from instruments or laboratory data systems to minimize transcription errors.
** How are these records integrated with any billing or other medical practice system?
*** If integrated with the EMR and EHR record systems, does this application apply granular control over who can access these records and what information they are able to see?
 
For an enterprise resource planning (ERP) system...
* Davis has indicated that a generally accepted definition of an ERP system is a "complex, modular software integrated across the company with capabilities for most functions in the organization."<ref name="DavisEnt08">{{cite book |chapter=Enterprise Resource Planning Under Open Source Software |title=Enterprise Resource Planning for Global Economies: Managerial Issues and Challenges |author=Davis, Ashley |editor=Ferran, Carlos; Salim, Ricardo |publisher=IGI Global |location=Hershey, PA |year=2008 |doi=10.4018/978-1-59904-531-3.ch004 |isbn=9781599045313}}</ref> I believe this translates as "good luck," considering the complexity of the systems an ERP is designed to model and all of the functional requirements that go into that. It is perhaps for that reason that successful ERP implementations generally take several years.
** ERP systems generally must be integrated with other informatics systems within the organization. What types of interfaces does this system support?
*** Is their definition of plug-and-play that you just have to configure the addresses and fields to exchange?
*** Is their definition of interface that the system can read in information in a specified format and export the same, leaving you to write the middle ware program to translate the formats between the two systems?
 
From the above, it is easy to see why researchers have encountered difficulty in developing a fixed method that can be used to evaluate anything. At this point it is quite acceptable to group similar functions together, as this is a high level survey to identify which systems will definitely ''not'' be suitable, so we can focus our researches on those that might be. Many researchers, such as Sarrab and Rehman, summarize system functionality as "achieving the user's requirements, correct output as user's expectations and verify that the software functions appropriately as needed."<ref name="SarrabSel13">{{cite journal |title=Selection Criteria of Open Source Software: First Stage for Adoption |journal=International Journal of Information Processing and Management |author=Sarrah, Mohamed; Rehman, Osama M. Hussain |volume=4 |issue=4 |pages=51–58 |year=2013 |doi=10.4156/ijipm.vol4.issue4.6}}</ref>
 
Suggested ratings are:
* Zero – Application does not support required functionality.
* One – Application supports the majority of functionality to at least an useable extent.
* Two – Application meets or exceeds all functional requirements.
 
====Community====
Many of the researchers that I've encountered have indicated that Community is the most critical factor of a FLOSS project. There are a number of reasons for this. First, the health and sustainability of a FLOSS project is indicated by a large and diverse community that is both active and responsive.<ref name="Silva10Quest09" /> Additionally, the core programmers in a FLOSS project are generally few, and it is the size of the project community that determines how well reviewed the application is, ensuring quality control of the projects code. Finally, the size of the project community correlates with the lifetime of the project.
 
Suggested ratings are:
* Zero – No community exists. No development activity is observable. Project is dead.
* One - Community is small and perhaps insular. May consist of just one or two programmers with perhaps a small number of satellite users.
* Two – Community is large and dynamic, with many contributors and active communication between the core developers and the rest of the community. Community is responsive to outside inquiries.
 
====System cost====
Since the main goal of this survey procedure is to evaluate FLOSS products, the base system cost for the software will normally be low, frequently $0.00. However, this is not the only cost you need to consider. Many of these costs could potentially be placed under multiple headings, depending on how your organization is structured. No matter how it itemizes them, there will be additional costs. Typical items to consider include the following:
* Cost of supporting software – e.g. Does it require a commercial data base such as Oracle or some other specialized commercial software component?
* Cost of additional hardware – e.g. Does the system require the purchase of additional servers or storage systems? Custom hardware interfaces?
* Cost of training – e.g. How difficult or intuitive is the system to operate? This will impact the cost of training that users must receive. Keep in mind this cost will exist whether you are dealing with an Open Source or proprietary system. Are costs for system manuals and other required training material included? Some proprietary systems don't, or might perhaps send a single hard copy of the manuals. Who will perform the training? Whether you hire someone from outside or have some of your own people do it, there will be a cost, as you would be pulling people away from from their regular jobs.
* Cost of support – e.g. Is support through a commercial organization or the Open Source development group? If the former, what are their contract costs? While harder to evaluate, what is the turnaround time from when you request the support? Immediate? Days? Sometime? The amount of time you have to wait for a problem to be fixed, is definitely a 'cost', whether it means your system is dead in the water or just not as efficient and productive as it could be.
 
The primary issue here is to be realistic in your evaluation. It's hard to believe that anyone would assume that there were no associated costs with using FLOSS, or proprietary software for that matter, but apparently there are. Foote does a good job of exploring and disproving this belief, showing all of the items that go into figuring the total cost of ownership (TCO), which should be representative of FLOSS applications.<ref name="FooteTheMyth10">{{cite journal |title=The Myth of Free: The Hidden Costs of Open Source Software |journal=Dalhousie Journal of Interdisciplinary Management |author=Foote, Amanda |volume=6 |issue=Spring 2010 |pages=1–9 |year=2010 |doi=10.5931/djim.v6i1.31}}</ref> I wouldn't call them hidden costs, at least not with the FLOSS systems, just costs that are overlooked, as are so many other things when people focus on a single central item. To quote Robert Heinlein, "TANSTAAFL!" (There ain't no such thing as a free lunch!)<ref name="HeinleinMoon66">{{cite book |author=Heinlein, Robert A. |title=The Moon Is a Harsh Mistress |publisher=Tom Doherty Associates |location=New York, NY |year=1997 |origyear=1966 |pages=8–9 |isbn=9780312863555}}</ref>
 
The important thing here is to pay attention to all of the interactions taking place. For example, if you wished to interface a piece of equipment to your FLOSS application, remember to factor in the cost of the interface. Despite what some advertisers think, data bits don't just disappear from one place and magically appear in another. It is very easy to lose track of where the costs are. Keep in mind that many items, such as training, you will have a cost either way you go. If a proprietary vendor says they will provide free training, you can be assured that the cost for it is included in the contract. But if you take that route, be very careful to read the contract thoroughly, as not all vendors include any training at all. It would be very easy to end up having to pay for "optional" training.
 
Suggested ratings are:
* Zero – Installation and support costs are excessive and greatly exceed any available budget.
* One – System may require purchase of additional hardware or customization. These costs, along with training costs, are within potential budget.
* Two – Installation and support costs are relatively minor, with no additional hardware required. System design is relatively intuitive with in-depth documentation and active support from the community.
 
====Popularity====
This heading can be somewhat confusing in terms of how it is interpreted, even though most of the recommendations we looked at include it. Popularity is sometimes considered to be similar to market share. That is, of the number of people using a specific application in a given class of Open Source applications, what percentage do they represent out of all people running applications in the class? If the majority of people are using a single system, this might indicate that it is the better system, or it might just indicate that other systems are newer and, even if potentially better, people haven't migrated over to them yet. An alternate approach to examining it is to ask how many times the application has been downloaded. In general, the larger the market share or the number of downloads, the more likely that a given product is to be usable. This is not an absolute, as people may have downloaded the application for testing and then rejected it or downloaded it simply to game the system, but it is a place to start. The point of this question usually isn't really to determine how popular a particular application is, but rather to ensure that it is being used and it is a living, as opposed to an abandoned, project. Be leery of those applications with just a few downloads. If there is a large group of people using the application, there s a higher probability that the application works as claimed.
 
At the same time, learning who some of the other users of this application are can give you some perception of how well it actually works. As Silva reminds us, "the best insight you can get into a product is from another user who has been using it for a while."<ref name="Silva10Quest09" />
 
Suggested ratings are:
* Zero – No other discernible users or reality of listed users is questionable.
* One – Application is being used by multiple groups, but represents only a small fraction of its "market." Appears to be little 'buzz' about the application.
* Two – Application appears to be widely used and represents a significant fraction of its "market." This rating is enhanced if listed users include major commercial organizations.
 
====Product support====
Product support can be a critically important topic for any application. Whether you are selecting a proprietary application or a FLOSS one, it is vitally important to ensure that you will have reliable support available. Just because you purchased a proprietary program will not ensure that you have the support you need. Some vendors include at least limited support in their contracts, others don't. However, over the years I've found that even purchasing a separate support contract doesn't ensure that the people who answer the phone will be able to help you. When making the final decision, don't make assumptions: research!
 
Support can be broken down into several different sub-categories:
* User Manuals – Do they exist? What is their quality?
* System Managers Manuals – Do they exist? What is their quality?
* Application Developers Manuals – Do they exist? What is their quality?
* System Design Manuals – Do they exist? What degree of detail do they provide?
* For data base related projects, is an accurate and detailed entity relationship (ERD) diagram included?
* Have any third party books been written about this application? Are they readily available, readable, and easy to interpret?
* Is product support provided directly by the application development community?
* Is product support provided by an independent Users Group, separate from the development group?
* Is commercial product support available?
** If so, what are their rates?
** Are on-site classes available?
**  Are on-line training classes available?
* A frequently overlooked type of product support is how well documented the program code is. Are embedded comments sparse or frequent and meaningful? Are the names of program variables and functions arbitrary or meaningful?
 
Another factor in evaluating product support is whether you have anyone on your team with the expertise to understand it. That is not a derogatory statement, depending on the issue, someone might have to modify an associated data base, the application code, or the code in a required library, whether to correct an error or add functionality. Do any of your people have expertise in that language? Would someone have to start from scratch or do you have the budget to hire an outside consultant? Even if you have no desire to modify the code, having someone on the project that understands the language used can be a big help in discerning how the program works, as well as determining how meaningful the program comments are.
 
Suggested ratings are:
* Zero – Limited or no support available. Documentation essentially non-existent, source code minimally documented, no user group support, and erratic response from the developers.
* One – Documentation scattered and of poor quality. Support from user group discouraged and no commercial support options exist.
* Two - Excellent documentation, including user, system manager, and developer documentation. Enthusiastic support from the user community and developers. Commercial third-part support available for those desiring it. Third party books may also have been released documenting the use of this product.
 
====Maintenance====
This is another item that can be interpreted in multiple ways. One way to look at it is how quickly the developers respond to any bug report. Depending on the particular FLOSS project, you may actually be able to review the problem logs for the system and see what the average response time was between a bug being reported and the problem resolved. In some cases this might be hours or days, in others it is never resolved. To be fair, don't base your decision on a single instance in the log file, as some bugs are much easier to find and fix than others. However, a constant stream of open bugs or bugs that have only been closed after months or years should make you leery.
 
To others this question is to determine whether development is still taking place on the project or if it is dead. Alternately, it is like asking if anybody is maintaining the system and correcting bugs when they are discovered. There are several ways of addressing this issue. Examining the problem logs described above is one way of checking for project activity, another is the check the release dates for different versions of the application. Are releases random or on a temporal schedule? How long has it been since the application was last updated? If the last release date was over a year or two ago, this is cause for concern and should trigger a closer look. Just because there hasn't been a recent release does not mean that the project has been abandoned. If the development of the app has advanced to the point where it is stable and no other changes need to be made you may not see a recent release because none is needed. However, the later is very rare, both because bugs can be so insidious and because a lot of programmers can't resist just tweaking things, to make them a wee bit better. If a project is inactive, but everything else regarding the project looks good, it might be possible to work with the developers to revitalize it. While this course of action is feasible, it is important to realize that it is taking on a great deal of responsibility and an unknown amount of expense. The latter is particularly true as you may be having to assign one or more developers to work on the project full time.
 
Suggested ratings are:
* Zero – No releases, change log activity, or active development discussion in message forums in over two years.
* One – No releases, change log activity or active development discussions in message forums for between one and two years.
* Two – A new version has been released within the year, change logs show recent development activity, and there is active development discussion in the message forums.
 
====Reliability====
Reliability is the degree to which you can rely on the application to function properly. Of course, the exact definition becomes somewhat more involved. The reliability of a system is defined as the ability of an application to operate properly under a specified set of conditions for a specified period of time. Fleming states that "One aspect of this characteristic is fault tolerance that is the ability of a system to withstand component failure. For example if the network goes down for 20 seconds then comes back the system should be able to recover and continue functioning."<ref name="FlemingISO14">{{cite web |url=http://www.sqa.net/iso9126.html |title=ISO 9126 Software Quality Characteristics |author=Fleming, Ian |work=SQA Definition |year=2014 |accessdate=18 June 2015}}</ref>
 
Because of its nature, the reliability of a system is hard to measure, as you are basically waiting to see how frequently it goes down. While we'd like to aim for never, one should probably be satisfied if the system recovered properly after the failure. In most instances, unless you are actually testing a system under load, the best that you can hope for is to observe indicators from which you can infer its reliability. As a generality, the more mature a given code base is, the more reliable it is, but keep in mind that this is a generality, there are always incidents that can occur to destabilize every thing. Unfortunately, it frequently feels as if the problem turns out to be something that you would swear was totally unrelated. Face it, Murphy is just cleverer than you.<ref name="TaylorMurphy">{{cite web |url=http://davetgc.com/Murphys_Law.html |title=Murphy's Laws |author=Taylor, Dave |work=Dave Taylor's Educational & Guidance Counseling Services |year=2015 |accessdate=08 August 2015}}</ref>
 
Wheeler also reminds us that "Problem reports are not necessarily a sign of poor reliability - people often complain about highly reliable programs, because their high reliability often leads both customers and engineers to extremely high expectations."<ref name="WheelerHow11" /> One thing that can be very reassuring is to see that the community takes reliability seriously by continually testing the system during development.
 
Suggested ratings are:
* Zero – Error or bug tracking logs show a high incident of serious system problems. Perhaps worse, no logs of reported problems are kept at all, particularly for systems that have been in release for less than a year.
* One – Error logs show relatively few repeating or serious problems, particularly if these entries correlate with entries in the change logs indicating that a particular problem has been corrected. System has been in release for over a year.
* Two - Error logs are maintained, but show relatively few bug reports, with the majority of them being minor. A version of the system, using the same code base, has been in release for over two years. Developers both distribute and run a test suite to confirm proper system operation.
 
====Performance====
Performance of an application is always a concern. Depending on what the application is trying to do and how the developers coded the functions, you may encounter a program that works perfectly, but is just too unresponsive to use. Sometimes this is a matter of hardware, other times it is just inefficient coding, such as making sequential calls to a data base to return part of a block of data, rather than making a single call to return all of the block at once. Performance and Scalability are usually closely linked.
 
You might be able to obtain some information on the systems actual performance from the project web site, but it is hard to tell if this is for representative or selected data. Reviewing the project mailing list may provide a more accurate indication of the systems performance or any performance problems encountered. Testing the system under your working conditions is the only way to make certain what the systems actual performance is. Unfortunately, the steps involved in setting up such a test system require much more effort than a high level survey will allow. If any user reviews exist, they may give an insight into the systems performance. Locating other users through the project message board might be a very useful resource as well, particularly if they handle the same projected work loads that you are expecting.
 
It is difficult to define performance ratings without having knowledge of what the application is supposed to do. However, for systems that interact with a human operator, the time lapse between when a function is initiated and when the system responds can be suggestive. If the project maintains a test suite, particularly one containing sample data, reviewing its processing time can give an insight to the systems performance as well. Response delays of even a few seconds in frequently executed functions will not only kill the overall process performance, but result in users resistive to using the system.
 
Suggested ratings are:
* Zero – A system designed to be interactive fails to respond in an acceptable time frame. For many types of applications it is reasonable to expect an almost instantaneous response, particularly for screen management functions. It is not reasonable for a system to take over a minute, or even 5 seconds to switch screens or acknowledge an input, particularly in regards to frequently executed functions such as results entry, modification, or review.. A system that batch processes data maxes out under data loads below that of your current system.
* One – A system designed to be interactive appears to lag behind human entry for peripheral functions, but frequently accessed functions, such as results entry, modification, or review appear to respond almost instantaneously. A system that batch processes data maxes out under your existing data loads.
* Two – System is highly responsive, showing no annoying delayed responses. A system that batch processes data can process several times your current data load before maxing out.
 
====Scalability====
Scalability ensures that the application will operate over the data scale range you will be working with. In general, it means that if you test the system functionality with a low data load, the application will "scale up" to handle larger data loads. This may be handled by expanding from a single processor to a larger cluster or parallel processing system. Note that for a given application, throwing more hardware at it may not resolve the problem, as the application needs to be designed to take advantage of that additional hardware. Another caveat is to carefully examine the flow of data through your system. The processor is not the only place you can encounter road blocks limiting scalability. Other possibilities include how quickly the system can access the needed data. If the system is processing the data faster than it can access it, adding more computer power will not resolve the problem. The limiting issues might be the bandwidth of your communication lines, the access speed of the devices that the data is stored on, or contention for needed resources with other applications. As with many aspects of selecting a system and getting it up and running, making assumptions is the real killer.
 
Silva indicates that many Open Source applications are built on the LAMP (Linux, Apache, MySQL, PHP/Perl/Python) technology stack and that this is one of the most scalable configurations available.<ref name="Silva10Quest09" /> However, you should ensure that there is evidence that the application has been successfully tested that way. While performing surveys and testing is never a good time to start making assumptions. A look at the applications user base will likely identify someone who can provide this feedback.
 
Suggested ratings are:
* Zero – System does not support scaling, whether due to application design or restriction of critical resources, such as rate of data access.
* One – System supports limited scaling, but overhead or resource contention, such as a data bottleneck, results in a quick performance fall off.
* Two - System is balanced and scales well, supporting large processing clusters or cloud operations, without any restrictive resource pinch-points.
 
====Usability====
Usability means pretty much what it says. The concern here is not how well the program works, but rather, how easy is it to learn and use. The interface should be clear, intuitive, and help guide the user through the programs operation. Despite Steve Jobs, there is a limit to how intuitive an interface can be, thus the operation of the interface should be clearly documented in the user manual. Ideally the system will support a good context sensitive help system as well. The best help systems may also provide multimedia support, so that the system can actually show you how something should be done, rather than trying to tell you. I've found that frequently a good video can be worth well more than a thousand words! No matter how much time is spent writing the text for a manual or help system, it will always be unclear to somebody, if only because of the diversity of the backgrounds of people using it.
 
The interface between the operator and the computer may vary with the purpose of the application. While with new applications you are more likely to encounter a graphic user interface (GUI), there are still instances where you may encounter a command-line interface. Both types of interfaces have their advantages and there are many times when something is actually easier to do with a command-line interface. The important thing to remember is that it ''is'' your interface with the system. It should be easy to submit commands to the system and interpret its response without having to hunt through a lot of extraneous information. This is normally best done by keeping the interface as clean and uncluttered as possible. As Abran, et al. have pointed out, the usability of a given interface varies with "the nature of the user, the task and the environment."<ref name="AbranUsa03">{{cite journal |title=Usability Meanings and Interpretations in ISO Standards |journal=Software Quality Journal |author=Abran, Alain; Khelifi, Adel; Suryn, Witold; Seffah, Ahmed |volume=11 |issue=4 |pages=325–338 |year=2003 |doi=10.1023/A:1025869312943}}</ref>
 
If you would prefer a somewhat drier set of definitions, Abran, et al., also extracted the definitions of usability from a variety of ISO standards and they are included in the following table:
 
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! ISO Usability Definitions
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|"The capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions." (ISO/IEC9126-1, 2000)
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use." (ISO 9241-11, 1998)
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|"The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component." (IEEE Std. 610.12-1990)
|-
|}
|}
<blockquote>'''Table 3.''': ISO Usability Definitions<ref name="AbranUsa03" /></blockquote>
 
Fleming translates this into the somewhat more colloquial statement: “Usability only exists with regard to functionality and refers to the ease of use for a given function.” For those interested in learning more about the usability 'debate', I suggest that you check out Andreasen, et al.<ref name="AndreasenUsa06">{{cite journal |title=Usability in open source software development: Opinions and practice |journal=Information Technology and Control |author=Andreasen, M.S.; Nielsen, H.V.; Schrøder, S.O.; Stage, J. |volume=35 |issue=3A |pages=303-312 |year=2006 |url=http://itc.ktu.lt/itc353/Stage353.pdf |format=PDF}}</ref> and especially Saxena and Dubey.<ref name="SaxenaImpact13">{{cite journal |title=Impact of Software Design Aspects on Usability |journal=International Journal of Computer Applications |author=Saxena, S.; Dubey, S.K. |volume=61 |issue=22 |pages=48-53 |year=January 2013 |doi=10.5120/10233-5043 |url=http://www.ijcaonline.org/archives/volume61/number22/10233-5043}}</ref>
 
In addition to usability, it has been highly recommended to me that, if budget exists, it is also helpful to have a user experience (UX) specialist on the review team as well.<ref name="AllUXDefs">{{cite web |url=http://www.allaboutux.org/ux-definitions |work=All About UX |author=AllAboutUX.org volunteers |date=08 October 2010 |accessdate=08 August 2015}}</ref><ref name="GubeWhat10">{{cite web |url=http://www.smashingmagazine.com/2010/10/what-is-user-experience-design-overview-tools-and-resources/ |title=What Is User Experience Design? Overview, Tools And Resources |author=Gube, Jacob |work=Smashing Magazine |publisher=Smashing Magazine GmbH |date=05 October 2010 |accessdate=08 August 2105}}</ref>
 
Suggested ratings are:
* Zero – System is difficult to use, frequently requiring switching between multiple screens or menus to perform a simple function. Operation of system discourages use and can actively antagonize users.
* One – System is useable, but relatively unintuitive regarding how to perform a function. Both control and output displays tend to be cluttered, increasing the effort required to operate the system and interpret its output.
* Two – System is relatively intuitive and designed to help guide the user through its operation. Ideally, this is complemented with a context sensitive help system to minimize any uncertainties in operation, particularly for any rarely used functions.
 
====Security====
This heading overlaps with Functionality and is usually difficult to assess from a high-level evaluation. While it is unlikely that you will observe any obvious security issues during a survey of this type, there are indicators that can provide a hint as to how much the applications designers and developers were concerned with security.
 
The simplest approach is to simply take a look at whether they've done anything that shows a concern for possible security vulnerabilities. The following are a few potential indicators that you can look for, but just being observant when seeing a demonstration can also tell you a lot.
 
* Is there any mention of security in the systems documentation? Does it describe any potential holes that you need to guard against or configuration changes you might make to your system environment to reduce any risks.
* Do the manuals describe any type of procedure for reporting bugs or observed system issues?
* If you have access to the developers, discuss any existing process for reporting and tracking security issues.
* Check their error logs and see if any security related issues are listed. If they are, what was the turnaround time to have them repaired, or were they repaired?
* If the developers are security conscious, they will almost certainly want to prove to whomever received the program that it hadn't been modified by a third party. The basic way of doing this is by separately sending you what is known as an MD5 hash. This is a distinctive number generated by another program from your applications code. If you generate a new MD5 hash from the code you receive, these numbers should match. If they don't match, that means that something in the code has been altered. For developers with more concern, they might generate a cryptographic signature incorporating the code. This will tell you who sent the code, as well as indicate whether the program was altered.
* Depending on the type of program, does it allow a 21 CFR part 11 compliant implementation or conform to a similar standard?
* Depending on the type of application, does it include a detailed audit trail and security logs?
 
Suggested ratings are:
* Zero – System shows no concern with security or operator tracking. Anyone can walk up to it and execute a function without having to log in. System doesn't support even a minimal audit trail. Any intermediate files are easily accessible and modifiable outside of the system.
* One - System shows some attempt at user control, but supports only a minimal audit trail. It may support a user table, but fails to follow best practices by allowing user records to be deleted. Audit trail is modifiable by power users.
* Two – Maintaining system security is emphasized in the user documentation. A detailed audit trail is maintained that logs all system changes and user activities. Application is distributed along with an MD5 hash or incorporated into an electronic signature by the developer.
 
====Flexibility/Customizability====
The goal of this topic is to identify how easily the functionality of this application can be altered or how capable it is of handling situations outside of its design parameters. Systems are generally designed to be either configurable or customizable, sometimes with a combination of both.
 
* Configurability - This refers to how much, or how easily, the functionality of the system can be altered by changing configuration settings. Configurable changes do not require any changes to the application code and generally simplify future application upgrades.
* Customizability - This refers to whether the functionality of the system must be altered by modifying the applications code. As we are targeting open source systems, the initial assumption might be that they are all customizable, however, this can be affected by the type of license that the application is released under. More practically, how easily an application can be customized depends on how well it is designed and documented. While in theory you might be able to customize a system, if it is a mass of spaghetti code and poorly compartmentalized, it might be a nightmare to do. In any case, if you customize the system code, you may not be able to take advantage of any system upgrades without having to recreate the customizations in them.
* Extendability – While you won't find this term in most definitions, it is a hybrid system that is both configurable and customizable. It is normally configured using the same approaches as a standard configurable system. However, the ability to be upgraded remains by feeding any code customizations through an Application Program Interface (API). As long as this API is maintained between upgrades, any extension modules should continue to work.
 
In addition, a well designed application is usually modular, which makes program changes easier. In an ideal world, any application that you may have to customize will be specifically designed to make customization simple. There are a variety of ways of doing this. Perhaps the easiest, for an application that is designed to be modular, would be to support optional software plug-in modules that added extra functionality. Unless these were "off-the-shelf" modules, you would need to confirm that there was appropriate documentation regarding their design and use. This would most likely be done via an API, as discussed above. Depending on the systems, you could transfer data through the API or have one system control another. The caveat again being that you need to have thorough documentation of the API and its capabilities.
 
In the majority of situations, I strongly encourage you to stick with a configurable system, if you can find one that meets your needs. Customizing a system is rarely justified, unless you are working situation. While almost everyone feels that their needs are unique, the reality is that a vell designed configurable system can generally meet your needs.
 
Suggested ratings are:
* Zero – Application does not support configuration and shows evidence of being difficult to customize. The later being indicated by use of spaghetti code rather than modular design, poorly named variables and functions, along with cryptic or no embedded comments. In worst case situations, the source code has been deliberately obfuscated to make the system even less customizable.
* One – Application supports minor configuration capabilities or is moderately difficult to modify. The later might be due to minimal application documentation or poor programming practices, but not deliberate obfuscation.
* Two – Application is highly configurable and accompanied by detailed documentation guiding the user through its configuration. Code is clearly documented and commented. It also follows good programming practices with highly modularized functionality, simplifying customization of the programs source code, ideally via an API.
 
====Interoperability/Integration====
Determine whether this software will work with the rest of the systems that you plan to use. Exactly what to check for is up to you, as you are the only one who has any idea what you will be doing with it. The following is a list of possible items that might conceivably fall under this heading:
 
* Does it understand the data and control protocols to talk to and control external equipment, whether a drill press, telescope, or sewing machine? (as appropriate)
* Is it designed to conform to standards, both electronic and physical, to avoid being locked into a single supplier.
* Does it handle localization to avoid conflicts with local systems?
 
Suggested ratings are:
* Zero – System provides no support for integration with other applications. File formats and communication protocols used are not documented.
* One – System is not optimized for either interoperability or integration with other systems. However, it does use standard protocols so that other applications can interpret its activities. Application likely does not include an API or any existing API is undocumented.
* Two – Application is optimized for interoperability and integration with other systems. All interfaces and protocols, particularly for any existing API, are clearly documented and accompanied with sample code.
 
====Legal/License issues====
This section refers to the type of license that the application was released under and the associated legal and functional implications. With proprietary software many people never bother to read the software license, either because they don't care or think they have no choice but to accept them. Be that as it may, when selecting a FLOSS application, it is wise to take the time to read the accompanying license, as it can make a big difference in what you can do with the software. First, if the software has no license, legally you have no right to even download the software, let alone run it.<ref name="ReitzChoosing14">{{cite web |url=http://docs.python-guide.org/en/latest/writing/license/ |title=Choosing a License |author=Reitz, Kenneth |work=The Hitchhiker's Guide to Python! |date=2014 |accessdate=13 May 2015}}</ref><ref name="AtwoodPick07">{{cite web |url=http://blog.codinghorror.com/pick-a-license-any-license/ |title=Pick a License, Any License |author=Atwood, Jeff |work=Coding Horror: Programming and Human Factors |date=03 April 2007 |accessdate=13 May 2015}}</ref>
 
While you have no control over which license the application was released under, you definitely control whether you wish to use it under the terms of the license. Which types of licenses are acceptable strongly depends on what you plan to do with the application? Do you intend to use the application as is or do you plan to modify it? If the later, what do you pla to do with the modified code? Do you want to integrate this FLOSS application with another, either proprietary or Open Source? Does this license clash with theirs? Your right to do any of these things is controlled by the license, so it must be considered very carefully, both in the light of what you want to do now and what you might want to do in the future.
 
One of the first things to do is to confirm that the application even is Open Source, just being able to see the source code is insufficient. To qualify as Open Source the license must comply with the 10 points listed in the Open Source Definition maintained by the Open Source Initiative (OSI).<ref name="OSIDef15" /> Pulling just the headers, their web site lists these as the required criteria:
* Free redistribution
* Source code
* Derived works
* Integrity of the author's source code
* No discrimination against persons or groups
* No discrimination against fields of endeavor
* Distribution of licenses
* License must not be specific to a product
* License must not restrict other software
* License must be technology-neutral
 
While somewhat cryptic to look at cold, each of the headings is associated with a longer definition, which primarily boils down to the freedom to use, modify, and redistribute the software. For those wanting to know the justification for each item, there is also an annotated version of this definition.<ref name="OSIAnnoDef15">{{cite web |url=http://opensource.org/docs/definition.php |title=The Open Source Definition (Annotated) |publisher=Open Source Initiative |date=2015 |accessdate=28 May 2015}}</ref> At present, OSI recognizes 71 distinct Open Source licenses, not counting the WTFPL.<ref name="WTFPLHome">{{cite web |url=http://www.wtfpl.net/ |title=WTFPL – Do What the Fuck You Want to Public License |author=Hocevar, Sam |work=WTFPL.net |date=2015 |accessdate=16 June 2015}}</ref>
 
One of the functions of the OSI is to review perspective licenses to determine whether they meet these criteria and are indeed Open Source. The OSI web site maintains a list of popular licenses, along with links to all approved licenses<ref name="OSILic15">{{cite web |url=http://opensource.org/licenses |title=Licenses & Standards |publisher=Open Source Initiative |date=2015 |accessdate=13 May 2015}}</ref>, sorted by name or category. These lists include full copies of the licenses. While clearer than most legal documents, they can still be somewhat confusing, particularly if you are trying to select one. If the definitions seem to blur, you might want to check out the Software Licenses Explained in Plain English web page maintained by TL;DRLegal.<ref name="TLDRLegalHome">{{cite web |url=https://tldrlegal.com/ |title=TL;DRLegal - Software Licenses Explained in Plain English |publisher=FOSSA, Inc |date=2015 |accessdate=16 June 2015}}</ref> As long as the license is classified as an Open Source license and you aren't planning to modify it yourself or integrate it into other systems, you probably won't have any problem. However, if you have any uncertainty at all it might be worth making the investment to discuss the license with an intellectual property lawyer who is familiar with OSS/FS before you inadvertently commit your organization to terms that conflict with their plans.
 
If your plans include the possible creation of complementary software, I suggest a quick read of Wheeler's essay on selecting a license.<ref name="WheelerMake14">{{cite web |url=http://www.dwheeler.com/essays/gpl-compatible.html |title=Make Your Open Source Software GPL-Compatible. Or Else |author=Wheeler, David A. |work=dwheeler.com |date=16 February 2014 |accessdate=20 March 2015}}</ref> The potential problem here is that most Open Source and proprietary applications contain multiple libraries or sub-applications, each with their own license. Depending on which licenses the original developers used, they may be compatible with other applications you wish to use, or they may be incompatible. The following figure illustrates some of these complications.
 
:[[File:Fig3 Joyce 2015.png|700px]]
{{clear}}
<blockquote>'''Illustration 3.''': License compatibility between common FOSS software licenses according to David A. Wheeler (2007):<br />the vector arrows denote an one directional compatibility, therefore better compatibility on the left side than on the right side.<ref name="WheelerTheFree07">{{cite web |url=http://www.dwheeler.com/essays/floss-license-slide.html |title=The Free-Libre / Open Source Software (FLOSS) License Slide |author=Wheeler, David A. |work=dwheeler.com |date=27 September 2007 |accessdate=28 May 2015}}</ref></blockquote>
 
Just to be absolutely clear, the license of FLOSS and proprietary applications generally disavow any type of warranty that the program will work and disclaims any liability for any damage or injuries that result from the programs use. Now, having said that, you may be able to purchase a warranty separately, just don't anticipate any legal recourse in the event of a system failure.
 
Suggested ratings are:
* Zero – Application does not include a license or license terms are unacceptable or incompatible with those of other applications being used.
* One – Application includes a license, but contains potential conflicts with other licenses or allowable use that will need to be carefully reviewed.
* Two – License is fully open, allowing you to freely use the software.
 
===Completing the evaluation===
====Caveats====
Other than independent reviews, one of the best ways to obtain some of the above information is direct testing of a system. Usually that is impractical in a survey situation because of the time it would take to install and configure an instance of the application. However, a new factor has entered the picture which may change this. Docker is a new service that combines an application, along with all of its dependencies into a single distributable package, which they term a Container. Because everything required is in the container, it is guaranteed to run the same on any system. Docker specifically states that their containers "are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure."<ref name="DockerWhatIs">{{cite web |url=https://www.docker.com/whatisdocker |title=What Is Docker? |work=Docker.com |publisher=Docker, Inc |accessdate=18 June 2015}}</ref> Using only a fraction of the resources that a virtual machine (VM) would require, you can easily run multiple containers on a laptop and switch back and forth while testing.<ref name="KnorrWhen15">{{cite web |url=http://www.infoworld.com/article/2900333/cloud-computing/scenes-from-the-docker-revolution.html |title=When will we see Docker in production? |author=Knorr, Eric |work=InfoWorld |publisher=InfoWorld, Inc |date=23 March 2015 |accessdate=18 June 2015}}</ref>
 
While the Docker web site contains repositories for a number of different containers, there are multiple web sites that also host containers configured with a variety of applications. If you can locate one that contains the application you are surveying, this makes it a simple matter to try it out. Probably the easiest way to check is to run a web search containing the terms "docker", "container", and the name of the application that you are seeking.
 
====Screening tabulation====
While you can perform this application survey in many ways, to keep your defined scoring criteria in front of the evaluators and to simplify scoring the survey, it might be prudent, to generate a document such as the following table with columns representing the criteria being evaluated, your rating definitions, and the numeric rating that your evaluators actually assign to the system. If you have decided to take the approach of using weighting factors instead of adjusting your definitions, you will also need to include columns for your weighting factors and a column containing the results of the evaluation after applying the weighting factor.
 
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! Criteria
  ! Rating = 0
  ! Rating = 1
  ! Rating = 2
  ! Score
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|System functionality
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Community
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|System cost
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Popularity
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Product support
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Maintenance
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Reliability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Performance
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Scalability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Usability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Security
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Flexibility/Customizability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Interoperability/Integration
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Legal/License issues
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
  | style="background-color:#EEEEEE; padding-left:10px; padding-right:10px;" colspan="4"|Summary Score
  | style="background-color:white; padding-left:10px; padding-right:10px;"|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
|-
|}
|}
<blockquote>'''Table 4.''': Potential screening criteria to filter prospective FLOSS applications. Rows can be<br />added or dropped as required by your needs. For example, if it is important that the<br />application be coded in a specific programming language or combination of languages, you<br />could add a row for this. The most likely anticipated change would be to subdivide the<br />System Functionality row so that specific information on the functionality of various system<br />components can be displayed. To minimize the risk of confusion, you can transfer your<br />selected scoring criteria for each rating into the cell corresponding to that rating and the item<br />being evaluated, though ths would admittedly generate a long check list.</blockquote>
 
====Demonstration surveys====
To demonstrate how this survey procedure is intended to be used, this section will apply the defined protocol to a block of Open Source LIMS applications suitable for use in a chemical [[laboratory]]. Note that in this instance, we are using LIMS to refer to a jaboratory information management system, not a labor information management system, a legislative information management system, or any other permutation that matches the LIMS acronym. The specific criteria used will quite likely be different from the ones used in your screening as the types of systems or the specific functionality required will be different.
 
For the purpose of this demonstration, we will not attempt to screen all of the Open Source LIMS available. Instead, we will select a subset of the systems that have been announced and attempt to apply our protocol to them, hopefully screening out the systems not meeting our requirements, so that the number of in-depth evaluations can be reduced. Part of the difficulty here is the broad range of fields that the term LIMS covers. There are general purpose LIMS, which can be configured to handle a wide range of samples, as well as targeted LIMS, which are designed to fill a particular niche. As such, you can find LIMS targeting specific areas as drinking water/waste water analysis, mining, radioisotopes, proteomics, and genetic analysis. Whether you make it a formal part of the screening process or isolate them while collecting the applications to be surveyed, you will need to filter out the systems which will not handle the type of samples you are dealing with. This issue is particularly prominent with LIMS, but will likely be encountered when screening other types of software as well. The following are the systems that we will include in this attempt:
 
* [[Bika LIMS]]
* eyeLIMS
* [[Open-LIMS]]
 
For simplicity in comparing results, I've reordered the screening table so that the first column contains the criteria being evaluated and the other columns correspond to the evaluation results for the LIMS being evaluated, with the bottom row reserved for the corresponding score summary. I have also added two subdivisions under System Functionality for the Programming Language and the Operating System used. Depending on the types of systems you are surveying, you will likely be including additional subdivisions. Scoring can be handled several ways. In this example, you might list the programming language in the appropriate cell. Depending on whether you have a team member competent in that language, you can use the results to set the value for the main criteria, e.g. System Functionality. Alternately, while you will record the information for programming language in your survey notebook, you can insert an actual numerical result into the corresponding cell, so that the sub-criteria can be evaluated separately, or used to generate a value for the main criteria field.
 
Normally, the best approach starting out is to check for existing reviews of systems, but here we have a problem in finding any. While references to Bika LIMS were common, and it was referenced in a number of scientific papers, actual reviews of the product were hard to come by and were usually for older versions.<ref name="EmadeenBika10">{{cite web |url=http://www.goomedic.com/bika-free-open-source-lims-laboratory-information-management-system-for-windows-linux-and-mac-osx.html |title=Bika : Free , Open source LIMS – Laboratory Information Management System for Windows , Linux and Mac OSX |author=Emadeen, Hamza |work=Goomedic.com |date=01 August 2010 |accessdate=21 July 2015}}</ref> No reviews were found for eyeLIMS and for Open-LIMS; the closest thing I found to an impartial review has been two postings on Joel Limardo's LIMSexpert blog.<ref name="LimardoOpen11">{{cite web |url=http://www.limsexpert.com/cgi-bin/bixchange/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1304895280&title=Open-LIMS%20Analysis%20Results |title=Open-LIMS Analysis Results |author=Limardo, J. |work=LIMSExpert.com |publisher=ForwardPhase Technologies, LLC |date=2011 |accessdate=07 February 2015}}</ref><ref name="LimardoScoring11">{{cite web |url=http://www.limsexpert.com/cgi-bin/bixchange/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1305075855&title=Scoring%20Open-LIMS |title=Scoring Open-LIMS |author=Limardo, J. |work=LIMSExpert.com |publisher=ForwardPhase Technologies, LLC |date=2011 |accessdate=07 February 2015}}</ref>
 
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! Criteria
  ! Bika LIMS 3.1.8
  ! eyeLIMS
  ! Open-LIMS*
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|System functionality
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0 (Project appears dead, no activity since 2008)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1 (System functional for a very specific application only)
|-
  | style="background-color:white; padding-left:30px; padding-right:10px;"|Programming language
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Python and PLONE CMS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|eyeOS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|PHP
|-
  | style="background-color:white; padding-left:30px; padding-right:10px;"|Operating system
  | style="background-color:white; padding-left:10px; padding-right:10px;"|platform-independent
  | style="background-color:white; padding-left:10px; padding-right:10px;"|eyeOS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Any OS supporting PostgreSQL and PHP
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Community
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|System cost
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Popularity
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Product support
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Maintenance
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Reliability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Performance
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Scalability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Usability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Security
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Flexibility/Customizability
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Interoperability/Integration
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2? (Supports import and export of CSV files)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
  | style="background-color:white; padding-left:10px; padding-right:10px;"|0
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Legal/License issues
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2 (AGPL 3.0 and GPL1)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2 (GNU Affero GPL v3)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2 (GNU GPL 3.0)
|-
  | style="background-color:#EEEEEE; padding-left:10px; padding-right:10px;"|Summary Score
  | style="background-color:#EEEEEE; padding-left:10px; padding-right:10px;"|22
  | style="background-color:#EEEEEE; padding-left:10px; padding-right:10px;"|2
  | style="background-color:#EEEEEE; padding-left:10px; padding-right:10px;"|10
|-
|}
|}
Based on the above Summary Scores, we would definitely filter out both eyeLIMS and Open-LIMS, while Bika LIMS would justify a more in-depth evaluation.
 
&#42; Note: The rating for Open-LIMS may need to be revisited, as it appears that major development work on this system has been taken over by Joel Limardo of ForwardPhase Technologies, so several of the rated parameters may be subject to major shifts. It is currently unclear whether this is a joint project with the original developer or a fork.
 
==Summary==
In support of a project to prepare published evaluations of various FLOSS applications, we have reviewed the FLOSS literature, focusing particularly on any assessment or evaluation documents. While many described proposed evaluation methods, none of them appear to be particularly popular or have developed an active community around them. Several review papers on the topic, while identifying multiple methods and their advantages, found flaws in all of them, particularly in terms of being able to perform a quality assessment on any FLOSS application. Many of the described systems were explicitly focused on a single class of FLOSS applications, such as Library Management Systems.
 
By consolidating suggestions and procedures from a number of these papers, we synthesized a general survey process to allow us to quickly assess the status of any given type of FLOSS applications, allowing us to triage them and identify the most promising candidates for in- depth evaluation. Note that this process is designed for performing high-level surveys, it is not designed to perform the in-depth evaluations required for product selection.
 
As a minor aside, in the course of researching this article I was surprised by the high percentage of the published papers on FLOSS which were published in classic subscription journals, as opposed to any of the various Open Source journals available.<ref name=GloverAca13">{{cite journal |title=Academia Should Embrace Open Access Scholarly Publishing |journal=Open Journal of Accounting |author=Glover, H.D. |volume=2 |issue=4 |pages=95–96 |year=2013 |doi=10.4236/ojacct.2013.24012}}</ref> Somehow it seems like a curious disconnect not to publish articles on Open Source Software in Open Source journals. Whether this is just habit of submission or due to more considered reasons would be interesting to know.
 
==Glossary==
The glossary for this article can be found [[User:Shawndouglas/sandbox/sublevel5|here]].


==References==
==References==
{{Reflist|colwidth=30em}}
{{Reflist|colwidth=30em}}
==Notes==
This article has not officially been published in a journal. However, this presentation is largely faithful to the original paper, with only a few minor changes to presentation and content. This article is being made available for the first time under the [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International] license, the same license used on this wiki.
<!--Place all category tags here-->
[[Category:LIMSwiki journal articles (all)‎]]
[[Category:LIMSwiki journal articles on software‎‎]]

Latest revision as of 19:33, 17 February 2023

Sandbox begins below

  • Discussion and practical use of artificial intelligence (AI) in the laboratory is, perhaps to the surprise of some, not a recent phenomena. In the mid-1980s, researchers were developing computerized AI systems able "to develop automatic decision rules for follow-up analysis of [clinical laboratory] tests depending on prior information, thus avoiding the delays of traditional sequential testing and the costs of unnecessary parallel testing."[1] In fact, discussion of AI in general was ongoing even in the mid-1950s.[2][3]
  • Hiring demand for laboratorians with AI experience (2015–18) has historically been higher in non-healthcare industries, such as manufacturing, mining, and agriculture, shedding a light on how AI adoption in the clinical setting may be lacking. According to the Brookings Institute, "Even for the relatively-skilled job postings in hospitals, which includes doctors, nurses, medical technicians, research lab workers, and managers, only approximately 1 in 1,250 job postings required AI skills." They add: "AI adoption may be slow because it is not yet useful, or because it may not end up being as useful as we hope. While our view is that AI has great potential in health care, it is still an open question."[4]
  • Today, AI is being practically used in not only clinical diagnostic laboratories but also clinical research labs, life science labs, and research and development (R&D) labs, and more. Practical uses of AI can be found in:
clinical research labs[5]
hospitals[5][6]
medical diagnostics labs[6][7][8][9][10][11]
chromatography labs[11]
biology and life science labs[12]
medical imaging centers[13]
ophthalmology clinics[14]
reproduction clinics[15][16][17]
digital pathology labs[18]
material testing labs[19][20][21]
chemical experimentation and molecular discovery labs[21][22][23]
quantum physics labs[24]
  • What's going on in these labs?
Materials science: The creation of "a modular robotic platform driven by a model-based optimization algorithm capable of autonomously optimizing the optical and electronic properties of thin-film materials by modifying the film composition and processing conditions ..."[19]
Materials science: "Most of the applications of [machine learning (ML)] in chemical and materials sciences, as we have said, feature supervised learning algorithms. The goal there is to supplement or replace traditional modeling methods, at the quantum chemical or classical level, in order to predict the properties of molecules or materials directly from their structure or their chemical composition ... Our research group was applying the same idea on a narrower range of materials, trying to confirm that for a given chemical composition, geometrical descriptors of a material’s structure could lead to accurate predictions of its mechanical features."[20]
Life science: "In biological experiments, we generally cannot as easily declare victory, but we can use the systems biology approach of cycling between experimentation and modelling to see which sequences, when tested, are most likely to improve the model. In artificial intelligence, this is called active learning, and it has some similarity to the way in which we as humans learn as infants: we get some help from parents and teachers, but mainly model the world around us by exploring it and interacting with it. Ideally then, we would recreate such an environment for our machine learning algorithms in the laboratory, where we start with an initial ‘infant’ model of a certain regulatory system or protein function and let the computer decide what sequence designs to try out – a deep learning version of the ‘robot scientist’. Microbes are ideal organisms for such an approach, given the ease and speed with which they can be grown and genetically manipulated. Combined with laboratory automation, many microbial experiments can (soon) be performed with minimal human intervention, ranging from strain construction and screening, such as operated by Amyris, Gingko, Transcriptic, etc., to full-genome engineering or even the design of microbial ecologies."[12]
Digital pathology: "The collaboration combines two AI solutions, VistaPath’s Sentinel, the world’s first automated tissue grossing platform, and Gestalt’s AI Requisition Engine (AIRE), a leading-edge AI algorithm for accessioning, to raise the bar in AI-driven pathology digitization. Designed to make tissue grossing faster and more accurate, VistaPath’s Sentinel uses a high-quality video system to assess specimens and create a gross report 93% faster than human technicians with 43% more accuracy. It not only improves on quality by continuously monitoring the cassette, container, and tissue to reduce mislabeling and specimen mix-up, but also increases traceability by retaining original images for downstream review."[25]
Chemistry and molecular science: "The benefits of combining automated experimentation with a layer of artificial intelligence (AI) have been demonstrated for flow reactors, photovoltaic films, organic synthesis, perovskites and in formulation problems. However, so far no approaches have integrated mobile robotics with AI for chemical experiments. Here, we built Bayesian optimization into a mobile robotic workflow to conduct photocatalysis experiments within a ten-dimensional space."[22]
Chemistry and immunology: "Chemistry and immunology laboratories are particularly well-suited to leverage machine learning because they generate large, highly structured data sets, Schulz and others wrote in a separate review paper. Labor-intensive processes used for interpretation and quality control of electrophoresis traces and mass spectra could benefit from automation as the technology improves, they said. Clinical chemistry laboratories also generate digital images—such as urine sediment analysis—that may be highly conducive to semiautomated analyses, given advances in computer vision, the paper noted."[26]
Clinical research: "... retrospective analysis of existing patient data for descriptive and clustering purposes [and] automation of knowledge extraction, ranging from text mining, patient selection for trials, to generation of new research hypotheses ..."[5]
Clinical research: "AI ... offers a further layer to the laboratory system by analyzing all experimental data collected by experiment devices, whether it be a sensor or a collaborative robot. From data collected, AI is able to produce hypotheses and predict which combination of materials or temperature is desired for the experiment. In short, this system will allow scientists to be aided by a highly intelligent system which is constantly monitoring and analyzing the experimental output. In this way, AI will help an experiment from its inception to conclusion."[27]
Clinical research/medical diagnostics: "Artificial intelligence (AI) in the laboratory is primarily used to make sense of big data, the almost impossibly large sets of data that biologists and pharmaceutical R&D teams are accustomed to working with. AI algorithms can parse large amounts of data in a short amount of time and turn that data into visualizations that viewers can easily understand. In certain data-intensive fields, such as genomic testing and virus research, AI algorithms are the best way to sort through the data and do some of the pattern recognition work."[28]
Medical diagnostics: Development and implementation of clinical decision support systems [5][6]
Medical diagnostics: "Finally, in the laboratory, AI reduces the number of unnecessary blood samples when diagnosing infection. Instead of the 'gold standard blood sample' that takes 24-72 hours, the algorithm can predict the outcome of the blood sample with almost 80% accuracy based on demographics, vital signs, medications, and laboratory and radiology results. These are all examples of how Artificial Intelligence can be used to test better and faster with information that already exists. This saves time and costs."[10]
Medical diagnostics: "Chang sees two overarching classes of AI models: those that tackle internal challenges in the lab, such as how to deliver more accurate results to clinicians; and those that seek to identify cohorts of patients and care processes to close quality gaps in health delivery systems. The lab, however, 'isn’t truly an island,' said Michelle Stoffel, MD, PhD, associate chief medical information officer for laboratory medicine and pathology at M Health Fairview and the University of Minnesota in Minneapolis. 'When other healthcare professionals are working with electronic health records or other applications, there could be AI-driven tools, or algorithms used by an institution’s systems that may draw on laboratory data.'"[26]
Medical diagnostics: AI is used for the formulation of reference ranges, improvement of quality control, and automated interpretation of results. "Continuous monitoring of specimen acceptability, collection and transport can result in the prompt identification and correction of problems, leading to improved patient care and a reduction in unnecessary redraws and delays in reporting results."[8]
Reproduction science: "The field of AI is the marriage of humans and computers while reproductive medicine combines clinical medicine and the scientific laboratory of embryology. The application of AI has the potential to disconnect healthcare professionals from patients through algorithms, automated communication, and clinical imaging. However, in the embryology laboratory, AI, with its focus on gametes and embryos, can avoid the same risk of distancing from the patient. Areas of application of AI in the laboratory would be to enhance and automate embryo ranking through analysis of images, the ultimate goal being to predict successful implantation. Might such a trend obviate the need for embryo morphological assessment, time-lapse imaging and preimplantation genetic testing for aneuploidy (PGT-A), including mosaicism. Additionally, AI could assist with automation through analysis of testicular sperm samples searching for viable gametes, embryo grading uniformity."[15]
Chromatography-heavy sciences: " A great example of this is AI in the Liquid Chromatography Mass Spectrometry (LC-MS) field. LC-MS is a great tool used to measure various compounds in the human body, including everything from hormone levels to trace metals. One of the ways AI has already integrated with LC-MS is how it cuts down on the rate limiting steps of LC-MS, which more often than not are sample prep and LC separations. One system that Physicians Lab has made use of is parallel processing using SCIEX MPX 2.0 High Throughput System. This system can couple parallel runs with one LCMS instrument, resulting in twice the speed with no loss to accuracy. It can do this by staggering two runs either using the same method, or different methods entirely. What really makes this system great is its ability to automatically detect carryover and inject solvent blanks to clean the instrument. The system will then continue its analyzing, while automatically reinjecting samples that may be affected by the carryover. It will also flag high concentration without user input, allowing for easy detection of possibly faulty samples. This allows it to operate without users from startup to shut down. Some of the other ways that it can be used to increase efficiency are by using integrated network features to work on anything from streamlining management to increased throughput."[11]
Most any lab: "Predictive analytics, for example, is one tool that the Pistoia Alliance is using to better understand laboratory instruments and how they might fail over time... With the right data management strategies and careful consideration of metadata, how to best store data so that it can be used in future AI and ML workflows is essential to the pursuit of AI in the laboratory. Utilizing technologies such as LIMS and ELN enables lab users to catalogue data, providing context and instrument parameters that can then be fed into AI or ML systems. Without the correct data or with mismatched data types, AI and ML will not be possible, or at the very least, could provide undue bias trying to compare data from disparate sources."[29]
Most any lab: "When the actionable items are automatically created by Optima, the 'engine' starts working. An extremely sophisticated algorithm is able to assign the tasks to the resources, both laboratory personnel and instruments, according to the system configuration. Optima, thanks to a large amount of time dedicated to research the best way to automate this critical process, is able to automate most of the lab resource scheduling."[30]
  • A number of challenges exist in the realm of effectively and securely implementing AI in the laboratory. This includes:
Ethical and privacy challenges[5][26][31]
Algorithmic limitations[4]
Data access limitations, including "where to get it, how to share it, and how to know when you have enough to train a machine-learning system that will produce good results"[4][26][32][33]
Data integration and transformation issues[5][33]
Regulatory barriers[4][7]
Misaligned incentives[4]
Lack of knowledgeable/skilled talent[5][26][32][33]
Cost of skilled talent and infrastructure for maintaining and updating AI systems[26]
Legacy systems running outdated technologies[32]
Lack of IT systems or specialized software systems[33]
Lack of standardized, best practices-based methods of validating algorithms[26]
Failure to demonstrate real-world performance[7]
Failure to meet the needs of the professionals using it[7]
  • Given those challenges, some considerations should be made about implementing AI-based components in the laboratory. Examples include:
Clinical diagnostics: "From an industry and regulatory perspective, however, only the intended uses supported from the media manufacturer can be supported from AI applications, unless otherwise justified and substantive evidence is presented for additional claims support. This means strict adherence to specimen type and incubation conditions. Considering that the media was initially developed for human assessment using the well-trained microbiologist eye, and not an advanced imaging system with or without AI, this paradigm should shift to allow advancements in technology to challenge the status-quo of decreasing media read-times especially, as decreased read-times assist with laboratory turnaround times and thus patient management. Perhaps with an increasing body of evidence to support any proposed indications for use, either regulatory positions should be challenged, or manufacturers of media and industry AI-development specialists should work together to advance the field with new indications for use.
While the use of AI in the laboratory setting can be highly beneficial there are still some issues to be addressed. The first being phenotypically distinct single organism polymorphisms that may be interpreted by AI as separate organisms, as may also be the case for a human assessment, as well as small colony variant categorization. As detailed earlier, the broader the inputs, the greater the generalization of the model, and the higher the likelihood of algorithm accuracy. In that respect, understanding and planning around these design constraints is critical for ultimate deployment of algorithms. Additionally, expecting an AI system to correctly categorize “contamination” is a difficult task as often this again seemingly innocuous decision is dependent on years of experience and understanding the specimen type and the full clinical picture with detailed clinical histories. In this respect, a fully integrated AI-LIS system where all data is available may assist, but it is currently not possible to gather this granular detail needed to make this assessment reliable."[9]
Clinical diagnostics and pathology: "Well, if I’ve learned anything in my research into this topic, it’s that AI implementation needs to be a two-way street. First, any company who is active in this space must reach out to pathologists and laboratory medicine professionals to understand their daily workflows, needs, and pain points in as much detail as possible. Second, pathologists, laboratory medicine professionals, and educators must all play their important part – willingly offering their time and expertise when it is sought or proactively getting involved. And finally, it’s clear that there is an imbalanced focus on certain issues – with privacy, respect, and sustainability falling by the wayside."[31]
Healthcare: "While we are encouraged by the promise shown by AI in healthcare, and more broadly welcome the use of digital technologies in improving clinical outcomes and health system productivity, we also recognize that caution must be exercised when introducing any new healthcare technology. Working with colleagues across the NHS Transformation Directorate, as well as the wider AI community, we have been developing a framework to evaluate AI-enabled solutions in the health and care policy context. The aim of the framework is several-fold but is, at its core, a tool with which to highlight to healthcare commissioners, end users, patients and members of the public the considerations to be mindful when introducing AI to healthcare settings."[34]
Most any lab: A code of AI ethics should address objectivity, privacy, transparency, accountability, and sustainability in any AI implementation.[31]
Most any lab: "Another approach is to implement an AI program alongside a manual process, assessing its performance along the way, as a means to ease into using the program. 'I think one of the most impactful things that laboratorians can do today is to help make sure that the lab data that they’re generating is as robust as possible, because these AI tools rely on new training sets, and their performance is really only going to be as good as the training data sets they’re given,' Stoffel said."[26]

References

  1. Berger-Hershkowitz, H.; Neuhauser, D. (1987). "Artificial intelligence in the clinical laboratory". Cleveland Clinic Journal of Medicine 54 (3): 165–166. doi:10.3949/ccjm.54.3.165. ISSN 0891-1150. PMID 3301059. https://www.ccjm.org/content/54/3/165. 
  2. Minsky, M. (17 December 1956). Heuristic Aspects of the Artificial Intelligence Problem. Ed Services Technical Information Agency. https://books.google.com/books?hl=en&lr=&id=fvWNo6_IZGUC&oi=fnd&pg=PA1. Retrieved 16 February 2023. 
  3. Minsky, Marvin (1 January 1961). "Steps toward Artificial Intelligence". Proceedings of the IRE 49 (1): 8–30. doi:10.1109/JRPROC.1961.287775. ISSN 0096-8390. http://ieeexplore.ieee.org/document/4066245/. 
  4. 4.0 4.1 4.2 4.3 4.4 Goldfarb, A.; Teodoridis, F. (9 March 2022). "Why is AI adoption in health care lagging?". Series: The Economics and Regulation of Artificial Intelligence and Emerging Technologies. Brookings Institute. https://www.brookings.edu/research/why-is-ai-adoption-in-health-care-lagging/. Retrieved 17 February 2023. 
  5. 5.0 5.1 5.2 5.3 5.4 5.5 5.6 Damiani, A.; Masciocchi, C.; Lenkowicz, J.; Capocchiano, N. D.; Boldrini, L.; Tagliaferri, L.; Cesario, A.; Sergi, P. et al. (7 December 2021). "Building an Artificial Intelligence Laboratory Based on Real World Data: The Experience of Gemelli Generator". Frontiers in Computer Science 3: 768266. doi:10.3389/fcomp.2021.768266. ISSN 2624-9898. https://www.frontiersin.org/articles/10.3389/fcomp.2021.768266/full. 
  6. 6.0 6.1 6.2 University of California, San Francisco; Adler-Milstein, Julia; Aggarwal, Nakul; University of Wisconsin-Madison; Ahmed, Mahnoor; National Academy of Medicine; Castner, Jessica; Castner Incorporated et al. (29 September 2022). "Meeting the Moment: Addressing Barriers and Facilitating Clinical Adoption of Artificial Intelligence in Medical Diagnosis". NAM Perspectives 22 (9). doi:10.31478/202209c. PMC PMC9875857. PMID 36713769. https://nam.edu/meeting-the-moment-addressing-barriers-and-facilitating-clinical-adoption-of-artificial-intelligence-in-medical-diagnosis. 
  7. 7.0 7.1 7.2 7.3 Government Accountability Office (GAO); National Academy of Medicine (NAM) (September 2022). "Artificial Intelligence in Health Care: Benefits and Challenges of Machine Learning Technologies for Medical Diagnostics" (PDF). Government Accountability Office. https://www.gao.gov/assets/gao-22-104629.pdf. Retrieved 16 February 2023. 
  8. 8.0 8.1 Wen, Xiaoxia; Leng, Ping; Wang, Jiasi; Yang, Guishu; Zu, Ruiling; Jia, Xiaojiong; Zhang, Kaijiong; Mengesha, Birga Anteneh et al. (24 September 2022). "Clinlabomics: leveraging clinical laboratory data by data mining strategies" (in en). BMC Bioinformatics 23 (1): 387. doi:10.1186/s12859-022-04926-1. ISSN 1471-2105. PMC PMC9509545. PMID 36153474. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-022-04926-1. 
  9. 9.0 9.1 DeYoung, B.; Morales, M.; Giglio, S. (4 August 2022). "Microbiology 2.0–A “behind the scenes” consideration for artificial intelligence applications for interpretive culture plate reading in routine diagnostic laboratories". Frontiers in Microbiology 13: 976068. doi:10.3389/fmicb.2022.976068. ISSN 1664-302X. PMC PMC9386241. PMID 35992715. https://www.frontiersin.org/articles/10.3389/fmicb.2022.976068/full. 
  10. 10.0 10.1 Schut, M. (1 December 2022). "Get better with bytes". Amsterdam UMC. https://www.amsterdamumc.org/en/research/news/get-better-with-bytes.htm. Retrieved 16 February 2023. 
  11. 11.0 11.1 11.2 Albano, V.; Morris, C.; Kent, T. (6 December 2019). "Calculations to Diagnosis: The Artificial Intelligence Shift That’s Already Happening". Physicians Lab. https://physicianslab.com/calculations-to-diagnosis-the-artificial-intelligence-shift-thats-already-happening/. Retrieved 16 February 2023. 
  12. 12.0 12.1 de Ridder, Dick (1 January 2019). "Artificial intelligence in the lab: ask not what your computer can do for you" (in en). Microbial Biotechnology 12 (1): 38–40. doi:10.1111/1751-7915.13317. PMC PMC6302702. PMID 30246499. https://onlinelibrary.wiley.com/doi/10.1111/1751-7915.13317. 
  13. Brandao-de-Resende, C.; Bui, M.; Daneshjou, R. et al. (11 October 2022). "AI Webinar: Clinical Adoption of AI Across Image Producing Specialties". Society for Imaging Informatics in Medicine. https://siim.org/page/22w_clinical_adoption_of_ai. 
  14. He, Mingguang; Li, Zhixi; Liu, Chi; Shi, Danli; Tan, Zachary (1 July 2020). "Deployment of Artificial Intelligence in Real-World Practice: Opportunity and Challenge" (in en). Asia-Pacific Journal of Ophthalmology 9 (4): 299–307. doi:10.1097/APO.0000000000000301. ISSN 2162-0989. https://journals.lww.com/10.1097/APO.0000000000000301. 
  15. 15.0 15.1 Trolice, Mark P.; Curchoe, Carol; Quaas, Alexander M (1 July 2021). "Artificial intelligence—the future is now" (in en). Journal of Assisted Reproduction and Genetics 38 (7): 1607–1612. doi:10.1007/s10815-021-02272-4. ISSN 1058-0468. PMC PMC8260235. PMID 34231110. https://link.springer.com/10.1007/s10815-021-02272-4. 
  16. European Society of Human Reproduction and Embryology (6 July 2022). "Annual Meeting 2022: Artificial intelligence in embryology and ART". Focus on Reproduction. https://www.focusonreproduction.eu/article/ESHRE-News-22AI. Retrieved 16 February 2023. 
  17. Hinckley, M. (17 March 2021). "Applying AI (Artificial Intelligence) in the Lab for Better IVF Success". Reproductive Science Center Blog. Reproductive Science Center of the Bay Area. https://rscbayarea.com/blog/applying-ai-for-better-ivf-success. Retrieved 16 February 2023. 
  18. Yousif, M.; McClintock, D.S.; Yao, K. (2021). "Artificial intelligence is the key driver for digital pathology adoption". Clinical Laboratory Int. PanGlobal Media. https://clinlabint.com/artificial-intelligence-is-the-key-driver-for-digital-pathology-adoption/. Retrieved 16 February 2023. 
  19. 19.0 19.1 MacLeod, B. P.; Parlane, F. G. L.; Morrissey, T. D.; Häse, F.; Roch, L. M.; Dettelbach, K. E.; Moreira, R.; Yunker, L. P. E. et al. (15 May 2020). "Self-driving laboratory for accelerated discovery of thin-film materials" (in en). Science Advances 6 (20): eaaz8867. doi:10.1126/sciadv.aaz8867. ISSN 2375-2548. PMC PMC7220369. PMID 32426501. https://www.science.org/doi/10.1126/sciadv.aaz8867. 
  20. 20.0 20.1 Chibani, Siwar; Coudert, François-Xavier (1 August 2020). "Machine learning approaches for the prediction of materials properties" (in en). APL Materials 8 (8): 080701. doi:10.1063/5.0018384. ISSN 2166-532X. http://aip.scitation.org/doi/10.1063/5.0018384. 
  21. 21.0 21.1 Mullin, R. (28 March 2021). "The lab of the future is now". Chemical & Engineering News 99 (11). Archived from the original on 06 May 2022. https://web.archive.org/web/20220506192926/http://cen.acs.org/business/informatics/lab-future-ai-automated-synthesis/99/i11. Retrieved 16 February 2023. 
  22. 22.0 22.1 Burger, Benjamin; Maffettone, Phillip M.; Gusev, Vladimir V.; Aitchison, Catherine M.; Bai, Yang; Wang, Xiaoyan; Li, Xiaobo; Alston, Ben M. et al. (9 July 2020). "A mobile robotic chemist" (in en). Nature 583 (7815): 237–241. doi:10.1038/s41586-020-2442-2. ISSN 0028-0836. https://www.nature.com/articles/s41586-020-2442-2.epdf?sharing_token=HOkIS6P5VIAo2_l3nRELmdRgN0jAjWel9jnR3ZoTv0Nw4yZPDO1jBpP52iNWHbb8TakOkK906_UHcWPTvNxCmzSMpAYlNAZfh29cFr7WwODI2U6eWv38Yq2K8odHCi-qwHcEDP18OjAmH-0KgsVgL5CpoEaQTCvbmhXDSyoGs6tIMe1nuABTeP58z6Ck3uULcdCtVQ66X244FsI7uH8GnA%3D%3D&tracking_referrer=cen.acs.org. 
  23. Lemonick, S. (6 April 2020). "Exploring chemical space: Can AI take us where no human has gone before?". Chemical & Engineering News 98 (13). Archived from the original on 29 July 2020. https://web.archive.org/web/20200729004137/https://cen.acs.org/physical-chemistry/computational-chemistry/Exploring-chemical-space-AI-take/98/i13. Retrieved 16 February 2023. 
  24. Doctrow, B. (16 December 2019). "Artificial intelligence in the laboratory". PNAS Science Sessions. https://www.pnas.org/post/podcast/artificial-intelligence-laboratory. Retrieved 16 February 2023. 
  25. VistaPath (28 July 2022). "VistaPath Launches New Collaboration with Gestalt Diagnostics to Further Accelerate Pathology Digitization". PR Newswire. Cision US Inc. https://www.prnewswire.com/news-releases/vistapath-launches-new-collaboration-with-gestalt-diagnostics-to-further-accelerate-pathology-digitization-301594718.html. Retrieved 17 February 2023. 
  26. 26.0 26.1 26.2 26.3 26.4 26.5 26.6 26.7 Blum, K. (1 January 2023). "A Status Report on AI in Laboratory Medicine". Clinical Laboratory News. American Association for Clinical Chemistry. https://www.aacc.org/cln/articles/2023/janfeb/a-status-report-on-ai-in-laboratory-medicine. Retrieved 17 February 2023. 
  27. Chubb, P. (3 November 2020). "How disruptive technology is helping laboratories combat COVID-19". Datafloq. https://datafloq.com/read/disruptive-technologies-lab-help-us-prepare-future-pandemics/. Retrieved 16 February 2023. 
  28. Stewart, B. (18 March 2021). "Using LIMS for Data Visualization". CSols Insights. CSols, Inc. https://www.csolsinc.com/insights/published-articles/using-lims-for-data-visualization/. Retrieved 17 February 2023. 
  29. "Data Analytics". Scientific Computing World - Building a Smart Laboratory 2020. Europa Science Ltd. 29 January 2021. https://www.scientific-computing.com/feature/data-analytics-0. Retrieved 17 February 2023. 
  30. Optima Team (15 December 2020). "The concept of machine learning applied to lab resources scheduling". Optima Blog. Optima PLC Tracking Tools S.L. https://www.optima.life/blog/the-concept-of-machine-learning-applied-to-lab-resources-scheduling/. Retrieved 17 February 2023. 
  31. 31.0 31.1 31.2 Lee, G.F. (10 October 2022). "The Robot May See You Now: It’s time to stop and think about the ethics of artificial intelligence". The Pathologist. https://thepathologist.com/outside-the-lab/the-robot-may-see-you-now. Retrieved 17 February 2023. 
  32. 32.0 32.1 32.2 Sherwood, L. (10 February 2022). "SLAS 2022: Barriers remain to AI adoption in life sciences". LabPulse.com Showcasts. Science and Medicine Group. https://www.labpulse.com/showcasts/slas/2022/article/15300130/slas-2022-barriers-remain-to-ai-adoption-in-life-sciences. Retrieved 17 February 2023. 
  33. 33.0 33.1 33.2 33.3 Bellini, Claudia; Padoan, Andrea; Carobene, Anna; Guerranti, Roberto (25 November 2022). "A survey on Artificial Intelligence and Big Data utilisation in Italian clinical laboratories" (in en). Clinical Chemistry and Laboratory Medicine (CCLM) 60 (12): 2017–2026. doi:10.1515/cclm-2022-0680. ISSN 1434-6621. https://www.degruyter.com/document/doi/10.1515/cclm-2022-0680/html. 
  34. Chada, Bharadwaj V; Summers, Leanne (10 October 2022). "AI in the NHS: a framework for adoption" (in en). Future Healthcare Journal: fhj.2022–0068. doi:10.7861/fhj.2022-0068. ISSN 2514-6645. PMC PMC9761451. PMID 36561823. https://www.rcpjournals.org/lookup/doi/10.7861/fhj.2022-0068.