Difference between revisions of "User:Shawndouglas/sandbox/sublevel4"

From LIMSWiki
Jump to navigationJump to search
(Added content. Saving and adding more.)
 
(441 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Infobox journal article
<div class="nonumtoc">__TOC__</div>
|name        =
{{ombox
|image        =
| type     = notice
|alt          = <!-- Alternative text for images -->
| style    = width: 960px;
|caption      =
| text     = This is sublevel4 of my sandbox, where I play with features and test MediaWiki code. If you wish to leave a comment for me, please see [[User_talk:Shawndouglas|my discussion page]] instead.<p></p>
|title_full  = Generalized Procedure for Screening Free Software and Open Source Software Applications
|journal     =
|authors      = Joyce, John
|affiliations = Arcana Informatica; Scientific Computing
|contact     = Email:  
|editors      =
|pub_year    = 2015
|vol_iss      =
|pages        =
|doi          =
|issn        =
|license      = [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International]
|website      =
|download    =
}}
}}
==Abstract==
Free Software and [[:Category:Open-source software|Open Source Software projects]] have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process we have initiated a project to generate a library of in-depth reviews of Free Software and Open Source Software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we will examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we will synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure will be executed against a selected class of applications.


==Introduction==
==Sandbox begins below==
There is much confusion regarding Free Software and Open Source Software and many people use these terms interchangeably, however, to some the connotations associated with the terms is highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with Open Source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).


When discussing Free Software, we are not explicitly discussing software for which no fee is charged, rather we are referring to free in terms of liberty. To quote the Free Software Foundation (FSF)<ref name="FSFWhat15">{{cite web |url=http://www.gnu.org/philosophy/free-sw.html |title=What is free software? |work=GNU Project |publisher=Free Software Foundation, Inc |date=2015 |accessdate=17 June 2015}}</ref>:
*Discussion and practical use of [[artificial intelligence]] (AI) in the [[laboratory]] is, perhaps to the surprise of some, not a recent phenomena. In the mid-1980s, researchers were developing computerized AI systems able "to develop automatic decision rules for follow-up analysis of &#91;[[clinical laboratory]]&#93; tests depending on prior information, thus avoiding the delays of traditional sequential testing and the costs of unnecessary parallel testing."<ref>{{Cite journal |last=Berger-Hershkowitz |first=H. |last2=Neuhauser |first2=D. |date=1987 |title=Artificial intelligence in the clinical laboratory |url=https://www.ccjm.org/content/54/3/165 |journal=Cleveland Clinic Journal of Medicine |volume=54 |issue=3 |pages=165–166 |doi=10.3949/ccjm.54.3.165 |issn=0891-1150 |pmid=3301059}}</ref> In fact, discussion of AI in general was ongoing even in the mid-1950s.<ref name="MinskyHeuristic56">{{cite book |url=https://books.google.com/books?hl=en&lr=&id=fvWNo6_IZGUC&oi=fnd&pg=PA1 |title=Heuristic Aspects of the Artificial Intelligence Problem |author=Minsky, M. |publisher=Ed Services Technical Information Agency |date=17 December 1956 |accessdate=16 February 2023}}</ref><ref>{{Cite journal |last=Minsky |first=Marvin |date=1961-01 |title=Steps toward Artificial Intelligence |url=http://ieeexplore.ieee.org/document/4066245/ |journal=Proceedings of the IRE |volume=49 |issue=1 |pages=8–30 |doi=10.1109/JRPROC.1961.287775 |issn=0096-8390}}</ref>


<blockquote>A program is free software if the program's users have the four essential freedoms:
*Hiring demand for laboratorians with AI experience (2015–18) has historically been higher in non-healthcare industries, such as manufacturing, mining, and agriculture, shedding a light on how AI adoption in the clinical setting may be lacking. According to the Brookings Institute, "Even for the relatively-skilled job postings in hospitals, which includes doctors, nurses, medical technicians, research lab workers, and managers, only approximately 1 in 1,250 job postings required AI skills." They add: "AI adoption may be slow because it is not yet useful, or because it may not end up being as useful as we hope. While our view is that AI has great potential in health care, it is still an open question."<ref name=":11">{{Cite web |last=Goldfarb, A.; Teodoridis, F. |date=09 March 2022 |title=Why is AI adoption in health care lagging? |work=Series: The Economics and Regulation of Artificial Intelligence and Emerging Technologies |url=https://www.brookings.edu/research/why-is-ai-adoption-in-health-care-lagging/ |publisher=Brookings Institute |accessdate=17 February 2023}}</ref>


* The freedom to run the program as you wish, for any purpose (freedom 0).
*Today, AI is being practically used in not only clinical diagnostic laboratories but also clinical research labs, life science labs, and research and development (R&D) labs, and more. Practical uses of AI can be found in:
* The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
* The freedom to redistribute copies so you can help your neighbor (freedom 2).
* The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.</blockquote>


This does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also 'free software' or 'freeware' that is given away at no charge, or gratis, but without the source code, this would not be considered Free Software under the FSF definition.
:clinical research labs<ref name=":0">{{Cite journal |last=Damiani |first=A. |last2=Masciocchi |first2=C. |last3=Lenkowicz |first3=J. |last4=Capocchiano |first4=N. D. |last5=Boldrini |first5=L. |last6=Tagliaferri |first6=L. |last7=Cesario |first7=A. |last8=Sergi |first8=P. |last9=Marchetti |first9=A. |last10=Luraschi |first10=A. |last11=Patarnello |first11=S. |date=2021-12-07 |title=Building an Artificial Intelligence Laboratory Based on Real World Data: The Experience of Gemelli Generator |url=https://www.frontiersin.org/articles/10.3389/fcomp.2021.768266/full |journal=Frontiers in Computer Science |volume=3 |pages=768266 |doi=10.3389/fcomp.2021.768266 |issn=2624-9898}}</ref>
:hospitals<ref name=":0" /><ref name=":1">{{Cite journal |last=University of California, San Francisco |last2=Adler-Milstein |first2=Julia |last3=Aggarwal |first3=Nakul |last4=University of Wisconsin-Madison |last5=Ahmed |first5=Mahnoor |last6=National Academy of Medicine |last7=Castner |first7=Jessica |last8=Castner Incorporated |last9=Evans |first9=Barbara J. |last10=University of Florida |last11=Gonzalez |first11=Andrew A. |date=2022-09-29 |title=Meeting the Moment: Addressing Barriers and Facilitating Clinical Adoption of Artificial Intelligence in Medical Diagnosis |url=https://nam.edu/meeting-the-moment-addressing-barriers-and-facilitating-clinical-adoption-of-artificial-intelligence-in-medical-diagnosis |journal=NAM Perspectives |volume=22 |issue=9 |doi=10.31478/202209c |pmc=PMC9875857 |pmid=36713769}}</ref>
:medical diagnostics labs<ref name=":1" /><ref name=":12">{{Cite web |last=Government Accountability Office (GAO); National Academy of Medicine (NAM) |date=September 2022 |title=Artificial Intelligence in Health Care: Benefits and Challenges of Machine Learning Technologies for Medical Diagnostics |url=https://www.gao.gov/assets/gao-22-104629.pdf |format=PDF |publisher=Government Accountability Office |accessdate=16 February 2023}}</ref><ref name=":13">{{Cite journal |last=Wen |first=Xiaoxia |last2=Leng |first2=Ping |last3=Wang |first3=Jiasi |last4=Yang |first4=Guishu |last5=Zu |first5=Ruiling |last6=Jia |first6=Xiaojiong |last7=Zhang |first7=Kaijiong |last8=Mengesha |first8=Birga Anteneh |last9=Huang |first9=Jian |last10=Wang |first10=Dongsheng |last11=Luo |first11=Huaichao |date=2022-09-24 |title=Clinlabomics: leveraging clinical laboratory data by data mining strategies |url=https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-022-04926-1 |journal=BMC Bioinformatics |language=en |volume=23 |issue=1 |pages=387 |doi=10.1186/s12859-022-04926-1 |issn=1471-2105 |pmc=PMC9509545 |pmid=36153474}}</ref><ref name=":7">{{Cite journal |last=DeYoung |first=B. |last2=Morales |first2=M. |last3=Giglio |first3=S. |date=2022-08-04 |title=Microbiology 2.0–A “behind the scenes” consideration for artificial intelligence applications for interpretive culture plate reading in routine diagnostic laboratories |url=https://www.frontiersin.org/articles/10.3389/fmicb.2022.976068/full |journal=Frontiers in Microbiology |volume=13 |pages=976068 |doi=10.3389/fmicb.2022.976068 |issn=1664-302X |pmc=PMC9386241 |pmid=35992715}}</ref><ref name=":5">{{Cite web |last=Schut, M. |date=01 December 2022 |title=Get better with bytes |url=https://www.amsterdamumc.org/en/research/news/get-better-with-bytes.htm |publisher=Amsterdam UMC |accessdate=16 February 2023}}</ref><ref name="AlbanoCal19">{{cite web |url=https://physicianslab.com/calculations-to-diagnosis-the-artificial-intelligence-shift-thats-already-happening/ |title=Calculations to Diagnosis: The Artificial Intelligence Shift That’s Already Happening |author=Albano, V.; Morris, C.; Kent, T. |work=Physicians Lab |date=06 December 2019 |accessdate=16 February 2023}}</ref>
:chromatography labs<ref name="AlbanoCal19" />
:biology and life science labs<ref name=":6">{{Cite journal |last=de Ridder |first=Dick |date=2019-01 |title=Artificial intelligence in the lab: ask not what your computer can do for you |url=https://onlinelibrary.wiley.com/doi/10.1111/1751-7915.13317 |journal=Microbial Biotechnology |language=en |volume=12 |issue=1 |pages=38–40 |doi=10.1111/1751-7915.13317 |pmc=PMC6302702 |pmid=30246499}}</ref>
:medical imaging centers<ref name="Brandao-de-ResendeAIWeb22">{{cite web |url=https://siim.org/page/22w_clinical_adoption_of_ai |title=AI Webinar: Clinical Adoption of AI Across Image Producing Specialties |author=Brandao-de-Resende, C.; Bui, M.; Daneshjou, R. et al. |publisher=Society for Imaging Informatics in Medicine |date=11 October 2022}}</ref>
:ophthalmology clinics<ref>{{Cite journal |last=He |first=Mingguang |last2=Li |first2=Zhixi |last3=Liu |first3=Chi |last4=Shi |first4=Danli |last5=Tan |first5=Zachary |date=2020-07 |title=Deployment of Artificial Intelligence in Real-World Practice: Opportunity and Challenge |url=https://journals.lww.com/10.1097/APO.0000000000000301 |journal=Asia-Pacific Journal of Ophthalmology |language=en |volume=9 |issue=4 |pages=299–307 |doi=10.1097/APO.0000000000000301 |issn=2162-0989}}</ref>
:reproduction clinics<ref name=":9">{{Cite journal |last=Trolice |first=Mark P. |last2=Curchoe |first2=Carol |last3=Quaas |first3=Alexander M |date=2021-07 |title=Artificial intelligence—the future is now |url=https://link.springer.com/10.1007/s10815-021-02272-4 |journal=Journal of Assisted Reproduction and Genetics |language=en |volume=38 |issue=7 |pages=1607–1612 |doi=10.1007/s10815-021-02272-4 |issn=1058-0468 |pmc=PMC8260235 |pmid=34231110}}</ref><ref name="ESHREArti22">{{cite web |url=https://www.focusonreproduction.eu/article/ESHRE-News-22AI |title=Annual Meeting 2022: Artificial intelligence in embryology and ART |author=European Society of Human Reproduction and Embryology |work=Focus on Reproduction |date=06 July 2022 |accessdate=16 February 2023}}</ref><ref name="HinckleyApply21">{{cite web |url=https://rscbayarea.com/blog/applying-ai-for-better-ivf-success |title=Applying AI (Artificial Intelligence) in the Lab for Better IVF Success |author=Hinckley, M. |work=Reproductive Science Center Blog |publisher=Reproductive Science Center of the Bay Area |date=17 March 2021 |accessdate=16 February 2023}}</ref>
:digital pathology labs<ref name="YousifArt21">{{cite web |url=https://clinlabint.com/artificial-intelligence-is-the-key-driver-for-digital-pathology-adoption/ |title=Artificial intelligence is the key driver for digital pathology adoption |author=Yousif, M.; McClintock, D.S.; Yao, K. |work=Clinical Laboratory Int |publisher=PanGlobal Media |date=2021 |accessdate=16 February 2023}}</ref>
:material testing labs<ref name=":2">{{Cite journal |last=MacLeod |first=B. P. |last2=Parlane |first2=F. G. L. |last3=Morrissey |first3=T. D. |last4=Häse |first4=F. |last5=Roch |first5=L. M. |last6=Dettelbach |first6=K. E. |last7=Moreira |first7=R. |last8=Yunker |first8=L. P. E. |last9=Rooney |first9=M. B. |last10=Deeth |first10=J. R. |last11=Lai |first11=V. |date=2020-05-15 |title=Self-driving laboratory for accelerated discovery of thin-film materials |url=https://www.science.org/doi/10.1126/sciadv.aaz8867 |journal=Science Advances |language=en |volume=6 |issue=20 |pages=eaaz8867 |doi=10.1126/sciadv.aaz8867 |issn=2375-2548 |pmc=PMC7220369 |pmid=32426501}}</ref><ref name=":3">{{Cite journal |last=Chibani |first=Siwar |last2=Coudert |first2=François-Xavier |date=2020-08-01 |title=Machine learning approaches for the prediction of materials properties |url=http://aip.scitation.org/doi/10.1063/5.0018384 |journal=APL Materials |language=en |volume=8 |issue=8 |pages=080701 |doi=10.1063/5.0018384 |issn=2166-532X}}</ref><ref name="MullinTheLab21">{{Cite journal |last=Mullin, R. |date=28 March 2021 |title=The lab of the future is now |url=http://cen.acs.org/business/informatics/lab-future-ai-automated-synthesis/99/i11 |journal=Chemical & Engineering News |volume=99 |issue=11 |archiveurl=https://web.archive.org/web/20220506192926/http://cen.acs.org/business/informatics/lab-future-ai-automated-synthesis/99/i11 |archivedate=06 May 2022 |accessdate=16 February 2023}}</ref>
:chemical experimentation and molecular discovery labs<ref name="MullinTheLab21" /><ref name=":4">{{Cite journal |last=Burger |first=Benjamin |last2=Maffettone |first2=Phillip M. |last3=Gusev |first3=Vladimir V. |last4=Aitchison |first4=Catherine M. |last5=Bai |first5=Yang |last6=Wang |first6=Xiaoyan |last7=Li |first7=Xiaobo |last8=Alston |first8=Ben M. |last9=Li |first9=Buyi |last10=Clowes |first10=Rob |last11=Rankin |first11=Nicola |date=2020-07-09 |title=A mobile robotic chemist |url=https://www.nature.com/articles/s41586-020-2442-2.epdf?sharing_token=HOkIS6P5VIAo2_l3nRELmdRgN0jAjWel9jnR3ZoTv0Nw4yZPDO1jBpP52iNWHbb8TakOkK906_UHcWPTvNxCmzSMpAYlNAZfh29cFr7WwODI2U6eWv38Yq2K8odHCi-qwHcEDP18OjAmH-0KgsVgL5CpoEaQTCvbmhXDSyoGs6tIMe1nuABTeP58z6Ck3uULcdCtVQ66X244FsI7uH8GnA%3D%3D&tracking_referrer=cen.acs.org |journal=Nature |language=en |volume=583 |issue=7815 |pages=237–241 |doi=10.1038/s41586-020-2442-2 |issn=0028-0836}}</ref><ref name="LemonickExplore20">{{Cite journal |last=Lemonick, S. |date=06 April 2020 |title=Exploring chemical space: Can AI take us where no human has gone before? |url=https://cen.acs.org/physical-chemistry/computational-chemistry/Exploring-chemical-space-AI-take/98/i13 |journal=Chemical & Engineering News |volume=98 |issue=13 |archiveurl=https://web.archive.org/web/20200729004137/https://cen.acs.org/physical-chemistry/computational-chemistry/Exploring-chemical-space-AI-take/98/i13 |archivedate=29 July 2020 |accessdate=16 February 2023}}</ref>
:quantum physics labs<ref name="DoctrowArti19">{{cite web |url=https://www.pnas.org/post/podcast/artificial-intelligence-laboratory |title=Artificial intelligence in the laboratory |author=Doctrow, B. |work=PNAS Science Sessions |date=16 December 2019 |accessdate=16 February 2023}}</ref>


The Open Source Initiative (OSI), originally formed to promote Free Software, which they referred to as Open Source Software (OSS) to make it sound more business friendly. The OSI defines Open Source Software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines<ref name="OSIDef15">{{cite web |url=http://opensource.org/osd |title=The Open Source Definition |publisher=Open Source Initiative |date=2015 |accessdate=17 June 2015}}</ref>:
*What's going on in these labs?


* Free redistribution
:'''Materials science''': The creation of "a modular robotic platform driven by a model-based optimization algorithm capable of autonomously optimizing the optical and electronic properties of thin-film materials by modifying the film composition and processing conditions ..."<ref name=":2" />
* Source code Included
:'''Materials science''': "Most of the applications of [machine learning (ML)] in chemical and materials sciences, as we have said, feature supervised learning algorithms. The goal there is to supplement or replace traditional modeling methods, at the quantum chemical or classical level, in order to predict the properties of molecules or materials directly from their structure or their chemical composition ... Our research group was applying the same idea on a narrower range of materials, trying to confirm that for a given chemical composition, geometrical descriptors of a material’s structure could lead to accurate predictions of its mechanical features."<ref name=":3" />
* Must allow derived works
:'''Life science''': "In biological experiments, we generally cannot as easily declare victory, but we can use the systems biology approach of cycling between experimentation and modelling to see which sequences, when tested, are most likely to improve the model. In artificial intelligence, this is called active learning, and it has some similarity to the way in which we as humans learn as infants: we get some help from parents and teachers, but mainly model the world around us by exploring it and interacting with it. Ideally then, we would recreate such an environment for our machine learning algorithms in the laboratory, where we start with an initial ‘infant’ model of a certain regulatory system or protein function and let the computer decide what sequence designs to try out – a deep learning version of the ‘robot scientist’. Microbes are ideal organisms for such an approach, given the ease and speed with which they can be grown and genetically manipulated. Combined with laboratory automation, many microbial experiments can (soon) be performed with minimal human intervention, ranging from strain construction and screening, such as operated by Amyris, Gingko, Transcriptic, etc., to full-genome engineering or even the design of microbial ecologies."<ref name=":6" />
* Must preserve the integrity of the authors source code
:'''Digital pathology''': "The collaboration combines two AI solutions, VistaPath’s Sentinel, the world’s first automated tissue grossing platform, and Gestalt’s AI Requisition Engine (AIRE), a leading-edge AI algorithm for accessioning, to raise the bar in AI-driven pathology digitization. Designed to make tissue grossing faster and more accurate, VistaPath’s Sentinel uses a high-quality video system to assess specimens and create a gross report 93% faster than human technicians with 43% more accuracy. It not only improves on quality by continuously monitoring the cassette, container, and tissue to reduce mislabeling and specimen mix-up, but also increases traceability by retaining original images for downstream review."<ref>{{Cite web |last=VistaPath |date=28 July 2022 |title=VistaPath Launches New Collaboration with Gestalt Diagnostics to Further Accelerate Pathology Digitization |work=PR Newswire |url=https://www.prnewswire.com/news-releases/vistapath-launches-new-collaboration-with-gestalt-diagnostics-to-further-accelerate-pathology-digitization-301594718.html |publisher=Cision US Inc |accessdate=17 February 2023}}</ref>
* License must not discriminate against persons or groups
:'''Chemistry and molecular science''': "The benefits of combining automated experimentation with a layer of artificial intelligence (AI) have been demonstrated for flow reactors, photovoltaic films, organic synthesis, perovskites and in formulation problems. However, so far no approaches have integrated mobile robotics with AI for chemical experiments. Here, we built Bayesian optimization into a mobile robotic workflow to conduct photocatalysis experiments within a ten-dimensional space."<ref name=":4" />
* License must not discriminate against fields of endeavor
:'''Chemistry and immunology''': "Chemistry and immunology laboratories are particularly well-suited to leverage machine learning because they generate large, highly structured data sets, Schulz and others wrote in a separate review paper. Labor-intensive processes used for interpretation and quality control of electrophoresis traces and mass spectra could benefit from automation as the technology improves, they said. Clinical chemistry laboratories also generate digital images—such as urine sediment analysis—that may be highly conducive to semiautomated analyses, given advances in computer vision, the paper noted."<ref name=":8">{{Cite web |last=Blum, K. |date=01 January 2023 |title=A Status Report on AI in Laboratory Medicine |work=Clinical Laboratory News |url=https://www.aacc.org/cln/articles/2023/janfeb/a-status-report-on-ai-in-laboratory-medicine |publisher=American Association for Clinical Chemistry |accessdate=17 February 2023}}</ref>
* Distribution of licenses
:'''Clinical research''': "... retrospective analysis of existing patient data for descriptive and clustering purposes [and] automation of knowledge extraction, ranging from text mining, patient selection for trials, to generation of new research hypotheses ..."<ref name=":0" />
* License must not be specific to a production
:'''Clinical research''': "AI ... offers a further layer to the laboratory system by analyzing all experimental data collected by experiment devices, whether it be a sensor or a collaborative robot. From data collected, AI is able to produce hypotheses and predict which combination of materials or temperature is desired for the experiment. In short, this system will allow scientists to be aided by a highly intelligent system which is constantly monitoring and analyzing the experimental output. In this way, AI will help an experiment from its inception to conclusion."<ref>{{Cite web |last=Chubb, P. |date=03 November 2020 |title=How disruptive technology is helping laboratories combat COVID-19 |url=https://datafloq.com/read/disruptive-technologies-lab-help-us-prepare-future-pandemics/ |publisher=Datafloq |accessdate=16 February 2023}}</ref>
* License must not restrict other software
:'''Clinical research/medical diagnostics''': "Artificial intelligence (AI) in the laboratory is primarily used to make sense of big data, the almost impossibly large sets of data that biologists and pharmaceutical R&D teams are accustomed to working with. AI algorithms can parse large amounts of data in a short amount of time and turn that data into visualizations that viewers can easily understand. In certain data-intensive fields, such as genomic testing and virus research, AI algorithms are the best way to sort through the data and do some of the pattern recognition work."<ref>{{Cite web |last=Stewart, B. |date=18 March 2021 |title=Using LIMS for Data Visualization |work=CSols Insights |url=https://www.csolsinc.com/insights/published-articles/using-lims-for-data-visualization/ |publisher=CSols, Inc |accessdate=17 February 2023}}</ref>
* License must be technology neutral
:'''Medical diagnostics''': Development and implementation of [[Clinical decision support system|clinical decision support systems]] <ref name=":0" /><ref name=":1" />
:'''Medical diagnostics''': "Finally, in the laboratory, AI reduces the number of unnecessary blood samples when diagnosing infection. Instead of the 'gold standard blood sample' that takes 24-72 hours, the algorithm can predict the outcome of the blood sample with almost 80% accuracy based on demographics, vital signs, medications, and laboratory and radiology results. These are all examples of how Artificial Intelligence can be used to test better and faster with information that already exists. This saves time and costs."<ref name=":5" />
:'''Medical diagnostics''': "Chang sees two overarching classes of AI models: those that tackle internal challenges in the lab, such as how to deliver more accurate results to clinicians; and those that seek to identify cohorts of patients and care processes to close quality gaps in health delivery systems. The lab, however, 'isn’t truly an island,' said Michelle Stoffel, MD, PhD, associate chief medical information officer for laboratory medicine and pathology at M Health Fairview and the University of Minnesota in Minneapolis. 'When other healthcare professionals are working with electronic health records or other applications, there could be AI-driven tools, or algorithms used by an institution’s systems that may draw on laboratory data.'"<ref name=":8" />
:'''Medical diagnostics''': AI is used for the formulation of reference ranges, improvement of quality control, and automated interpretation of results. "Continuous monitoring of specimen acceptability, collection and transport can result in the prompt identification and correction of problems, leading to improved patient care and a reduction in unnecessary redraws and delays in reporting results."<ref name=":13" />
:'''Reproduction science''': "The field of AI is the marriage of humans and computers while reproductive medicine combines clinical medicine and the scientific laboratory of embryology. The application of AI has the potential to disconnect healthcare professionals from patients through algorithms, automated communication, and clinical imaging. However, in the embryology laboratory, AI, with its focus on gametes and embryos, can avoid the same risk of distancing from the patient. Areas of application of AI in the laboratory would be to enhance and automate embryo ranking through analysis of images, the ultimate goal being to predict successful implantation. Might such a trend obviate the need for embryo morphological assessment, time-lapse imaging and preimplantation genetic testing for aneuploidy (PGT-A), including mosaicism. Additionally, AI could assist with automation through analysis of testicular sperm samples searching for viable gametes, embryo grading uniformity."<ref name=":9" />
:'''Chromatography-heavy sciences''': " A great example of this is AI in the Liquid Chromatography Mass Spectrometry (LC-MS) field. LC-MS is a great tool used to measure various compounds in the human body, including everything from hormone levels to trace metals. One of the ways AI has already integrated with LC-MS is how it cuts down on the rate limiting steps of LC-MS, which more often than not are sample prep and LC separations. One system that Physicians Lab has made use of is parallel processing using SCIEX MPX 2.0 High Throughput System. This system can couple parallel runs with one LCMS instrument, resulting in twice the speed with no loss to accuracy. It can do this by staggering two runs either using the same method, or different methods entirely. What really makes this system great is its ability to automatically detect carryover and inject solvent blanks to clean the instrument. The system will then continue its analyzing, while automatically reinjecting samples that may be affected by the carryover. It will also flag high concentration without user input, allowing for easy detection of possibly faulty samples. This allows it to operate without users from startup to shut down. Some of the other ways that it can be used to increase efficiency are by using integrated network features to work on anything from streamlining management to increased throughput."<ref name="AlbanoCal19" />
:'''Most any lab''': "Predictive analytics, for example, is one tool that the Pistoia Alliance is using to better understand laboratory instruments and how they might fail over time... With the right data management strategies and careful consideration of metadata, how to best store data so that it can be used in future AI and ML workflows is essential to the pursuit of AI in the laboratory. Utilizing technologies such as LIMS and ELN enables lab users to catalogue data, providing context and instrument parameters that can then be fed into AI or ML systems. Without the correct data or with mismatched data types, AI and ML will not be possible, or at the very least, could provide undue bias trying to compare data from disparate sources."<ref>{{Cite web |date=29 January 2021 |title=Data Analytics |work=Scientific Computing World - Building a Smart Laboratory 2020 |url=https://www.scientific-computing.com/feature/data-analytics-0 |publisher=Europa Science Ltd |accessdate=17 February 2023}}</ref>
:'''Most any lab''': "When the actionable items are automatically created by Optima, the 'engine' starts working. An extremely sophisticated algorithm is able to assign the tasks to the resources, both laboratory personnel and instruments, according to the system configuration. Optima, thanks to a large amount of time dedicated to research the best way to automate this critical process, is able to automate most of the lab resource scheduling."<ref>{{Cite web |last=Optima Team |date=15 December 2020 |title=The concept of machine learning applied to lab resources scheduling |work=Optima Blog |url=https://www.optima.life/blog/the-concept-of-machine-learning-applied-to-lab-resources-scheduling/ |publisher=Optima PLC Tracking Tools S.L |accessdate=17 February 2023}}</ref>


Open Source Software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.
*A number of challenges exist in the realm of effectively and securely implementing AI in the laboratory. This includes:


In an attempt to placate both groups when discussing the same software application, the term Free/Open Source Software (F/OSS) was developed. Since the term Free was still tending to confuse some people, the term libre, which connotes freedom, was added resulting in the term Free/Libre Open Source Software (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all Free Software fits the Open Source Software definition, while not all Open Source Software fits the Free Software definition. However, any Open Source Software that is not also Free Software is the exception, rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.
:Ethical and privacy challenges<ref name=":0" /><ref name=":8" /><ref name=":10" />
:Algorithmic limitations<ref name=":11" />
:Data access limitations, including "where to get it, how to share it, and how to know when you have enough to train a machine-learning system that will produce good results"<ref name=":11" /><ref name=":8" /><ref name=":14">{{Cite web |last=Sherwood, L. |date=10 February 2022 |title=SLAS 2022: Barriers remain to AI adoption in life sciences |work=LabPulse.com Showcasts |url=https://www.labpulse.com/showcasts/slas/2022/article/15300130/slas-2022-barriers-remain-to-ai-adoption-in-life-sciences |publisher=Science and Medicine Group |accessdate=17 February 2023}}</ref><ref name=":15">{{Cite journal |last=Bellini |first=Claudia |last2=Padoan |first2=Andrea |last3=Carobene |first3=Anna |last4=Guerranti |first4=Roberto |date=2022-11-25 |title=A survey on Artificial Intelligence and Big Data utilisation in Italian clinical laboratories |url=https://www.degruyter.com/document/doi/10.1515/cclm-2022-0680/html |journal=Clinical Chemistry and Laboratory Medicine (CCLM) |language=en |volume=60 |issue=12 |pages=2017–2026 |doi=10.1515/cclm-2022-0680 |issn=1434-6621}}</ref>
:Data integration and transformation issues<ref name=":0" /><ref name=":15" />
:Regulatory barriers<ref name=":11" /><ref name=":12" />
:Misaligned incentives<ref name=":11" />
:Lack of knowledgeable/skilled talent<ref name=":0" /><ref name=":8" /><ref name=":14" /><ref name=":15" />
:Cost of skilled talent and infrastructure for maintaining and updating AI systems<ref name=":8" />
:Legacy systems running outdated technologies<ref name=":14" />
:Lack of IT systems or specialized software systems<ref name=":15" />
:Lack of standardized, best practices-based methods of validating algorithms<ref name=":8" />
:Failure to demonstrate real-world performance<ref name=":12" />
:Failure to meet the needs of the professionals using it<ref name=":12" />


The reality is that since both groups trace their history back to the same origins, the practical differences between an application being Free Software or Open Source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in Open Source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us.<ref name="SchießleFree12">{{cite web |url=https://fsfe.org/freesoftware/basics/comparison.en.html |title=Free Software, Open Source, FOSS, FLOSS - same same but different |author=Schießle, Björn |publisher=Free Software Foundation Europe |date=12 August 2012 |accessdate=5 June 2015}}</ref> In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With in excess of 300,000 FLOSS applications currently registered in SourceForge.net<ref name="DateRepOSS12">{{cite web |url=http://events.linuxfoundation.org/images/stories/pdf/lceu2012_date.pdf |format=PDF |title=RepOSS: A Flexible OSS Assessment Repository |publisher=Northeast Asia OSS Promotion Forum WG3 |date=05 November 2012 |accessdate=05 May 2015}}</ref> and over 10 million repositories on GitHub<ref name="Doll10Mil13">{{cite web |url=https://github.com/blog/1724-10-millionrepositories |title=10 Million Repositories |author=Doll, Brian |publisher=GitHub, Inc |date=23 December 2013 |accessdate=08 August 2015}}</ref>, there are generally multiple options accessible for any class of application, be it a [[Laboratory information management system|Laboratory Information Management System]] (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an Open Source application<ref name=SarrabTheTech13">{{cite journal |title=The Technical, Non-technical Issues and the Challenges of Migration to Free and Open Source Software |journal=IJCSI International Journal of Computer Science Issues |author=Sarrab, Mohamed; Elsabir, Mahmoud; Elgamel, Laila |volume=10 |issue=2.3 |year=March 2013 |url=http://ijcsi.org/papers/IJCSI-10-2-3-464-469.pdf |format=PDF}}</ref> and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers with no indication of the vitality or independent evaluation of the project.
*Given those challenges, some considerations should be made about implementing AI-based components in the laboratory. Examples include:


What is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While once they've made a tentative selection, the organization will need to perform their own testing to confirm that the selected application meets their specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.
:'''Clinical diagnostics''': "From an industry and regulatory perspective, however, only the intended uses supported from the media manufacturer can be supported from AI applications, unless otherwise justified and substantive evidence is presented for additional claims support. This means strict adherence to specimen type and incubation conditions. Considering that the media was initially developed for human assessment using the well-trained microbiologist eye, and not an advanced imaging system with or without AI, this paradigm should shift to allow advancements in technology to challenge the status-quo of decreasing media read-times especially, as decreased read-times assist with laboratory turnaround times and thus patient management. Perhaps with an increasing body of evidence to support any proposed indications for use, either regulatory positions should be challenged, or manufacturers of media and industry AI-development specialists should work together to advance the field with new indications for use.
 
:While the use of AI in the laboratory setting can be highly beneficial there are still some issues to be addressed. The first being phenotypically distinct single organism polymorphisms that may be interpreted by AI as separate organisms, as may also be the case for a human assessment, as well as small colony variant categorization. As detailed earlier, the broader the inputs, the greater the generalization of the model, and the higher the likelihood of algorithm accuracy. In that respect, understanding and planning around these design constraints is critical for ultimate deployment of algorithms. Additionally, expecting an AI system to correctly categorize “contamination” is a difficult task as often this again seemingly innocuous decision is dependent on years of experience and understanding the specimen type and the full clinical picture with detailed clinical histories. In this respect, a fully integrated AI-LIS system where all data is available may assist, but it is currently not possible to gather this granular detail needed to make this assessment reliable."<ref name=":7" />
:[[File:Fig1 Joyce 2015.png|500px]]
:'''Clinical diagnostics and pathology''': "Well, if I’ve learned anything in my research into this topic, it’s that AI implementation needs to be a two-way street. First, any company who is active in this space must reach out to pathologists and laboratory medicine professionals to understand their daily workflows, needs, and pain points in as much detail as possible. Second, pathologists, laboratory medicine professionals, and educators must all play their important part – willingly offering their time and expertise when it is sought or proactively getting involved. And finally, it’s clear that there is an imbalanced focus on certain issues – with privacy, respect, and sustainability falling by the wayside."<ref name=":10">{{Cite web |last=Lee, G.F. |date=10 October 2022 |title=The Robot May See You Now: It’s time to stop and think about the ethics of artificial intelligence |work=The Pathologist |url=https://thepathologist.com/outside-the-lab/the-robot-may-see-you-now |accessdate=17 February 2023}}</ref>
{{clear}}
:'''Healthcare''': "While we are encouraged by the promise shown by AI in healthcare, and more broadly welcome the use of digital technologies in improving clinical outcomes and health system productivity, we also recognize that caution must be exercised when introducing any new healthcare technology. Working with colleagues across the NHS Transformation Directorate, as well as the wider AI community, we have been developing a framework to evaluate AI-enabled solutions in the health and care policy context. The aim of the framework is several-fold but is, at its core, a tool with which to highlight to healthcare commissioners, end users, patients and members of the public the considerations to be mindful when introducing AI to healthcare settings."<ref>{{Cite journal |last=Chada |first=Bharadwaj V |last2=Summers |first2=Leanne |date=2022-10-10 |title=AI in the NHS: a framework for adoption |url=https://www.rcpjournals.org/lookup/doi/10.7861/fhj.2022-0068 |journal=Future Healthcare Journal |language=en |pages=fhj.2022–0068 |doi=10.7861/fhj.2022-0068 |issn=2514-6645 |pmc=PMC9761451 |pmid=36561823}}</ref>
<blockquote>'''Illustration 1.''': This diagram, originally by Chao-Kuei and updated by several others since,<br />explains the different categories of software. It's available as a Scalable Vector Graphic and<br />as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2<br />or later, or the Creative Commons Attribution-Share Alike v2.0 or later</blockquote>
:'''Most any lab''': A code of AI ethics should address objectivity, privacy, transparency, accountability, and sustainability in any AI implementation.<ref name=":10" />
 
:'''Most any lab''': "Another approach is to implement an AI program alongside a manual process, assessing its performance along the way, as a means to ease into using the program. 'I think one of the most impactful things that laboratorians can do today is to help make sure that the lab data that they’re generating is as robust as possible, because these AI tools rely on new training sets, and their performance is really only going to be as good as the training data sets they’re given,' Stoffel said."<ref name=":8" />
The primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end it we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.
 
You can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, commercial is referring to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior quality, to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS ''is'' commercial software.<ref name="WheelerFree11">{{cite web |url=http://www.dwheeler.com/essays/commercial-floss.html |title=Free-Libre / Open Source Software (FLOSS) is Commercial Software |author=Wheeler, David A. |work=dwheeler.com |date=14 June 2011 |accessdate=28 May 2015}}</ref>
 
The final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.
 
==Results==
===Literature review===
A search of the literature returns thousands of papers related to Open Source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between Open Source projects is mentioned in a number of these papers and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large scale support in the industry. Stol and Babar have published a framework for comparing evaluation methods targeting Open Source software and include a comparison of 20 of them.<ref name="StolAComp10">{{cite book |chapter=A Comparison Framework for Open Source Software Evaluation Methods |title=Open Source Software: New Horizons |author=Stol, Klaas-Jan; Ali Babar, Muhammad |editor=Ågerfalk, P.J.; Boldyreff, C.; González-Barahona, J.M.; Madey, G.R.; Noll, J |publisher=Springer |year=2010 |pages=389–394 |isbn=9783642132445 |doi=10.1007/978-3-642-13244-5_36}}</ref> They noted that web sites that simply consisted of a suggestion list for selecting an Open Source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, and other factors involved.<ref name="FritzAClass94">{{cite book |title=A Classification And Summary Of Software Evaluation And Selection Methodologies |author=Fritz, Catherine A.; Carter, Bradley D. |publisher=Department of Computer Science, Mississippi State University |location=Mississippi State, MS |date=23 August 1994 |url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.55.4470}}</ref>
 
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! No.
  ! Name
  ! Year
  ! Orig
  ! Method
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|1
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Capgemini Open Source Maturity Model
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2003
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Evaluation Framework for Open Source Software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2004
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|3
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Model for Comparative Assessment of Open Source Products
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2004
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|4
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Navica Open Source Maturity Model
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2004
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|5
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Woods and Guliani's OSMM
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2005
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|6
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Business Readiness Rating (OpenBRR)<ref name="OpenBRROpen2005">{{cite web |url=http://docencia.etsit.urjc.es/moodle/file.php/125/OpenBRR_Whitepaper.pdf |format=PDF |title=OpenBRR, Business Readiness Rating for Open Source: A Proposed Open Standard to Facilitate Assessment and Adoption of Open Source Software |publisher=OpenBRR |year=2005 |accessdate=13 April 2015}}</ref><ref name="WassermanTheBus06">{{cite web |url=http://www.openbrr.org/comoworkshop/papers/WassermanPalChan_EFOSS06.pdf |archiveurl=http://web.archive.org/web/20070111113722/http://www.openbrr.org/comoworkshop/papers/WassermanPalChan_EFOSS06.pdf |format=PDF |title=The Business Readiness Rating: a Framework for Evaluating Open Source |work=Proceedings of the Workshop on Evaluation Frameworks for Open Source Software (EFOSS) at the Second International Conference on Open Source Systems |author=Wasserman, A.I.; Pal, M.; Chan, C. |location=Lake Como, Italy |pages=1–5 |date=10 June 2006 |archivedate=11 January 2007 |accessdate=15 April 2015}}</ref>
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2005
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R/I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|7
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Atos Origin Method for Qualification and Selection of Open Source Software (QSOS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2006
  | style="background-color:white; padding-left:10px; padding-right:10px;"|I
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|8
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Evaluation Criteria for Free/Open Source Software Products
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2006
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|9
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Quality Model for OSS Selection
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|10
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Selection Process of Open Source Software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|11
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Observatory for Innovation and Technological transfer on Open Source software (OITOS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|12
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Framework for OS Critical Systems Evaluation (FOCSE)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|13
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Balanced Scorecards for OSS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|14
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Business Quality Rating (OpenBQR)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|15
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Evaluating OSS through Prototyping
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2007
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|16
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Comprehensive Approach for Assessing Open Source Projects
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|17
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Software Quality Observatory for Open Source Software (SQO-OSS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Yes
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|18
  | style="background-color:white; padding-left:10px; padding-right:10px;"|An operational approach for selecting open source components in a software development project<ref name="MajchrowskiAnOp08">{{cite book |chapter=An Operational Approach for Selecting Open Source Components in a Software Development Project |title=Software Process Improvement |author=Majchrowski, Annick; Deprez, Jean-Christophe |editor=O'Connor, R.; Baddoo, N.; Smolander, K.; Messnarz, R. |publisher=Springer |year=2008 |pages=176–188 |isbn=9783540859369 |doi=10.1007/978-3-540-85936-9_16}}</ref>
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|19
  | style="background-color:white; padding-left:10px; padding-right:10px;"|QualiPSo trustworthiness model
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2008
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|20
  | style="background-color:white; padding-left:10px; padding-right:10px;"|OpenSource Maturity Model (OMM)<ref name="PetrinjaIntro09">{{cite book |chapter=Introducing the Open Source Maturity Model |title=ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development, 2009 |author=Petrinja, E.; Nambakam, R.; Sillitti, A. |publisher=IEEE |year=2009 |pages=37–41 |doi=10.1109/FLOSS.2009.5071358 |isbn=9781424437207 }}</ref>
  | style="background-color:white; padding-left:10px; padding-right:10px;"|2009
  | style="background-color:white; padding-left:10px; padding-right:10px;"|R
  | style="background-color:white; padding-left:10px; padding-right:10px;"|No
|-
|}
|}
<blockquote>'''Table 1.''': Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar.<ref name="StolAComp10" /> The selection<br />procedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described<br /> process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)<br />or just a list of evaluation criteria (No).</blockquote>
 
Extensive comparisons between some of these methods have also been published, such as Deprez's and Alexandre's comparative assessment of the OpenBRR and QSOS techniques.<ref name="Deprezcomp08">{{cite book |chapter=Comparing Assessment Methodologies for Free/Open Source Software: OpenBRR and QSOS |title=Product-Focused Software Process Improvement |author=Deprez,Jean-Christophe; Alexandre, Simon |editor=Jedlitschka, Andreas; Salo, Outi |publisher=Springer |year=2008 |pages=189-203 |isbn=9783540695660 |doi=10.1007/978-3-540-69566-0_17}}</ref> Wasserman and Pal have also published a paper under the title of ''Evaluating Open Source Software'', which appears to be more of an updated announcement and in-depth description of the Business Readiness Rating (BRR) framework.<ref name="WassermanEval10">{{cite web |url=http://oss.sv.cmu.edu/readings/EvaluatingOSS_Wasserman.pdf |archiveurl=https://web.archive.org/web/20150218173146/http://oss.sv.cmu.edu/readings/EvaluatingOSS_Wasserman.pdf |format=PDF |title=Evaluating Open Source Software |author=Wasserman, Anthony I.; Pal, Murugan |publisher=Carnegie Mellon University - Silicon Valley |date=2010 |archivedate=18 February 2015 |accessdate=31 May 2015}}</ref> Jadhav and Sonar have also examined the issue of both evaluating and selecting software packages. They include a helpful analysis of the strengths and weaknesses of the various techniques.<ref name="JadhavEval09">{{cite journal |title=Evaluating and selecting software packages: A review |journal=Information and Software Technology |author=Jadhav, Anil S.; Sonar, Rajendra M. |volume=51 |issue=3 |year=March 2009 |pages=555–563 |doi=10.1016/j.infsof.2008.09.003}}</ref> Perhaps more importantly, they clearly point out that there is no common list of evaluation criteria. While the majority of the articles they reviewed listed the criteria used, Jadhav and Sonar indicated that these criteria frequently did not include a detailed definition, which required each evaluator to use their own, sometimes conflicting, interpretation.
 
Since the publication of Stol's and Babar's paper, additional evaluation methods have been published. Of particular interest are a series of papers by Pani, et al. describing their proposed FAME (Filter, Analyze, Measure and Evaluate) methodology.<ref name="PaniAMeth10">{{cite web |title=FAME, A Methodology for Assessing Software Maturity |work=Atti della IV Conferenza Italiana sul Software Libero |author=Pani, F.E.; Sanna, D. |date=11 June 2010 |location=Cagliari, Italy}}</ref><ref name="PaniTheFAMEApp10">{{cite book |chapter=The FAME Approach: An Assessing Methodology |title=Proceedings of the 9th WSEAS International Conference on Telecommunications and Informatics |author=Pani, F.E.; Concas, G.; Sanna, D.; Carrogu, L. |editor=Niola, V.; Quartieri, J.; Neri, F.; Caballero, A.A.; Rivas-Echeverria, F.; Mastorakis, N. |publisher=WSEAS |location=Stevens Point, WI |year=2010 |url=http://www.wseas.us/e-library/conferences/2010/Catania/TELE-INFO/TELE-INFO-10.pdf |format=PDF |isbn=9789549260021}}</ref><ref name="PaniTheFAMEtool10">{{cite journal |title=The FAMEtool: an automated supporting tool for assessing methodology |journal=WSEAS Transactions on Information Science and Applications |author=Pani, F.E.; Concas, G.; Sanna, S.; Carrogu, L. |volume=7 |issue=8 |pages=1078–1089 |year=August 2010 |url=http://www.wseas.us/e-library/transactions/information/2010/88-137.pdf |format=PDF}}</ref><ref name="PaniTrans10">{{cite book |chapter=Transferring FAME, a Methodology for Assessing Open Source Solutions, from University to SMEs |title=Management of the Interconnected World |author=Pani, F.E.; Sanna, D.; Marchesi, M.; Concas, G. |editor= D'Atri, A.; De Marco, M.; Braccini, A.M.; Cabiddu, F. |publisher=Springer |year=2010 |pages=495–502 |isbn=9783790824049 |doi=10.1007/978-3-7908-2404-9_57}}</ref> In their ''Transferring FAME'' paper, they emphasized that all of the evaluation frameworks previously described in the published literature were frequently not easy to apply to real environments, as they were developed using an analytic research approach which incorporated a multitude of factors.<ref name="PaniTrans10" />
 
Their stated design objective with FAME is to reduce the complexity of performing the application evaluation, particularly for small organizations. As specified ”The goals of FAME methodology are to aid the choice of high-quality F/OSS products, with high probability to be sustainable in the long term, and to be as simple and user friendly as possible.” They further state that “The main idea behind FAME is that the users should evaluate which solution amongst those available is more suitable to their needs by comparing technical and economical factors, and also taking into account the total cost of individual solutions and cash outflows. It is necessary to consider the investment in its totality and not in separate parts that are independent of one another.”<ref name="PaniTrans10" />
 
This paper breaks the FAME methodology into four activities:
# Identify the constraints and risks of the projects
# Identify user requirements and rank
# Identify and rank all key objectives of the project
# Generate a priority framework to allow comparison of needs and features
 
Their paper includes a formula for generating a score from the information collected. The evaluated system with the highest 'major score', ''Pjtot'', indicates the system selected. While it is a common practice to define an analysis process which condenses all of the information gathered into a single score, I highly caution against blindly accepting such a score. FAME, as well as a number or the other assessment methodologies, is designed for iterative use. The logical purpose of this is to allow the addition of factors initially overlooked into your assessment, as well as to change the weighting of existing factors as you reevaluate their importance. However, this feature means that it is also very easy to unconsciously, or consciously, skew the results of the evaluation to select any system you wish. Condensing everything down into a single value also strips out much of the information that you have worked so hard to gather. Note that you can generate the same result score using significantly different input values. While of value, selecting a system based on just the highest score could potentially leave you with a totally unworkable system.
 
Pani, et al. also describe a FAMEtool to assist in this data gathering and evaluation.<ref name="PaniTheFAMEtool10" /> However a general web search, as well as a review of their FAME papers revealed no indication of how to obtain this resource. While this paper includes additional comparisons with other FLOSS analysis methodologies and there are some hints suggesting that the FAMEtool is being provided as a web service, I have found no URL specified for it. As of now, I have received no responses from the research team via either e-mail or Skype, regarding FAME, the FAMEtool, or feedback on its use.
 
During this same time frame Soto and Ciolkowski also published papers describing the QualOSS Open Source Assessment Model and compared it to a number of the procedures in Stol's and Barbar's table.<ref name="SotoTheQual09">{{cite book |chapter=The QualOSS open source assessment model measuring the performance of open source communities |title=3rd International Symposium on Empirical Software Engineering and Measurement, 2009 |author=Soto, M.; Ciolkowski, M. |publisher=IEEE |year=2009 |pages=498-501 |doi=10.1109/ESEM.2009.5314237 |isbn=9781424448425}}</ref><ref name="SotoTheQualProc09">{{cite book |chapter=The QualOSS Process Evaluation: Initial Experiences with Assessing Open Source Processes |title=Software Process Improvement |author=Soto, M.; Ciolkowski, M. |editor=O'Connor, R.; Baddoo, N.; Cuadrado-Gallego, J.J.; Rejas Muslera, R.; Smolander, K.; Messnarz, R. |publisher=Springer |year=2009 |pages=105–116 |isbn=9783642041334 |doi=10.1007/978-3-642-04133-4_9}}</ref> Their focus was primarily on three process perspectives: product quality, process maturity, and sustainability of the development community. Due to the lack of anything more than a rudimentary process perspective examination, they felt that the following OSS project assessment models were unsatisfactory: QSOS, CapGemni OSMM, Navica OSMM, and OpenBRR. They position QualOSS as an extension of the tralatitious CMMI and SPICE process maturity models. While there are multiple items in the second paper that are worth incorporating into an in-depth evaluation process, they do not seem suitable for what is intended as a quick survey.
 
Another paper, published by Haaland and Groven also compared a number of Open Source quality models. To this paper's credit, the authors devoted a significant amount of space to discussing the different definitions of quality and how the target audience of a tool might affect which definition was used.<ref name="HaalandFree10">{{cite web |url=http://publications.nr.no/directdownload/publications.nr.no/5444/Haaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf |format=PDF |title=Free/Libre Open Source Quality Models - a comparison between two approaches |work=4th FLOSS International Workshop on Free/Libre Open Source Software |author=Haaland, Kirsten; Groven, Arne-Kristian; Glott, Ruediger; Tannenberg, Anna |location=Jena, Germany |pages=1–17 |date=01 July 2010 |accessdate=15 April 2015}}</ref> Like Stohl and Babar, they listed a number of the quality assessment models to choose from, including OSMM, QSOS, OpenBRR, and others. For their comparison, they selected OpenBRR and QualOSS. They appear to have classified OpenBRR as a first generation tool with a “User view on quality” and QualOSS as a second generation tool with a “business point of view”. An additional variation is that OpenBRR is primarily a manual tool while QualOSS is primarily an automated tool. Their analysis in this article clearly demonstrates the steps involve in using these tools and in highlighting where they are objective and subjective. While they were unable to answer their original question as to whether the first or second generation tools did a better job of evaluation, to me they answered the following even more important, but unasked question. As they proceeded through their evaluation, it became apparent as to how much the questions defined in the methods could affect the results of the evaluations. Even though the authors might have considered the questions to be objective, I could readily see how some of these questions could be interpreted in alternate ways. My takeaway is an awareness of the potential danger of using rigid tools, as they can skew the accuracy of the evaluation results depending on exactly what you want the evaluated application to do and how you plan to use it. These models can be very useful guides, but they should not be used to replace a carefully considered evaluation, as there will always be factors influencing the selection decision which did not occur to anyone when the specifications were being written.
 
Hauge, et al. have noted that despite the development of several normative methods of assessment, empirical studies have failed to show wide spread adoption of these methods.<ref name="HaugeAnEmp09">{{cite book |chapter=An empirical study on selection of Open Source Software - Preliminary results |title=ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development, 2009 |author=Hauge, O.; Osterlie, T.; Sorensen, C.-F.; Gerea, M. |publisher=IEEE |year=2009 |pages=42-47 |doi=10.1109/FLOSS.2009.5071359 |isbn=9781424437207}}</ref> From their survey of a number of Norwegian software companies, they have noticed a tendency for selectors to skip the in-depth search for what they call the 'best fit' application and fall back on what they refer to as a 'first fit'. This is an iterative procedure with the knowledge gained from the failure of one set of component tests being incorporated into the evaluation of the next one. Their recommendation is for researchers to stop attempting to develop either general evaluation schemas or normative selection methods which would be applicable to any software application and instead focus on identifying situationally sensitive factors which can be used as evaluation criteria. This is a very rational approach as all situations, even if evaluating the same set of applications, are going to be different, as each user's needs are different.
 
Ayalal, et al. have performed a study to try to more accurately determine why more people don't take advantage of the various published selection methodologies.<ref name="AyalaTowards11">{{cite book |chapter=Towards Improving OSS Products Selection – Matching Selectors and OSS Communities Perspectives |title=Open Source Systems: Grounding Research |author=Ayala, Claudia; Cruzes, Daniela S.; Franch, Xavier; Conradi, Reidar |editor=Hissam, S.; Russo, B.; de Mendonça Neto, M.G.; Kon, F. |publisher=Springer |year=2011 |pages=244–258 |isbn=9783642244186 |doi=10.1007/978-3-642-24418-6_17}}</ref> While they looked at a number of factors and identified several possible problems, one of the biggest factors was the difficulty in obtaining the needed information for the evaluation. Based on the projects they studied, many did not provide a number of the basic pieces of information required for the evaluation, or perhaps worse, required extensive examination of the project web site and documentation to retrieve the required information. From her paper, it sounded as if this issue was more of a communication breakdown than an attempt to hide any of the information, not that this had any impact on the inaccessibility of the information.
 
In addition to the low engagement rates for the various published evaluation methods, another concern is the viability of the sponsoring organizations. One of the assessment papers indicated that the published methods with the smallest footprint, or the easiest to use, appeared to be FAME and the OpenBRR. I have already mentioned my difficulty obtaining additional information regarding FAME, and OpenBRR appears to be even more problematic. BRR was first registered on SourceForge in September of 2005<ref name="OBRRSource">{{cite web |url=https://sourceforge.net/projects/openbrr/ |title=Business Readiness Rating (BRR) |author=Chan, C.; enugroho; Wasserman, T. |publisher=SourceForge |date=17 April 2013 |accessdate=21 April 2015}}</ref>, and an extensive Request For Comments from the founding members of the BRR consortium (SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel Corporation) was released.<ref name="OpenBRROpen2005" /> In 2006, in contrast to typical Open Source development groups, the OpenBRR group announced the formation of an OpenBRR Corporate Community group. Peter Galli's story indicates that "the current plan is that membership will not be open to all."<ref name="GalliOpenBRR06">{{cite web |url=http://www.eweek.com/c/a/Linux-and-Open-Source/OpenBRR-Launches-Closed-OpenSource-Group |title=OpenBRR Launches Closed Open-Source Group |author=Galli, Peter |work=eWeek |publisher=QuinStreet, Inc |date=24 April 2006 |accessdate=13 April 2015}}</ref> He quotes Murugan Pal saying "membership will be on an invitation-only basis to ensure that only trusted participants are coming into the system." However, for some reason, at least some in the group "expressed concern and unhappiness about the idea of the information discussed not being shared with the broader open-source community."<ref name="GalliOpenBRR06" />
 
While the original Business Readiness Rating web site still exists, it is currently little more than a static web page.<ref name="OBRRArch">{{cite web |url=http://www.openbrr.org/ |archiveurl=https://web.archive.org/web/20141224233009/http://www.openbrr.org/ |title=Welcome to Business Readiness Rating: A FrameWork for Evaluating OpenSource Software |publisher=OpenBRR |archivedate=24 December 2014 |accessdate=14 April 2015}}</ref> It appears that some of the original information posted on the site is still there, you just have to know what its URL is to access it, as the original links on the web site have been removed. Otherwise, you may have to turn to the Internet Archive to retrieve some of their documentation. The lack of any visible activity regarding OpenBRR prompted a blog post from one graduate student in 2012 asking "What happened to OpenBRR (Business Readiness Rating for Open Source)?"<ref name="ArjonaWhat12">{{cite web |url=https://larjona.wordpress.com/2012/01/06/what-happened-to-openbrr-business-readiness-rating-for-open-source/ |title=What happened to OpenBRR (Business Readiness Rating for Open Source)? |author=Arjona, Laura |work=The Bright Side |date=06 January 2012 |accessdate=13 April 2015}}</ref>
 
It appears that at some point, any development activity regarding OpenBRR was morphed into OSSpal.<ref name="OSSpalHome">{{cite web |url=http://osspal.org/ |title=Welcome to OSSpal |publisher=OSSpal |accessdate=18 April 2015}}</ref> However, background information on this project is sparse as well. While the site briefly mentions that OSSpal incorporates a number of lessons learned from BRR, there is very little additional information regarding the group or the methods procedures. Their 'All Projects' tab provides a list of over 30 Open Source projects, but the majority simply show 'No votes yet' under the various headings. In fact, as of now, the only projects showing any input at all are for Ubuntu and Mozilla Firefox.
 
===Evaluation and selection recommendations===
At this point, we'll take a step back from the evaluation methodologies papers and examine some of the more general recommendations regarding evaluating and selecting FLOSS applications. The consistency of their recommendations may provide a more useful guide for an initial survey of FLOSS applications.
 
In ''TechRepublic'', de Silva recommends 10 questions to ask when selecting a FLOSS application.<ref name="Silva10Quest09">{{cite web |url=http://www.techrepublic.com/blog/10-things/10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise/ |title=10 questions to ask when selecting open source products for your enterprise |author=Silva, Chamindra de |work=TechRepublic |publisher=CBS Interactive |date=20 December 2009 |accessdate=13 April 2015}}</ref> While he provides a brief discourse on each question in his paper to ensure you understand the point of his question, I've collected the 10 questions from his article into the following list. Once we see what overlap, if any, are amongst our general recommendations, we'll address some of the consolidated questions in more detail.
 
# Are the open source license terms compatible with my business requirements?
# What is the strength of the community?
# How well is the product adopted by users?
# Can I get a warranty or commercial support if I need it?
# What quality assurance processes exist?
# How good is the documentation?
# How easily can the system be customized to my exact requirements?
# How is this project governed and how easily can I influence the road map?
# Will the product scale to my enterprise's requirements?
# Are there regular security patches?
 
Similarly, in ''InfoWorld'' Phipps lists seven questions you should have answered before even starting to select a software package.<ref name="Phipps7Quest15">{{cite web |url=http://www.infoworld.com/article/2872094/open-source-software/seven-questions-to-ask-any-open-source-project.html |title=7 questions to ask any open source project |author=Phipps, Simon |work=InfoWorld |publisher=InfoWorld, Inc |date=21 January 2015 |accessdate=10 April 2015}}</ref> His list of questions, pulled directly from his article are:
 
# Am I granted copyright permission?
# Am I free to use my chosen business model?
# Am I unlikely to suffer patent attack?
# Am I free to compete with other community members?
# Am I free to contribute my improvements?
# Am I treated as a development peer?
# Am I inclusive of all people and skills?
 
This list of questions shows a moderately different point of view, as it is not just about someone selecting an Open Source system, but looking to be involved in its direct development. Padin, of 8th Light, Inc., takes the viewpoint of a developer who might incorporate Open Source software into their projects.<ref name="PadinHow14">{{cite web |url=https://blog.8thlight.com/sandro-padin/2014/01/03/how-i-evaluate-open-source-software.html |title=How I Evaluate Open-Source Software |author=Padin, Sandro |publisher=8th Light, Inc |date=03 January 2014 |accessdate=01 June 2015}}</ref> The list of criteria pulled directly from his blog includes:
 
# Does it do what I need it to do?
# How much more do I need it to do?
# Documentation
# Easy to review source code
# Popularity
# Tests and specs
# Licensing
# Community
 
Metcalfe of OSS Watch lists his top tips as<ref name="MetcalfeTop13">{{cite web |url=http://oss-watch.ac.uk/resources/tips |title=Top tips for selecting open source software |author=Metcalfe, Randy |work=OSSWatch |publisher=University of Oxford |date=01 February 2004 |accessdate=23 March 2015}}</ref>:
 
# Reputation
# Ongoing effort
# Standards and interoperability
# Support (Community)
# Support (Commercial)
# Version
# Version 1.0
# Documentation
# Skill setting
# Project Development Development Model
# License
 
In his LIMSexpert blog, Joel Limardo of ForwardPhase Technologies, LLC lists the following as components to check when evaluating an Open Source application<ref name="LimardoDIY13">{{cite web |url=http://www.limsexpert.com/cgi-bin/bixchange/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1363288315&title=DIY%20Evaluation%20Process |title=DIY Evaluation Process |author=Limardo, J. |work=LIMSExpert.com |publisher=ForwardPhase Technologies, LLC |date=2013 |accessdate=07 February 2015}}</ref>:
 
* Check licensing
* Check code quality
* Test setup time
* Verify extensibility
* Check for separation of concerns
* Check for last updated date
* Check for dependence on outdated toolkits/frameworks
 
Perhaps the most referenced of the general articles on selecting FLOSS applications is David Wheeler's How to Evaluate Open Source Software / Free Software (OSS/FS) Programs.<ref name="WheelerHow11">{{cite web |url=http://www.dwheeler.com/oss_fs_eval.html |title=How to Evaluate Open Source Software / Free Software (OSS/FS) Programs |author=Wheeler, David A. |work=dwheeler.com |date=05 August 2011 |accessdate=19 March 2015}}</ref> The detailed functionality to consider will vary with the types of applications being compared, but there are a number of general features that are relevant to almost any type of application. While we will cover them in more detail later, Wheeler categorizes the features to consider as the following:
 
* System functionality
* System cost – direct and in-direct
* Popularity of application, i.e. its market share for that type of application
* Varieties of product support available
* Maintenance of application, i.e, is development still taking place
* Reliability of application
* Performance of application
* Scalability of application
* Usability of application
* Security of application
* Adaptability/customizability of application
* Interoperability of application
* Licensing and other legal issues
 
While a hurried glance might suggest a lot of diversity in the features these various resources suggest, a closer look at the meaning of what they are saying show a repetitive series of concerns. The primary significant differences between the functionality lists suggested is actually due more to how wide a breadth of the analysis process the authors are considering, as well as the underlying features that they are concerned with.
 
With a few additions, the high-level screening template described in the rest of this communication is based on Wheeler's previously mentioned document describing his recommended process for evaluating open source software and free software programs. Structuring the items thus will make it easier to locate the corresponding sections in his document, which includes many useful specific recommendations, as well as a great deal of background information to help you understand the why of the topic. I highly recommend reading it and following up on some of the links he provides. I will also include evaluation suggestions from several of the previously mentioned procedures where appropriate.
 
Wheeler defines four basic steps to this evaluation process, as listed below:
 
* Identify candidate applications
* Read existing product reviews
* Compare attributes of these applications to your needs
* Analyze the applications best matching your needs in more depth
 
Wheeler categorizes this process with the acronym IRCA. In this paper we will be focusing on the IRC components of this process. To confirm the efficacy of this protocol we will later apply it to several classes of Open Source applications and examine the output of the protocol.
 
====Identify needs====
Realistically, before you can perform a survey of applications to determine which ones best match your needs, you must determine what your needs actually are. The product of determining these needs is frequently referred to as the User Requirements Specification (URS).<ref name="VOURS15">{{cite web |url=http://www.validation-online.net/user-requirements-specification.html |title=User Requirements Specification (URS) |work=validation-online.net |publisher=Validation Online |accessdate=08 August 2015}}</ref><ref name="OKeeffeHow15">{{cite web |url=http://www.askaboutgmp.com/296-how-to-create-a-bullet-proof-urs |title=How to Create a Bullet-Proof User Requirement Specification (URS) |author=O'Keefe, Graham |publisher=askaboutgmp |date=01 March 2015 |accessdate=08 August 2015}}</ref> This document can be generated in several ways, including having all of the potential users submit a list of the functions and capabilities that they feel is important. While the requirements document can be created by a single person, it is generally best to make it a group effort with multiple reviews of the draft document and including all of the users who will be working with the application. The reason for this is to ensure that an important requirement is not missed, When a requirement is missed, it is frequently due to the requirement being so basic that it never occurs to anyone that it specifically needed to be included in the requirements document. Admittedly, a detailed URS is not required at the survey level, but it is worth having if only to identify, by their implications, other features that might be significant.
 
Needs will, of course, vary with the type of application you are looking for and what you are planning to do with it. Keep in mind that the URS is a living document, subject to change through this whole process. Developing a URS is generally an iterative process, since as you explore systems, you may well see features that you hadn't considered that you find desirable. This process will also be impacted by whether the application to be selected will be used in a regulated environment. If it is, there will be existing documents that describe the minimum functionality that must be present in the system. Even if it is not to be used in a regulated environment, documents exist for many types of systems that describe the recommended functional requirements that would be expected for that type of system.
 
For a clarifying example, if you were attempting to select a Laboratory Information Management System (LIMS), you can download checklists and standards of typical system requirements from a variety of sources.<ref name="ASTMStandsLIMS">{{cite web |url=http://www.astm.org/Standards/E1578.htm |title=ASTM E1578-13, Standard Guide for Laboratory Informatics |publisher=ASTM International |location=West Conshohocken, PA |doi=10.1520/E1578 |date=2013 |accessdate=14 March 2015}}</ref><ref name="ASURCheck">{{cite web |url=http://www.autoscribeinformatics.com/services/user-requirements |title=User Requirements Checklist |publisher=Autoscribe Informatics |accessdate=10 April 2015}}</ref><ref name="LIIComplete15">{{cite web |url=http://www.limsbook.com/the-complete-guide-to-lims-laboratory-informatics-2015-edition/ |title=The Complete Guide to LIMS & Laboratory Informatics – 2015 Edition |editor=Laboratory Informatics Institute |publisher=LabLynx, Inc |date=2015 |accessdate=10 April 2015}}</ref><ref name="FDAP1115">{{cite web |url=http://www.fda.gov/regulatoryinformation/guidances/ucm125067.htm |title=Part 11, Electronic Records; Electronic Signatures — Scope and Application |publisher=U.S. Food and Drug Administration |date=26 August 2015 |accessdate=10 June 2015}}</ref> These will provide you with examples of the questions to ask, but you will have to determine which ones are important to, or required for, your particular effort.
 
Depending on the use to which this application is to be applied, you may be subject to other specific regulatory requirements as well. Which regulations may vary, since the same types of analysis performed for different industries fall under different regulatory organizations. This aspect is further complicated by the fact that you may be affected by more than one countries' regulations if your analysis are applicable to products being shipped to other countries. While some specific regulations may have changed since its publication, an excellent resource to orient you to the diverse factors that must be considered is Siri Segalstad's book ''International IT Regulations and Compliance''.<ref name="SegalstadInt08">{{cite book |title=International IT Regulations and Compliance: Quality Standards in the Pharmaceutical and Regulated Industries |author=Segalstad, Siri H. |publisher=John Wiley & Sons, Inc |year=2008 |pages=338 |isbn=9780470758823 |url=http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470758821.html}}</ref> My understanding is that an updated version of this book is currently in preparation. Keep in mind that while regulatory requirements that you must meet will vary, these regulations by and large also describe best practices, or at least the minimal allowed practices. These requirements are not put in place arbitrarily (generally) or to make things difficult for you, but to ensure the quality of the data produced. As such, any deviations should be carefully considered, whether working in a regulated environment or not. Proper due diligence would be to determine which regulations and standards would apply to your operation.
 
For a LIMS, an example of following best practices is to ensure that the application has a full and detailed [[audit trail]]. An audit trail allows you to follow the processing of items through your system, determining who did what and when. In any field where it might become important to identify the actions taken during a processing step, an audit trail should be mandatory. While your organization's operations may not fall under the FDA's [[21 CFR Part 11]] regulations, which address data access and security, including audit trails, it is still extremely prudent that the application you select complies with them. If it does not, then almost anyone could walk up to your system and modify data, either deliberately or accidentally, and you would have no idea of who made the changes or what changes they made. For that matter, you might not even be able to tell a change was made at all, which likely will raise concerns both inside and outside of your organization. This would obviously cause major problems if they became a hinge issue for any type of liability law suit.
 
For this screening procedure, you do not have to have a fully detailed URS, but it is expedient to have a list of your make-or-break issues. This list will be used later for comparing systems and determining which ones justify a more in-depth evaluation.
 
====Identify candidates====
To evaluate potential applications against your functional criteria, you must initially generate a list of potential systems. While this might sound easy, generating a comprehensive list frequently proves to be a challenge. When initiating the process, you must first determine the type of system that you are looking for, be it a LIMS, a hospital management system, a data base, et al. At this point, you should be fairly open in building your list of candidates. By that, I mean that you should be careful not to select applications based solely on the utilization label applied to them. The same piece of software can frequently be applied to solve multiple problems, so you should cast a wide net and not automatically reject a system because the label you were looking for hadn't been applied to it. While the label may give you a convenient place to start searching, it is much more important to look for the functionality that you need, not what the system is called. In any case, many times the applied labels are vague and mean very different things to different people.
 
There are a variety of ways to generate your candidate list. A good place to start is simply talking with colleagues in your field. Have they heard of or use a FLOSS application of the appropriate type that they like? Another way is to just flip through journals and trade magazines that cover your field. Any sufficiently promising applications are likely to be mentioned there. Many of the trade magazines will have a special annual issue that covers equipment and software applicable to their field. It is difficult to generate a list of all potential resources, as many of these trade publications are little known outside of their field. Also keep in mind that with the continued evolution of the World Wide Web, many of these trade publications also have associated web sites that you can scan or search. The table below includes just a minor fraction of these sites that are available. (We would welcome the suggestion of any additional resource sites that you are aware of. Please e-mail the fields covered, the resource name, and either its general URL or the URL of the specific resource section to the corresponding editor.)
 
{|
| STYLE="vertical-align:top;"|
{| class="wikitable" border="1" cellpadding="5" cellspacing="0" width="100%"
|-
  ! Field
  ! Resource Name
  ! URL
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Astronomy
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Tech Support Alert
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.techsupportalert.com/best-free-astronomy-software.htm http://www.techsupportalert.com/best-free-astronomy-software.htm]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Business intelligence reporting
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Technology Innovation Management Review
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://timreview.ca/article/288 http://timreview.ca/article/288]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Community radio
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Prometheus Radio Project
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://prometheusradio.org/Free_Open_Source_Tools_C_R http://prometheusradio.org/Free_Open_Source_Tools_C_R]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Comprehensive range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Black Duck KnowledgeBase
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://www.blackducksoftware.com/products/knowledgebase https://www.blackducksoftware.com/products/knowledgebase]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Comprehensive range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|The Directory of Open Access Repositories - OpenDOAR
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.opendoar.org/ http://www.opendoar.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Data storage
  | style="background-color:white; padding-left:10px; padding-right:10px;"|InfoStor
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.infostor.com/nas/58-top-open-source-storage-project-1.html http://www.infostor.com/nas/58-top-open-source-storage-project-1.html]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Digital audio editing
  | style="background-color:white; padding-left:10px; padding-right:10px;"|25 Free Digital Audio Editors You Should Know
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.hongkiat.com/blog/25-free-digital-audio-editors/ http://www.hongkiat.com/blog/25-free-digital-audio-editors/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Digital video editing
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Best free video editing software: 20 top programs in 2015
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.techradar.com/us/news/software/applications/best-free-video-editing-software-9-top-programs-you-should-download-1136264 http://www.techradar.com/us/news/software/applications/best-free-video-editing-software-9-top-programs-you-should-download-1136264]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of Android apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia – Android applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_free_and_open-source_Android_applications https://en.wikipedia.org/wiki/List_of_free_and_open-source_Android_applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Launchpad, The Canonical Group
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://launchpad.net/ http://launchpad.net/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Microsoft CodePlex
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://www.codeplex.com/ https://www.codeplex.com/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Hub, Black Duck Software, Inc.
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://www.openhub.net/ https://www.openhub.net/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps (but site is deprecated)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Google Code
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://code.google.com/p/support/ https://code.google.com/p/support/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps (focus on infrastructure)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source Guide by Smile
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.open-source-guide.com/en http://www.open-source-guide.com/en]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of apps (primarily targeting Samsung products)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Samsung Open Source Release Center
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensource.samsung.com/reception.do http://opensource.samsung.com/reception.do]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of commercial open source apps and services
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia – Commercial open source applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_commercial_open-source_applications_and_services https://en.wikipedia.org/wiki/List_of_commercial_open-source_applications_and_services]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of GNU apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Savannah GNU
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://savannah.gnu.org/ http://savannah.gnu.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of non-GNU apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Savannah Non-GNU
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://savannah.nongnu.org http://savannah.nongnu.org]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Diverse range of top apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Projects and Applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensource.com/resources/projects-and-applications http://opensource.com/resources/projects-and-applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Drug discovery
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Drug Discovery Today
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.sciencedirect.com/science/article/pii/S1359644605036925 http://www.sciencedirect.com/science/article/pii/S1359644605036925]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Electronic engineering
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Education Engineering (EDUCON), 2010 IEEE
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5492430 http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5492430]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Embroidery
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Machine Embroidery Portal
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.k2g2.org/portal:machine_embroidery http://www.k2g2.org/portal:machine_embroidery]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Enterprise resource planning (ERP)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|A Comparison of Open Source ERP Systems
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.big.tuwien.ac.at/system/theses/20/papers.pdf?1298476232 http://www.big.tuwien.ac.at/system/theses/20/papers.pdf?1298476232]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Enterprise resource planning (ERP)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source ERP Site
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.open-source-erp-site.com/list-of-open-source-erps.html http://www.open-source-erp-site.com/list-of-open-source-erps.html]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geographic information systems (GIS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source GIS
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensourcegis.org/ http://opensourcegis.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geophysics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geopsy project
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.geopsy.org/ http://www.geopsy.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Geophysics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia - Geophysics software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/Comparison_of_free_geophysics_software https://en.wikipedia.org/wiki/Comparison_of_free_geophysics_software]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|GIS for libraries
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Library Hi Tech
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.emeraldinsight.com/doi/abs/10.1108/07378831011026742 http://www.emeraldinsight.com/doi/abs/10.1108/07378831011026742]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Help desk
  | style="background-color:white; padding-left:10px; padding-right:10px;"|CIO
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.cio.com.au/article/320110/5_open_source_help_desk_apps_watch/ http://www.cio.com.au/article/320110/5_open_source_help_desk_apps_watch/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Highly diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Code NASA
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://code.nasa.gov/#/ https://code.nasa.gov/#/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Highly diverse range of apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|NASA Open Source Software
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://ti.arc.nasa.gov/opensource/ http://ti.arc.nasa.gov/opensource/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Laboratory informatics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|LIMSwiki - Laboratory informatics software (open source)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.limswiki.org/index.php/Category:Laboratory_informatics_software_%28open_source%29 http://www.limswiki.org/index.php/Category:Laboratory_informatics_software_%28open_source%29]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Learning management systems (LMS)
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source LMS – 10 Alternatives to Moodle
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://barrysampson.com/2009/04/08/open-source-lms-10-alternatives-to-moodle/ http://barrysampson.com/2009/04/08/open-source-lms-10-alternatives-to-moodle/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Library science
  | style="background-color:white; padding-left:10px; padding-right:10px;"|FOSS4LIB
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://foss4lib.org/packages https://foss4lib.org/packages]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Limited range of iOS apps
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia - Open-source iOS applications
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_free_and_open-source_iOS_applications https://en.wikipedia.org/wiki/List_of_free_and_open-source_iOS_applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medical and laboratory informatics
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Healthcare Freeware
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.healthcarefreeware.com/lim.htm http://www.healthcarefreeware.com/lim.htm]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medical office records
  | style="background-color:white; padding-left:10px; padding-right:10px;"|ZDNet
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.zdnet.com/article/free-and-open-source-healthcare-software-for-your-practice/ http://www.zdnet.com/article/free-and-open-source-healthcare-software-for-your-practice/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|'Open' Health IT Solutions for the Public & Private Sectors
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://drive.google.com/file/d/0B_TuX9zE68eWcVM1dkdDQTVCdnM/edit https://drive.google.com/file/d/0B_TuX9zE68eWcVM1dkdDQTVCdnM/edit]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|50 Successful Open Source Projects That Are Changing Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://nursingassistantguides.com/2009/50-successful-open-source-projects-that-are-changing-medicine/ http://nursingassistantguides.com/2009/50-successful-open-source-projects-that-are-changing-medicine/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Datamation
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.datamation.com/open-source/50-open-source-replacements-for-health-care-software-1.html http://www.datamation.com/open-source/50-open-source-replacements-for-health-care-software-1.html]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medicine
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Medfloss.org
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.medfloss.org/ http://www.medfloss.org/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|MIDI controllers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source MIDI Controllers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://punkmanufacturing.com/wiki/open-source-midi-controllers http://punkmanufacturing.com/wiki/open-source-midi-controllers]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Music
  | style="background-color:white; padding-left:10px; padding-right:10px;"|20 great free and open source music making programs
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://www.musicradar.com/tuition/tech/20-great-free-and-open-source-music-making-programs-582934 http://www.musicradar.com/tuition/tech/20-great-free-and-open-source-music-making-programs-582934]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Pattern management for textiles
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Tools for computer generated patterns
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://xxxclairewilliamsxxx.wordpress.com/tools-to-create-and-explore-digital-patterns/ https://xxxclairewilliamsxxx.wordpress.com/tools-to-create-and-explore-digital-patterns/]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Personal information management
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Wikipedia - Personal information managers
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[https://en.wikipedia.org/wiki/List_of_personal_information_managers#Open_source_applications https://en.wikipedia.org/wiki/List_of_personal_information_managers#Open_source_applications]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Photography
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source Photography
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensourcephotography.org/software-list http://opensourcephotography.org/software-list]
|-
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Project management
  | style="background-color:white; padding-left:10px; padding-right:10px;"|Open Source Photography
  | style="background-color:white; padding-left:10px; padding-right:10px;"|[http://opensourcephotography.org/software-list http://opensourcephotography.org/software-list]
|-
|}
|}


==References==
==References==
{{Reflist|colwidth=30em}}
{{Reflist|colwidth=30em}}
==Notes==
This article has not officially been published in a journal. However, this presentation is faithful to the original paper, with only a few minor changes to presentation. This article is being made available for the first time under the [https://creativecommons.org/licenses/by-sa/4.0/ Creative Commons Attribution-ShareAlike 4.0 International] license, the same license used on this wiki.
<!--Place all category tags here-->
[[Category:LIMSwiki journal articles (all)‎]]
[[Category:LIMSwiki journal articles on software‎‎]]

Latest revision as of 19:33, 17 February 2023

Sandbox begins below

  • Discussion and practical use of artificial intelligence (AI) in the laboratory is, perhaps to the surprise of some, not a recent phenomena. In the mid-1980s, researchers were developing computerized AI systems able "to develop automatic decision rules for follow-up analysis of [clinical laboratory] tests depending on prior information, thus avoiding the delays of traditional sequential testing and the costs of unnecessary parallel testing."[1] In fact, discussion of AI in general was ongoing even in the mid-1950s.[2][3]
  • Hiring demand for laboratorians with AI experience (2015–18) has historically been higher in non-healthcare industries, such as manufacturing, mining, and agriculture, shedding a light on how AI adoption in the clinical setting may be lacking. According to the Brookings Institute, "Even for the relatively-skilled job postings in hospitals, which includes doctors, nurses, medical technicians, research lab workers, and managers, only approximately 1 in 1,250 job postings required AI skills." They add: "AI adoption may be slow because it is not yet useful, or because it may not end up being as useful as we hope. While our view is that AI has great potential in health care, it is still an open question."[4]
  • Today, AI is being practically used in not only clinical diagnostic laboratories but also clinical research labs, life science labs, and research and development (R&D) labs, and more. Practical uses of AI can be found in:
clinical research labs[5]
hospitals[5][6]
medical diagnostics labs[6][7][8][9][10][11]
chromatography labs[11]
biology and life science labs[12]
medical imaging centers[13]
ophthalmology clinics[14]
reproduction clinics[15][16][17]
digital pathology labs[18]
material testing labs[19][20][21]
chemical experimentation and molecular discovery labs[21][22][23]
quantum physics labs[24]
  • What's going on in these labs?
Materials science: The creation of "a modular robotic platform driven by a model-based optimization algorithm capable of autonomously optimizing the optical and electronic properties of thin-film materials by modifying the film composition and processing conditions ..."[19]
Materials science: "Most of the applications of [machine learning (ML)] in chemical and materials sciences, as we have said, feature supervised learning algorithms. The goal there is to supplement or replace traditional modeling methods, at the quantum chemical or classical level, in order to predict the properties of molecules or materials directly from their structure or their chemical composition ... Our research group was applying the same idea on a narrower range of materials, trying to confirm that for a given chemical composition, geometrical descriptors of a material’s structure could lead to accurate predictions of its mechanical features."[20]
Life science: "In biological experiments, we generally cannot as easily declare victory, but we can use the systems biology approach of cycling between experimentation and modelling to see which sequences, when tested, are most likely to improve the model. In artificial intelligence, this is called active learning, and it has some similarity to the way in which we as humans learn as infants: we get some help from parents and teachers, but mainly model the world around us by exploring it and interacting with it. Ideally then, we would recreate such an environment for our machine learning algorithms in the laboratory, where we start with an initial ‘infant’ model of a certain regulatory system or protein function and let the computer decide what sequence designs to try out – a deep learning version of the ‘robot scientist’. Microbes are ideal organisms for such an approach, given the ease and speed with which they can be grown and genetically manipulated. Combined with laboratory automation, many microbial experiments can (soon) be performed with minimal human intervention, ranging from strain construction and screening, such as operated by Amyris, Gingko, Transcriptic, etc., to full-genome engineering or even the design of microbial ecologies."[12]
Digital pathology: "The collaboration combines two AI solutions, VistaPath’s Sentinel, the world’s first automated tissue grossing platform, and Gestalt’s AI Requisition Engine (AIRE), a leading-edge AI algorithm for accessioning, to raise the bar in AI-driven pathology digitization. Designed to make tissue grossing faster and more accurate, VistaPath’s Sentinel uses a high-quality video system to assess specimens and create a gross report 93% faster than human technicians with 43% more accuracy. It not only improves on quality by continuously monitoring the cassette, container, and tissue to reduce mislabeling and specimen mix-up, but also increases traceability by retaining original images for downstream review."[25]
Chemistry and molecular science: "The benefits of combining automated experimentation with a layer of artificial intelligence (AI) have been demonstrated for flow reactors, photovoltaic films, organic synthesis, perovskites and in formulation problems. However, so far no approaches have integrated mobile robotics with AI for chemical experiments. Here, we built Bayesian optimization into a mobile robotic workflow to conduct photocatalysis experiments within a ten-dimensional space."[22]
Chemistry and immunology: "Chemistry and immunology laboratories are particularly well-suited to leverage machine learning because they generate large, highly structured data sets, Schulz and others wrote in a separate review paper. Labor-intensive processes used for interpretation and quality control of electrophoresis traces and mass spectra could benefit from automation as the technology improves, they said. Clinical chemistry laboratories also generate digital images—such as urine sediment analysis—that may be highly conducive to semiautomated analyses, given advances in computer vision, the paper noted."[26]
Clinical research: "... retrospective analysis of existing patient data for descriptive and clustering purposes [and] automation of knowledge extraction, ranging from text mining, patient selection for trials, to generation of new research hypotheses ..."[5]
Clinical research: "AI ... offers a further layer to the laboratory system by analyzing all experimental data collected by experiment devices, whether it be a sensor or a collaborative robot. From data collected, AI is able to produce hypotheses and predict which combination of materials or temperature is desired for the experiment. In short, this system will allow scientists to be aided by a highly intelligent system which is constantly monitoring and analyzing the experimental output. In this way, AI will help an experiment from its inception to conclusion."[27]
Clinical research/medical diagnostics: "Artificial intelligence (AI) in the laboratory is primarily used to make sense of big data, the almost impossibly large sets of data that biologists and pharmaceutical R&D teams are accustomed to working with. AI algorithms can parse large amounts of data in a short amount of time and turn that data into visualizations that viewers can easily understand. In certain data-intensive fields, such as genomic testing and virus research, AI algorithms are the best way to sort through the data and do some of the pattern recognition work."[28]
Medical diagnostics: Development and implementation of clinical decision support systems [5][6]
Medical diagnostics: "Finally, in the laboratory, AI reduces the number of unnecessary blood samples when diagnosing infection. Instead of the 'gold standard blood sample' that takes 24-72 hours, the algorithm can predict the outcome of the blood sample with almost 80% accuracy based on demographics, vital signs, medications, and laboratory and radiology results. These are all examples of how Artificial Intelligence can be used to test better and faster with information that already exists. This saves time and costs."[10]
Medical diagnostics: "Chang sees two overarching classes of AI models: those that tackle internal challenges in the lab, such as how to deliver more accurate results to clinicians; and those that seek to identify cohorts of patients and care processes to close quality gaps in health delivery systems. The lab, however, 'isn’t truly an island,' said Michelle Stoffel, MD, PhD, associate chief medical information officer for laboratory medicine and pathology at M Health Fairview and the University of Minnesota in Minneapolis. 'When other healthcare professionals are working with electronic health records or other applications, there could be AI-driven tools, or algorithms used by an institution’s systems that may draw on laboratory data.'"[26]
Medical diagnostics: AI is used for the formulation of reference ranges, improvement of quality control, and automated interpretation of results. "Continuous monitoring of specimen acceptability, collection and transport can result in the prompt identification and correction of problems, leading to improved patient care and a reduction in unnecessary redraws and delays in reporting results."[8]
Reproduction science: "The field of AI is the marriage of humans and computers while reproductive medicine combines clinical medicine and the scientific laboratory of embryology. The application of AI has the potential to disconnect healthcare professionals from patients through algorithms, automated communication, and clinical imaging. However, in the embryology laboratory, AI, with its focus on gametes and embryos, can avoid the same risk of distancing from the patient. Areas of application of AI in the laboratory would be to enhance and automate embryo ranking through analysis of images, the ultimate goal being to predict successful implantation. Might such a trend obviate the need for embryo morphological assessment, time-lapse imaging and preimplantation genetic testing for aneuploidy (PGT-A), including mosaicism. Additionally, AI could assist with automation through analysis of testicular sperm samples searching for viable gametes, embryo grading uniformity."[15]
Chromatography-heavy sciences: " A great example of this is AI in the Liquid Chromatography Mass Spectrometry (LC-MS) field. LC-MS is a great tool used to measure various compounds in the human body, including everything from hormone levels to trace metals. One of the ways AI has already integrated with LC-MS is how it cuts down on the rate limiting steps of LC-MS, which more often than not are sample prep and LC separations. One system that Physicians Lab has made use of is parallel processing using SCIEX MPX 2.0 High Throughput System. This system can couple parallel runs with one LCMS instrument, resulting in twice the speed with no loss to accuracy. It can do this by staggering two runs either using the same method, or different methods entirely. What really makes this system great is its ability to automatically detect carryover and inject solvent blanks to clean the instrument. The system will then continue its analyzing, while automatically reinjecting samples that may be affected by the carryover. It will also flag high concentration without user input, allowing for easy detection of possibly faulty samples. This allows it to operate without users from startup to shut down. Some of the other ways that it can be used to increase efficiency are by using integrated network features to work on anything from streamlining management to increased throughput."[11]
Most any lab: "Predictive analytics, for example, is one tool that the Pistoia Alliance is using to better understand laboratory instruments and how they might fail over time... With the right data management strategies and careful consideration of metadata, how to best store data so that it can be used in future AI and ML workflows is essential to the pursuit of AI in the laboratory. Utilizing technologies such as LIMS and ELN enables lab users to catalogue data, providing context and instrument parameters that can then be fed into AI or ML systems. Without the correct data or with mismatched data types, AI and ML will not be possible, or at the very least, could provide undue bias trying to compare data from disparate sources."[29]
Most any lab: "When the actionable items are automatically created by Optima, the 'engine' starts working. An extremely sophisticated algorithm is able to assign the tasks to the resources, both laboratory personnel and instruments, according to the system configuration. Optima, thanks to a large amount of time dedicated to research the best way to automate this critical process, is able to automate most of the lab resource scheduling."[30]
  • A number of challenges exist in the realm of effectively and securely implementing AI in the laboratory. This includes:
Ethical and privacy challenges[5][26][31]
Algorithmic limitations[4]
Data access limitations, including "where to get it, how to share it, and how to know when you have enough to train a machine-learning system that will produce good results"[4][26][32][33]
Data integration and transformation issues[5][33]
Regulatory barriers[4][7]
Misaligned incentives[4]
Lack of knowledgeable/skilled talent[5][26][32][33]
Cost of skilled talent and infrastructure for maintaining and updating AI systems[26]
Legacy systems running outdated technologies[32]
Lack of IT systems or specialized software systems[33]
Lack of standardized, best practices-based methods of validating algorithms[26]
Failure to demonstrate real-world performance[7]
Failure to meet the needs of the professionals using it[7]
  • Given those challenges, some considerations should be made about implementing AI-based components in the laboratory. Examples include:
Clinical diagnostics: "From an industry and regulatory perspective, however, only the intended uses supported from the media manufacturer can be supported from AI applications, unless otherwise justified and substantive evidence is presented for additional claims support. This means strict adherence to specimen type and incubation conditions. Considering that the media was initially developed for human assessment using the well-trained microbiologist eye, and not an advanced imaging system with or without AI, this paradigm should shift to allow advancements in technology to challenge the status-quo of decreasing media read-times especially, as decreased read-times assist with laboratory turnaround times and thus patient management. Perhaps with an increasing body of evidence to support any proposed indications for use, either regulatory positions should be challenged, or manufacturers of media and industry AI-development specialists should work together to advance the field with new indications for use.
While the use of AI in the laboratory setting can be highly beneficial there are still some issues to be addressed. The first being phenotypically distinct single organism polymorphisms that may be interpreted by AI as separate organisms, as may also be the case for a human assessment, as well as small colony variant categorization. As detailed earlier, the broader the inputs, the greater the generalization of the model, and the higher the likelihood of algorithm accuracy. In that respect, understanding and planning around these design constraints is critical for ultimate deployment of algorithms. Additionally, expecting an AI system to correctly categorize “contamination” is a difficult task as often this again seemingly innocuous decision is dependent on years of experience and understanding the specimen type and the full clinical picture with detailed clinical histories. In this respect, a fully integrated AI-LIS system where all data is available may assist, but it is currently not possible to gather this granular detail needed to make this assessment reliable."[9]
Clinical diagnostics and pathology: "Well, if I’ve learned anything in my research into this topic, it’s that AI implementation needs to be a two-way street. First, any company who is active in this space must reach out to pathologists and laboratory medicine professionals to understand their daily workflows, needs, and pain points in as much detail as possible. Second, pathologists, laboratory medicine professionals, and educators must all play their important part – willingly offering their time and expertise when it is sought or proactively getting involved. And finally, it’s clear that there is an imbalanced focus on certain issues – with privacy, respect, and sustainability falling by the wayside."[31]
Healthcare: "While we are encouraged by the promise shown by AI in healthcare, and more broadly welcome the use of digital technologies in improving clinical outcomes and health system productivity, we also recognize that caution must be exercised when introducing any new healthcare technology. Working with colleagues across the NHS Transformation Directorate, as well as the wider AI community, we have been developing a framework to evaluate AI-enabled solutions in the health and care policy context. The aim of the framework is several-fold but is, at its core, a tool with which to highlight to healthcare commissioners, end users, patients and members of the public the considerations to be mindful when introducing AI to healthcare settings."[34]
Most any lab: A code of AI ethics should address objectivity, privacy, transparency, accountability, and sustainability in any AI implementation.[31]
Most any lab: "Another approach is to implement an AI program alongside a manual process, assessing its performance along the way, as a means to ease into using the program. 'I think one of the most impactful things that laboratorians can do today is to help make sure that the lab data that they’re generating is as robust as possible, because these AI tools rely on new training sets, and their performance is really only going to be as good as the training data sets they’re given,' Stoffel said."[26]

References

  1. Berger-Hershkowitz, H.; Neuhauser, D. (1987). "Artificial intelligence in the clinical laboratory". Cleveland Clinic Journal of Medicine 54 (3): 165–166. doi:10.3949/ccjm.54.3.165. ISSN 0891-1150. PMID 3301059. https://www.ccjm.org/content/54/3/165. 
  2. Minsky, M. (17 December 1956). Heuristic Aspects of the Artificial Intelligence Problem. Ed Services Technical Information Agency. https://books.google.com/books?hl=en&lr=&id=fvWNo6_IZGUC&oi=fnd&pg=PA1. Retrieved 16 February 2023. 
  3. Minsky, Marvin (1 January 1961). "Steps toward Artificial Intelligence". Proceedings of the IRE 49 (1): 8–30. doi:10.1109/JRPROC.1961.287775. ISSN 0096-8390. http://ieeexplore.ieee.org/document/4066245/. 
  4. 4.0 4.1 4.2 4.3 4.4 Goldfarb, A.; Teodoridis, F. (9 March 2022). "Why is AI adoption in health care lagging?". Series: The Economics and Regulation of Artificial Intelligence and Emerging Technologies. Brookings Institute. https://www.brookings.edu/research/why-is-ai-adoption-in-health-care-lagging/. Retrieved 17 February 2023. 
  5. 5.0 5.1 5.2 5.3 5.4 5.5 5.6 Damiani, A.; Masciocchi, C.; Lenkowicz, J.; Capocchiano, N. D.; Boldrini, L.; Tagliaferri, L.; Cesario, A.; Sergi, P. et al. (7 December 2021). "Building an Artificial Intelligence Laboratory Based on Real World Data: The Experience of Gemelli Generator". Frontiers in Computer Science 3: 768266. doi:10.3389/fcomp.2021.768266. ISSN 2624-9898. https://www.frontiersin.org/articles/10.3389/fcomp.2021.768266/full. 
  6. 6.0 6.1 6.2 University of California, San Francisco; Adler-Milstein, Julia; Aggarwal, Nakul; University of Wisconsin-Madison; Ahmed, Mahnoor; National Academy of Medicine; Castner, Jessica; Castner Incorporated et al. (29 September 2022). "Meeting the Moment: Addressing Barriers and Facilitating Clinical Adoption of Artificial Intelligence in Medical Diagnosis". NAM Perspectives 22 (9). doi:10.31478/202209c. PMC PMC9875857. PMID 36713769. https://nam.edu/meeting-the-moment-addressing-barriers-and-facilitating-clinical-adoption-of-artificial-intelligence-in-medical-diagnosis. 
  7. 7.0 7.1 7.2 7.3 Government Accountability Office (GAO); National Academy of Medicine (NAM) (September 2022). "Artificial Intelligence in Health Care: Benefits and Challenges of Machine Learning Technologies for Medical Diagnostics" (PDF). Government Accountability Office. https://www.gao.gov/assets/gao-22-104629.pdf. Retrieved 16 February 2023. 
  8. 8.0 8.1 Wen, Xiaoxia; Leng, Ping; Wang, Jiasi; Yang, Guishu; Zu, Ruiling; Jia, Xiaojiong; Zhang, Kaijiong; Mengesha, Birga Anteneh et al. (24 September 2022). "Clinlabomics: leveraging clinical laboratory data by data mining strategies" (in en). BMC Bioinformatics 23 (1): 387. doi:10.1186/s12859-022-04926-1. ISSN 1471-2105. PMC PMC9509545. PMID 36153474. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-022-04926-1. 
  9. 9.0 9.1 DeYoung, B.; Morales, M.; Giglio, S. (4 August 2022). "Microbiology 2.0–A “behind the scenes” consideration for artificial intelligence applications for interpretive culture plate reading in routine diagnostic laboratories". Frontiers in Microbiology 13: 976068. doi:10.3389/fmicb.2022.976068. ISSN 1664-302X. PMC PMC9386241. PMID 35992715. https://www.frontiersin.org/articles/10.3389/fmicb.2022.976068/full. 
  10. 10.0 10.1 Schut, M. (1 December 2022). "Get better with bytes". Amsterdam UMC. https://www.amsterdamumc.org/en/research/news/get-better-with-bytes.htm. Retrieved 16 February 2023. 
  11. 11.0 11.1 11.2 Albano, V.; Morris, C.; Kent, T. (6 December 2019). "Calculations to Diagnosis: The Artificial Intelligence Shift That’s Already Happening". Physicians Lab. https://physicianslab.com/calculations-to-diagnosis-the-artificial-intelligence-shift-thats-already-happening/. Retrieved 16 February 2023. 
  12. 12.0 12.1 de Ridder, Dick (1 January 2019). "Artificial intelligence in the lab: ask not what your computer can do for you" (in en). Microbial Biotechnology 12 (1): 38–40. doi:10.1111/1751-7915.13317. PMC PMC6302702. PMID 30246499. https://onlinelibrary.wiley.com/doi/10.1111/1751-7915.13317. 
  13. Brandao-de-Resende, C.; Bui, M.; Daneshjou, R. et al. (11 October 2022). "AI Webinar: Clinical Adoption of AI Across Image Producing Specialties". Society for Imaging Informatics in Medicine. https://siim.org/page/22w_clinical_adoption_of_ai. 
  14. He, Mingguang; Li, Zhixi; Liu, Chi; Shi, Danli; Tan, Zachary (1 July 2020). "Deployment of Artificial Intelligence in Real-World Practice: Opportunity and Challenge" (in en). Asia-Pacific Journal of Ophthalmology 9 (4): 299–307. doi:10.1097/APO.0000000000000301. ISSN 2162-0989. https://journals.lww.com/10.1097/APO.0000000000000301. 
  15. 15.0 15.1 Trolice, Mark P.; Curchoe, Carol; Quaas, Alexander M (1 July 2021). "Artificial intelligence—the future is now" (in en). Journal of Assisted Reproduction and Genetics 38 (7): 1607–1612. doi:10.1007/s10815-021-02272-4. ISSN 1058-0468. PMC PMC8260235. PMID 34231110. https://link.springer.com/10.1007/s10815-021-02272-4. 
  16. European Society of Human Reproduction and Embryology (6 July 2022). "Annual Meeting 2022: Artificial intelligence in embryology and ART". Focus on Reproduction. https://www.focusonreproduction.eu/article/ESHRE-News-22AI. Retrieved 16 February 2023. 
  17. Hinckley, M. (17 March 2021). "Applying AI (Artificial Intelligence) in the Lab for Better IVF Success". Reproductive Science Center Blog. Reproductive Science Center of the Bay Area. https://rscbayarea.com/blog/applying-ai-for-better-ivf-success. Retrieved 16 February 2023. 
  18. Yousif, M.; McClintock, D.S.; Yao, K. (2021). "Artificial intelligence is the key driver for digital pathology adoption". Clinical Laboratory Int. PanGlobal Media. https://clinlabint.com/artificial-intelligence-is-the-key-driver-for-digital-pathology-adoption/. Retrieved 16 February 2023. 
  19. 19.0 19.1 MacLeod, B. P.; Parlane, F. G. L.; Morrissey, T. D.; Häse, F.; Roch, L. M.; Dettelbach, K. E.; Moreira, R.; Yunker, L. P. E. et al. (15 May 2020). "Self-driving laboratory for accelerated discovery of thin-film materials" (in en). Science Advances 6 (20): eaaz8867. doi:10.1126/sciadv.aaz8867. ISSN 2375-2548. PMC PMC7220369. PMID 32426501. https://www.science.org/doi/10.1126/sciadv.aaz8867. 
  20. 20.0 20.1 Chibani, Siwar; Coudert, François-Xavier (1 August 2020). "Machine learning approaches for the prediction of materials properties" (in en). APL Materials 8 (8): 080701. doi:10.1063/5.0018384. ISSN 2166-532X. http://aip.scitation.org/doi/10.1063/5.0018384. 
  21. 21.0 21.1 Mullin, R. (28 March 2021). "The lab of the future is now". Chemical & Engineering News 99 (11). Archived from the original on 06 May 2022. https://web.archive.org/web/20220506192926/http://cen.acs.org/business/informatics/lab-future-ai-automated-synthesis/99/i11. Retrieved 16 February 2023. 
  22. 22.0 22.1 Burger, Benjamin; Maffettone, Phillip M.; Gusev, Vladimir V.; Aitchison, Catherine M.; Bai, Yang; Wang, Xiaoyan; Li, Xiaobo; Alston, Ben M. et al. (9 July 2020). "A mobile robotic chemist" (in en). Nature 583 (7815): 237–241. doi:10.1038/s41586-020-2442-2. ISSN 0028-0836. https://www.nature.com/articles/s41586-020-2442-2.epdf?sharing_token=HOkIS6P5VIAo2_l3nRELmdRgN0jAjWel9jnR3ZoTv0Nw4yZPDO1jBpP52iNWHbb8TakOkK906_UHcWPTvNxCmzSMpAYlNAZfh29cFr7WwODI2U6eWv38Yq2K8odHCi-qwHcEDP18OjAmH-0KgsVgL5CpoEaQTCvbmhXDSyoGs6tIMe1nuABTeP58z6Ck3uULcdCtVQ66X244FsI7uH8GnA%3D%3D&tracking_referrer=cen.acs.org. 
  23. Lemonick, S. (6 April 2020). "Exploring chemical space: Can AI take us where no human has gone before?". Chemical & Engineering News 98 (13). Archived from the original on 29 July 2020. https://web.archive.org/web/20200729004137/https://cen.acs.org/physical-chemistry/computational-chemistry/Exploring-chemical-space-AI-take/98/i13. Retrieved 16 February 2023. 
  24. Doctrow, B. (16 December 2019). "Artificial intelligence in the laboratory". PNAS Science Sessions. https://www.pnas.org/post/podcast/artificial-intelligence-laboratory. Retrieved 16 February 2023. 
  25. VistaPath (28 July 2022). "VistaPath Launches New Collaboration with Gestalt Diagnostics to Further Accelerate Pathology Digitization". PR Newswire. Cision US Inc. https://www.prnewswire.com/news-releases/vistapath-launches-new-collaboration-with-gestalt-diagnostics-to-further-accelerate-pathology-digitization-301594718.html. Retrieved 17 February 2023. 
  26. 26.0 26.1 26.2 26.3 26.4 26.5 26.6 26.7 Blum, K. (1 January 2023). "A Status Report on AI in Laboratory Medicine". Clinical Laboratory News. American Association for Clinical Chemistry. https://www.aacc.org/cln/articles/2023/janfeb/a-status-report-on-ai-in-laboratory-medicine. Retrieved 17 February 2023. 
  27. Chubb, P. (3 November 2020). "How disruptive technology is helping laboratories combat COVID-19". Datafloq. https://datafloq.com/read/disruptive-technologies-lab-help-us-prepare-future-pandemics/. Retrieved 16 February 2023. 
  28. Stewart, B. (18 March 2021). "Using LIMS for Data Visualization". CSols Insights. CSols, Inc. https://www.csolsinc.com/insights/published-articles/using-lims-for-data-visualization/. Retrieved 17 February 2023. 
  29. "Data Analytics". Scientific Computing World - Building a Smart Laboratory 2020. Europa Science Ltd. 29 January 2021. https://www.scientific-computing.com/feature/data-analytics-0. Retrieved 17 February 2023. 
  30. Optima Team (15 December 2020). "The concept of machine learning applied to lab resources scheduling". Optima Blog. Optima PLC Tracking Tools S.L. https://www.optima.life/blog/the-concept-of-machine-learning-applied-to-lab-resources-scheduling/. Retrieved 17 February 2023. 
  31. 31.0 31.1 31.2 Lee, G.F. (10 October 2022). "The Robot May See You Now: It’s time to stop and think about the ethics of artificial intelligence". The Pathologist. https://thepathologist.com/outside-the-lab/the-robot-may-see-you-now. Retrieved 17 February 2023. 
  32. 32.0 32.1 32.2 Sherwood, L. (10 February 2022). "SLAS 2022: Barriers remain to AI adoption in life sciences". LabPulse.com Showcasts. Science and Medicine Group. https://www.labpulse.com/showcasts/slas/2022/article/15300130/slas-2022-barriers-remain-to-ai-adoption-in-life-sciences. Retrieved 17 February 2023. 
  33. 33.0 33.1 33.2 33.3 Bellini, Claudia; Padoan, Andrea; Carobene, Anna; Guerranti, Roberto (25 November 2022). "A survey on Artificial Intelligence and Big Data utilisation in Italian clinical laboratories" (in en). Clinical Chemistry and Laboratory Medicine (CCLM) 60 (12): 2017–2026. doi:10.1515/cclm-2022-0680. ISSN 1434-6621. https://www.degruyter.com/document/doi/10.1515/cclm-2022-0680/html. 
  34. Chada, Bharadwaj V; Summers, Leanne (10 October 2022). "AI in the NHS: a framework for adoption" (in en). Future Healthcare Journal: fhj.2022–0068. doi:10.7861/fhj.2022-0068. ISSN 2514-6645. PMC PMC9761451. PMID 36561823. https://www.rcpjournals.org/lookup/doi/10.7861/fhj.2022-0068.