Journal:Generalized procedure for screening free software and open-source software applications/Completing the evaluation

From LIMSWiki
Revision as of 16:19, 19 January 2016 by Shawndouglas (talk | contribs) (→‎Abstract: Internal link)
Jump to navigationJump to search
Full article title Generalized Procedure for Screening Free Software and Open Source Software Applications
Author(s) Joyce, John
Author affiliation(s) Arcana Informatica; Scientific Computing
Primary contact Email:
Year published 2015
Distribution license Creative Commons Attribution-ShareAlike 4.0 International
Website Print-friendly version
Download PDF (Note: Inline references fail to load in PDF version)


Free software and open-source software projects have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process, we have initiated a project to generate a library of in-depth reviews of free software and open-source software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure is executed against a selected class of applications.



Literature review

Initial evaluation and selection recommendations

In-depth evaluation

Completing the evaluation


Other than independent reviews, one of the best ways to obtain some of the above information is direct testing of a system. Usually that is impractical in a survey situation because of the time it would take to install and configure an instance of the application. However, a new factor has entered the picture which may change this. Docker is a new service that combines an application, along with all of its dependencies, into a single distributable package, which they call a container. Because everything required is in the container, it is guaranteed to run the same on any system. Docker specifically states that their containers "are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure."[1] Using only a fraction of the resources that a virtual machine (VM) would require, you can easily run multiple containers on a laptop and switch back and forth while testing.[2]

While the Docker web site contains repositories for a number of different containers, there are multiple web sites that also host containers configured with a variety of applications. If you can locate one that contains the application you are surveying, this makes it a simple matter to try it out. Probably the easiest way to check is to run a web search containing the terms "docker", "container", and the name of the application that you are seeking.

Screening tabulation

While you can perform this application survey in many ways, to keep your defined scoring criteria in front of the evaluators and to simplify scoring the survey, it might be prudent to generate a document such as the following table with columns representing the criteria being evaluated, your rating definitions, and the numeric rating that your evaluators actually assign to the system. If you have decided to take the approach of using weighting factors instead of adjusting your definitions, you will also need to include columns for your weighting factors and a column containing the results of the evaluation after applying the weighting factor.

Criteria Rating = 0 Rating = 1 Rating = 2 Score
System functionality                                                                    
System cost                                                                    
Product support                                                                    
Legal/License issues                                                                    
Summary Score                 

Table 4.: Potential screening criteria to filter prospective FLOSS applications. Rows can be added or dropped as required by your needs. For example, if it is important that the application be coded in a specific programming language or combination of languages, you could add a row for this. The most likely anticipated change would be to subdivide the "system functionality" row so that specific information on the functionality of various system components can be displayed. To minimize the risk of confusion, you can transfer your selected scoring criteria for each rating into the cell corresponding to that rating and the item being evaluated, though ths would admittedly generate a long check list.

Demonstration surveys

To demonstrate how this survey procedure is intended to be used, this section will apply the defined protocol to a block of open-source LIMS applications suitable for use in a chemical laboratory. Note that in this instance, we are using LIMS to refer to a laboratory information management system, not a labor information management system, a legislative information management system, or any other permutation that matches the LIMS acronym. The specific criteria used will quite likely be different from the ones used in your screening as the types of systems or the specific functionality required will be different.

For the purpose of this demonstration, we will not attempt to screen all of the open-source LIMS available. Instead, we will select a subset of the systems that have been announced and attempt to apply our protocol to them, hopefully screening out the systems not meeting our requirements so that the number of in-depth evaluations can be reduced. Part of the difficulty here is the broad range of fields that the term LIMS covers. There are general purpose LIMS, which can be configured to handle a wide range of samples, as well as targeted LIMS, which are designed to fill a particular niche. As such, you can find LIMS targeting specific areas as drinking water/waste water analysis, mining, radioisotopes, proteomics, and genetic analysis. Whether you make it a formal part of the screening process or isolate them while collecting the applications to be surveyed, you will need to filter out the systems which will not handle the type of samples you are dealing with. This issue is particularly prominent with LIMS, but it will likely be encountered when screening other types of software as well. The following are the systems that we will include in this attempt:

For simplicity in comparing results, I've reordered the screening table so that the first column contains the criteria being evaluated and the other columns correspond to the evaluation results for the LIMS being evaluated, with the bottom row reserved for the corresponding score summary. I have also added two subdivisions under "system functionality" for the programming language and the operating system used. Depending on the types of systems you are surveying, you will likely be including additional subdivisions. Scoring can be handled several ways. In this example, you might list the programming language in the appropriate cell. Depending on whether you have a team member competent in that language, you can use the results to set the value for the main criteria, e.g. system functionality. Alternately, while you will record the information for programming language in your survey notebook, you can insert an actual numerical result into the corresponding cell so that the sub-criteria can be evaluated separately or used to generate a value for the main criteria field.

Normally, the best starting approach is to check for existing reviews of systems, but here we have a problem in finding any. While references to Bika LIMS were common, and it was referenced in a number of scientific papers, actual reviews of the product were hard to come by and were usually for older versions.[3] No reviews were found for eyeLIMS and for Open-LIMS; the closest thing I found to an impartial review has been two postings on Joel Limardo's LIMSExpert blog.[4][5]

Criteria Bika LIMS 3.1.8 eyeLIMS Open-LIMS*
System functionality 2 0 (Project appears dead, no activity since 2008) 1 (System functional for a very specific application only)
Programming language Python and PLONE CMS eyeOS PHP
Operating system platform-independent eyeOS Any OS supporting PostgreSQL and PHP
Community 2 0 1
System cost 0 1
Popularity 2 0 0
Product support 2 0 1
Maintenance 2 0 1
Reliability 0
Performance 0
Scalability 0
Usability 2 0 1
Security 2 0 1
Flexibility/Customizability 2 0 1
Interoperability/Integration 2? (Supports import and export of CSV files) 0 0
Legal/License issues 2 (AGPL 3.0 and GPL1) 2 (GNU Affero GPL v3) 2 (GNU GPL 3.0)
Summary Score 22 2 10

Based on the above summary scores, we would definitely filter out both eyeLIMS and Open-LIMS, while Bika LIMS would justify a more in-depth evaluation.

* Note: The rating for Open-LIMS may need to be revisited, as it appears that major development work on this system has been taken over by Joel Limardo of ForwardPhase Technologies, so several of the rated parameters may be subject to major shifts. It is currently unclear whether this is a joint project with the original developer or a fork.





This article has not officially been published in a journal. However, this presentation is largely faithful to the original paper. The content has been edited for grammar, punctuation, and spelling. Additional error correction of a few reference URLs and types as well as cleaning up of the glossary also occurred. Redundancies and references to entities that don't offer open-source software were removed from the FLOSS examples in Table 2. DOIs and other identifiers have been added to the references to make them more useful. This article is being made available for the first time under the Creative Commons Attribution-ShareAlike 4.0 International license, the same license used on this wiki.