User:Shawndouglas/sandbox/sublevel4

Abstract

Free Software and Open Source Software projects have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process we have initiated a project to generate a library of in-depth reviews of Free Software and Open Source Software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we will examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we will synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure will be executed against a selected class of applications.

Introduction

There is much confusion regarding Free Software and Open Source Software and many people use these terms interchangeably, however, to some the connotations associated with the terms is highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with Open Source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).

When discussing Free Software, we are not explicitly discussing software for which no fee is charged, rather we are referring to free in terms of liberty. To quote the Free Software Foundation (FSF):

A program is free software if the program's users have the four essential freedoms:

• The freedom to run the program as you wish, for any purpose (freedom 0).
• The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

This does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also 'free software' or 'freeware' that is given away at no charge, or gratis, but without the source code, this would not be considered Free Software under the FSF definition.

The Open Source Initiative (OSI), originally formed to promote Free Software, which they referred to as Open Source Software (OSS) to make it sound more business friendly. The OSI defines Open Source Software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines:

• Free redistribution
• Source code Included
• Must allow derived works
• Must preserve the integrity of the authors source code
• License must not discriminate against persons or groups
• License must not discriminate against fields of endeavor
• Distribution of licenses
• License must not be specific to a production
• License must not restrict other software
• License must be technology neutral

Open Source Software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.

In an attempt to placate both groups when discussing the same software application, the term Free/Open Source Software (F/OSS) was developed. Since the term Free was still tending to confuse some people, the term libre, which connotes freedom, was added resulting in the term Free/Libre Open Source Software (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all Free Software fits the Open Source Software definition, while not all Open Source Software fits the Free Software definition. However, any Open Source Software that is not also Free Software is the exception, rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.

The reality is that since both groups trace their history back to the same origins, the practical differences between an application being Free Software or Open Source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in Open Source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us. In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With in excess of 300,000 FLOSS applications currently registered in SourceForge.net and over 10 million repositories on GitHub, there are generally multiple options accessible for any class of application, be it a Laboratory Information Management System (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an Open Source application and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers with no indication of the vitality or independent evaluation of the project.

What is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While once they've made a tentative selection, the organization will need to perform their own testing to confirm that the selected application meets their specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.

Illustration 1.: This diagram, originally by Chao-Kuei and updated by several others since,
explains the different categories of software. It's available as a Scalable Vector Graphic and
as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2
or later, or the Creative Commons Attribution-Share Alike v2.0 or later

The primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end it we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.

You can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, commercial is referring to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior quality, to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS is commercial software.

The final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.

Results

A search of the literature returns thousands of papers related to Open Source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between Open Source projects is mentioned in a number of these papers and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large scale support in the industry. Stol and Babar have published a framework for comparing evaluation methods targeting Open Source software and include a comparison of 20 of them. They noted that web sites that simply consisted of a suggestion list for selecting an Open Source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, and other factors involved.

No. Name Year Orig Method 1 Capgemini Open Source Maturity Model 2003 I Yes 2 Evaluation Framework for Open Source Software 2004 R No 3 A Model for Comparative Assessment of Open Source Products 2004 R Yes 4 Navica Open Source Maturity Model 2004 I Yes 5 Woods and Guliani's OSMM 2005 I No 6 Open Business Readiness Rating (OpenBRR) 2005 R/I Yes 7 Atos Origin Method for Qualification and Selection of Open Source Software (QSOS) 2006 I Yes 8 Evaluation Criteria for Free/Open Source Software Products 2006 R No 9 A Quality Model for OSS Selection 2007 R No 10 Selection Process of Open Source Software 2007 R Yes 11 Observatory for Innovation and Technological transfer on Open Source software (OITOS) 2007 R Yes 12 Framework for OS Critical Systems Evaluation (FOCSE) 2007 R No 13 Balanced Scorecards for OSS 2007 R No 14 Open Business Quality Rating (OpenBQR) 2007 R Yes 15 Evaluating OSS through Prototyping 2007 R Yes 16 A Comprehensive Approach for Assessing Open Source Projects 2008 R No 17 Software Quality Observatory for Open Source Software (SQO-OSS) 2008 R Yes 18 An operational approach for selecting open source components in a software development project 2008 R No 19 QualiPSo trustworthiness model 2008 R No 20 OpenSource Maturity Model (OMM) 2009 R No

Table 1.: Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar. The selection
procedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described
process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)
or just a list of evaluation criteria (No).

Notes

This article has not officially been published in a journal. However, this presentation is faithful to the original paper, with only a few minor changes to presentation. This article is being made available for the first time under the Creative Commons Attribution-ShareAlike 4.0 International license, the same license used on this wiki.