Difference between revisions of "User:Shawndouglas/sandbox/sublevel4"

From LIMSWiki
Jump to navigationJump to search
Line 63: Line 63:


===3.2 Functionality considerations===
===3.2 Functionality considerations===
Technology aside, the lab will also want to consider the functionality of the various laboratory informatics solutions offered by vendors. Some laboratorians may already be familiar with the LIMS or ELN and understand what it is capable of doing for the lab. However, others may have little familiarity with such systems and have only vague concepts of how they function. Either way, it's useful for the lab to gain a clearer understanding of what such solutions can do. There are several ways to go about this, from reading through examples of user requirements documents to participating in demonstrations of software from various vendors. In fact, both methods remain useful, and a standard approach to acquisition will usually include both. In the end, the question for any prospective vendor to answer is "how can your solution meet our laboratory's requirements?" This is where functionality comes in.
The follow sections will look at functionality considerations first by introducing the user requirements specification (URS) as a means of learning more about system functionality. From there we'll look specifically at a LIMS and what its core and ancillary features look like, before finally re-addressing the URS from the perspective of how it can be implemented to full effect.


====3.2.1 The user requirements specification as a functionality guide====
====3.2.1 The user requirements specification as a functionality guide====
A specification is "a detailed precise presentation of something or of a plan or proposal for something."<ref name="MWSpec">{{cite web |url=https://www.merriam-webster.com/dictionary/specification |title=specification |work=Merriam-Webster |publisher=Merriam-Webster, Inc |accessdate=20 January 2023}}</ref> This concept of a specification as a presentation is critical to the laboratory seeking to find laboratory informatics software that fulfills their needs; they "present" their use case with the help of a requirements specification, and the vendor "presents" their ability (or inability) to comply through documentation and demonstration. However, even the most seasoned of presenters at conferences and the like still require quality preparation before the presentation. Similarly, your lab has work to do beforehand. This is where initial specification research comes into play for the lab.
A specification is "a detailed precise presentation of something or of a plan or proposal for something."<ref name="MWSpec">{{cite web |url=https://www.merriam-webster.com/dictionary/specification |title=specification |work=Merriam-Webster |publisher=Merriam-Webster, Inc |accessdate=20 January 2023}}</ref> This concept of a specification as a presentation is critical to the laboratory seeking to find laboratory informatics software that fulfills their needs; they "present" their use case with the help of a requirements specification, and the vendor "presents" their ability (or inability) to comply through documentation and demonstration. However, even the most seasoned of presenters at conferences and the like still require quality preparation before the presentation. Similarly, your lab has work to do beforehand. This is where initial specification research comes into play for the lab.


Your lab's requirements specification document will eventually be a critical component for effectively selecting a laboratory informatics solution. For those labs with limited knowledge of what such a solution is capable of doing, building a user requirements specification (URS) can be especially useful. There are numerous ways to approach the overall development of a URS. But why re-invent the wheel when others have already gone down that road? Sure, you could search for examples of such documents on the internet and customize them to your needs, or you and your team could brainstorm how a laboratory informatics solution should help your lab accomplish its goals. Or you could turn to a robust URS like [[Book:LIMSpec 2022 R2|LIMSpec 2022]].  
Your lab's requirements specification document will eventually be a critical component for effectively selecting a laboratory informatics solution. For those labs with limited knowledge of what such a solution is capable of doing, building a URS can be especially useful. There are numerous ways to approach the overall development of a URS. But why re-invent the wheel when others have already gone down that road? Sure, you could search for examples of such documents on the internet and customize them to your needs, or you and your team could brainstorm how a laboratory informatics solution should help your lab accomplish its goals. Or you could turn to a robust URS like [[Book:LIMSpec 2022 R2|LIMSpec 2022]].  


LIMSpec is a robust [[laboratory informatics]] requirements specification document that has evolved significantly over the years, and any lab can turn to LIMSpec to better visualize what the critical requirements of a LIMS (or other laboratory informatics solution) are. With the current version of LIMSpec having at its core standards such as [[ASTM E1578|ASTM E1578-18]] ''Standard Guide for Laboratory Informatics''<ref name="ASTME1578_18">{{cite web |url=https://www.astm.org/e1578-18.html |title=ASTM E1578-18 Standard Guide for Laboratory Informatics |publisher=ASTM International |date=23 August 2019 |accessdate=20 January 2023}}</ref> and ISO/IEC 17025 ''General requirements for the competence of testing and calibration laboratories''<ref name="ISO17025_17">{{cite web |url=https://www.iso.org/standard/66912.html |title=ISO/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories |publisher=International Organization for Standardization |date=November 2017 |accessdate=20 January 2023}}</ref>, the LIMSpec makes for a durable URS that, when used to acquire an informatics solution, can better help a laboratory choose appropriate functionality based upon current standards, regulations, guidance, and more.  
LIMSpec is a robust [[laboratory informatics]] requirements specification document that has evolved significantly over the years, and any lab can turn to LIMSpec to better visualize what the critical requirements of a LIMS (or other laboratory informatics solution) are. With the current version of LIMSpec having at its core standards such as [[ASTM E1578|ASTM E1578-18]] ''Standard Guide for Laboratory Informatics''<ref name="ASTME1578_18">{{cite web |url=https://www.astm.org/e1578-18.html |title=ASTM E1578-18 Standard Guide for Laboratory Informatics |publisher=ASTM International |date=23 August 2019 |accessdate=20 January 2023}}</ref> and ISO/IEC 17025 ''General requirements for the competence of testing and calibration laboratories''<ref name="ISO17025_17">{{cite web |url=https://www.iso.org/standard/66912.html |title=ISO/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories |publisher=International Organization for Standardization |date=November 2017 |accessdate=20 January 2023}}</ref>, the LIMSpec makes for a durable URS that, when used to acquire an informatics solution, can better help a laboratory choose appropriate functionality based upon current standards, regulations, guidance, and more.  


LIMSpec is divided into five distinct sections, with numerous subsections in each. These sections and subsections should be able to address most any requirement you have for your system. Of course, if something isn't covered by LIMSpec, you can always add additional requirements. More importantly, if you have little concept of what functionality a LIMS or SDMS should have, reviewing this URS can give you a leg up. Of course, the base LIMSpec can be used to develop your lab's custom URS to present to potential vendors. (More on that in a bit.)
LIMSpec is divided into five distinct sections, with numerous subsections in each. These sections and subsections should be able to address most any requirement you have for your system. Of course, if something isn't covered by LIMSpec, you can always add additional requirements. More importantly, if you have little concept of what functionality systems such as a LIMS or SDMS should have, reviewing this URS can give you a leg up, as can participating in vendor demonstrations. Of course, the base LIMSpec can be used to develop your lab's custom URS to present to potential vendors. (More on that in a bit.)


====3.2.2 Core functions and features====
====3.2.2 Core functions and features====
Line 122: Line 125:
*[[LIMS feature#Web client or portal|web client or portal]]
*[[LIMS feature#Web client or portal|web client or portal]]


===3.4 Putting a URS to work for your lab===
===3.3 Putting a URS to work for your lab===
During the initial research towards your URS, you won't have to include every requirement for when you approach potential vendors. Most vendors appreciate a more inviting approach that doesn't overwhelm, at least initially. You will want to go with a limited yet practical set of requirements carefully chosen because they matter to you and your laboratory the most. In the case of a lab seeking a solution to help them better comply with ISO/IEC 17025, one might want to make requirements related to the standard the practical starting point. (See Chapter 3, section 3.1.2, Table 1 for the full list.) Wherever you choose to start, you'll likely want to wait until after participating in several software demonstrations before even considering your URS to be complete. (More on that in 5.3.1.) This naturally leads us to a discussion about the request for information (RFI) process.
As stated prior, building a URS specific to your lab can easily start by examining an existing URS like LIMSpec, which covers most functionality a laboratory could need out of their informatics solution. However, LIMSpec is extensive in what it covers. During the initial research put towards your URS, know that you likely don't want to send the full, comprehensive URS to vendors, at least not in the initial discovery stages. Most vendors appreciate a more inviting approach that doesn't initially overwhelm. Early on you're better off to go with a limited yet practical set of requirements carefully chosen because they matter to you and your laboratory the most. For a food and beverage laboratory, support for molecular biology workflows, stability studies, and production management could be critical. In the case of a lab seeking a solution to help them better comply with ISO/IEC 17025, the lab might want to focus on requirements related to complying with the standard.  
 
However you choose to approach URS development and presentation, you'll likely want to wait until after participating in several vendors' software demonstrations before even considering your URS to be complete. The demo offers a unique and valuable opportunity to see in-person how data and information is added, edited, deleted, tracked, and protected within the context of the application; you can ask about how a function works and see it right then and there. Equally, it is an excellent time to compare notes with the vendor, particularly in regard to requirements your lab deems being critical. You can ask the vendor in real-time to answer questions about how a specific task is achieved, or how the system addresses specialized needs, and the vendor can ask you about your lab's system and workflow requirements and how you best envision them being implemented in the system.
 
Again, be careful about falling for the temptation of presenting a full URS or other specification document to the vendor during the demonstration. You'll want to wait until after participating in several software demonstrations to consider presenting your full specification document to the vendor, and that's assuming that you've grown enamored with their solution. By waiting to finalize your lab's URS until after the demos, a common error is avoided: too often labs think the first thing they must do is create a requirements list, then sit back and let the informatics vendors tell them how they meet it. Remember that even though most labs thoroughly understand their processes, they likely don't have as strong a grasp on the informatics portion of their processes and workflows. Participating in a demo before finalizing your list of specified requirements—or having only a minimal yet flexible requirements list during the demo—is a great way to later crosscheck the software features you have seen demonstrated to your lab's processes and any initial requirements specification you've made.<ref name="HammerHowTo19">{{cite web |url=https://www.striven.com/blog/erp-software-demo |title=How to Get the Most Value from an ERP Software Demo |author=Hammer, S. |work=The Takeoff |date=27 June 2019 |accessdate=02 February 2023}}</ref> After all, how can you effectively require specific functions of your laboratory informatics software if you don't fully know what such a system is capable of? After the demonstrations, you may end up adding several requirements to your final specifications document, which you later pass on to your potential vendors of choice for final confirmation.
 
In some cases, rather than approaching vendors, a lab may want to take a different exploratory route by issuing a [[request for information]] (RFI) or something similar. An RFI is an ideal means for learning more about a potential solution and how it can solve your problems, or for when you're not even sure how to solve your problem yet. However, like when approaching a vendor directly, the RFI should not be unduly long and tedious to complete for prospective vendors; it should be concise, direct, and honest. This means not only presenting a clear and humble vision of your own organization and its goals, but also asking just the right amount of questions to allow potential vendors to demonstrate their expertise and provide a clearer picture of who they are. Some take a technical approach to an RFI, using dense language and complicated spreadsheets for fact finding. However, as previously noted, you will want to limit the specified requirements in your RFI to those carefully chosen because they matter to you and your lab the most.<ref name="HolmesItsAMatch">{{cite web |url=https://allcloud.io/blog/its-a-match-how-to-run-a-good-rfi-rfp-or-rfq-and-find-the-right-partner/ |title=It's a Match: How to Run a Good RFI, RFP, or RFQ and Find the Right Partner |author=Holmes, T. |work=AllCloud Blog |accessdate=02 February 2023}}</ref>
 
Remember, an RFI is not meant to answer all of your questions. The RFI is meant as a means to help narrow down your search to a few quality candidates offering functionality that meets your needs, while also learning more about each other.<ref name="HolmesItsAMatch" /> Once the pool of potential software vendors is narrowed down, and you then participate in their demonstrations, you then can broadly add more requirements to the original collection of critical requirements from the RFI to ensure those vendors can meet all or most of your needs. That said, be cognizant that there may be no vendor that can meet each and every need of your lab. Your lab will have to make important decisions about which requirements are non-negotiable and which are more flexible. The vendors you engage with may be able to provide realistic advice in this regard, based upon your lab's requirements and their past experience with labs. As such, those vendors with real-world experience meeting the needs of laboratories like your may have a strong leg up on other vendors.
 
Ultimately, your URS may look similar to LIMSpec, or it may have a slightly different format. Many prospective buyers will develop a requirement specification in Microsoft Excel, but that has a few minor disadvantages. Regardless of format, you'll want to give plenty of space for vendors to submit a response to each requirement. With a more complete URS in hand, and your vendor list narrowed down to a few possibilities, you're ready to submit the full URS to those vendors. Other considerations come into play afterwards, depending how those vendors' respond to you URS. How well did the vendors respond to your laboratory's unique set of needs? Were there critical areas that one vendor could address with their off-the-shelf solution but another vendor would have to address with custom coding? Did any of the vendors meet your budget expectations? Have you followed up on any references and customer experiences the vendors provided to you?
 
Hopefully, once all the due diligence is done, and with a little luck, you've found a vendor that fits your technology and functionality needs, even if a few minor compromises had to be made along the way. However, there's obviously more than technology and system functionality to consider! How is their solution offered? What kind of licensing is used? What are the specifics of the maintenance, warranty, and support plans? The next section addresses that.





Revision as of 23:59, 2 February 2023

Sandbox begins below

Lion Lab Technician2 (16279873861).jpg

This buyer's guide is based off the LIMS Buyer's Guide, a former publication of the Laboratory Informatics Institute (LII), an open trade association that was associated with LabLynx, Inc.[1][2] In late 2013, the LII and LabLynx discontinued publishing a copyrighted version and chose to release future guides to the public domain via this wiki. Per the Creative Commons license and the copyright terms of this site, you are free to copy, adapt, distribute, and transmit this guide as long as you 1. give proper attribution and 2. distribute the work only under the same or a similar license.


1. Introduction

What exactly is a laboratory information management system (LIMS) or laboratory information system (LIS) anyway? Do I need one? What options are available and how do I compare them? Where does a user requirements specification (USR), request for information (RFI), request for proposal (RFP), or request for quotation (RFQ) come into play, and how does it fit into acquiring a laboratory informatics solution? These are questions laboratory professionals typically ponder upon finding themselves charged with the mission of finding software for their lab. It can be a daunting proposition, and there are few at least partially objective references to help with it all. This brief guide exists to, in part, meet that need.

At the core of this buyer's guide are several elements, including core information about evaluating, acquiring, and implementing laboratory software that fits your lab's needs. Also included here are laboratory informatics vendors who actually make their pricing partially or fully public, through one means or another. That pricing may be fully transparent and posted on the company website, or it may be a partial representation of the lowest possible price offered, as with those vendors who have a publicly viewable contract with the U.S. General Services Administration. While in the past vendors have refrained from providing public pricing, a more open information process may have its merits (particularly for potential buyers), though also not without its own set of caveats.[3] The theory—at least on paper—has been that prices should decrease as LIMS become commodities that labs can compare and contrast in a more competitive fashion. However, it remains to be seen if that will ever be the case.

Laboratories are, by their very nature, in the business of producing analytical data and reports, essentially information.[4] Everything else that occurs in the laboratory is largely just a means towards that central goal of producing timely, accurate, and unbiased information. As such, in a very real sense, information management is, by extension, also at the core of any lab. In a world where we use the latest technology for most of our daily tasks and pleasures, why do so many labs still rely on hand-written notes and spreadsheets? Why do they still spend thousands of dollars on a sophisticated analytical instrument yet hesitate when faced with purchasing an information management system? The primary reason has traditionally been the cost associated with acquiring an informatics solution, as well as not having the information technology resources to properly implement it.[5][6][7]

Software solutions like LIMS are increasingly becoming commodities, and potential buyers don't need to find the acquisition process as daunting as it used to be. As technology has improved, smaller LIMS companies have emerged, along with affordable cloud-based SaaS options that are flexible and reliable. This means any lab can put their resources where they belong: into its analytical information and its management.

This brief buyer's guide is here to help you take the first steps towards acquiring a laboratory informatics solution. Use these vendor profiles and recommendations to get a feel for what's out there and what makes the most sense. This guide contains information on a little bit of everything, from discovering what a LIMS is to maintaining and supporting that system. While the LIMS is the stand-out solution among the crowd for laboratories, this guide may also reference other systems like a LIS or electronic laboratory notebook (ELN).


2. How does laboratory software benefit the lab?

Laboratory, scientific, and healthcare informatics solutions can improve laboratory workflows while enhancing safety, quality, and compliance in a number of ways. A LIMS, for example, is often adopted to eliminate manual processes, improve sample management, increase productivity, and improve regulatory conformance.[8] The adoption of LIMS and other such systems to eliminate deficiencies and improve operations is often based in the disadvantages of paper-based processes—particularly in regulatory environments—and their inability at making their contents rapidly findable, searchable, modifiable, and secure.[9][10]

Even a fragmented mix of paper-based and electronic information sources can be a detriment to the traceability of or rapid accessibility to quality control samples, standard operating procedures (SOPs), calibration data, chain of custody data, and other vital aspects of analytical, sampling, and calibration testing in the lab. A well-implemented LIMS can reduce the silos of data and information, while at the same time make that information and data more secure and readily accessible. Given the demands placed on laboratories—by ISO/IEC 17025 and clients seeking labs certified to that standard, for example—to provide rapid proof of traceable sample movement and relevant quality control data, integrating a LIMS into the laboratory makes sense. The LIMS can act as the central integrator and audit trail for all that laboratory data and information.[11][12][13] Because the LIMS improves traceability—including through its automated interfaces with instruments and other data systems—real-time monitoring of supply chain issues, quality control data, instrument use, and more is further enabled, particularly when paired with configurable dashboards and alert mechanisms. By extension, labs can more rapidly act on insights gained from those real-time dashboards.[11]

There is a growing variety of laboratory informatics software providing these and other benefits to labs, allowing them to more rapidly adapt and remain competitive. Arguably, the most common solutions are the LIMS and LIS. The LIMS and LIS are similar, in that they focus on the entire laboratory process and improving the various aspects of it. Traditionally, the LIMS has been used in more non-clinical environments and the LIS in more clinical environments, but that distinction has been fading for well over a decade. These distinctions have blurred even more with the age of modular and platformed software solutions that are able to adapt to most any industry. For now, just know that these systems are a bit more holistic, placing the focus on improving your lab's workflows, which in turn strengthens traceability, analytical outcomes, and—in the case of clinical settings—patient outcomes. Outside of that, there are other more specialized solutions, including:

  • the ELN, which is used in many research and development environments (among others) as a means to securely and collaboratively document experiments and their results;
  • the laboratory execution system (LES), which is used in production-based environments to ensure the rigidity of a method and and the process' end result;
  • the chromatography data system (CDS), which is used to accurately collect, process, and visualize chromatography data; and
  • the scientific data management system (SDMS), which is used to collect disparate silos of data and information across the laboratory enterprise and make it more actionable.

While these systems can be found in many types of labs, some tendencies appear. LIMS, LIS, and ELN are top-level solutions that help the lab manage workflows, as well as aggregate and organize data and information, including analytical- and experiment-based results. In services labs, the ELN is less likely to appear. However, in the case of research labs, the ELN may function more top-level, with the LIMS acting as a supplementary system for managing test workflows before associated test results are loaded into the ELN. Support systems like the LES, CDS, and SDMS may also appear, filling specialized roles. The LES will appear in more regulated environments where laboratory procedures must be executed precisely, before then feeding results to a LIMS or ELN. The CDS will appear in labs doing heavy chromatography work, and it too may feed data into a LIMS or ELN. The SDMS appears in contexts where data and information produced by the lab don't fit in the structure of what a LIMS or ELN handles.


3. How do I find the right solution for my needs?

As indicated above, the type of solution you choose is based on your lab type (e.g., service, research, production), the tests predominately run (e.g., chromatography work, requiring a CDS), and the type of data and information generated and stored (e.g., non-standard files, requiring an SDMS). However, finding the right system to fit your labs needs isn't a straightforward process. While your lab may know its regulatory-, workflow-, and standard-driven requirements, perhaps critical personnel don't know much about data management solutions like LIMS, leaving a lab's key stakeholders intimidated by all the options. You'll first need to gauge your lab's informatics needs in order to determine which products are worth investigating further. Of course your lab's analysis requirements, reporting and data sharing constraints, instrument interfacing needs, barcoding and tracking requirements, quality assurance processes, etc. are certainly important factors. But vendors' systems vary in numerous ways, and other important factors exist. Of course, price is one consideration, although value is ultimately more important than a low price. Other important questions that get asked include:

  • Should we purchase software licenses or "rent" the software via a subscription-based model?
  • Does the software need to be on-site, or is a SaaS hosted option more practical?
  • Is a modular or complete system better for us?
  • What is the best licensing/rental scheme for us? Should we consider site, named user, concurrent user, or workstation licenses?
  • Is the company qualified and trustworthy?
  • What functionality is available to help our lab remain compliant with the a variety of standards and regulations?

3.1 Technology considerations

Your laboratory's workflow, instruments, data management requirements, budget, technological expertise, business goals, and risk tolerances will all play a role in deciding what technology to invest in. Your lab will also be interested in how that technology not only helps with workflow requirements but also better assists the lab with meeting the requirements placed upon them by various standards and regulations, as well as management.

As part of this discovery process, many questions may arise. Chief among them may be how to best match the laboratory's goals with one or more informatics solutions. Does the laboratory envision a small investment and modest goals, taking in a slow but steady flow of analytical requests, or do they envision expansive growth, expanding into multiple testing domains? If the lab is starting small but is confidently expecting to grow, technological investments early on may want to take into account future technologies that may shape data management and security processes. The lab will also want to account for how potential changes in standards and regulations may affect future expansion and technology needs.

Second, ask what kind of work will the lab be doing, and what regulatory responsibilities will guide hardware and software investment at the lab. If your lab will be conducting extractable and leachable testing, you'll be considering chromatography and spectroscopy instruments and software systems, as well as requirements for retaining analytical results for regulators. Those labs accredited to standards such as ISO/IEC 17025 will have additional needs for instruments and software systems; such labs' data management system will likely need to be sufficiently robust to meet the needs of the standard. For those labs in more regulated spaces, the system will even need to interface to government systems, or at a minimum report in the regulatory body's specific format. These and other considerations will need to be made.

Third, your laboratory's budget is always important. Does the budget allow for on-site hardware and software systems, with the personnel to maintain them? Is it easier to pay up-front or find a vendor willing to work with you on leasing or rental terms? Is their more value to be had from a slightly more expensive system able to provide the functionality that makes regulatory compliance more painless for the lab?

Finally, will the lab have someone on-site or on-call to resolve technology issues, including set-up and maintenance of software systems? If your lab will have little in the way of available tech help locally, you'll want to consider the distribution model you want to use for any installed software, i.e., you may want to consider software as a service (SaaS). An increasing number of software services are hosted using cloud computing, which when done well is an increasingly reliable option.[14] Having someone else host the software for you typically means the hosting provider will carry a non-trivial portion of responsibility for technology maintenance and security.

Speaking of security, you'll also want to consider the cybersecurity of not only your software solution but also your overall laboratory operations. Does your organization have a cybersecurity plan already in place, or has the decision to make one been postponed? What extra investment is required to ensure your sensitive data is secure? How do your cybersecurity goals compare to the demands of standards like ISO/IEC 17025? Remember that how you rank your cybersecurity preparedness and implement a cybersecurity plan will also guide your technology investment decisions.[15]

3.2 Functionality considerations

Technology aside, the lab will also want to consider the functionality of the various laboratory informatics solutions offered by vendors. Some laboratorians may already be familiar with the LIMS or ELN and understand what it is capable of doing for the lab. However, others may have little familiarity with such systems and have only vague concepts of how they function. Either way, it's useful for the lab to gain a clearer understanding of what such solutions can do. There are several ways to go about this, from reading through examples of user requirements documents to participating in demonstrations of software from various vendors. In fact, both methods remain useful, and a standard approach to acquisition will usually include both. In the end, the question for any prospective vendor to answer is "how can your solution meet our laboratory's requirements?" This is where functionality comes in.

The follow sections will look at functionality considerations first by introducing the user requirements specification (URS) as a means of learning more about system functionality. From there we'll look specifically at a LIMS and what its core and ancillary features look like, before finally re-addressing the URS from the perspective of how it can be implemented to full effect.

3.2.1 The user requirements specification as a functionality guide

A specification is "a detailed precise presentation of something or of a plan or proposal for something."[16] This concept of a specification as a presentation is critical to the laboratory seeking to find laboratory informatics software that fulfills their needs; they "present" their use case with the help of a requirements specification, and the vendor "presents" their ability (or inability) to comply through documentation and demonstration. However, even the most seasoned of presenters at conferences and the like still require quality preparation before the presentation. Similarly, your lab has work to do beforehand. This is where initial specification research comes into play for the lab.

Your lab's requirements specification document will eventually be a critical component for effectively selecting a laboratory informatics solution. For those labs with limited knowledge of what such a solution is capable of doing, building a URS can be especially useful. There are numerous ways to approach the overall development of a URS. But why re-invent the wheel when others have already gone down that road? Sure, you could search for examples of such documents on the internet and customize them to your needs, or you and your team could brainstorm how a laboratory informatics solution should help your lab accomplish its goals. Or you could turn to a robust URS like LIMSpec 2022.

LIMSpec is a robust laboratory informatics requirements specification document that has evolved significantly over the years, and any lab can turn to LIMSpec to better visualize what the critical requirements of a LIMS (or other laboratory informatics solution) are. With the current version of LIMSpec having at its core standards such as ASTM E1578-18 Standard Guide for Laboratory Informatics[17] and ISO/IEC 17025 General requirements for the competence of testing and calibration laboratories[18], the LIMSpec makes for a durable URS that, when used to acquire an informatics solution, can better help a laboratory choose appropriate functionality based upon current standards, regulations, guidance, and more.

LIMSpec is divided into five distinct sections, with numerous subsections in each. These sections and subsections should be able to address most any requirement you have for your system. Of course, if something isn't covered by LIMSpec, you can always add additional requirements. More importantly, if you have little concept of what functionality systems such as a LIMS or SDMS should have, reviewing this URS can give you a leg up, as can participating in vendor demonstrations. Of course, the base LIMSpec can be used to develop your lab's custom URS to present to potential vendors. (More on that in a bit.)

3.2.2 Core functions and features

You should expect the following functions to be demonstrated in a full-function LIMS:

3.2.3 Additional useful features

The following functions aren't necessary for all but useful for many:

3.3 Putting a URS to work for your lab

As stated prior, building a URS specific to your lab can easily start by examining an existing URS like LIMSpec, which covers most functionality a laboratory could need out of their informatics solution. However, LIMSpec is extensive in what it covers. During the initial research put towards your URS, know that you likely don't want to send the full, comprehensive URS to vendors, at least not in the initial discovery stages. Most vendors appreciate a more inviting approach that doesn't initially overwhelm. Early on you're better off to go with a limited yet practical set of requirements carefully chosen because they matter to you and your laboratory the most. For a food and beverage laboratory, support for molecular biology workflows, stability studies, and production management could be critical. In the case of a lab seeking a solution to help them better comply with ISO/IEC 17025, the lab might want to focus on requirements related to complying with the standard.

However you choose to approach URS development and presentation, you'll likely want to wait until after participating in several vendors' software demonstrations before even considering your URS to be complete. The demo offers a unique and valuable opportunity to see in-person how data and information is added, edited, deleted, tracked, and protected within the context of the application; you can ask about how a function works and see it right then and there. Equally, it is an excellent time to compare notes with the vendor, particularly in regard to requirements your lab deems being critical. You can ask the vendor in real-time to answer questions about how a specific task is achieved, or how the system addresses specialized needs, and the vendor can ask you about your lab's system and workflow requirements and how you best envision them being implemented in the system.

Again, be careful about falling for the temptation of presenting a full URS or other specification document to the vendor during the demonstration. You'll want to wait until after participating in several software demonstrations to consider presenting your full specification document to the vendor, and that's assuming that you've grown enamored with their solution. By waiting to finalize your lab's URS until after the demos, a common error is avoided: too often labs think the first thing they must do is create a requirements list, then sit back and let the informatics vendors tell them how they meet it. Remember that even though most labs thoroughly understand their processes, they likely don't have as strong a grasp on the informatics portion of their processes and workflows. Participating in a demo before finalizing your list of specified requirements—or having only a minimal yet flexible requirements list during the demo—is a great way to later crosscheck the software features you have seen demonstrated to your lab's processes and any initial requirements specification you've made.[19] After all, how can you effectively require specific functions of your laboratory informatics software if you don't fully know what such a system is capable of? After the demonstrations, you may end up adding several requirements to your final specifications document, which you later pass on to your potential vendors of choice for final confirmation.

In some cases, rather than approaching vendors, a lab may want to take a different exploratory route by issuing a request for information (RFI) or something similar. An RFI is an ideal means for learning more about a potential solution and how it can solve your problems, or for when you're not even sure how to solve your problem yet. However, like when approaching a vendor directly, the RFI should not be unduly long and tedious to complete for prospective vendors; it should be concise, direct, and honest. This means not only presenting a clear and humble vision of your own organization and its goals, but also asking just the right amount of questions to allow potential vendors to demonstrate their expertise and provide a clearer picture of who they are. Some take a technical approach to an RFI, using dense language and complicated spreadsheets for fact finding. However, as previously noted, you will want to limit the specified requirements in your RFI to those carefully chosen because they matter to you and your lab the most.[20]

Remember, an RFI is not meant to answer all of your questions. The RFI is meant as a means to help narrow down your search to a few quality candidates offering functionality that meets your needs, while also learning more about each other.[20] Once the pool of potential software vendors is narrowed down, and you then participate in their demonstrations, you then can broadly add more requirements to the original collection of critical requirements from the RFI to ensure those vendors can meet all or most of your needs. That said, be cognizant that there may be no vendor that can meet each and every need of your lab. Your lab will have to make important decisions about which requirements are non-negotiable and which are more flexible. The vendors you engage with may be able to provide realistic advice in this regard, based upon your lab's requirements and their past experience with labs. As such, those vendors with real-world experience meeting the needs of laboratories like your may have a strong leg up on other vendors.

Ultimately, your URS may look similar to LIMSpec, or it may have a slightly different format. Many prospective buyers will develop a requirement specification in Microsoft Excel, but that has a few minor disadvantages. Regardless of format, you'll want to give plenty of space for vendors to submit a response to each requirement. With a more complete URS in hand, and your vendor list narrowed down to a few possibilities, you're ready to submit the full URS to those vendors. Other considerations come into play afterwards, depending how those vendors' respond to you URS. How well did the vendors respond to your laboratory's unique set of needs? Were there critical areas that one vendor could address with their off-the-shelf solution but another vendor would have to address with custom coding? Did any of the vendors meet your budget expectations? Have you followed up on any references and customer experiences the vendors provided to you?

Hopefully, once all the due diligence is done, and with a little luck, you've found a vendor that fits your technology and functionality needs, even if a few minor compromises had to be made along the way. However, there's obviously more than technology and system functionality to consider! How is their solution offered? What kind of licensing is used? What are the specifics of the maintenance, warranty, and support plans? The next section addresses that.


4. What other considerations should I make, and how much will it cost?

Purchase vs. subscribe

In the past this was not an option. But much like the recent trend toward leasing cars rather than finding a large amount of money for up-front purchasing, labs can choose to pay only the cost of services (setup, training, report configuration, instrument interfaces, data migration, custom functions, etc.) and get started on a monthly subscription rather than buy licenses outright. When does this make sense? Subscriptions make sense primarily:

  • ...if a large lump sum is hard to get budgeted. If your business cash flow will support the regular subscription fee but finding license fees is more problematic, then a subscription may be right for you. But do the math. Calculate project costs over a reasonable period (e.g. five years) to make sure it is a value proposition. Be sure to include maintenance and support in your figures; this is often included in a subscription but not in a license.
  • ...if you may need to reduce the number of users. Once you buy licenses, they are yours. You can't "un-buy" them. But with a subscription you can raise and lower the number of users, workstations, etc. as you need to.
  • ...if you may need to bail. Business decisions often need to be dynamic. Your lab may decide to go into another area of analysis, and if your LIMS isn't versatile enough to support the change, you have potentially wasted a lot of money.

On the other hand, it may be important to you to have the LIMS source code. Some subscriptions allow you just as much access to it as if you had purchased licenses, while others may not give you the access you seek. Confirm this with the vendor. Alos, ask whether you get to keep an image of the database should you decide to end your subscription.

Onsite vs. SaaS

A small but growing number of LIMS vendors will actually host your system on their servers for you or cloud-host it elsewhere. We refer to software accessed via the Internet rather than your workstation or server as software as a service or SaaS. Most of us already make copious use of SaaS whenever we "Google" something. Cloudhosted SaaS is characterized by multiple load-balanced servers that allow resources to be strategically used, and virtualized servers that allow for the creation of custom environments.To decide if SaaS is for you or if you should go the traditional route, here are some points to consider:

  • If you have a small or overworked IT department, or none at all, then it may make sense to let the LIMS provider take care of those functions rather than invest in additional hardware, personnel, and other resources just to support your LIMS. If you are a large company with an extensive and capable IT department, then you may prefer the LIMS and its database to reside on premises.
  • IT techs cite security as a major reason to keep a LIMS on lab premises. The truth is, if the vendor uses a SAS-70 or SAS 70 Type II data center to host, with GxP SOPs, your system and data are probably a lot safer than on a typical business infrastructure. Ask the vendor.
  • If you decide to have your system hosted, ensure it's not by Bob and his buddy in their basement. The vendor needs to have been around awhile, have solid references, and feature good customer service.
  • A reputable SaaS host will guarantee you high availability, approaching 100% up time, with quick and responsive catastrophe response. Redundant components and infrastructure (power, cooling, etc.) allow them to do that.

Modular vs. complete

Some LIMS are offered as a collection of modules for you to select from to constitute your completed system, while others come complete with all the functionality available. Those whose LIMS are modular espouse the benefit of only paying for the functionality you need. Those whose LIMS come as a complete package say labs won't need to pay extra for any add-ons. Who's right? Well, it depends. If buying modules means you need one module for sample tracking and another for data entry, and still another to generate reports, then it may not be long before you run up a sizable bill just to get basic standard functionality, especially if the modules require hourly services to implement. If the modules tend to be industry-specific and complete, then they may make sense. Make sure you compare your needs with the product functionality and identify all costs associated with getting everything you need out of the software.

Named users vs. concurrent users

When comparing license fees, understand the difference between named users and concurrent users. If a vendor charges by named users, and your lab will have 30 people who will use the LIMS at any time, you will need 30 licenses. If the vendor charges by concurrent users, then you only need enough licenses to cover the number of users who are likely to be on the system at the same time. Typically in a lab with 30 staff, you might need a maximum of 20 concurrent user licenses. This is reduced even further if you have sites in other parts of the world whose work days differ.

The company

As important as the LIMS and its functions are to you, the company is at least as important. Make no mistake: this is a relationship you are entering into. This is not like selecting a piece of furniture. A LIMS is like a living, dynamic entity, and you'll need to interact with the vendor from time to time even with the most trouble-free system. Of course that interaction will be particularly intense in the beginning as they provide installation, provisioning, training, and other set-up services. Take your cue from your initial dealings with them. Just like in any relationship, they will be presenting their best side to you then. If the vendor return calls or emails late or fails to follow through with what they say they'll do, then you can bet it will be much worse once you are their customer. So yes, do the usual: research their years in business, size, staff qualifications, references, etc., but also ask yourself if you would be comfortable doing business with the vendor in the long term.


How much will it cost?

OK, now you understand what to look for in a company and its products. What you likely don't yet know: the price tag. Heck, most of us don't even know how LIMS vendors price their products or what is involved, much less how much they actually cost. In truth, there are three vital pricing components for any LIMS:

  1. licenses
  2. subscriptions
  3. services

The software itself never comprises the entire cost. LIMS are complex creatures, and your lab, even if it's small, is fairly complex, too. Let's go over what's involved and how much it's roughly going to cost.

Licenses

If the software has a purchased license type (as opposed to rented/subscription), then you will of course have to pay for those. Keep in mind what we said earlier about named vs. concurrent user pricing. Other methods include by site, by CPU or server, by workstation, or by unlimited user corporate level licensing. Arguably the lack of standardization in this area has contributed as much as anything to the vagueness that has surrounded LIMS pricing for so long. The linked vendor profiles in the next section feature pricing information for licenses for the included vendors. (Remember: the primary criterion for inclusion is publicly available pricing.) Review and compare, but make sure you factor in pricing method.

Subscriptions

These include two possible items:

  1. rented or SaaS LIMS
  2. annual maintenance, support, and warranty (MSW)

The cost of LIMS rental is equivalent to the licensed type, but a lump sum up front is not required. These can run anywhere from a couple of hundred dollars a month for a single user up to maybe $2000 or so for 20+ users. Just like purchased licenses, however, these can be priced by site, concurrent or named users, etc., so make sure you compare like with like or at least factor these considerations in as you shop. And your rental may be annual instead of monthly. In most cases it does include all IT services and maintenance, support, and warranty, including updates, at a specified level.

The second type of subscription cost is annual MSW, and you need to factor that into your budgeting if you are buying LIMS licenses. Typically it is priced at around 15 percent of the license fee and is available at graduated levels. A certain level may be standard for a certain number of licenses (for example, 10 hours of support and additional services available at $200 per hour for a 10-concurrent user LIMS), but you can buy a higher level of support and cheaper rate for additional services if you want to pay extra. One thing to keep in mind: with an MSW you will certainly need coverage as you go through your first year. If you think you can then drop it, think again. A modern LIMS should be built on technology that can give it a much longer life span than those in years past. That is dependent on staying updated. If you lose that update path, your LIMS will expire prematurely. If you decide later to renew MSW, you may find yourself liable for the missed years before the vendor will bring you current.

Services

Your LIMS is a function of the cost of the LIMS itself plus the services involved in its implementation plus, in the case of a licensed LIMS, annual MSW. Many first-time LIMS buyers neglect to factor in the cost of services when budgeting. As mentioned earlier, any LIMS will require services to get going, and you may want more if there are extras you need or want. Services break down more or less like this:

Basic implementation services

  • kickoff meeting (planning, coordination, communication procedures, etc.)
  • ƒtraining
  • setup (enter users, configure profiles, departments, tests, screens, etc.)
  • create main report(s)
  • go live support

Additional or optional services

  • instrument interfaces
  • additional reports
  • data migration from a previous system
  • interfaces to other systems or databases
  • special customizations
  • web portal configuration
  • validation
  • standards certification support

You may need other services. Rates for services vary from vendor to vendor, but a good rule of thumb for initial budgeting purposes is to figure service costs to be roughly equal to the licensing cost or to a year's worth of LIMS subscription.

 


5. Commercial vendors with public pricing

Finally, a primary criterion for inclusion in this guide is publicly available pricing information that can thusly be cited. If citeable public pricing is not available, the vendor will not be listed in this guide. Any vendors who remove pricing or no longer make it public will be removed from the vendor list.

To hide the contents of this section for easier reading of other sections, click the "Collapse" link to the right.

 


Website and submitted pricing

These vendors either post most of their pricing information for their solutions on their website or have submitted their pricing to the wiki admin directly. In all cases, be sure to contact the vendor to verify pricing, as it may change with little notice. Vendors are sorted by number of public videos demonstrating how their products work.

Vendor Pricing Has cloud or
SaaS offering?
# of demo
videos
LIMS? LIS? ELN? CDS
and/or
SDMS?
Additional notes
Research Innovations Limited Link Yes 90+ No No Yes No
LabKey Corporation Link Yes 86 Yes No Yes Yes
Biomatters Ltd. Link No 68 Yes No No No
Creliant Software Pvt. Ltd. Link Yes 60 No Yes No No
Vendor:LabArchives, LLC Link Yes 56 No No Yes No All but Enterprise pricing listed.
Labii, Inc. Link Yes 56 Yes No Yes No Product is combination LIMS and ELN
SLCLAB Informática SL Link Yes 38 Yes Yes No No
Labstep Ltd. Link Yes 28 No No Yes No
Terra Systems OÜ Link No 17 No Yes No No Pricing only available for MiniLIS, not other products.
MEDEORA GmbH Link Yes 16 Yes No No No
HealthScion Technologies Pvt. Ltd. Link Yes 15 Yes No No No LIMS is part of a larger overall platform.
LabLynx, Inc. Link Yes 15 Yes Yes Yes Yes Pricing supplied directly by company.
Labforward GmbH Link Yes 14 No No Yes No
Mestrelab Research S.L. Link Yes 14 No No Yes No
Autoscribe Informatics, Inc. Link Yes 13 Yes No No No Pricing supplied directly by company.
CloudLIMS.com, LLC Link Yes 11 Yes No No No
AgileBio Link No 9 Yes No Yes No
INTEGRIS LIMS GmbH Link No 9 Yes No No No Pricing available for "Small lab solution"; videos in German
Aiderbotics Corporation Link No 7 No No Yes No
WebPathLab, Inc. Link Yes 6 No Yes No No Pay-per-report pricing made public
Forney, LP Link Yes 4 Yes No No No
Vendor:LabSmart Healthcare Technologies Link Yes 4 No Yes No No
Simploud Limited Link No 4 Yes No No No
Mci IT Pty. Ltd. Link Yes 3 Yes No No No
The Edge Software Consultancy Ltd. Link Yes 3 Yes No Yes No
Third Wave Analytics, Inc. Link Yes 3 Yes No No No
BioInfoRx, Inc. Link Yes 2 Yes No No No
Biomed Systems Ltd. Link Yes 2 Yes No No No LIMS available with ELN as add-on.
Broughton Software Ltd. Link Yes 2 Yes No No No Pricing supplied directly by company.
ML Consulting Technologies, LLC Link Yes 2 Yes No No No Free quality control LIMS for small laboratories
Revol Software Solutions, LLC Link Yes 2 Yes No No No
AINZ Corp. Link Yes 1 Yes No No No Former product was PharmWare.
Blaze Systems Corporation Link No 1 Yes No No No Pricing supplied directly by company.
Bureau Conseils et Services SCRL Link Yes 1 Yes No No No Starting prices listed
Bytewize AB Link Yes 1 Yes No No No Pricing only available for O3 LimsXpress.
TeselaGen Biotechnology, Inc. Link Yes 1 Yes No No No Pricing only available for Community and Starter editions.
Vendor:Apex HealthWare, LLC Link Yes 0 No Yes No No Pricing only available for cloud-based POL version.
BiochemLab Solutions Link No 0 No No Yes No Free ad-based and paid ad-free versions of ELN exist.
ChemBytes Link No 0 No No Yes No Its successor to its free basic Espresso ELN will be Phoenix ELN,
yet to be released as free, open-source software.
Dorays Technologies Pvt. Ltd. Link Yes 0 No Yes No No
General LIMS, Inc. Link No 0 Yes No No No
Genetic Technologies, Inc. Link No 0 Yes No No No
J Street Technology, Inc. Link No 0 Yes No No No
LabLite, LLC Link No 0 Yes No No No Pricing supplied directly by company.
Labstudio (Pty) Ltd. Link Yes 0 Yes No No No
Microcline Projects Link Yes 0 Yes No No No
Minerva Data Link Yes 0 No Yes No No
Progeny Genetics, LLC Link Yes 0 Yes No No No
Wolfe Information Systems, Inc. Link No 0 Yes No No No

Free, minimally featured version, but no other public pricing

These vendors provide minimal pricing information, stating they have a free, minimally featured version of their software but not revealing any other pricing information for other versions of the same software. These are usually vendors offering a tiered solution, with the lowest tier being the free tier. Vendors are sorted by number of public videos demonstrating how their products work.

Vendor Pricing Has cloud or
SaaS offering?
# of demo
videos
LIMS? LIS? ELN? CDS
and/or
SDMS?
Additional notes
Benchling, Inc. Link Yes 36 No No Yes No Academic pricing is free, but rest of pricing was removed.
LabSpace, LLC Link Yes 0 No No Yes No Academic pricing is free, but rest of pricing was removed.

GSA pricing

These are vendors who have a publically viewable contract with the U.S. General Services Administration (GSA), and that contract displays prices for their solutions. As these are GSA prices, be aware they represent the lowest possible negotiated price. These prices aren't available to any entity outside of the Federal Government, and thus prices for non-government entities will be higher. If anything, they represent the lowest possible negotiated price. Vendors are sorted by number of public videos demonstrating how their products work.

Vendor Pricing Has cloud or
SaaS offering?
# of demo
videos
LIMS? LIS? ELN? CDS
and/or
SDMS?
Additional notes
Waters Corporation Link No 33 Yes No Yes Yes
LabLynx, Inc. Link Yes 15 Yes Yes Yes Yes
STARLIMS Corporation Link Yes 8 Yes No Yes Yes
LabVantage Solutions, Inc. Link Yes 6 Yes No Yes No
LabWare, Inc. Link Yes 6 Yes Yes Yes No
Promium, LLC Link Yes 2 Yes No No No
Xybion Corporation Link Yes 2 Yes No No No
Bentley Systems, Inc. Link No 0 Yes No No No
JusticeTrax, Inc. Link No 0 Yes No No No


6. Additional resources and help

LIMSforum

Formerly a LinkedIn-associated group, LIMSforum is a web portal for those interested in laboratory, scientific, and health informatics.

LIMSforum Career Opportunities

Formerly the LinkedIn-associated Lab Careers group, LIMSforum's career opportunities section is available for the viewing and posting of job openings for laboratory, scientific, and health lab careers.

LIMSforum Online Courses

Formerly LIMS University, the laboratory courses at LIMSforum provide free, open-access learning and teaching resources for those wanting to learn more about laboratory informatics.

LIMSfinder

LIMSfinder is a web portal for those looking for a LIMS and related information, services, products, news, events, resources, jobs, etc.

LIMSpec.com

LIMSpec.com provides a collection of datasheets — from lab requirements assessment to LIMS vendor and system questionnaires, validation documents, and more — for identifying LIMS needs and matching them with what's out there.

LIMSwiki informatics resource portal

The informatics resource portal here at LIMSwiki features a collection of as many useful online scientific and health informatics-related materials and research tools as possible, including books, journals, blogs, web portals, education programs, conferences, and more.

References

  1. "Laboratory Informatics Institute Established". Laboratory Informatics Institute, Inc. 17 July 2006. Archived from the original on 24 September 2013. https://web.archive.org/web/20130924062254/http://www.limsfinder.com/BlogDetail.aspx?id=31049_0_3_0_C. Retrieved 31 January 2023. 
  2. "The LIMSbook ...everything about LIMS". Laboratory Informatics Institute, Inc. Archived from the original on 02 Junw 2013. https://web.archive.org/web/20130602164430/https://www.limsbook.com/. Retrieved 31 January 2023. 
  3. Metrick, Gloria (15 August 2011). "Understanding Openness and Other Marketing Tactics in Laboratory Informatics and Other Industries". GeoMetrick Enterprises. http://outonalims.com/2011/08/15/understanding-openness-and-other-marketing-tactics-in-laboratory-informatics-and-other-industries/. Retrieved 06 September 2013.  [permanent dead link]
  4. Allen, Thomas J.; Cohen, Stephen I. (1969). "Information Flow in Research and Development Laboratories". Administrative Science Quarterly 14 (1): 12–19. doi:10.2307/2391357. http://www.jstor.org/stable/2391357. 
  5. California Health Care Foundation (19 March 2014). "ELINCS: The National Lab Data Standard for Electronic Health Records". https://www.chcf.org/project/elincs-the-national-lab-data-standard-for-electronic-health-records/. Retrieved 31 January 2023. 
  6. Japsen, B. (29 March 2014). "Despite Stimulus Dollars, Hundreds Of Hospitals Still Use Mostly Paper Records". Forbes. https://www.forbes.com/sites/brucejapsen/2014/03/29/despite-stimulus-dollars-hundreds-of-hospitals-still-use-mostly-paper-records/?sh=25c7b0364c19. Retrieved 31 January 2023. 
  7. Colangeli, Patrizia; Del Negro, Ercole; Molini, Umberto; Malizia, Sara; Scacchia, Massimo (1 December 2019). "“SILAB for Africa”: An Innovative Information System Supporting the Veterinary African Laboratories" (in en). Telemedicine and e-Health 25 (12): 1216–1224. doi:10.1089/tmj.2018.0208. ISSN 1530-5627. https://www.liebertpub.com/doi/10.1089/tmj.2018.0208. 
  8. "Astrix 2020 LIMS Market Research Survey Report" (PDF). Astrix Technology, LLC. March 2021. https://astrixinc.com/wp-content/uploads/2021/03/Astrix-2020-LIMS-Market-Research-Report.pdf. Retrieved 01 February 2023. 
  9. Liscouski, J. (December 2020). "Laboratory Technology Planning and Management: The Practice of Laboratory Systems Engineering". LIMSwiki. https://www.limswiki.org/index.php/LII:Laboratory_Technology_Planning_and_Management:_The_Practice_of_Laboratory_Systems_Engineering. Retrieved 01 February 2023. 
  10. Liscouski, J. (April 2021). "The Application of Informatics to Scientific Work: Laboratory Informatics for Newbies". LIMSwiki. https://www.limswiki.org/index.php/LII:The_Application_of_Informatics_to_Scientific_Work:_Laboratory_Informatics_for_Newbies. Retrieved 01 February 2023. 
  11. 11.0 11.1 Smith, K. (2 July 2019). "Integrated Informatics: Optimizing Food Quality and Safety by Building Regulatory Compliance into the Supply Chain". Food Safety Tech. https://foodsafetytech.com/feature_article/integrated-informatics-optimizing-food-quality-and-safety-by-building-regulatory-compliance-into-the-supply-chain/. Retrieved 20 January 2023. 
  12. McDermott, P. (31 July 2018). "How Digital Solutions Support Supply Chain Transparency and Traceability". Food Safety Tech. https://foodsafetytech.com/column/how-digital-solutions-support-supply-chain-transparency-and-traceability/. Retrieved 20 January 2023. 
  13. Evans, K. (15 November 2019). "The Digital Transformation of Global Food Security". Food Safety Tech. https://foodsafetytech.com/feature_article/the-digital-transformation-of-global-food-security/. Retrieved 20 January 2023. 
  14. Izrailevsky, Y.; Bell, C. (2018). "Cloud Reliability". IEEE Cloud Computing 5 (3): 39–44. doi:10.1109/MCC.2018.032591615. 
  15. Douglas, S.E. (July 2020). "Comprehensive Guide to Developing and Implementing a Cybersecurity Plan". LIMSwiki. 
  16. "specification". Merriam-Webster. Merriam-Webster, Inc. https://www.merriam-webster.com/dictionary/specification. Retrieved 20 January 2023. 
  17. "ASTM E1578-18 Standard Guide for Laboratory Informatics". ASTM International. 23 August 2019. https://www.astm.org/e1578-18.html. Retrieved 20 January 2023. 
  18. "ISO/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories". International Organization for Standardization. November 2017. https://www.iso.org/standard/66912.html. Retrieved 20 January 2023. 
  19. Hammer, S. (27 June 2019). "How to Get the Most Value from an ERP Software Demo". The Takeoff. https://www.striven.com/blog/erp-software-demo. Retrieved 02 February 2023. 
  20. 20.0 20.1 Holmes, T.. "It's a Match: How to Run a Good RFI, RFP, or RFQ and Find the Right Partner". AllCloud Blog. https://allcloud.io/blog/its-a-match-how-to-run-a-good-rfi-rfp-or-rfq-and-find-the-right-partner/. Retrieved 02 February 2023.