In 1970, Temple University Medical School's Marion Ball, M.A., an assistant professor in the Department of Medical Physics, conducted a survey of pathology directors in clinical laboratories that use computers. Asking their opinions about the advantages and disadvantages of computerized systems in the lab, she received responses from directors in 15 U.S. states, as well as from three other countries. Responses included:
The ability to rapidly prepare cumulative records and then to inspect them for possible errors through analysis trends has been proven to be of tremendous advantage in a number of laboratories. We can prevent errors in our analytical systems, but we are not prepared to prevent errors in the collecting of the sample, the mislabeling of the sample, or the accidental use of an incorrect sample. Thus, the ability to inspect data trends presents the only real tool that we currently have to pick out these kinds of errors. - Max E. Chilcote, Ph.D, Meyer Memorial Hospital Division
There is little argument about whether an operating computer system can be an advantage in a laboratory, but the most critical time is the installation and transition from a "manual" to a "computer" oriented laboratory. - Robert L. Habig, Duke University Medical Center
The most pressing future need for computerization of the laboratory lies in the area of medical diagnosis and guidance of the therapeutic management. This is where the physician's role for the future in the laboratory lies ... We will be gathering vast amounts of information on the health status of many individuals. We can then take advantage of large data processing computers to analyze this information and come up with patterns of disease states. - Leonard Jarett, M.D., Barnes Hospital
Reading about these potentials and opinions today, some 50 years later, we see both clear similarities and definite advances. For example, Habig's statement about transitioning from manual to more automated processes still rings true today: it can be nerve wracking and critical to get the transition right. Conversely, while the systems of decades past weren't able to "prevent errors in the collecting of the sample, the mislabeling of the sample, or the accidental use of an incorrect sample," modern laboratory informatics systems provide many assurances to sample management in the lab. In many cases, activities such as label generation, reporting, results analysis, workflow control, test ordering, and broad interoperability are commonplace in modern systems. And those systems continue to advance, with machine learning now becoming a part of laboratory data management and analysis.
We've come a long way since the 1960s, to a point where the question is no longer "can a computerized system help my lab?" but rather "how do I choose and implement an informatics system to help my lab?" What follows is information to help you with that question, while considering the technology, features, security, cost, implementation, and vendor guarantees that come with such a system.
- 1 2.1 Evaluation and selection
- 2 2.2 Implementation
- 3 2.3 MSW, updates, and other contracted services
- 4 2.4 How a user requirements specification fits into the entire process
- 5 References
2.1 Evaluation and selection
What exactly is a laboratory information system (LIS) or laboratory information management system (LIMS) anyway? Do I need one? What options are available and how do I compare them? What about a request for information (RFI), request for proposal (RFP), or request for quotation (RFQ)? These are questions laboratory professionals typically ponder upon finding themselves charged with the mission of finding software for their lab. For many the task can be a daunting proposition.
You may know the workflow-related needs of your laboratory, but perhaps you don't know much about data management solutions like LIS and LIMS, leaving you intimidated by all the options. You'll first need to gauge your lab's informatics needs in order to determine which products are worth investigating further. Of course your lab's analysis requirements, reporting and data sharing constraints, instrument interfacing needs, barcoding and tracking requirements, quality assurance processes, etc. are very important factors. But these systems vary in numerous ways, and other important factors exist. Price should certainly be considered, although value is ultimately more important than a low price. Other important question that get asked include:
- Should we purchase software licenses or "rent" the software via a subscription-based model?
- Does the software need to be on-site, or is a SaaS hosted option more practical?
- Is a modular or complete system better for us?
- What is the best licensing/rental scheme for us? Show we consider site, named user, concurrent user, or workstation licenses?
- Is the company qualified and trustworthy?
- What functionality is available to help our lab not only accomplish workflow tasks but also remain regulatory compliant?
These and other questions are addressed in this chapter.
2.1.1 Technology considerations
Your laboratory's workflow, instruments, data management requirements, budget, technological expertise, business goals, and risk tolerances will all play a role in deciding what technology to invest in. The physician office lab (POL), with its easy-to-use point-of-care testing and relatively simplified laboratory procedures, will invest significantly less into analyzers, instruments, and laboratory software than the molecular diagnostics laboratory, for example. As such, look at your laboratory's short- and long-term goals, budget, workflow, and regulatory requirements to gain a better understanding of what technology will be involved.
First, what are the laboratory's goals? Does the laboratory owner envision a small investment, taking in a slow but steady flow of simple clinical tests of human fluids, or expansive growth, expanding into multiple testing domains? If the lab is starting small but is confidently expecting to grow, technological investments early on may want to take into account future technologies that may shape data management procedures. Second, what kind of work will the lab be doing, and what regulatory responsibilities will guide hardware and software investment at the lab? If your lab will be testing medical cannabis for the state or province's associated program, you'll be considering chromatography and spectroscopy instruments, as well as regulatory requirements for complete track-and-trace activities, including reporting. The public health laboratory will likely have many more instruments to cover all its testing needs, and its data management system will likely need to be able to use the Centers for Disease Control and Prevention's PHIN Messaging System. Third, your laboratory's budget is ever important. Does the budget allow for on-site hardware and software systems, with the personnel to maintain them? Is it easier to pay up-front or find a vendor willing to work with you on leasing or rental terms? (We talk about other cost considerations a bit later.)
Finally, will the lab have someone on-site or on-call to resolve technological issues, including set-up and maintenance of software systems? If your lab will have little in the way of available tech help locally, you'll want to consider the distribution model you want to use for any installed software, i.e., you may want to consider software as a service. An increasing number of software services are hosted using cloud computing, which when done well is an increasingly reliable option. Having someone else host the software for you typically means the hosting provider will carry a huge portion of responsibility for technology maintenance and security. Speaking of security, you'll also want to consider the cybersecurity (addressed later) of not only your software solution but also your overall laboratory operations. Does your laboratory have a cybersecurity plan already in place, or has the decision to make one been postponed? What extra investment is required to ensure your sensitive data is secure? Remember that how you rank your cybersecurity preparedness and implement a cybersecurity plan will also guide your technology investment decisions.
22.214.171.124 Laboratory informatics options
A December 2019 survey by Medical Laboratory Observer, consisting of 273 respondents, is somewhat revealing in what a LIS or LIMS is being used for by a medical laboratory. Ninety-five percent of respondents indicated they use it to streamline their electronic order entry and result management, with medical data connectivity being the second most popular use. Automation tools, customer relationship management, scheduling, inventory management, revenue management, quality management, and reporting were all also mentioned as important to users. When asked to select from five choices (or provide some other reason) in regard to what their top priority was in selecting a LIS or LIMS, respondents indicated that most important was providing data analysis mechanisms for all types of pathology. See Table 1 below for all responses.
These responses help paint a picture of what a LIS or LIMS can do, but there's definitely more to it. (See the next subsection on features and functions.) And other systems are also being used in medical laboratories. The previously mentioned MLO survey indicated that 68 percent of respondents came from a hospital laboratory, highlighting their importance in the medical diagnostic laboratory demographic. As such, we'd be remiss to mention the hospital information system (HIS), a hospital-level information management system that often incorporates modular functionality similar to that of a LIS or LIMS. However, some such labs will have their own laboratory data management solution independent of the HIS.
The survey also made reference the the EMR. This software, along with the electronic health record (EHR), is most prevalent among health care systems and other ambulatory providers, including physicians. (As of 2017, approximately 86 percent of U.S. physicians have adopted EHRs.) These systems act as portable, longitudinal collections of patient and population data and a convenient tool for documenting, monitoring, and managing health care delivery. Medical diagnostic laboratory workflow typically sees test data from a LIS get transferred to the respective patient's record in the EHR.
Finally, you may also see electronic laboratory notebooks (ELN) in medical research labs. This software acts as an electronic substitute for the traditional laboratory notebook, assisting researchers with direct recording of experiment data, linking records, and protecting proprietary information. They can typically be integrated with other software systems as well.
Choosing the right software will largely depend on your laboratory type and what you wish to accomplish. We next review the base features of offerings like an LIS and LIMS, as well as the features required by sub-specialties of medical science.
2.1.2 Features and functions
A LIS or LIMS can have an extravagant list of features, or it may have minimal functionality. Software developers with competent and experienced personnel usually do well with a collection of the required base features, plus any industry-specific features a laboratory may need. But not all developers get it right.
What follows is a list of system functionality that is considered by a variety of experts to be vital to almost any medical diagnostic or research laboratory. Without this functionality, end users may at best grumble about additional workloads or more complicated procedures, and at worse be setting themselves up for major liability issues by not complying with regulations. Arguably, a few items such as mobile device support, voice recognition, and multilingual support may be negotiable, but if the system you are evaluating doesn't contain most of the below bullet-pointed functionality, you may want to look elsewhere.
Test, experiment, and patient management
- specimen log-in and management, with support for unique IDs
- batching support
- barcode and RFID support
- specimen tracking
- clinical decision support, including test ordering tools and duplicate test checks
- custom test management
- event and instrument scheduling
- templates, forms, and data fields that are configurable
- analytical tools, including data visualization, trend analysis, and data mining features
- data import and export
- robust query tools
- document and image management
- project and experiment management
- workflow management
- patient management
- case management
- physician and supplier management
Quality, security, and compliance
- quality assurance / quality control mechanisms, including tracking of nonconformance
- data normalization and validation
- results review and approval
- version control
- user qualification, performance, and training management
- audit trails and chain of custody support
- configurable and granular role-based security
- configurable system access and use (log-in requirements, account usage rules, account locking, etc.)
- electronic signature support
- configurable alarms and alerts
- data encryption and secure communication protocols
- data archiving and retention support
- configurable data backups
- environmental monitoring and control
Operations management and reporting
- customizable rich-text reporting, with multiple supported output formats
- synoptic reporting
- industry-compliant labeling
- email integration
- internal messaging system
- revenue management
- instrument interfacing and data management
- instrument calibration and maintenance tracking
- inventory and reagent management
- third-party software and database interfacing
- mobile device support
- voice recognition capability
- results portal for external parties
- integrated (or online) system help
- configurable language
In the following subsections, the subcategories of labs we looked at in the prior chapter are reviewed, specifically for functionality critical to their specialty. This functionality is supported by four to five citations from vendors and other academic sources.
- configure the system using templates for histology and cytology case types
- add, view, and link pre-generated organ maps and other diagrams
- add, view, and link custom annotated pathology imaging
- track abnormal results and provide trending reports for monitoring disease populations
- support blocks and slides as specimens, with predefined descriptions
- document grossing examinations
- print slides and cassettes
- provide case management, reporting, and test requisition
- provide specialty workflow for autopsy
- provide specialty workflow for gynecological cytology, including HPV + Pap co-testing for cervical cancer
- provide stain panels and histology worksheets
- support shared management of tissue samples among departments
- support EHR integration
- support polymerase chain reaction (PCR) workflow and reporting
- support for pathology-specific reflex testing
- provide option to combine same-day anatomical and clinical pathology results and reporting
- flag unusual cases for conference or committee reporting
- support pre-logging of evidence
- allow full documentation of a crime scene
- track storage, movement, and disposal of evidence and property using an ASTM-compliant log
- manage chain-of-custody transfers of evidence and samples
- provide quarantine protocol for samples and evidence
- provide forensic case management, including case status and court testimony
- manage agency interactions and information
- add, view, and link forensic imaging into case files
- manage field scheduling for fingerprinting, homicide casing, and lab work
- provide custom reporting for toxicology and controlled substance analyses
- provide support for DNA profile management
- provide support for convicted offender and other database integration
- support the use of personal identity verification and other forms of hardware-based (i.e., public key infrastructure or PKI) token authentication
Physician office lab:
The physician office laboratory (POL) is arguably a more simple version of the medical diagnostics lab, often depending on CLIA-waived and CLIA-certified point-of-care instruments for making diagnoses. As such, the data management requirements for a POL may not be as significant as those of a large-scale diagnostic laboratory. That said, a POL employing laboratory informatics will still need much of the same base functionality mentioned prior, and the system will still need to comply with data management and sharing regulations such as those found with HIPAA and CLIA.
Any POL performing sufficient volumes of testing to benefit from using a laboratory informatics solution may also want to consider the costs and drawbacks, if any, of interfacing to their EHR system, if they have one. In a case where the POL is in a position to consider both an LIS and an EHR at the same time, they should examine the features and potential integration of those products, and they should be sure to consider any future potential of integrating their systems with other external data management systems, including another reference laboratory.
In some cases, an EHR with some laboratory management functionality may make a solid alternative. If considering an EHR that includes some LIS functionality, be sure to clearly identify the functional requirements and demo the system thoroughly to ensure test and reporting workflows make sense. Finally, in cases where POL test volumes are low—coming from only one or a few instruments—and an LIS is not required, POL operators may want to simply consider a middleware option that smoothly facilitates the flow of instrument data to the EHR.
Integrative medicine lab:
If an integrative medicine laboratory is using a laboratory informatics solution, their requirements will be nearly identical to a standard medical diagnostic laboratory, meaning the base functionality mentioned prior will likely be suitable. If there is a major difference or required piece of additional functionality, it will have to do with a more extensive list of available tests and billing codes for them. This usually consists of expansions into nutritional, metabolic, and toxicity test types, as well as support for diagnostic imaging.
- provide specialty workflow for newborn screening
- provide surge capacity for high-priority analyses
- provide workflow and tools for managing microorganisms and toxins of elevated risk
- support most medical test protocols and specimen types
- support ELISA, DNA extraction, sequencing, and other molecular workflows
- support for a robust set of decision support rules for reflex testing
- support the Centers for Disease Control and Prevention's PHIN Messaging System
- support other electronic data exchange standards for critical community partners
- support customizable drug panels and tests
- support reference lab activities
- track prescribed medicines and associated history
- provide management for compounds and compound grouping
- provide medication-based compliance monitoring and interpretive reporting on it
- provide decision-support rules for pain management and toxicology
- provide toxicology-specific reporting formats
- manage drug court cases associated with testing
- manage inventory across multiple facilities
- manage donor and harvested tissues
- support positive patient identification (PPID)
- support the ISBT 128 standard for medical products of human origin
- support for both autologous and directed medical product management
- allow for emergency release of inventory
- allow for electronic crossmatch of human-based medical products
- manage medical product recall and documentation
- manage donor demographics, notification, scheduling, and history
- manage donation drives and other campaigns
- track bag and supply lot numbers
- track quality control testing
- monitor access to and environmental conditions of supply fridges
- provide workflow management for non-standard patients
- support antibody screening processes
- manage and track clinical trial kits
- manage multi-site logistics of specimens
- provide a reservation function for specimens
- manage clinical trials and their various functions, including recruitment, study protocols, treatment groups, metadata, multi-site master scheduling, consent checks, and other required reporting
- provide special access privileges to sponsors, monitors, and investigators
- support a wide variety of data transfer formats, including CDISC, ASCII, SAS, and XML
- provide patient management, including demographics, consent forms, clinical notation, and test results
- provide highly configurable "blinding" features for reports and the user interface
- track contracts, budgets, and other financials
- develop exclusion rules and monitor exclusions
- support testing for a wide variety of disciplines
- provide study-specific monitoring and alerts
- provide granular cumulative reporting
- provide study-specific project portals that allow review of documents, data visualizations, training material, and other study information
- manage sample collection kits
- manage informed consent documentation
- provide customized workflows for molecular and [DNA sequencing#High-throughput methods|next-generation sequencing]] (NGS) testing
- track specimen and aliquot lineage for cell lines, tissues, slides, etc.
- track nucleic acid quantity and quality of specimens
- support a wide array of molecular testing and associated data fields, including biochemical and molecular genetics, carrier screening, immunology, molecular profiling, prenatal and newborn testing, and pharmacogenetics
- provide custom workflows for FISH, PCR, gel eletrophoresis, cytogenetics, DNA sequencing, and more
- support specialty testing reimbursement and other revenue management unique to this lab type
- support single sign-on with imaging platforms
- provide color coding for turn-around time and other testing statuses
- provide cleanly formatted rich-text reports customized for molecular diagnostics
- add, view, and link custom annotated images
- interface with a wide array of chromatography and spectroscopy instruments
- optimize sample login and management for the industry, including clear differentiation between medical and recreational cannabis
- provide compliant test protocols, workflows, labels, and reporting for medical cannabis testing
- provide ability to interface with state-required compliance reporting systems
- support inventory reconciliation
- support disease testing, sexing, and genetic tracking of cannabis
- support stability testing
2.1.3 Cybersecurity considerations
From law firms to automotive manufacturers, the need to address cybersecurity is increasingly apparent. In 2018, the Center for Strategic & International Studies estimated that cybercrime causes close to $600 billion in damages to the global economy every year, though due to underreporting of crimes, that number may be much higher. That number also likely doesn't take into account lost business, fines, litigation, and intangible losses In the end, businesses of all sizes average about $200,000 in losses due to a cybersecurity incident, and nearly 60 percent of small and midsize businesses go bankrupt within six months because of it.
Medical diagnostic and research laboratories are no exception, regardless of business size. Even tiny labs whose primary digital footprint is a WordPress website advertising their lab are at risk, as hackers could still spread malware, steal user data, add the website to a bot network, hack the site for the learning experience, or even hack it just for fun. Even more importantly are those labs performing digital data management tasks that handle sensitive patient and proprietary data, requiring additional cybersecurity considerations.
A laboratory can integrate cybersecurity thinking into its laboratory informatics product selection in several ways. First, the lab should have a cybersecurity plan in place, or if not, it should be on the radar. This is a good resource to tap into in regards to deciding what cybersecurity considerations should be made for the software. Can the software help your lab meet your cybersecurity goals? What regulatory requirements for your lab are or are not covered by the software? Another tool to consider—which may have been used in any prior cybersecurity planning efforts—is a cybersecurity framework. Many, but not all, cybersecurity frameworks include a catalog of security controls. Each control is "a safeguard or countermeasure prescribed for an information system or an organization designed to protect the confidentiality, integrity, and availability of its information and to meet a set of defined security requirements." These controls give the implementing organization a concrete set of configurable goals to apply to their overall cybersecurity strategy. Other frameworks may be less oriented to security controls and more program-based or risk-based. Choosing the best frameworks will likely depend on multiple factors, including the organization's industry type, the amount of technical expertise within the organization, the budget, the organizational goals, the amount of buy-in from key organizational stakeholders, and those stakeholders' preferred approach.
Finally, having a cybersecurity plan that incorporates one or more cybersecurity frameworks gives the laboratory ample opportunity to apply stated goals and chosen security controls to the evaluation and selection process. In particular, a user requirements specification (URS) that incorporates cybersecurity considerations will certainly help a laboratory with meeting regulatory requirements while also protecting its data systems. A USR that is pre-built with cybersecurity controls in mind—such as LIMSpec, discussed later—makes the evaluation process even easier.
2.1.4 Regulatory compliance considerations
Without a doubt, it's vital that medical diagnostic and research laboratories operate within the bounds of a regulatory atmosphere, not only to better ensure the best patient outcomes but also to ensure the quality of test results, the privacy of patient information, and the safety of personnel. Maintaining regulatory compliance requires deliberate approaches to developing and enforcing processes and procedures, quality training, consistent communication, and knowledgeable personnel. It also requires a top-down appreciation and commitment to a culture of quality. From the Clinical Laboratory Improvement Amendments (CLIA) and Health Insurance Portability and Accountability Act (HIPAA) to 21 CFR Part 11 and the General Data Protection Regulation, laboratories have much to consider in regards to what regulations impact them.
That said, consider approaching the question of regulatory compliance from the standpoint of adopting standards. Consider first that the risks and consequences of performing a task poorly drives regulation and, more preferably, standardization, which in turn moves the "goalposts" of quality and security among organizations. In the case of regulations, those organization that get caught not conforming to the necessary regulations tend to suffer negative consequences, providing some incentive for them to improve organizational processes and procedures.
One of the downsides of regulations is that they can at times be "imprecise" or "disconnected" from what actually occurs within the organization and its information systems. Rather than focusing heavily on regulatory conformance, well-designed standards may, when adopted, provide a clearer path of opportunity for organizations to improve their operational culture and outcomes, particularly since standards are usually developed with a broader consensus of interested individuals with expertise in a given field. In turn, the organizations that adopt well-designed standards likely have a better chance of conforming to the regulations they must, and they'll likely have more interest in maintaining and improving the goalposts of quality and security in the lab.
Additionally, reputable software developers of laboratory informatics software will not only adopt their own industry standards for software development but also understand the standards and regulations that affect laboratories and research centers. In turn, the developed software should meet regulations and standards, help the laboratory comply with its regulations and standards, and be of reliably good quality.
If you're a potential buyer of a laboratory informatics solution, it may be that you know a bit about your laboratory's workflow and a few of the regulations and standards that influence how that workflow is conducted, but you're not entirely informed about all the regulations and standards that affect your lab. Turning to a URS such as LIMSpec—which was developed around laboratory regulations and standards—and reviewing the various statements contained within may be necessary to help further inform you. Additionally, as you investigate various informatics options, you can then use the requirements in the URS as a base for your laboratory's own requirements list. Using the categories and their subdivisions, you can then add those requirements that are unique to your laboratory and industry that are not sufficiently covered by the base URS. As you review the various options available to you and narrow down your search, your own list of requirements can be used as both as a personal checklist and as a requirements list you hand over to the vendor you query. And since your URS is based off the standards and regulations affecting your lab, you can feel more confident in your acquisition and its integration into your laboratory workflow.
2.1.5 Cost considerations
First, you'll want to be clear on what will be included in the sales agreement. Whether through an estimate or statement of work (SOW), it is important it includes exactly what is expected, being as specific as possible, since this will be the entire contractual obligation for both you the buyer and them the vendor. Note that line items may differ slightly from system to system, according to what features and functions are included by default with each vendor's solution and which, if any, are additional. Also keep in mind that any hourly amount in the the estimate or SOW is usually a best estimate; however, if sufficient attention to detailed requirements has been given, then it should be quite accurate, and in fact the final cost may even be below the quoted cost if you prioritize your own obligations so that the vendor's hours are used sparingly and efficiently.
The estimate or SOW should optimally include:
- licensing or subscription rates;
- required core items to meet federal, state, and local regulations;
- additional optional items and totals; and
- required services (implementation, maintenance and support, optional add-ons).
There are two primary ways to price a laboratory informatics solution: a one-time license fee or a subscription rate (cloud-hosted software as a service [SaaS]). If you have your own dedicated IT department and staff, you may prefer the former (although many system administrators are just as happy to let it be hosted elsewhere rather than add to their workload). Otherwise, a SaaS subscription may well be the better and more cost-effective way to go (since the primary IT cost is simply internet access). This item will be part of your up-front cost and, in the case of subscription, it will also figure into your first year and ongoing costs; otherwise only associated maintenance, support, and warranty (MSW) will figure in. Typically, your first year's subscription costs will be due at signing. More often, the vendor may require three months or even the first year up front, so be prepared to factor that into up-front costs. However, it still is almost always less expensive at the outset (and over time, if you factor in IT costs and annual MSW) than paying for a license fee.
In addition to the two types of software pricing, there are also sub-types. Generally these are based on the number of users (or, in some cases, "nodes," which are simply any entities that access the informatics system, including other systems, instruments, etc.). How these are counted can vary.
- Named users: This method bases pricing on the actual individual users of the system, even if they only log in sporadically. Users may not use each other's logins (this is a no-no regardless of pricing structure, for good laboratory practice and other regulatory reasons).
- Concurrent users: This bases pricing on the maximum number of users who will be logged in at any given time. You can define an unlimited number of named users in the system, each with their own login credentials. However, only the number of concurrent users specified in the license or subscription may be logged in at any one time. For example, you may have 10 staff, but due to work processes, shifts, etc., only up to six might ever be logged in simultaneously. Whereas this would require a named user license for 10, it would only require a concurrent user license for six.
- Unlimited users: In the case of very large labs (typically 30 to 50 and up), the license or subscription may simply be a flat fee that allows any number of users.
The line items in the estimate or SOW should reflect these nuances, as well as whether the listed costs are monthly or annual (for subscription services), hourly (typically for support and training), or a fixed one-time cost. Additionally, be cautious with fixed costs, as they typically represent one of two possible scenarios:
- Final fixed cost: In this case, the cost has been figured by the vendor so as to cover their worst-case hourly labor total. If a line item (e.g., an interface) is not "worst case," then you are overpaying.
- "Expandable" fixed cost: This is as bad as final fixed cost, and maybe even worse because it's almost a case of "bait-and-switch," popping up as a surprise. The initial "fixed cost" number is low, and additional hourly services are needed to actually deliver the item. This will have been provided for somewhere in the small print.
The bottom line is that everything in a laboratory informatics solution is really either licensing or hourly services. Just be careful if they are portrayed as anything else.
It is important to be clear which category each line item falls under when figuring costs: up-front (due upon signing), annual, or ongoing (e.g., SaaS subscription). It is useful to clearly lay out each and compute initial costs, as well as first-year and subsequent years' costings. For example, your initial obligation may be as little as your first year's subscription plus the first 40 hours of services. Different vendors have different policies, however, and you may be required to pay for your first full year's subscription and all services, or some other combination. Normally, though, any instrument interface or other service charges aren't due until the they are implemented, which may be a few weeks or even a month down the road. This may depend on your budget, complexity of the SOW, and urgency. Your first year's expenses will include everything, including initial license fees; all setup and training; any interfaces and additional configurations or customization; and first annual MSW. (If this isn't included in the SaaS subscription, then it usually commences on full system delivery). Afterwards, your subscription and MSW will be the only ongoing expenses (included as one in this example), unless you choose to have additional interfaces or other services performed at any time.
If you've ever worked through a system implementation process with a vendor, it was hopefully a smooth process. However, there are plenty of horror stories out there, highlighting the need of the laboratory to discuss in detail how a potential vendor will handle installation, validation, and training for the informatics solution. Does the vendor truly understand the industry and your needs? Does the vendor assign a project manager who will work with you from planning to go-live and beyond? Can they offer you references of other labs who have gone through implementation so you can compare notes with those labs? How much attention does the potential vendor give to related issues such as data integrity of migrated data? Do they have the means to properly handle your legacy data? And are they able to work with your schedule, even if it means implementing software at off-peak work hours?
As you finally get down to the ultimate decision on which vendor to work with, you may wish to start setting up an implementation checklist as part of your early project planning. Do you receive a help desk account as part of the implementation process, and if so, what information is included? If not, you'll need to keep track of specific details such as business associate agreement (BAA), sales agreement, scope documents, welcome letters, documentation, and approved staff who can utilize the vendor's support. You'll likely need to pass on other configuration details to the vendor, including timezone requirements, DNS and URL requirements, up-time monitors, and administrative account requirements. Finally, you'll want to ensure you and the vendor are on the same page concerning any additional customization, integration, and system validation requirements, ensuring the roll-out period is pain-free and efficient.
2.2.1 Internal and external integrations
In many cases, a vendor's solution will have integration capability built into the software, but occasionally such interfaces are separate from the main software. Today's interfaces are generally built on standardized communication tools, including messaging formats like Health Level 7 (HL7). The HL7 messaging standards are particularly important to laboratory data management because they define how information is packaged and communicated from one party to another. Such standards set the language, structure, and data types required for seamless integration of various systems and instruments. Health Level 7 describes the types of information communicated between such systems in the clinical environment as including "process control and status information for each device or analyzer, [as well as] each specimen, specimen container, and container carrier; information and detailed data related to patients, orders, and results; and information related to specimen flow algorithms and automated decision making."
You may also want your laboratory informatics solution to be able to communicate with other software and databases. This is often done using application programming interfaces (APIs) that depend on web services implementation protocols such as REST and SOAP. These messaging protocols actually allow for the creation of an API that receives communication requests and sends responses between two software systems. A more practical example is wanting your laboratory informatics solution to communicate with an enterprise resource planning (ERP) application. Perhaps the ERP system needs to create sample batches within the informatics solution, and when testing is done, have the results returned to the ERP. APIs and communication protocols make this happen.
2.3 MSW, updates, and other contracted services
The maintenance, support, and warranty (MSW) offered with the vendor's solution is almost as important as the solution itself. The laboratory informatics solution you acquire is more than than the software you operate: it's mission-critical and deserves having a reliable and responsive team with the necessary resources to ensure it remains operational. Downtime can negatively affect both immediate customer satisfaction and your reputation. As such, it's imperative you ask the vendor about the details of its MSW, making sure you understand what is and isn't covered, as well as how much it will cost. Cost-wise, industry norms are anywhere from 15% to 25% of either the license fee or total contract, levied annually to provide this coverage. Alternatively, it may simply be included with your subscription. The MSW will include a specified number of support and maintenance hours or guarantees. The actual warranty should be unlimited for as long as the MSW or subscription is kept current.
Maintenance includes any and all work necessary to keep your system working as designed. It should include updates, patches, or fixes, and most if not all upgrades. (Note, however, a major upgrade to a totally new edition may not be covered, but it may come at a negotiable, significantly lower cost.) The support aspect of MSW generally consists of a specified number of hours dedicated more to helping you with the operation of the system rather than "fixing" anything. Support includes guidance on training, password or login support, and more. Finally, with any professional application you also expect to have a warranty. The warranty should cover anything that doesn't work that otherwise should for the designated period of time. That includes any standard features and functions, as well as any additional ones that were delivered and signed off on, and any other work performed by the vendor or its representatives. However, a typical warranty does not cover anything that was working fine, but upon being manipulated in a way beyond normal operation the functionality ceased. In these cases, you'll probably have to pay to get it fixed.
Beyond the MSW, additional updates and services related to the system may also be required. No matter how well it is pre-configured, any professional laboratory informatics solution will require some amount of standard setup to reflect your particular lab. This includes adding lab branding and demographics for reports and certificates; entering users, their roles, and access permissions; adding and/or modifying tests and workflows; renaming fields; adding or hiding fields; setting up a web portal; and implementing interfaces. Equally indispensable is proper training for both users and administrators. And of course you may later find that you would like additional features or functions. These and other services may prove particularly useful to the laboratory with little in the way of IT and systems expertise. As such, the vendor may provide one or more of the following as a billable service for the laboratory:
- initial implementation meeting (e.g., initial planning, identify delta, set schedule)
- project management
- requirements gathering and documentation
- initial setup
- user and administrator training
- configuration and customization
- interface development and implementation
- custom screen and field development
- custom functionality development
- custom reports and labels
- custom triggers and alerts
- validation or acceptance testing (to a third-party standard or certification, or to agreed manufacturer specs)
2.4 How a user requirements specification fits into the entire process
Merriam-Webster defines a "specification" as "a detailed precise presentation of something or of a plan or proposal for something." In other words, an existing or theoretical product, concept, or idea is presented in detail for a particular audience. In a broad sense, detailing the specifics about a project, concept, or idea to others is just common sense. This applies just as well to the world of software development, where a software requirements specification is essential for preventing the second most commonly cited reason for project failure: poor requirements management.
In fact, the ISO/IEC/IEEE 29148:2018 standard (a conglomeration of what was formerly IEEE 830 and other standards) is in place to help specify "the required processes implemented in the engineering activities that result in requirements for systems and software products" and provide guidelines for how to apply those requirements. The standard describes the characteristics that make up quality software requirement development, including aspects such as:
- correctly describing system behavior;
- effectively removing ambiguity from the language used;
- completely covering the system behavior and features;
- accurately prioritizing and ranking the requirements; and
- unequivocally ensuring the requirements are testable, modifiable, and traceable.
A requirement typically comes in the form of a statement that begins with "the system/user/vendor shall/should ..." and focuses on a provided service, reaction to input, or expected behavior in a given situation. The statement may be abstract (high-level) or specific and detailed to a precise function. The statement may also be of a functional nature, describing functionality or services in detail, or of a non-functional nature, describing the constraints of a given functionality or service and how it's rendered. An example of a functional software requirement could be "the user shall be able to query either all of the initial set of databases or select a subset from it." This statement describes specific functionality the system should have. On the other hand, a non-functional requirement, for example, may state "the system's query tool shall conform to the ABC 123-2014 standard." The statement describes a constraint placed upon the system's query functionality.
This is where a requirements specification shines, not only for the software developer but also for those acquiring the software. A set of development requirements, compiled in the form of a software requirements specification, can serve to strengthen the software development process. For those acquiring the software, a set of user requirements, compiled in the form of a user requirements specification (URS), can be used for the selection and acquisition of software or a service. In the case of the URS, the acquiring business can approach this several ways. The simple way would be to essentially take the vendor at the word in regards to what they say their system can and can't do, agreeing formally to their description and taking responsibility that it will cover all the applicable regulations required by your business. However, this method isn't comprehensive and leaves the business open to not being able to fully meet its goals.
The other method has the URS be specific to your business' needs. The process is more work but leaves less to chance. Developing your own URS isn't always straightforward. Often times, the developed document turns into a mix of "wishlist" requirements from potential and active clients, as well as regulation-mandated requirements. The wishlist items aren't necessarily ignored by developers, but the URS should in fact clearly prioritize requirements as "nice to have" or "essential to system operation," or something in between. Whatever the URS looks like in the end, it's ultimately up to the vendor to be able to demonstrate how the software does and does not meet its requirements.
In the latter half of this guide, you'll be given an opportunity to see an example of a URS for the medical diagnostic and research industries in the form of LIMSpec, an evolving set of software requirements specifications for laboratory informatics systems. Built from requirements found in ASTM E1578-18 Standard Guide for Laboratory Informatics, as well as dozens of other standards and regulations, the LIMSpec examples we provide will demonstrate how a URS is put to use, while also showing you how an informatics system can help you laboratory better meet regulatory requirements.
- Krieg, A.F. (1974). "Chapter 30: Clinical Laboratory Computerization". In Davidsohn, I.; Henry, J.B.. Clinical Diagnosis by Laboratory Methods. W.B. Saunders Company. pp. 1340–58. ISBN 0721629229.
- Flynn, F.V. (1965). "Computer-assisted processing of bio-chemical test data". In Atkins, H.J.B.. Progress in Medical Computing. Blackwell Science Ltd. p. 46. ISBN 0632001801.
- Williams, G.Z. (1964). "The Use of Data Processing and Automation in Clinical Pathology". Military Medicine 129 (6): 502–9. doi:10.1093/milmed/129.6.502.
- Hicks, G.P.; Gieschen, M.M.; Slack, W.V. et al. (1966). "Routine Use of a Small Digital Computer in the Clinical Laboratory". JAMA 196 (11): 973–78. doi:10.1001/jama.1966.03100240107021.
- Straumfjord, J.V.; Spraberry, M.N.; Biggs, H.G.; Noto, T.A. (1967). "Electronic Data Processing System for Clinical Laboratories: A System Used for All Laboratory Sections". American Journal of Clinical Pathology 47 (5_ts): 661–76. doi:10.1093/ajcp/47.5_ts.661.
- Ball, M.J. (1970). "A Survey of Field Experience in Clinical Laboratory Computerization". Laboratory Medicine 1 (11): 25–27, 49–51. doi:10.1093/labmed/1.11.25.
- Jones, R.G.; Johnson, O.A.; Batstone, G. (2014). "Informatics and the Clinical Laboratory". The Clinical Biochemist Reviews 35 (3): 177–92. PMC 25336763. PMID PMC4204239. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=25336763.
- Burton, R. (19 July 2018). "NHS Laboratories Need Data Science". Towards Data Science. https://towardsdatascience.com/nhs-laboratories-need-data-science-c93f7983302c. Retrieved 28 January 2020.
- Cuff, J. (18 June 2018). "Augmenting Pathology Labs with Big Data and Machine Learning". The Next Platform. https://www.nextplatform.com/2018/06/19/augmenting-pathology-labs-with-big-data-and-machine-learning/. Retrieved 28 January 2020.
- Izrailevsky, Y.; Bell, C. (2018). "Cloud Reliability". IEEE Cloud Computing 5 (3): 39–44. doi:10.1109/MCC.2018.032591615.
- Silva, B. (19 December 2019). "IT solutions in the clinical lab". Medical Laboratory Observer. https://www.mlo-online.com/information-technology/article/21117759/it-solutions-in-the-clinical-lab. Retrieved 01 February 2020.
- "Office-based Physician Electronic Health Record Adoption, Health IT Quick-Stat #50". Office of the National Coordinator for Health Information Technology. January 2019. https://dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php. Retrieved 01 February 2020.
- Perrotta, P.L.; Karcher, D.S. (2016). "Validating Laboratory Results in Electronic Health Records: A College of American Pathologists Q-Probes Study". Archives of Pathology and Laboratory Medicine 140 (9): 926–31. doi:10.5858/arpa.2015-0320-CP. PMC PMC5513146. PMID 27575266. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5513146.
- Menzel, J.; Weil, P.; Bittihn, P. et al. (2013). "Requirement analysis for an electronic laboratory notebook for sustainable data management in biomedical research". Studies in Health Technologies and Informatics 192: 1108. doi:10.3233/978-1-61499-289-9-1108. PMID 23920882.
- Guerrero, S.; Dujardin, G.; Cabrera-Andrade, A. et al. (2016). "Analysis and Implementation of an Electronic Laboratory Notebook in a Biomedical Research Institute". PLoS One 11 (8): e0160428. doi:10.1371/journal.pone.0160428. PMC PMC4968837. PMID 27479083. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC4968837.
- Association of Public Health Laboratories (May 2019). "Laboratory Information Systems Project Management: A Guidebook for International Implementations" (PDF). APHL. https://www.aphl.org/aboutAPHL/publications/Documents/GH-2019May-LIS-Guidebook-web.pdf. Retrieved 30 January 2020.
- Kyobe, S.; Musinguzi, H.; Lwanga, N. et al. (2017). "Selecting a Laboratory Information Management System for Biorepositories in Low- and Middle-Income Countries: The H3Africa Experience and Lessons Learned". Biopreservation and Biobanking 15 (2): 111–15. doi:10.1089/bio.2017.0006. PMC PMC5397240. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC5397240.
- List, M.; Schmidt, S.; Trojnar, J. et al. (2014). "Efficient sample tracking with OpenLabFramework". Scientific Reports 4: 4278. doi:10.1038/srep04278. PMC PMC3940979. PMID 24589879. http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pmcentrez&artid=PMC3940979.
- Splitz, A.R.; Balis, U.J.; Friedman, B.A. et al. (20 September 2013). "LIS Functionality Assessment Toolkit". Association for Pathology Informatics. https://www.pathologyinformatics.org/lis_toolkit.php. Retrieved 30 January 2020.
- "Anatomic Pathology". LabWare, Inc. 2020. https://www.labware.com/en/p/Industries/Healthcare/Hospitals-and-Reference-Labs/Anatomic-Pathology. Retrieved 30 January 2020.
- "Sunquest CoPathPlus". Sunquest Information Systems, Inc. 2020. https://www.sunquestinfo.com/software-and-services/copathplus/. Retrieved 30 January 2020.
- "NovoPath: Software Advancing Patient Diagnostics". NovoPath, Inc. 2020. https://www.novopath.com/. Retrieved 30 January 2020.
- "TD HistoCyto Livextens". Technidata SAS. 2020. https://www.technidata-web.com/en-gb/solutions/disciplines/histopathology. Retrieved 30 January 2020.
- "Starlims Forensic LIMS". Abbot. 2020. https://www.informatics.abbott/us/en/industries/forensics. Retrieved 30 January 2020.
- "Forensics & Medical Examiner". LabLynx, Inc. 2020. https://www.lablynx.com/forensics/. Retrieved 30 January 2020.
- "LIMS Solution for Forensics". Thermo Fisher Scientific. 2020. https://www.thermofisher.com/order/catalog/product/INF-12000-FORENSIC#/INF-12000-FORENSIC. Retrieved 30 January 2020.
- "Laboratory Information Management System". Porter Lee Corporation. 2020. http://www.porterlee.com/lims.html. Retrieved 30 January 2020.
- Bralley, J.A.; Lord, R.S. (2008). "Chapter 1: Basic Concepts". Laboratory Evaluations for Integrative and Functional Medicine (2nd ed.). MetaMetrix Institute. pp. 1–16. ISBN 0967394945. https://books.google.com/books?id=CpXVAwgOv7sC&pg=PT11.
- "Public Health". LabWare, Inc. 2020. https://www.labware.com/en/p/Industries/Healthcare/Public-Health. Retrieved 30 January 2020.
- "Public Health Laboratories". Orchard Software Corporation. 2020. https://www.orchardsoft.com/public-health-labs.html. Retrieved 30 January 2020.
- "ApolloLIMS for Public Health Labs". Common Cents Systems, Inc. 2020. https://www.apollolims.com/lab-expertise/public-health/. Retrieved 30 January 2020.
- "Easytox: Features". AP Easy Software Solutions. 2020. https://easytoxicology.com/features/. Retrieved 30 January 2020.
- "Starfruit Toxicology". Data Unlimited International, Inc. 2020. http://www.duii.com/products/starfruit-toxicology/. Retrieved 30 January 2020.
- "Toxicology LIMS". LabLynx, Inc. 2020. https://lablynx.com/toxicology/. Retrieved 30 January 2020.
- "Online Data Management". DTPM, Inc. 2020. https://www.dtpm.com/online-data-management/. Retrieved 30 January 2020.
- "Pain Management & Toxicology Laboratories". Orchard Software Corporation. 2020. https://www.orchardsoft.com/pain-management-toxicology-support.html. Retrieved 30 January 2020.
- "Sunquest Blood Bank". Sunquest Information Systems, Inc. 2020. https://www.sunquestinfo.com/software-and-services/blood-bank-management/. Retrieved 30 January 2020.
- "Sunquest Blood Bank". SCC Soft Computer. 2020. https://www.softcomputer.com/products-services/blood-services/. Retrieved 30 January 2020.
- "Hemasoft: We Care". Hemasoft Software SL. 2020. http://www.hemasoft.com/. Retrieved 30 January 2020.
- "TD BloodBank Livextens". Technidata SAS. 2020. https://www.technidata-web.com/en-gb/solutions/disciplines/blood-banking. Retrieved 30 January 2020.
- "Supporting Services - Data management". Eurofins Scientific. 2020. https://www.eurofins.com/biopharma-services/central-lab/support-services/data-management/. Retrieved 30 January 2020.
- "Starlims Clinical Research LIMS". Abbot. 2020. https://www.informatics.abbott/us/en/industries/clinical-research. Retrieved 30 January 2020.
- "Clinical Research". LabWare, Inc. 2020. https://www.labware.com/en/p/Industries/Biobanking-and-Clinical-Research/Clinical-Research. Retrieved 30 January 2020.
- "ClinTrak Lab". Medpace, Inc. 2020. https://www.medpace.com/capabilities/technology/laboratory-information-management/. Retrieved 30 January 2020.
- "Sunquest Mitogen LIMS". Sunquest Information Systems, Inc. 2020. https://www.sunquestinfo.com/software-and-services/lims/. Retrieved 30 January 2020.
- "Molecular Diagnostics". XIFIN, Inc. 2020. https://www.xifin.com/industry-solutions/laboratory/molecular-diagnostics. Retrieved 30 January 2020.
- "nucleoLIS Ē.finity". XIFIN, Inc. 2020. https://psychesystems.com/enterprise-laboratory-information-software/nucleolis-molecular-lab-testing-software/. Retrieved 30 January 2020.
- Myers, C.; Swadley, M.; Carter, A.B. (2018). "Laboratory Information Systems and Instrument Software Lack Basic Functionality for Molecular Laboratories". Journal of Molecular Diagnostics 20 (5): 591–99. doi:10.1016/j.jmoldx.2018.05.011.
- "Cannabis LIMS for QA and Medical Research". LabLynx, Inc. 2020. https://www.lablynx.com/cannabis/. Retrieved 30 January 2020.
- "Cannabis Testing". Khemia Software, Inc. 2020. https://khemia.com/industries/cannabis-testing/. Retrieved 30 January 2020.
- "LabWare Cannabis". LabWare, Inc. 2020. https://www.labware.cloud/cannabis. Retrieved 30 January 2020.
- "LIMS for Cannabis & Hemp Analysis Labs". Junction Concepts. 2020. https://qbench.net/qbench-lims-cannabis-testing-labs/. Retrieved 30 January 2020.
- "Selecting a LIMS for the Cannabis Industry". LabCompare. CompareNetworks, Inc. 27 November 2018. https://www.labcompare.com/10-Featured-Articles/354722-Selecting-a-LIMS-for-the-Cannabis-Industry/. Retrieved 30 January 2020.
- Sobowale, J. (01 March 2017). "Law firms must manage cybersecurity risks". ABA Journal. American Bar Association. http://www.abajournal.com/magazine/article/managing_cybersecurity_risk/. Retrieved 09 January 2020.
- Watney, C.; Draffin, C. (November 2017). "Addressing new challenges in automotive cybersecurity" (PDF). R Street Policy Study No. 118. R Street Institute. https://www.rstreet.org/wp-content/uploads/2018/04/118-1.pdf. Retrieved 09 January 2020.
- Lewis, J.A. (21 February 2018). "Economic Impact of Cybercrime". Center for Strategic & International Studies. https://www.csis.org/analysis/economic-impact-cybercrime. Retrieved 09 January 2020.
- "BLOG: Cost of Cyber Crime to Small Businesses". Virginia SBDC Blog. Virginia SBDC. 30 May 2017. https://www.virginiasbdc.org/blog-cost-of-cyber-crime-to-small-businesses/. Retrieved 09 January 2020.
- "Hiscox Cyber Readiness Report 2019" (PDF). Hiscox Ltd. April 2019. https://www.hiscox.com/documents/2019-Hiscox-Cyber-Readiness-Report.pdf. Retrieved 09 January 2020.
- Galvin, J. (07 May 2018). "60 Percent of Small Businesses Fold Within 6 Months of a Cyber Attack. Here's How to Protect Yourself". Inc.com. https://www.inc.com/joe-galvin/60-percent-of-small-businesses-fold-within-6-months-of-a-cyber-attack-heres-how-to-protect-yourself.html. Retrieved 09 January 2020.
- Grima, M. (14 November 2019). "Top reasons why WordPress websites get hacked (and how you can stop it)". WP White Security. https://www.wpwhitesecurity.com/why-malicious-hacker-target-wordpress/. Retrieved 09 January 2020.
- Moen, D. (19 April 2016). "What Hackers Do With Compromised WordPress Sites". Wordfence Blog. Defiant, Inc. https://www.wordfence.com/blog/2016/04/hackers-compromised-wordpress-sites/. Retrieved 09 January 2020.
- Talaleve, A. (May 2019). "Website Hacking Statistics (Updated 2019)". WebARX. https://www.webarxsecurity.com/website-hacking-statistics-2018-february/. Retrieved 09 January 2020.
- "security control". Computer Security Resource Center. National Institute of Standards and Technology. 2019. https://csrc.nist.gov/glossary/term/security-control. Retrieved 10 January 2020.
- Ciocoui, C.N.; Dobrea, R.C. (2010). "Chapter 1. The Role of Standardization in Improving the Effectiveness of Integrated Risk Management". In Nota, G.. Advances in Risk Management. IntechOpen. doi:10.5772/9893. ISBN 9789535159469.
- "Data Standardization: A Call to Action" (PDF). JPMorgan Chase & Co. May 2018. https://www.jpmorganchase.com/corporate/news/document/call-to-action.pdf. Retrieved 14 December 2019.
- Wagner, M. (10 October 2019). "7 Software Implementation Challenges and How to Solve Them". WalkMe Blog. WalkMe Ltd. https://blog.walkme.com/7-software-implementation-challenges/. Retrieved 10 January 2020.
- Mura, A. (12 July 2018). "Bullet-Proof Software Implementation Plan: Challenges and Tactics". Userlane Digital Adoption Blog. Userlane GmbH. https://blog.userlane.com/software-implementation-plan/. Retrieved 10 January 2020.
- Sinard, J. (2006). Practical pathology informatics: Demystifying informatics for the practicing anatomic pathologist. Springer Science+Business Media. ISBN 9780387280585. http://www.springer.com/medicine/pathology/book/978-0-387-28057-8.
- MLO Staff (01 August 2012). "Interfacing the LIS". Medical Laboratory Observer. Endeavor Business Media, LLC. https://www.mlo-online.com/home/article/13004490/interfacing-the-lis. Retrieved 10 January 2020.
- Kim, Katherine (July 2005). "Creating Clinical Data Standards in Health Care: Five Case Studies" (PDF). California HealthCare Foundation. Archived from the original on 14 January 2017. https://web.archive.org/web/20170114055221/http://www.kathykim.com/sitebuildercontent/sitebuilderfiles/ClinicalDataStandardsInHealthCare.pdf. Retrieved 10 January 2020.
- Health Level Seven International (2011). "HL7 version 2.7 standard: Chapter 13 - Clinical laboratory automation". http://www.hl7.org/implement/standards/product_brief.cfm?product_id=203. Retrieved 10 January 2020.
- Monus, A. (January 2019). "SOAP vs REST vs JSON comparison [2019"]. Raygun. https://raygun.com/blog/soap-vs-rest-vs-json/. Retrieved 10 January 2020.
- LabVantage Solutions (07 January 2018). "A Quick Guide to LIMS Web Services". LabVantage Solutions, Inc. https://www.labvantage.com/a-quick-guide-to-lims-web-services/. Retrieved 10 January 2020.
- Grand, A.; Geda, E.; Mignone, A. et al. (2019). "One tool to find them all: A case of data integration and querying in a distributed LIMS platform". Database 2019: baz004. doi:10.1093/database/baz004.
- Scavo, F. (08 February 2005). "High Software Maintenance Fees and What to Do About Them". Computer Economics. https://www.computereconomics.com/article.cfm?id=1033. Retrieved 10 January 2020.
- Gordon-Byrne, G. (2014). "Maintenance in the Digital World". IT Performance Improvement. Taylor & Francis, LLC. http://www.ittoday.info/ITPerformanceImprovement/Articles/2014-08GordonByrne2.html. Retrieved 10 January 2020.
- "specification". Merriam-Webster. Merriam-Webster, Inc. https://www.merriam-webster.com/dictionary/specification. Retrieved 09 January 2020.
- Bieg, D.P. (August 2014). "Introduction" (PDF). Requirements Management: A Core Competency for Project and Program Success. Project Management Institute. p. 3. https://www.pmi.org/-/media/pmi/documents/public/pdf/learning/thought-leadership/pulse/requirements-management.pdf. Retrieved 09 January 2020.
- "ISO/IEC/IEEE 29148:2018". International Organization for Standardization. November 2018. https://www.iso.org/standard/72089.html. Retrieved 09 January 2020.
- Seibert, P. (28 July 2011). "How do you write software requirements? What are software requirements? What is a software requirement?". HubTechInsider. https://hubtechinsider.wordpress.com/2011/07/28/how-do-you-write-software-requirements-what-are-software-requirements-what-is-a-software-requirement/. Retrieved 09 January 2020.
- Memon, A. (Spring 2010). "Software Requirements: Descriptions and specifications of a system" (PDF). University of Maryland. https://www.cs.umd.edu/~atif/Teaching/Spring2010/Slides/3.pdf. Retrieved 09 January 2020.
- Schmitt, S. (2018). "User Requirements Specifications–How Difficult Can It Be?". Pharmaceutical Technology 42 (11): 58. http://www.pharmtech.com/user-requirements-specifications-how-difficult-can-it-be-0. Retrieved 09 January 2020.
- Aasem, M.; Ramzan, M.; Jaffar, A. (2010). "Analysis and optimization of software requirements prioritization techniques". Proceedings from the 2010 International Conference on Information and Emerging Technologies: 1–6. doi:10.1109/ICIET.2010.5625687.
- Hirsch, J. (22 November 2013). "10 Steps To Successful Requirements Gathering". Phase2 Technology, LLC. https://www.phase2technology.com/blog/successful-requirements-gathering. Retrieved 09 January 2020.
- Burris, E. (2007). "Requirements Specification". CS451R, University of Missouri–Kansas City. University of Missouri–Kansas City. http://sce2.umkc.edu/BIT/burrise/pl/requirements/. Retrieved 09 January 2020.