Journal:Design and implementation of a clinical laboratory information system in a low-resource setting

From LIMSWiki
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
Full article title Design and implementation of a clinical laboratory information system in a low-resource setting
Journal African Journal of Laboratory Medicine
Author(s) Mtonga, Timothy M.; Choonara, Faheema E.; Espino, Jeremy U.; Kachaje, Chimwemwe; Kapundi, Kenneth;
Mengezi, Takondwa E.; Mumba, Soyapi L.; Douglas, Gerald P.
Author affiliation(s) University of Pittsburgh, Kamuzu Central Hospital, Baobab Health Trust
Primary contact Email: tmm113 at pitt dot edu
Year published 2019
Volume and issue 8(1)
Article # a841
DOI 10.4102/ajlm.v8i1.841
ISSN 2225-2010
Distribution license Creative Commons Attribution 4.0 International
Website https://ajlmonline.org/index.php/ajlm/article/view/841/1391
Download None

Abstract

Background: Reducing laboratory errors presents a significant opportunity for both cost reduction and healthcare quality improvement. This is particularly true in low-resource settings where laboratory errors are further exacerbated by poor infrastructure and shortages in a trained workforce. informatics interventions can be used to address some of the sources of laboratory errors.

Objectives: This article describes the development process for a clinical laboratory information system (LIS) that leverages informatics interventions to address problems in the laboratory testing process at a hospital in a low-resource setting.

Methods: We designed interventions using informatics methods for previously identified problems in the laboratory testing process at a clinical laboratory in a low-resource setting. First, we reviewed a pre-existing LIS functionality assessment toolkit and consulted with laboratory personnel. This provided requirements that were developed into a LIS with interventions designed to address the problems that had been identified. We piloted the LIS at the Kamuzu Central Hospital in Lilongwe, Malawi.

Results: We implemented a series of informatics interventions in the form of a LIS to address sources of laboratory errors and support the entire laboratory testing process. Custom hardware was built to support the ordering of laboratory tests and review of laboratory test results.

Conclusion: Our experience highlights the potential of using informatics interventions to address systemic problems in the laboratory testing process in low-resource settings. Implementing these interventions may require innovation of new hardware to address various contextual issues. We strongly encourage thorough testing of such innovations to reduce the risk of failure when implemented.

Keywords: low-resource setting, laboratory testing, laboratory information system, Malawi, informatics interventions

Introduction

Laboratory testing plays a vital role in clinical decision-making. It is estimated that up to 70% of medical decisions in high-resource healthcare settings are made based on clinical laboratory test results.[1][2] Even though access to clinical laboratory services is comparatively lower in low-resource settings, studies show that clinicians in low-resource settings also make most decisions based on laboratory testing.[3][4] Despite the importance of laboratory test results in clinical decision-making, little effort has been made in low-resource settings to improve the entire laboratory testing process, which starts when the test is first ordered and ends when the results are interpreted and a clinical decision is made.[5]

Laboratory errors include a wide variety of mistakes in the testing process and have no universally accepted definition. We define a laboratory error as any event or mistake that leads to failure to perform a laboratory test, misdiagnosis of a laboratory test, or delayed reporting of laboratory test results. In 2001, it was estimated that laboratory errors accounted for $200 to $400 million in American healthcare expenditures per annum.[6] Since then, the rate of utilization of laboratory services has increased, making the reduction of laboratory errors a significant opportunity for cost reduction and healthcare quality improvement.

Recent studies have tried to categorize errors using phases of the total testing process, which comprises pre-analytical, analytical, and post-analytical phases.[7] The pre-analytical phase covers all activities from when the test is ordered to when the specimen is delivered to the laboratory for testing. The analytical phase covers the activities involved in the actual testing of the specimen, and the post-analytical phase involves the reporting and interpretation of the laboratory result. Among the phases of the total testing process, it has been observed that most laboratory errors happen outside of the analytical phase.[8] An example of an error outside the analytical phase is the mislabeling of a specimen, which could happen during the drawing of a sample in the pre-analytical phase. While error rates vary between health facilities, it is estimated that 32% to 75% of all laboratory errors happen in the pre-analytical phase.[9] Error rates in the analytical phase are estimated in the range of 13% to 32% and in the post-analytical phase in the range of 9% to 31%.[9]

Informatics interventions may be useful in reducing such laboratory errors. Examples of such interventions are computer-aided ordering of laboratory tests, barcode labeling of specimen tubes, and automated reporting of laboratory test results. These interventions are often provided using computer systems that allow physicians to order diagnostic tests, medications, and other procedures, commonly referred to as computerized provider order entry (CPOE)[10], which is often a part of a larger electronic health record system. However, such comprehensive electronic health record systems have low penetration in low-resource settings where the burden of disease is high and laboratory errors are further exacerbated by poor infrastructure, shortages in trained workforce, and informational challenges.[1][11]

Although laboratory information systems (LIS) have been shown to help reduce laboratory errors, little information is available on the implementation of these in low-resource settings. Furthermore, most descriptions of LIS implementations in low-resource settings focus on the analytical phase of the total testing process.[12] In this article, we describe preliminary work in developing a LIS that addresses problems using informatics interventions to support all phases of the total testing process in a low-resource setting with no pre-existing CPOE system.

Methods

Ethical considerations

The work associate with this article followed all ethical standards for research without direct contact with human or animal subjects.

Setting

We implemented a LIS at the Kamuzu Central Hospital (KCH) in Lilongwe, Malawi, between January and March 2015. The Kamuzu Central Hospital is a 750-bed government-operated referral hospital. The laboratory at KCH comprises eight departments: microbiology, parasitology, serology, hematology, molecular biology, blood bank, flow cytometry, and biochemistry. These departments perform laboratory tests for both outpatients and inpatients at the hospital and conducted 242,242 tests between July 1, 2010 and June 30, 2011.[13] The system described in this article was piloted in the outpatient tuberculosis clinic of the hospital and the microbiology department of the laboratory starting on March 31, 2015.

User requirements and system capabilities

Requirements for the LIS were provided by laboratory technicians in the form of user stories. A user story is a succinct way of representing a task that a user will want to perform using an information resource.[14] It includes the role of the user, the task or action, and the benefit, goal, or achievement. An example of a user story in this context is:

As a laboratory technician, I want to know which specimen was drawn first so that I can prioritize it for analysis to reduce the number of non-viable specimens.

We compiled a consolidated list of user stories for each phase of the total testing process. We used that list to define a set of functionality requirements from the laboratory technicians.

To ensure that no core functionality was omitted from the specifications, we leveraged the Laboratory Information System Functionality Assessment Toolkit (LIS-FAT) developed by the Association of Pathology Informatics. This assessment toolkit provides 850 declarative statements that describe the functions that a LIS should possess.[15] An example of a functionality statement from LIS-FAT is:

A laboratory information system should provide intelligent sample labeling that groups samples based on the test to be done and prints them out.

The LIS-FAT was originally intended for use as a LIS evaluation checklist. However, in our implementation, we repurposed it to define capabilities for the proposed system. Furthermore, we recognized that the LIS-FAT was primarily developed for use in a setting with adequate resources, and some aspects of it may not be well suited for a low-resource setting. We therefore assessed the LIS-FAT functionality statements and selected those that focused on direct user needs and were most applicable in a low-resource setting. Special effort was made to ensure that major functional categories of the LIS-FAT were not overlooked. This resulted in a customized LIS-FAT applicable to a low-resource setting, with the declarative statements describing the core requirements for LIS in this setting.

To elucidate the dependencies that could drive the design phase, all functionality statements created in this step were grouped into high, medium, and low priority categories by a group of laboratory management personnel. This helped determine the order in which the functionality would be implemented to ensure that the most important functionality was implemented first.

System design and development

Laboratory information system software can be commercial, open-source, or home-grown. We chose to build on existing open-source LIS software and customize it based on our functional requirements. Before any functionality was implemented, we conducted a design validation study of two open-source LIS software systems to determine the extent to which they implemented the required functionality for the KCH laboratory.[16] These systems were Open Enterprise Laboratory Information System (OpenELIS) and Basic Laboratory Information System (C4G BLIS).[12][17] We assessed and ranked the systems based on the number of functionality requirements that they satisfied for the LIS implementation at KCH. A functionality requirement was considered satisfied if the LIS had a feature that could be used to achieve the goal of that requirement. The choice between the systems was based on the total number of required functions that each of the systems possessed. The system with the most functionality requirements was selected as the foundation upon which the LIS implementation at KCH would be built, and a comprehensive design was made around it to ensure that all functional requirements were met.

We also realized that information systems frequently emphasize the collection and use of data for reporting purposes and streamlining workflow. However, the use of such systems does not necessarily result in improved outcomes. To maximize the value of the LIS, we considered problems identified in the laboratory testing process and described in previous publications.[11][13] Targeted informatics interventions were developed and incorporated into the system’s design to address each of these problems.

Upon completion of the system design, a team of three developers (C.K., K.K., T.M.M.) iteratively implemented and integrated the remaining functionality over eight weeks from mid-January to mid-March 2015. During this time, clinicians and laboratory personnel provided initial feedback, which we used to refine the user interfaces for the new features.

Results

User requirements and system capabilities

A list of 34 user stories was compiled and mapped into functionality statements for the KCH LIS implementation. An additional 41 statements were added from our review of the LIS-FAT statements. The selected LIS-FAT statements had direct user benefits in keeping with the user stories provided by laboratory technicians and were most suitable for LIS implementation in low-resource settings. These 75 statements formed the core functionality requirements for the LIS implementation at KCH.

System design and development

In our design validation study, we independently assessed C4G BLIS and OpenELIS against our set of 75 functionality requirements for the KCH LIS. C4G BLIS met 25 (33%) of the functionality requirements and OpenELIS met 22 (29.3%). A detailed breakdown of the functionality that each system had is presented in Table 1.


Tab1 Mtonga AfricanJofLabMed2019 8-1.jpg

Table 1. Functionality assessment of two open-source laboratory information systems for the Kamuzu Central Hospital laboratory testing process, Malawi, 2015

Following the design validation study, C4G BLIS was selected as the base software for the LIS implementation at KCH. To support the pre-analytical and post-analytical phases, we built clinician-facing laboratory order entry and results reporting software modules. Since functionality was already delineated by phases, each phase could be easily conceptualized as an independent component in a larger system. The decision to adopt a modular approach was further driven by the understanding that modules are easier to maintain than a single monolithic system. With the modular approach, any software module can be easily replaced should a more suitable alternative be identified. For example, if another type of LIS software was chosen to replace the customized C4G BLIS at KCH, it could be easily integrated because of the modular approach. This provides flexibility for future improvements of the system. Each module also addressed specific challenges with targeted informatics interventions. A summary of these is provided in Table 2.


Tab2 Mtonga AfricanJofLabMed2019 8-1.jpg

Table 2. Problems in the laboratory testing process and interventions implemented in the laboratory information system to address them at the Kamuzu Central Hospital, Malawi, 2015

Developing bedside solutions

To facilitate the bedside use of the LIS, we designed custom hardware in the form of a mobile workstation that clinicians could use for ordering laboratory tests and reviewing laboratory results. The mobile workstation is equipped with a low-cost nine-inch tablet computer, a barcode scanner, and a thermal label printer, as shown in Figure 1. To provide complete mobility, the workstation is powered by batteries and does not need to be plugged in to a power outlet during use. The batteries are charged between ward rounds when the mobile workstation is not in use. The mobile workstation also provides room for the clinicians and nurses to easily carry around all medical supplies and consumables required for specimen collection during ward rounds. The provision of space for medical supplies was made as a value addition for the medical personnel and eliminated the need for a separate cart for medical supplies.


Fig1 Mtonga AfricanJofLabMed2019 8-1.jpg

Fig 1. A mobile workstation equipped with a tablet computer, label printer and barcode scanner, Kamuzu Central Hospital, Malawi, 2015

To provide visibility into the status of laboratory tests and results at each stage of the testing process, we also built a dashboard application. This application runs on Raspberry Pi mini-computers connected to 23-inch screens that are mounted in relevant work areas both in the laboratory as well as in the hospital wards. On the screen, we display context-specific work lists such that each user only sees the processes in which they are involved and on which they must act. For example, the dashboard in the microbiology department only shows specimens that require microbiological tests and not any other specimens. A screenshot of the dashboard is provided in Figure 2.


Fig2 Mtonga AfricanJofLabMed2019 8-1.jpg

Fig 2. An example of a nursing station dashboard at the Kamuzu Central Hospital, Malawi, 2015

Figure 3 depicts how the dashboard application, C4G BLIS and the laboratory order entry and results reporting modules are integrated.


Fig3 Mtonga AfricanJofLabMed2019 8-1.jpg

Fig 3. Architecture of laboratory information system implementation at the Kamuzu Central Hospital, Malawi, 2015

Laboratory testing workflow with the laboratory information system

The testing process begins with the clinical decision to order a laboratory test to assess the patient’s condition. To initiate an order, the clinician uses a unique national patient identifier in the form of a barcode to open the patient’s record in the order entry module. Patient identifiers are issued to patients on arrival at the hospital after completion of a one-time patient registration process, which generates an adhesive label to be affixed to a patient’s personal health passport. A health passport is a paper-based continuity of care document kept by the patient. The framework for uniquely identifying patients at KCH has been described in detail elsewhere.[18] Scanning the patient’s barcode opens the patient’s summary in the order entry module displaying the patient’s past test orders and their status, including test results when available. From this page, the clinician can place new test orders and initiate the pre-analytical phase of the total testing process.

In addition to test ordering, the order entry module also maintains and displays an up-to-date catalog of all the tests that are currently being offered at the facility. This intervention helps prevent ordering of tests that are unavailable in the laboratory. Once the test order has been placed through the order entry module, a Health Level 7 message is sent to the C4G BLIS server via a Health Level 7 message router to record the test order. The C4G BLIS server responds by issuing an accession number for the specimen, which is printed on a label in barcode and human readable form together with other test order details during specimen collection. The label is then manually affixed to the specimen container. This accession number is used to track the specimen throughout the testing process. The time at which the specimen label is printed is used in the system as the approximate time when the specimen was collected.

Once the order has been placed, it appears on the relevant nursing station dashboard as well as the laboratory reception dashboard. This was done to provide a visual cue for the laboratory receptionist to anticipate incoming specimens from the ward while at the same time reminding nursing station staff that they need to collect and transport the specimens to the laboratory. Once a specimen has been collected, the dashboard displays its viability based on how long it has been since it was drawn. This serves as a reminder for the nursing staff to bring the specimens to the laboratory on time. The specimen will continue to show on the nursing station dashboard until it has been received at the laboratory reception.

When the specimen arrives at the laboratory, the laboratory receptionist scans the barcode and performs a visual inspection of the specimen container and test order documentation. Based on this, the receptionist determines whether the specimen should be accepted or rejected. For example, a specimen can be rejected if it is no longer viable depending on when it was drawn and the type of test that was ordered. When a specimen is rejected, a notification appears on the nursing station dashboard informing the nursing staff to redraw the specimen. If the specimen is accepted at laboratory reception, it is sent to the appropriate laboratory department for analysis and an entry is added to that department’s dashboard. This is the beginning of the analytical phase. The department dashboard acts as a dynamic worklist informing laboratory technicians of tests that need to be run and results that have yet to be recorded.

Once a specimen has been analyzed, the laboratory technician enters the result using a touchscreen workstation in the laboratory department to complete the analytical phase of the testing process. The nursing staff are notified of the new result through the nursing station dashboard and can now print out the test result and affix it to the patient’s health passport or medical chart for review by the clinician.

Discussion

In this article, we have described the process used to implement a LIS in a low-resource setting, specifically at KCH in Lilongwe, Malawi. We aimed to demonstrate a problem-driven approach that implements individual informatics interventions to support the laboratory testing process in a low-resource setting. We demonstrated this by piloting a system that supported the entire testing process for outpatient tuberculosis screening tests at KCH.

An example of a problem that was addressed in this implementation is the incorrect use of specimen containers for various tests. To address this problem, a picture of the correct container and required specimen volume for each test type was presented to the users during specimen drawing as an electronic job aid. These two interventions addressed the cause of 84% of all untestable specimens at KCH that we reported in a previous article.[13]

While the pilot implementation in the outpatient tuberculosis clinic achieved our main goal, it also limited our ability to measure the impact of the interventions. For instance, sputum is the only specimen type collected in the outpatient tuberculosis clinic at KCH. Therefore, we could not measure the impact of the specimen container decision support on specimen viability. Furthermore, since this ward serves outpatients, the laboratory turnaround time for these tests is not an accurate measure of process efficiency as the review of the results depends on when the patient returns to the hospital, and not when the actual test result itself becomes available. Despite these limitations, there are several points worth noting from this work.

A set of 75 functionality requirements (Online Appendix 1) for LIS in low-resource settings was produced as part of this project. This contains significantly fewer requirements than the 850 statements found in the LIS-FAT document, which are more appropriate for high-resource settings. These 75 requirements can be used by other implementers in low-resource settings to design and evaluate their own LIS.

A further point can be made about the actual design of the implementation. The use of distinct modules separated by the phases of the total testing process offers several benefits. Not only can the modules be more easily maintained, they can also be easily replaced should better alternatives be identified. This offers significant benefits going forward as the implementation is not tightly coupled to any single piece of software.

This implementation further highlights the benefits that open-source software provides with regard to systems implementation in low-resource settings. Software development takes time and is expensive. However, using existing open-source software has the potential to vastly reduce both effort and cost. For instance, design and development took only 10 weeks because existing software was reused for some parts of our system. This could have been significantly longer if everything was built from scratch. Therefore, we recommend that other implementers in low-resource settings find ways of making use of the many open-source products in the health informatics community as this can reduce their effort and expenditure.

Using cheaper alternatives is a common approach to cost reduction. In this implementation, we did this by using tablets that cost $60.00 for the workstations instead of the $650.00 touchscreen clinical workstations that we have used in the past. The tablets presented a significant price reduction and other desirable qualities like the ability to easily be mounted on the mobile workstation. There seemed to be no significant problems with the tablets during the testing phase. However, when we deployed them in the hospital, the tablets often stopped responding and would occasionally power down during use without warning. This led to the loss of all current work that clinical staff had done related to the current patient or specimen and was very disruptive to the workflow.

Our experience with the tablets emphasizes the need for rigor in the testing of new hardware before deployment. In the next iteration, we will address this by replacing the tablets with Raspberry Pi mini-computers and 10.1-inch touchscreen displays. We have comprehensively tested this solution and believe that these new computers will perform more reliably than the tablets and will not significantly inflate the expenditure on the project as they cost less than $200.00 each.[19]

The main limitation of this implementation was our inability to measure the impact that the interventions had on various laboratory key performance indicators such as turnaround times for laboratory tests. In addition, we did not deploy the mobile workstation in the inpatient wards for use during ward rounds. This was mainly due to dependencies that had to be met before deploying the system to inpatient wards at KCH. In the future, we intend to perform field usability evaluations for the mobile workstations and problem impact studies to quantify the effect of the various interventions on laboratory key performance indicators.

Lessons learned from this pilot have informed the continuing scale-up of LIS implementations in Malawi. Those lessons learned include:

  • An electronic laboratory information system that supports the entire testing process can work in the absence of an electronic medical records system.
  • Leveraging individual interventions to solve systemic process challenges can provide benefits to clinical staff that incentivize them to continue using the interventions after implementation.
  • Pre-existing functionality assessment toolkits can provide a foundation for building better tools to improve workflows and processes.

A revised version of this system has now been deployed in three central hospitals and four district hospitals.[20] Revisions have focused on improving operations in the analytical phase by interfacing instruments to the LIS. Future efforts will focus on maximizing the benefits in the pre-analytical and post-analytical phases where most laboratory errors occur.

Acknowledgements

The authors would like to thank the clinical and laboratory staff at Kamuzu Central Hospital (KCH), and the Laboratory Information Management Systems Program Secretariat.

Competing interests

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

Authors’ contributions

T.M.M. drafted the manuscript and assisted in the design and development of all the required software components. F.E.C. coordinated the implementation in the Kamuzu Central Hospital (KCH) laboratory and influenced the design through original contributions and critical feedback. J.U.E. and S.L.M. analysed open-source laboratory information system software for potential use at KCH. C.K. and K.K. assisted in the design and development of all the required software components. T.E.M. coordinated project activities and oversaw implementation. G.P.D. conceived the original idea and designed custom hardware for the implementation. All authors critically revised the manuscript.

Sources of support

This project was supported by the President’s Emergency Plan for AIDS Relief through the Centers for Disease Control and Prevention under the terms of 3U2GGH000729-02S1.

The authors acknowledge the United States Centers for Disease Control and Prevention which funded the development of the LIS at KCH through their cooperative agreements with the Malawi Ministry of Health and Baobab Health Trust.

Data availability statement

All software described in this article has been made available on GitHub. The list of functionality statements for laboratory information systems in low-resource settings has been included as supplemental material (Online Appendix 1). No further data were created or analysed in this study.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the funding agencies.

References

  1. 1.0 1.1 Becich, M.J. (2000). "Information management: moving from test results to clinical information". Clinical Leadership & Management Review 14 (6): 296–300. PMID 11210218. 
  2. Hallworth, M.J. (2011). "The '70% claim': what is the evidence base?". Annals of Clinical Biochemistry 48 (Pt. 6): 487–8. doi:10.1258/acb.2011.011177. PMID 22045648. 
  3. Wilson, M.L.; Fleming, K.A.; Kuti, M.A. et al. (2018). "Access to pathology and laboratory medicine services: A crucial gap". Lancet 391 (10133): 1927–38. doi:10.1016/S0140-6736(18)30458-6. PMID 29550029. 
  4. Moyo, K.;, Porter, C.; Chilima, B. et al. (2015). "Use of laboratory test results in patient management by clinicians in Malawi". African Journal of Laboratory Medicine 4 (1): 277. doi:10.4102/ajlm.v4i1.277. PMC PMC4870597. PMID 27213139. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4870597. 
  5. Price, C.P.; John, A.S.; Christenson, R. et al. (2016). "Leveraging the real value of laboratory medicine with the value proposition". Clinical Chimica Acta 462: 183–6. doi:10.1016/j.cca.2016.09.006. PMID 27649855. 
  6. Bologna, L.; Hardy, G.; Mutter, M. (2001). "Reducing specimen and medication error with handheld technology". Proceedings of the 2001 HIMSS Annual Conference. 
  7. Plebani, M. (2009). "Exploring the iceberg of errors in laboratory medicine". Clinical Chimica Acta 404 (1): 16–23. doi:10.1016/j.cca.2009.03.022. PMID 19302995. 
  8. Plebani, M. (2006). "Errors in clinical laboratories or errors in laboratory medicine?". Clinical Chemistry and Laboratory Medicine 44 (6): 750–9. doi:10.1515/CCLM.2006.123. PMID 16729864. 
  9. 9.0 9.1 The Lewin Group (May 2008). "Laboratory Medicine: A National Status Report" (PDF). Centers for Disease Control and Prevention. https://www.cdc.gov/labbestpractices/pdfs/2007-status-report-laboratory_medicine_-_a_national_status_report_from_the_lewin_group_updated_2008-9.pdf. 
  10. Baron, J.M.; Dighe, A.S. (2011). "Computerized provider order entry in the clinical laboratory". Journal of Pathology Informatics 2: 35. doi:10.4103/2153-3539.83740. PMC PMC3162747. PMID 21886891. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3162747. 
  11. 11.0 11.1 Petrose, L.G.; Fisher, A.M.; Douglas, G.P. et al. (2016). "Assessing Perceived Challenges to Laboratory Testing at a Malawian Referral Hospital". American Journal of Tropical Medicine and Hygiene 94 (6): 1426-32. doi:10.4269/ajtmh.15-0867. PMC PMC3162747. PMID 27022150. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3162747. 
  12. 12.0 12.1 Vempala, S.; Chopra, N.; Rajagopal, A. et al. (2016). "C4G BLIS: Health Care Delivery via Iterative Collaborative Design in Resource-constrained Settings". Proceedings of the Eighth International Conference on Information and Communication Technologies and Development: 21. doi:10.1145/2909609.2909657. 
  13. 13.0 13.1 13.2 Driessen, J.; Limula, H.; Gadabu, O.J. et al. (2015). "Informatics solutions for bridging the gap between clinical and laboratory services in a low-resource setting". African Journal of Laboratory Medicine 4 (1): 176. doi:10.4102/ajlm.v4i1.176. 
  14. Cohn, M. (2004). User Stories Applied: For Agile Software Development (1st ed.). Addison-Wesley Professional. ISBN 9780321205681. 
  15. Tuthill, J.M.; Friedman, B.A.; Balis, U.J. et al. (2014). "The laboratory information system functionality assessment tool: Ensuring optimal software support for your laboratory". Journal of Pathology Informatics 5 (1): 7. doi:10.4103/2153-3539.127819. PMC PMC3986538. PMID 24741466. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3986538. 
  16. Friedman, C.P.; Wyatt, J., ed. (2006). Evaluation Methods in Biomedical Informatics. Springer-Verlag. doi:10.1007/0-387-30677-3. ISBN 9780387306773. 
  17. Monu, R. (27 May 2010). "Design and implementation of a basic laboratory information system for resource-limited settings". SMARTech: Georgia Tech Theses and Dissertations. http://hdl.handle.net/1853/34792. 
  18. Douglas, G.P.; Gadabu, O.J.; Joukes, S. et al. (2010). "Using touchscreen electronic medical record systems to support and monitor national scale-up of antiretroviral therapy in Malawi". PLoS Medicine 7 (8): e1000319. doi:10.1371/journal.pmed.1000319. PMC PMC2919419. PMID 20711476. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2919419. 
  19. Mtonga, T.; Abaye, M.; Rosko, S.C.; Douglas, G.P. (2018). "A comparative usability study of two touchscreen clinical workstations for use in low resource settings". Journal of Health Informatics in Africa 5 (2): 78–84. doi:10.12856/JHIA-2018-v5-i2-209. 
  20. Baobab Health. "Baobab Health Electronic Medical Records Systems Foot Print". http://www.baobabhealth.org/#work. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added. The article mentions "Online Appendix 1," but it could not be found on the journal website. Neither is it clear which GitHub repository applies to their developed software, which the authors claim is on GitHub.