Journal:Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results

Full article title	Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results
Journal	BMC Medical Informatics and Decision Making
Author(s)	Jin, Di; Wang, Dezhi; Wang, Jiajia; Li, Bijuan; Cheng, Yating; Mo, Nanxun; Deng, Xiaoyan; Tao, Ran
Author affiliation(s)	Jinan Kingmed Center for Clinical Laboratory, Guangzhou Medical University
Primary contact	Email: Online form
Year published	2021
Volume and issue	21
Article #	174
DOI	10.1186/s12911-021-01545-3
ISSN	1472-6947
Distribution license	Creative Commons Attribution 4.0 International
Website	https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-021-01545-3
Download	https://bmcmedinformdecismak.biomedcentral.com/track/pdf/10.1186/s12911-021-01545-3.pdf (PDF)

Abstract

Background: For laboratory informatics applications, validation of the autoverification function is one of the critical steps to confirm its effectiveness before use. It is crucial to verify whether the programmed algorithm follows the expected logic and produces the expected results. This process has always relied on the assessment of human–machine consistency and is mostly a manually recorded and time-consuming activity with inherent subjectivity and arbitrariness that cannot guarantee a comprehensive, timely, and continuous effectiveness evaluation of the autoverification function. To overcome these inherent limitations, we independently developed and implemented a laboratory information system (LIS)-based validation system for autoverification.

Methods: We developed a correctness verification and integrity validation method (hereinafter referred to as the "new method") in the form of a human–machine dialog. The system records personnel review steps and determines whether the human–machine review results are consistent. Laboratory personnel then analyze the reasons for any inconsistency according to system prompts, add to or modify rules, reverify, and finally improve the accuracy of autoverification.

Results: The validation system was successfully established and implemented. For a dataset consisting of 833 rules for 30 assays, 782 rules (93.87%) were successfully verified in the correctness verification phase, and 51 rules were deleted due to execution errors. In the integrity validation phase, 24 projects were easily verified, while the other six projects still required the additional rules or changes to the rule settings. Taking the Hepatitis B virus test as an example, from the setting of 65 rules to the automated releasing of 3,000 reports, the validation time was reduced from 452 (manual verification) to 275 hours (new method), a reduction in validation time of 177 hours. Furthermore, 94.6% (168/182) of laboratory users believed the new method greatly reduced the workload, effectively controlled the report risk, and felt satisfied. Since 2019, over 3.5 million reports have been automatically reviewed and issued without a single clinical complaint.

Conclusion: To the best of our knowledge, this is the first report to realize autoverification validation as a human–machine interaction. The new method effectively controls the risks of autoverification, shortens time consumption, and improves the efficiency of laboratory verification.

Keywords: autoverification, correctness verification, integrity validation, human–computer interaction, risk management, laboratory information system

Background

Autoverification—the use of automated computer-based rules to initially validate laboratory test results—is a powerful tool for the batch processing of test results and has been widely used in recent years. It has obvious advantages in reducing reporting errors, shortening turnaround time (TAT), and improving audit efficiency.

Current status and challenges

Our self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, molecular diagnostics, and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its validation process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875 and International Organization for Standardization's ISO 15189:2012 requirement 5.9.2b both require that autoverification systems undergo functional verification before use.

According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results—manually recorded consistency—and reached a conclusion after a statistical analysis of the results. The manual verification method is less difficult to operate but has the following limitations:

1. Massive validation workload: Based on the requirements of WS/T 616-2018 (a China Health Organization recommended standard) for validation of the autoverification of quantitative clinical laboratory test results, every test and every sample type involved in the autoverification procedure should be tested; the validation time should be no less than three months and/or the number of reports released should be no less than 50,000; and periodic verification should be performed every year for no less than 10 working days and/or for no less than 5,000 reports. The validation workload is large, and it is difficult to rely on manual comparison and recording, which greatly increases the postanalytical workload.
2. Reporting risk: During manual verification, personnel are prone to inertia or judgment errors. The lack of a system control mechanism for this kind of validation can generate reporting risks and directly affect clinical diagnosis and treatment.

Therefore, there is an urgent need to design a verification method that minimizes the workload and systematically controls risks. We report a rule verification system with a small workload and ease of operation that can be used as a reference for self-built and automatic test auditing for laboratories and manufacturers.

Methods

System design

Based on the Clinical and Laboratory Standards Institute's (CLSI) AUTO10 standard and current review processes, we established an autoverification system including 11 rule categories. Technicians set the rules according to audit requirements and rule categories. Each item can set multiple rules, including limited range check, combined mode judgment, Delta check, sampling time validity judgment, sample abnormality judgement (e.g., hemolysis, lipemia), and quality control check. The autoverification system determines whether the report is abnormal according to the rules. Tests that do not trigger contradiction mode are displayed in green, while failed tests (triggering rules, contradictory modes set by the rules) are displayed in red, and the cause of the contradiction is indicated. If all the tests in the report are green, the barcode of the report is also green. If any test in the report is red, the report shows a red barcode, which signals a warning in the system.

According to the above steps, the autoverification system displays colors and abnormal prompts after judging the rules in a process called automatic early warning. The automatic warning is only for judgment and is not involved in the decision to issue a report. Based on this, the system automatically sends out a report with a green barcode in a process called automated reporting. Automatic early warning and automatic reporting comprise autoverification. This system is especially useful in the review of complex diagnostic projects (e.g., molecular diagnostics, pathological testing). These projects prompt absurd values from personnel. For some moderately complex projects (e.g., biochemical, blood), the combination of report reviewing, automatic warning, and automated reporting is equivalent to the autoverification system in a large number of literature reports and laboratory information system (LIS) automatic reports. The autoverification process used by our laboratory is shown in Fig. 1.

Figure 1. The autoverification process. Single test results must meet all the warning rules at the same time. The autoverification algorithm can identify those samples requiring manual review that do not meet the laboratory’s criteria for autoverification. If the automated reporting switch is not activated, then reports that pass the automatic warning step are manually issued. If the automated reporting switch is turned on and all tests on the report pass their warning rules, then the system automatically releases the report.

Validation scheme

On the premise that automatic audits are divided into automatic warnings and automatic reports, we divide the verification system into two stages. The first stage is called correctness verification, which verifies that the operation of the rules is consistent with the expectations set by the personnel. If there is a problem, the responsible party may be the program development department. The second stage is called integrity validation. Based on the results from the first stage, this stage verifies whether the set rules include all the elements from the personnel’s audit report. The functional design of the two-stage system is shown in Table 1.

Table 1. Two validation methods designed for two parts of the autoverification system Phase Object Validation method Explanation Inconsistent solutions Automatic warning Warning rules Correctness verification To verify that the warning rules behave as expected and produce the expected outcome If the warning rule setting is wrong, delete and reset the rules Automated reporting Laboratory tests Integrity validation To confirm that the laboratory test results that pass the automatic warning can be reported automatically Add more warning rules according to the laboratory report criteria

Correctness verification

The correctness verification phase confirms whether the execution of a single rule is correct. It is implemented as follows: (1) For newly added rules, the system adds the label "Pending Verification." (2) When the report is reviewed, the system displays the rule judgment result, and a purple color block is displayed to remind the staff to judge whether the execution result of the "Pending Verification" rule is correct. (3) The staff input the judgment result. (4) The system changes the rule status according to the staff input. If it is consistent, the rule label is set to "verified," prompting the personnel to continue to the next stage of verification. If it is inconsistent, the staff are prompted to delete the rule. Figure 2 is a diagram representing this correctness verification process using the example of C-reactive protein (CRP). Figure 3 shows an example of the correctness verification interface.

Figure 2. Schematic diagram of the correctness verification using the example of C-reactive protein (CRP). The CRP test result was 1.8 mg/l and passed quality control. The autoverification system searched all the rules for the CRP and hit two of them, No. 001879 and No. 002009. The No. 001879 rule (verified) checks whether the CRP result has passed the quality control. The No. 002009 rule (pending verification) intercepts the results greater than or equal to 5. Therefore, when No. 002009 is triggered, the warning information of the sample appears purple, indicating that the technician needs to confirm whether the warning result is consistent with the manual judgment. In the correctness verification interface (as shown in the subsequent Fig. 3), the system provides two options, the human–machine judgment is consistent or the system judges incorrectly. The technician can confirm that the rule is performing correctly and change its status to “verified.”

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. For this version, a definition of "autoverification" was added to the introductory sentence of the background.