Difference between revisions of "Journal:Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results"
Shawndouglas (talk | contribs) (Saving and adding more.) |
Shawndouglas (talk | contribs) (Saving and adding more.) |
||
| Line 45: | Line 45: | ||
According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results—manually recorded consistency—and reached a conclusion after a statistical analysis of the results.<ref name="WangDesign19" /><ref name="RandellStrat18" /><ref name="PalmieriTheDev18">{{cite journal |title=The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform |journal=Clinica Chimica Acta |author=Palmieri, R.; Falbo, R.; Caoowllini, F. et al. |volume=485 |pages=275–81 |year=2018 |doi=10.1016/j.cca.2018.07.001 |pmid=29981288}}</ref><ref name="SediqDesign14">{{cite journal |title=Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile |journal=Annals of Saudi Medicine |author=Sediq, A.M.-E., Abdel-Azeez, A.G.H. |volume=34 |issue=5 |pages=427–32 |year=2014 |doi=10.5144/0256-4947.2014.427 |pmid=25827700 |pmc=PMC6074554}}</ref> The manual verification method is less difficult to operate but has the following limitations: | According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results—manually recorded consistency—and reached a conclusion after a statistical analysis of the results.<ref name="WangDesign19" /><ref name="RandellStrat18" /><ref name="PalmieriTheDev18">{{cite journal |title=The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform |journal=Clinica Chimica Acta |author=Palmieri, R.; Falbo, R.; Caoowllini, F. et al. |volume=485 |pages=275–81 |year=2018 |doi=10.1016/j.cca.2018.07.001 |pmid=29981288}}</ref><ref name="SediqDesign14">{{cite journal |title=Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile |journal=Annals of Saudi Medicine |author=Sediq, A.M.-E., Abdel-Azeez, A.G.H. |volume=34 |issue=5 |pages=427–32 |year=2014 |doi=10.5144/0256-4947.2014.427 |pmid=25827700 |pmc=PMC6074554}}</ref> The manual verification method is less difficult to operate but has the following limitations: | ||
# Massive validation workload: Based on the requirements of WS/T 616-2018 (a China Health Organization recommended standard)<ref name="WS/T616-2018">{{cite web |url=https://www.chinesestandard.net/PDF/English.aspx/WST616-2018 |title=WS/T 616-2018 (WST 616-2018) |work=Chinese Standard |date=20 August 2018}}</ref> for validation of the autoverification of quantitative [[clinical laboratory]] test results, every test and every sample type involved in the autoverification procedure should be tested; the validation time should be no less than three months and/or the number of reports released should be no less than 50,000; and periodic verification should be performed every year for no less than 10 working days and/or for no less than 5,000 reports. The validation workload is large, and it is difficult to rely on manual comparison and recording, which greatly increases the postanalytical workload. | |||
# Reporting risk: During manual verification, personnel are prone to inertia or judgment errors. The lack of a system control mechanism for this kind of validation can generate reporting risks and directly affect clinical diagnosis and treatment.<ref name="WangDesign19" /> | |||
Therefore, there is an urgent need to design a verification method that minimizes the workload and systematically controls risks. We report a rule verification system with a small workload and ease of operation that can be used as a reference for self-built and automatic test auditing for laboratories and manufacturers. | |||
==Methods== | |||
===System design=== | |||
Based on the Clinical and Laboratory Standards Institute's (CLSI) AUTO10 standard<ref name="CLSIAUTO10">{{cite web |url=https://clsi.org/standards/products/automation-and-informatics/documents/auto10/ |title=AUTO10 Autoverification of Clinical Laboratory Test Results, 1st Edition |publisher=Clinical and Laboratory Standards Institute |date=31 October 2006}}</ref> and current review processes, we established an autoverification system including 11 rule categories. Technicians set the rules according to audit requirements and rule categories. Each item can set multiple rules, including limited range check, combined mode judgment, Delta check, sampling time validity judgment, [[Sample (material)|sample]] abnormality judgement (e.g., hemolysis, lipemia), and [[quality control]] check. The autoverification system determines whether the report is abnormal according to the rules. Tests that do not trigger contradiction mode are displayed in green, while failed tests (triggering rules, contradictory modes set by the rules) are displayed in red, and the cause of the contradiction is indicated. If all the tests in the report are green, the [[barcode]] of the report is also green. If any test in the report is red, the report shows a red barcode, which signals a warning in the system. | |||
According to the above steps, the autoverification system displays colors and abnormal prompts after judging the rules in a process called automatic early warning. The automatic warning is only for judgment and is not involved in the decision to issue a report. Based on this, the system automatically sends out a report with a green barcode in a process called automated reporting. Automatic early warning and automatic reporting comprise autoverification. This system is especially useful in the review of complex diagnostic projects (e.g., molecular diagnostics, pathological testing). These projects prompt absurd values from personnel. For some moderately complex projects (e.g., biochemical, blood), the combination of report reviewing, automatic warning, and automated reporting is equivalent to the autoverification system in a large number of literature reports and [[laboratory information system]] (LIS) automatic reports. The autoverification process used by our laboratory is shown in Fig. 1. | |||
Revision as of 19:30, 9 June 2021
| Full article title |
Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results |
|---|---|
| Journal | BMC Medical Informatics and Decision Making |
| Author(s) | Jin, Di; Wang, Dezhi; Wang, Jiajia; Li, Bijuan; Cheng, Yating; Mo, Nanxun; Deng, Xiaoyan; Tao, Ran |
| Author affiliation(s) | Jinan Kingmed Center for Clinical Laboratory, Guangzhou Medical University |
| Primary contact | Email: Online form |
| Year published | 2021 |
| Volume and issue | 21 |
| Article # | 174 |
| DOI | 10.1186/s12911-021-01545-3 |
| ISSN | 1472-6947 |
| Distribution license | Creative Commons Attribution 4.0 International |
| Website | https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-021-01545-3 |
| Download | https://bmcmedinformdecismak.biomedcentral.com/track/pdf/10.1186/s12911-021-01545-3.pdf (PDF) |
 |
This article should be considered a work in progress and incomplete. Consider this article incomplete until this notice is removed. |
Abstract
Background: For laboratory informatics applications, validation of the autoverification function is one of the critical steps to confirm its effectiveness before use. It is crucial to verify whether the programmed algorithm follows the expected logic and produces the expected results. This process has always relied on the assessment of human–machine consistency and is mostly a manually recorded and time-consuming activity with inherent subjectivity and arbitrariness that cannot guarantee a comprehensive, timely, and continuous effectiveness evaluation of the autoverification function. To overcome these inherent limitations, we independently developed and implemented a laboratory information system (LIS)-based validation system for autoverification.
Methods: We developed a correctness verification and integrity validation method (hereinafter referred to as the "new method") in the form of a human–machine dialog. The system records personnel review steps and determines whether the human–machine review results are consistent. Laboratory personnel then analyze the reasons for any inconsistency according to system prompts, add to or modify rules, reverify, and finally improve the accuracy of autoverification.
Results: The validation system was successfully established and implemented. For a dataset consisting of 833 rules for 30 assays, 782 rules (93.87%) were successfully verified in the correctness verification phase, and 51 rules were deleted due to execution errors. In the integrity validation phase, 24 projects were easily verified, while the other six projects still required the additional rules or changes to the rule settings. Taking the Hepatitis B virus test as an example, from the setting of 65 rules to the automated releasing of 3,000 reports, the validation time was reduced from 452 (manual verification) to 275 hours (new method), a reduction in validation time of 177 hours. Furthermore, 94.6% (168/182) of laboratory users believed the new method greatly reduced the workload, effectively controlled the report risk, and felt satisfied. Since 2019, over 3.5 million reports have been automatically reviewed and issued without a single clinical complaint.
Conclusion: To the best of our knowledge, this is the first report to realize autoverification validation as a human–machine interaction. The new method effectively controls the risks of autoverification, shortens time consumption, and improves the efficiency of laboratory verification.
Keywords: autoverification, correctness verification, integrity validation, human–computer interaction, risk management, laboratory information system
Background
Autoverification—the use of automated computer-based rules to initially validate laboratory test results[1]—is a powerful tool for the batch processing of test results and has been widely used in recent years. It has obvious advantages in reducing reporting errors, shortening turnaround time (TAT), and improving audit efficiency.[1][2][3][4][5]
Current status and challenges
Our self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, molecular diagnostics, and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its validation process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875[6] and International Organization for Standardization's ISO 15189:2012 requirement 5.9.2b[7] both require that autoverification systems undergo functional verification before use.
According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results—manually recorded consistency—and reached a conclusion after a statistical analysis of the results.[2][4][8][9] The manual verification method is less difficult to operate but has the following limitations:
- Massive validation workload: Based on the requirements of WS/T 616-2018 (a China Health Organization recommended standard)[10] for validation of the autoverification of quantitative clinical laboratory test results, every test and every sample type involved in the autoverification procedure should be tested; the validation time should be no less than three months and/or the number of reports released should be no less than 50,000; and periodic verification should be performed every year for no less than 10 working days and/or for no less than 5,000 reports. The validation workload is large, and it is difficult to rely on manual comparison and recording, which greatly increases the postanalytical workload.
- Reporting risk: During manual verification, personnel are prone to inertia or judgment errors. The lack of a system control mechanism for this kind of validation can generate reporting risks and directly affect clinical diagnosis and treatment.[2]
Therefore, there is an urgent need to design a verification method that minimizes the workload and systematically controls risks. We report a rule verification system with a small workload and ease of operation that can be used as a reference for self-built and automatic test auditing for laboratories and manufacturers.
Methods
System design
Based on the Clinical and Laboratory Standards Institute's (CLSI) AUTO10 standard[11] and current review processes, we established an autoverification system including 11 rule categories. Technicians set the rules according to audit requirements and rule categories. Each item can set multiple rules, including limited range check, combined mode judgment, Delta check, sampling time validity judgment, sample abnormality judgement (e.g., hemolysis, lipemia), and quality control check. The autoverification system determines whether the report is abnormal according to the rules. Tests that do not trigger contradiction mode are displayed in green, while failed tests (triggering rules, contradictory modes set by the rules) are displayed in red, and the cause of the contradiction is indicated. If all the tests in the report are green, the barcode of the report is also green. If any test in the report is red, the report shows a red barcode, which signals a warning in the system.
According to the above steps, the autoverification system displays colors and abnormal prompts after judging the rules in a process called automatic early warning. The automatic warning is only for judgment and is not involved in the decision to issue a report. Based on this, the system automatically sends out a report with a green barcode in a process called automated reporting. Automatic early warning and automatic reporting comprise autoverification. This system is especially useful in the review of complex diagnostic projects (e.g., molecular diagnostics, pathological testing). These projects prompt absurd values from personnel. For some moderately complex projects (e.g., biochemical, blood), the combination of report reviewing, automatic warning, and automated reporting is equivalent to the autoverification system in a large number of literature reports and laboratory information system (LIS) automatic reports. The autoverification process used by our laboratory is shown in Fig. 1.
References
- ↑ 1.0 1.1 Li, J.; Cheng, B; Ouyang, H. et al. (2018). "Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests". Annals of Clinical Biochemistry 55 (2): 254–63. doi:10.1177/0004563217712291. PMID 28490181.
- ↑ 2.0 2.1 2.2 Wang, Z.; Peng, C.; Kang, H. et al. (2019). "Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory". BMC Medical Informatics and Decision Making 19 (1): 123. doi:10.1186/s12911-019-0848-2. PMC PMC6609390. PMID 31269951. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6609390.
- ↑ Wu, J.; Pan, M.; Ouyang, H. et al. (2018). "Establishing and Evaluating Autoverification Rules with Intelligent Guidelines for Arterial Blood Gas Analysis in a Clinical Laboratory". SLS Technology 23 (6): 631–40. doi:10.1177/2472630318775311. PMID 29787327.
- ↑ 4.0 4.1 Randell, E.W.; Short, G; Lee, N. et al. (2018). "Strategy for 90% autoverification of clinical chemistry and immunoassay test results using six sigma process improvement". Data in Brief 18: 1740-1749. doi:10.1016/j.dib.2018.04.080. PMC PMC5998219. PMID 29904674. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5998219.
- ↑ Randell, E.W.; Short, G; Lee, N. et al. (2018). "Autoverification process improvement by Six Sigma approach: Clinical chemistry & immunoassay". Clinical Biochemistry 55: 42–8. doi:10.1016/j.clinbiochem.2018.03.002. PMID 29518383.
- ↑ College of American Pathologists (21 August 2017). "Laboratory General Checklist - CAP Accreditation Program" (PDF). https://elss.cap.org/elss/ShowProperty?nodePath=/UCMCON/Contribution%20Folders/DctmContent/education/OnlineCourseContent/2017/LAP-TLTM/checklists/cl-gen.pdf.
- ↑ "ISO 15189:2012 Medical laboratories — Requirements for quality and competence". International Organization for Standardization. November 2012. https://www.iso.org/standard/56115.html.
- ↑ Palmieri, R.; Falbo, R.; Caoowllini, F. et al. (2018). "The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform". Clinica Chimica Acta 485: 275–81. doi:10.1016/j.cca.2018.07.001. PMID 29981288.
- ↑ Sediq, A.M.-E., Abdel-Azeez, A.G.H. (2014). "Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile". Annals of Saudi Medicine 34 (5): 427–32. doi:10.5144/0256-4947.2014.427. PMC PMC6074554. PMID 25827700. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6074554.
- ↑ "WS/T 616-2018 (WST 616-2018)". Chinese Standard. 20 August 2018. https://www.chinesestandard.net/PDF/English.aspx/WST616-2018.
- ↑ "AUTO10 Autoverification of Clinical Laboratory Test Results, 1st Edition". Clinical and Laboratory Standards Institute. 31 October 2006. https://clsi.org/standards/products/automation-and-informatics/documents/auto10/.
Notes
This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. For this version, a definition of "autoverification" was added to the introductory sentence of the background.
