Difference between revisions of "Journal:Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results"
Shawndouglas (talk | contribs) (Saving and adding more.) |
Shawndouglas (talk | contribs) (Saving and adding more.) |
||
| Line 42: | Line 42: | ||
===Current status and challenges=== | ===Current status and challenges=== | ||
Our self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, [[molecular diagnostics]], and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its [[Software verification and validation|validation]] process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875<ref name="CAPLab17">{{cite web |url=https://elss.cap.org/elss/ShowProperty?nodePath=/UCMCON/Contribution%20Folders/DctmContent/education/OnlineCourseContent/2017/LAP-TLTM/checklists/cl-gen.pdf |format=PDF |title=Laboratory General Checklist - CAP Accreditation Program |author=College of American Pathologists |date=21 August 2017}}</ref> and International Organization for Standardization's [[ISO 15189|ISO 15189:2012]] requirement 5.9.2b<ref name="ISO15189:2012">{{cite web |url=https://www.iso.org/standard/56115.html |title=ISO 15189:2012 Medical laboratories — Requirements for quality and competence |publisher=International Organization for Standardization |date=November 2012}}</ref> both require that autoverification systems undergo functional verification before use. | Our self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, [[molecular diagnostics]], and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its [[Software verification and validation|validation]] process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875<ref name="CAPLab17">{{cite web |url=https://elss.cap.org/elss/ShowProperty?nodePath=/UCMCON/Contribution%20Folders/DctmContent/education/OnlineCourseContent/2017/LAP-TLTM/checklists/cl-gen.pdf |format=PDF |title=Laboratory General Checklist - CAP Accreditation Program |author=College of American Pathologists |date=21 August 2017}}</ref> and International Organization for Standardization's [[ISO 15189|ISO 15189:2012]] requirement 5.9.2b<ref name="ISO15189:2012">{{cite web |url=https://www.iso.org/standard/56115.html |title=ISO 15189:2012 Medical laboratories — Requirements for quality and competence |publisher=International Organization for Standardization |date=November 2012}}</ref> both require that autoverification systems undergo functional verification before use. | ||
According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results—manually recorded consistency—and reached a conclusion after a statistical analysis of the results.<ref name="WangDesign19" /><ref name="RandellStrat18" /><ref name="PalmieriTheDev18">{{cite journal |title=The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform |journal=Clinica Chimica Acta |author=Palmieri, R.; Falbo, R.; Caoowllini, F. et al. |volume=485 |pages=275–81 |year=2018 |doi=10.1016/j.cca.2018.07.001 |pmid=29981288}}</ref><ref name="SediqDesign14">{{cite journal |title=Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile |journal=Annals of Saudi Medicine |author=Sediq, A.M.-E., Abdel-Azeez, A.G.H. |volume=34 |issue=5 |pages=427–32 |year=2014 |doi=10.5144/0256-4947.2014.427 |pmid=25827700 |pmc=PMC6074554}}</ref> The manual verification method is less difficult to operate but has the following limitations: | |||
Revision as of 19:14, 9 June 2021
| Full article title |
Development and implementation of an LIS-based validation system for autoverification toward zero defects in the automated reporting of laboratory test results |
|---|---|
| Journal | BMC Medical Informatics and Decision Making |
| Author(s) | Jin, Di; Wang, Dezhi; Wang, Jiajia; Li, Bijuan; Cheng, Yating; Mo, Nanxun; Deng, Xiaoyan; Tao, Ran |
| Author affiliation(s) | Jinan Kingmed Center for Clinical Laboratory, Guangzhou Medical University |
| Primary contact | Email: Online form |
| Year published | 2021 |
| Volume and issue | 21 |
| Article # | 174 |
| DOI | 10.1186/s12911-021-01545-3 |
| ISSN | 1472-6947 |
| Distribution license | Creative Commons Attribution 4.0 International |
| Website | https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-021-01545-3 |
| Download | https://bmcmedinformdecismak.biomedcentral.com/track/pdf/10.1186/s12911-021-01545-3.pdf (PDF) |
 |
This article should be considered a work in progress and incomplete. Consider this article incomplete until this notice is removed. |
Abstract
Background: For laboratory informatics applications, validation of the autoverification function is one of the critical steps to confirm its effectiveness before use. It is crucial to verify whether the programmed algorithm follows the expected logic and produces the expected results. This process has always relied on the assessment of human–machine consistency and is mostly a manually recorded and time-consuming activity with inherent subjectivity and arbitrariness that cannot guarantee a comprehensive, timely, and continuous effectiveness evaluation of the autoverification function. To overcome these inherent limitations, we independently developed and implemented a laboratory information system (LIS)-based validation system for autoverification.
Methods: We developed a correctness verification and integrity validation method (hereinafter referred to as the "new method") in the form of a human–machine dialog. The system records personnel review steps and determines whether the human–machine review results are consistent. Laboratory personnel then analyze the reasons for any inconsistency according to system prompts, add to or modify rules, reverify, and finally improve the accuracy of autoverification.
Results: The validation system was successfully established and implemented. For a dataset consisting of 833 rules for 30 assays, 782 rules (93.87%) were successfully verified in the correctness verification phase, and 51 rules were deleted due to execution errors. In the integrity validation phase, 24 projects were easily verified, while the other six projects still required the additional rules or changes to the rule settings. Taking the Hepatitis B virus test as an example, from the setting of 65 rules to the automated releasing of 3,000 reports, the validation time was reduced from 452 (manual verification) to 275 hours (new method), a reduction in validation time of 177 hours. Furthermore, 94.6% (168/182) of laboratory users believed the new method greatly reduced the workload, effectively controlled the report risk, and felt satisfied. Since 2019, over 3.5 million reports have been automatically reviewed and issued without a single clinical complaint.
Conclusion: To the best of our knowledge, this is the first report to realize autoverification validation as a human–machine interaction. The new method effectively controls the risks of autoverification, shortens time consumption, and improves the efficiency of laboratory verification.
Keywords: autoverification, correctness verification, integrity validation, human–computer interaction, risk management, laboratory information system
Background
Autoverification—the use of automated computer-based rules to initially validate laboratory test results[1]—is a powerful tool for the batch processing of test results and has been widely used in recent years. It has obvious advantages in reducing reporting errors, shortening turnaround time (TAT), and improving audit efficiency.[1][2][3][4][5]
Current status and challenges
Our self-developed autoverification system has been used for six years in many disciplines, such as biochemistry, immunology, hematology, microbiology, molecular diagnostics, and pathology. To date, 25,487 rules have been set. The system judges test results 1.1 million times a day and provides audit recommendations for 250,000 report forms, accounting for 87% of the total number of report forms. Approximately 80,000 reports are automatically generated every day. To ensure the effectiveness and safety of the autoverification system, its validation process is very important. The College of American Pathologists' laboratory accreditation checklist item GEN.43875[6] and International Organization for Standardization's ISO 15189:2012 requirement 5.9.2b[7] both require that autoverification systems undergo functional verification before use.
According to published studies, in laboratories that use autoverification, the majority of laboratories have performed personnel-based and automatic system audits with the same results—manually recorded consistency—and reached a conclusion after a statistical analysis of the results.[2][4][8][9] The manual verification method is less difficult to operate but has the following limitations:
References
- ↑ 1.0 1.1 Li, J.; Cheng, B; Ouyang, H. et al. (2018). "Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests". Annals of Clinical Biochemistry 55 (2): 254–63. doi:10.1177/0004563217712291. PMID 28490181.
- ↑ 2.0 2.1 Wang, Z.; Peng, C.; Kang, H. et al. (2019). "Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory". BMC Medical Informatics and Decision Making 19 (1): 123. doi:10.1186/s12911-019-0848-2. PMC PMC6609390. PMID 31269951. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6609390.
- ↑ Wu, J.; Pan, M.; Ouyang, H. et al. (2018). "Establishing and Evaluating Autoverification Rules with Intelligent Guidelines for Arterial Blood Gas Analysis in a Clinical Laboratory". SLS Technology 23 (6): 631–40. doi:10.1177/2472630318775311. PMID 29787327.
- ↑ 4.0 4.1 Randell, E.W.; Short, G; Lee, N. et al. (2018). "Strategy for 90% autoverification of clinical chemistry and immunoassay test results using six sigma process improvement". Data in Brief 18: 1740-1749. doi:10.1016/j.dib.2018.04.080. PMC PMC5998219. PMID 29904674. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5998219.
- ↑ Randell, E.W.; Short, G; Lee, N. et al. (2018). "Autoverification process improvement by Six Sigma approach: Clinical chemistry & immunoassay". Clinical Biochemistry 55: 42–8. doi:10.1016/j.clinbiochem.2018.03.002. PMID 29518383.
- ↑ College of American Pathologists (21 August 2017). "Laboratory General Checklist - CAP Accreditation Program" (PDF). https://elss.cap.org/elss/ShowProperty?nodePath=/UCMCON/Contribution%20Folders/DctmContent/education/OnlineCourseContent/2017/LAP-TLTM/checklists/cl-gen.pdf.
- ↑ "ISO 15189:2012 Medical laboratories — Requirements for quality and competence". International Organization for Standardization. November 2012. https://www.iso.org/standard/56115.html.
- ↑ Palmieri, R.; Falbo, R.; Caoowllini, F. et al. (2018). "The development of autoverification rules applied to urinalysis performed on the AutionMAX-SediMAX platform". Clinica Chimica Acta 485: 275–81. doi:10.1016/j.cca.2018.07.001. PMID 29981288.
- ↑ Sediq, A.M.-E., Abdel-Azeez, A.G.H. (2014). "Designing an autoverification system in Zagazig University Hospitals Laboratories: Preliminary evaluation on thyroid function profile". Annals of Saudi Medicine 34 (5): 427–32. doi:10.5144/0256-4947.2014.427. PMC PMC6074554. PMID 25827700. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6074554.
Notes
This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. For this version, a definition of "autoverification" was added to the introductory sentence of the background.
