Journal:The impact of electronic health record (EHR) interoperability on immunization information system (IIS) data quality

Full article title	The impact of electronic health record (EHR) interoperability on immunization information system (IIS) data quality
Journal	Online Journal of Public Health Informatics
Author(s)	Woinarowicz, Mary A.; Howell, Molly
Author affiliation(s)	North Dakota Department of Health, Division of Disease Control
Primary contact	Email: mary dot woinarowicz at nd dot gov
Year published	2016
Volume and issue	8(2)
Page(s)	e184
DOI	10.5210/ojphi.v8i2.6380
ISSN	1947-2579
Distribution license	Custom open-access
Website	http://ojphi.org/ojs/index.php/ojphi/article/view/6380
Download	http://journals.uic.edu/ojs/index.php/ojphi/article/download/6380/5638 (PDF)

Abstract

Objectives: To evaluate the impact of electronic health record (EHR) interoperability on the quality of immunization data in the North Dakota Immunization Information System (NDIIS).

Methods: NDIIS "doses administered" data was evaluated for completeness of the patient and dose-level core data elements for records that belong to interoperable and non-interoperable providers. Data was compared at three months prior to EHR interoperability enhancement to data at three, six, nine and 12 months post-enhancement following the interoperability go live date. Doses administered per month and by age group, timeliness of vaccine entry and the number of duplicate clients added to the NDIIS was also compared, in addition to immunization rates for children 19–35 months of age and adolescents 11–18 years of age.

Results: Doses administered by both interoperable and non-interoperable providers remained fairly consistent from pre-enhancement through 12 months post-enhancement. Comparing immunization rates for infants and adolescents, interoperable providers had higher rates both pre- and post-enhancement than non-interoperable providers for all vaccines and vaccine series assessed. The overall percentage of doses entered into the NDIIS within one month of administration varied slightly between interoperable and non-interoperable providers; however, there were significant changes between the percentage of doses entered within one day and within one week with the percentage entered within one day increasing and within one week decreasing with interoperability. The number of duplicate client records created by interoperable providers increased from 94 duplicates pre-enhancement to 10,552 at 12 months post-enhancement, while the duplicates from non-interoperable providers only increased from 300 to 637 over the same period. Of the 40 core data elements in the NDIIS, there was some difference in completeness between the interoperable versus non-interoperable providers. Only middle name, sex, county, phone number, mother’s maiden name, vaccine manufacturer, lot number and expiration date were significantly (>=5%) different between the two provider groups.

Conclusions: Interoperability with provider EHRs has had an impact on NDIIS data quality. Timeliness of data entry has improved and overall doses administered have remained fairly consistent, as have the immunization rates for the providers assessed. There are more technical and non-technical interventions that will need to be accomplished by NDIIS staff and vendor to help reduce the negative impact of duplicate record creation, as well as data completeness.

Keywords: Immunization information systems, electronic health records, meaningful use, HL7, data quality

Introduction

Immunization information systems (IISs) are confidential, population-based systems that record immunization administration data from participating providers, provide consolidated immunization histories at the point of care and provide aggregate data on vaccinations for use in surveillance to increase immunization rates and reduce vaccine-preventable disease. In 1995, the Centers for Disease Control and Prevention (CDC) compiled a list of required and optional core data elements for IIS. The National Vaccine Advisory Committee (NVAC) reviewed and updated the IIS core data elements as part of their Initiative on Immunization Registries. The CDC incorporated the NVAC recommendations and again updated the IIS core data elements in 2012 to correspond with the IIS functional standards and IIS Strategic Plan for 2013-2017. The purpose of the core data elements is to help standardize the capture of data in the IIS and to facilitate the consistent exchange of data between the IIS, electronic health record (EHR) systems and other IIS.

When the American Recovery and Reinvestment Act (ARRA) was enacted in 2009, one of its goals was to promote the adoption and increase the “meaningful use” of EHRs. The Centers for Medicare and Medicaid Services (CMS), in coordination with the Office of the National Coordinator for Health Information Technology (ONC), have set different criteria that must be met by participating providers in order to receive an incentive payment. One of the public health reporting criteria included in all three stages of meaningful use is the electronic exchange of data between an EHR and an IIS. This electronic exchange of data is referred to as interoperability. According to the Healthcare Information and Management Systems Society (HIMSS), “interoperability is the ability of different information technology systems … to … exchange data, and use the information that has been exchanged.”

In 2010, the North Dakota Department of Health (NDDoH) received ARRA funding to establish real-time, bidirectional interoperability connections between the North Dakota Immunization Information System (NDIIS) and the EHRs of the state’s providers responsible for the highest volume of childhood (< 6 years) immunizations. Prior to 2010, the NDIIS was not interoperable with any providers. All providers entered immunizations directly into the NDIIS, in addition to entry into their EHR. As of August 31, 2013, the NDIIS was interoperable with 186 individual provider practices that represented more than half of all doses administered to children 18 and younger in the NDIIS.

The quality of data in an IIS is vital to its ability to determine a patient’s immunization status, calculate immunizations due, assess immunization coverage or generate reminder and recall notices. Without timely and accurate data entered into the NDIIS, it cannot support its basic functionality or provide a benefit to its users or the people in its jurisdiction. The NDIIS has historically maintained high data quality and provider and patient participation by having a number of data quality checks in place that help ensure a high level of data quality. However, these checks are only visible to providers manually entering data directly into the NDIIS. Providers submitting data electronically (interoperable) do not see these same validation checks or warnings if data is not complete in their EHRs.

Other IIS have evaluated the completeness of immunization records in their IIS by comparing the IIS immunization records to the records in provider offices and other facilities in order to evaluate the impact of EHR interoperability. They have found that their IIS records have had more complete immunization information since it is a consolidated record and not just record of immunizations administered at one provider practice. North Dakota requires reporting of childhood immunizations to the IIS. According to the 2013 IIS Annual Report, the NDIIS has 96.8% of North Dakota children <6 years of age, 83.4% of adolescents 11-17 years of age and 77.9% of adults 19 years of age and older participating in the NDIIS. Since the NDIIS already has excellent immunization record completeness, the NDDoH wanted to examine other areas of data quality related to the immunization record.

The objective of this study is to evaluate the impact of EHR interoperability on NDIIS data quality by comparing data entry, timeliness and data completeness, as well as, immunization rates for records belonging to providers submitting electronic data (i.e., interoperable) to those for providers manually entering data directly into the NDIIS (i.e., non-interoperable).

Methods

In the NDIIS, a “provider” refers to a clinic or immunization practice, not to a specific individual practitioner. A client record in the NDIIS is assigned to a provider practice if they were the last provider to enter a non-influenza vaccine in that client’s record. Data for all clients in the NDIIS with doses administered between August 1, 2011 and August 31, 2014 was extracted from the NDIIS for evaluation. Extracted data elements included client first, middle and last name; birthday; sex; birth state/country; race; ethnicity; address; phone number; mother’s first, middle and last name; vaccine type; vaccination date (dose date); lot number and vaccine-level VFC eligibility IIS core data elements.

All data extracted was separated into one of two groups based on assigned provider: records belonging to interoperable providers (IPs) and non-interoperable providers (NIPs). An IP is defined as a provider whose EHR submits data to the NDIIS electronically (i.e., without human intervention) using messaging standards set by Health Level 7 (HL7) International and as defined by the HL7 implementation guide for immunization messaging. A NIP is defined as a provider who manually enters data directly into the NDIIS user interface in addition to documenting in their EHR or paper records. They do not submit data to the NDIIS electronically. Pre-enhancement (before interoperability) data included doses administered between August 1 and October 31, 2011, prior to the first health system EHR connection to the NDIIS. Post-enhancement (after interoperability) data included doses administered between September 1, 2013 and August 31, 2014, three through 12 months after the final provider EHRs were connected to the NDIIS.

The overall data entry analysis evaluated the total number of doses administered by IPs and NIPs per month and by age group for clients five years of age and younger, six to 10 years, 11-18 years, 19-59 years and 60 years and older. Doses for each month were summed for three months pre-enhancement and three, six, nine and 12 months post-enhancement. The data entry analysis also evaluated the number of duplicate client records added by IPs and NIPs at three months pre-enhancement and three, six, nine and 12 months post-enhancement, as well as, the timeliness of all doses entered into the NDIIS less than one day after the dose administration date, within one week (i.e., one to seven days) of administration and within one month (i.e., 30 days) for the three months pre-enhancement and three, six, nine and 12 months post-enhancement.

Completeness of IIS core data elements for client and dose records that belong to IPs and were entered into the NDIIS three months pre-enhancement and at three, six, nine and twelve months post-enhancement were calculated and compared to the completeness of records for NIPs.

Immunization rates for children 19–35 months of age for the 4:3:1:3:3:1:4 (4 diphtheria, tetanus and acellular pertussis [DTaP], 3 polio, 1 measles, mumps and rubella [MMR], 3 hepatitis B, 3 haemophilus influenzae type B [Hib], 1 varicella and 4 pneumococcal conjugate [PCV]) vaccine series and for adolescents 11–18 years of age for one dose of tetanus, diphtheria and acellular pertussis (Tdap) and meningococcal conjugate (MCV4) vaccines, two doses of varicella and three doses of human papillomavirus (HPV) vaccines were calculated and compared for clients that belong to IPs versus NIPs for three months pre-enhancement and three, six, nine and 12 months post-enhancement. SAS 9.3, (SAS Institute, Inc., Cary, North Carolina) and Microsoft Excel 2010 (Microsoft Corp., Redmond, Washington) were used to analyze all NDIIS data.

Results

Doses administered to the 6–10 year, 11–18 year, 19–59 year and 60 years and older age groups by both IPs and NIPs remained fairly consistent from pre-enhancement to three, six, nine and 12 months post-enhancement. NIPs had a higher percentage of doses administered for all age groups with the exception of the 0–5 year age group. Pre-enhancement NIPs entered 53% of the doses in the NDIIS compared to 47% for IPs. Post-enhancement, IPs entered 53% of doses at three months, 55% at six months, 56% at nine and 12 months (Figure 1).

Figure 1. Comparison of percentage of doses administered to clients in the NDIIS by interoperable and non-interoperable providers. Data obtained from the NDIIS for doses added between August 1, 2011 and August 31, 2014.

Looking at the timeliness of data entry, the overall percentage of doses entered into the NDIIS within one month of administration varied slightly (less than 2%) over the three months preenhancement and 4.6% over the 12 months post-enhancement. There were, however, some significant changes between the percentage of doses entered within one day and within one week. The percentage of doses entered within one day increased from 54.6% at the start of the pre-enhancement period to 79.5% at the end of the twelve months post-enhancement, while the doses entered within one week of administration decreased from 38.6% to 14.9% over the same time period (Figure 2).

Figure 2. Overall timeliness of dose data entry by all providers in the NDIIS. Data obtained from the NDIIS for dose records added each month between August 1, 2011 and August 31, 2014.

Pre-enhancement, the number of duplicate client records created in the NDIIS by NIPs was more than three times the number created by IPs (300 vs. 94). Post-enhancement, the number of duplicate client records created by NIPs increased from 377 at three months to 637 at 12 months, whereas the number of duplicate client records created by IPs increased to 1,695 at three months post enhancement and continued to increase with 9,883 duplicates created at nine months and 10,552 at 12 months (Figure 3).

Figure 3. Comparison of the number of duplicate client records added to the NDIIS by interoperable and non-interoperable providers. Data obtained from the NDIIS for duplicate client records added to the NDIIS between August 1, 2011 and August 31, 2014.

Of the 40 core data elements in the NDIIS, completeness for only three elements (middle name, county and phone number) were significantly (>=5%) higher for NIPs compared to IPs pre-enhancement. Completeness for only two data elements (sex and mother’s maiden name) were significantly higher for IPs pre-enhancement. For all other data elements, there was no significant difference in completeness between the interoperable versus NIPs. By 12 months post-enhancement, there were five data elements (middle name, phone number, vaccine manufacturer, lot number and expiration date) significantly higher for NIPs, and completeness for mother’s maiden name was significantly higher for IPs. Additionally, completeness for sex showed almost no difference between IPs and NIPs post-enhancement, and the percentage difference in completeness for county was only 2% post-enhancement versus 5% pre-enhancement (Table 1).

Table 1. Comparison of the Completeness of IIS Core Data Elements in the NDIIS for Interoperable and Non-Interoperable Providers. Data obtained from the NDIIS for records added between August 1, 2011 and August 31, 2014. IIS Core Data Element Pre-Enhancement IPs Pre-Enhancement NIPs Difference Post-Enhancement IPs Post-Enhancement NIPs Difference MIDDLE NAME 72.4% 84.1% -11.7% 64.1% 80.3% -16.2% SEX 99.9% 94.9% 5.0% 96.9% 96.8% 0.1% MOTHER'S MAIDEN NAME 46.8% 40.0% 6.8% 55.8% 44.1% 11.7% COUNTY 93.8% 98.4% -4.6% 96.4% 98.7% -2.3% PHONE NUMBER 73.5% 81.2% -7.7% 69.3% 82.6% -13.3% VACCINE MANUFACTURER 71.0% 70.4% 0.6% 68.0% 84.4% -16.4% LOT NUMBER 70.1% 69.0% 1.1% 64.7% 83.0% -18.3% EXPIRATION DATE 70.1% 69.0% 1.1% 64.7% 83.0% -18.3%

Comparing immunization rates for infants and adolescents, IPs had higher rates both pre- and post-enhancement than NIPs for all vaccines and vaccine series assessed. The rate for the infant 4:3:1:3:3:1:4 series was 82% for IPs pre-enhancement and 77% for NIPs, for a difference of only 5% between the two groups. By 12 months post-enhancement, there was a difference of 12% with the IPs still having a higher percentage of their infants up-to-date with the complete series (70%) when compared to the infants of NIPs (58%) (Table 2).

Table 2. Comparison of the Percentage of North Dakota Infants (19 – 35 months) at Interoperable and Non-Interoperable Providers Up-To-Date with Recommended Immunizations (4:3:1:3:3:1:4 series). Data obtained from the NDIIS for client records as of October 31, 2011, November 30, 2013, February 28, 2014, May 31, 2014 and August 31, 2014. Interoperability Status Time Period (months) IPs NIPs Difference Pre-Enhancement 3 82% 77% 5% Post-Enhancement 3 78% 72% 6% 6 72% 66% 6% 9 72% 59% 12% 12 70% 58% 13%

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.