Journal:Digitalization of calibration data management in the pharmaceutical industry using a multitenant platform
|Full article title||Digitalization of calibration data management in the pharmaceutical industry using a multitenant platform|
|Author(s)||Mustapää, Tuukka; Nummiluikki, Juho; Viitala, Raine|
|Author affiliation(s)||Aalto University, Beamex Oy Ab|
|Primary contact||Email: tuukka dot mustapaa at aalto dot fi|
|Volume and issue||12(15)|
|Distribution license||Creative Commons Attribution 4.0 International|
The global quality infrastructure (QI) has been established and is maintained to ensure the safety of products and services for their users. One of the cornerstones of the QI is metrology, i.e., the science of measurement, as quality management systems (QMS) commonly rely on measurements for evaluating quality. For this reason, calibration procedures and management of the data related to them are of the utmost importance for quality management in the process industry, made a particularly high priority by regulatory authorities. To overcome the relatively low level of digitalization in metrology, machine-interpretable data formats such as digital calibration certificates (DCC) are being developed.
In this paper, we analyze the current calibration processes in the pharmaceutical industry, and the requirements defined for them in the relevant standards and regulations. For digitalizing the calibration-related data exchange, a multitenant cloud- and platform-based method is presented. To test and validate the approach, a proof of concept (POC) implementation of the platform is developed with a focus on ease and cost-efficiency of deployment and use, while also ensuring the preservation of traceability and data integrity. The POC is based on two industrial use cases involving organizations with different roles in the metrology infrastructure. From testing this POC, the presented approach proves to be an efficient method for organizing the calibration data exchange in industrial use.
Keywords: digitalization, quality management, pharmaceutical industry, digital calibration certificate, metrology, traceability, platform
Whenever a product or service is brought to market, it is crucial that the product or service in question is safe for consumers and the environment. For these purposes, a global quality infrastructure (QI) consisting of both public and private organizations has been established. The global cooperation of the QI is coordinated by the International Network on Quality Infrastructure (INetQI), which defines QI as "the system comprising the organizations (public and private) together with the policies, relevant legal and regulatory framework, and practices needed to support and enhance the quality, safety and environmental soundness of goods, services and processes." The cornerstones of QI are metrology, standardization, accreditation, conformity assessment, and market surveillance, which have their own sub-infrastructures, such as metrology infrastructure (MI), which is tied to its own international organizations. Due to the differences in roles of the organizations and regulatory frameworks in the QI, there are organizations and regulatory bodies covering the different parts of the QI on national or regional levels. In turn, due to this division, there are possibilities for overlapping or conflicting regulations, which regulatory bodies ideally aim to avoid.
Currently, as significant efforts are being made to digitalize the QI, the differences in current practices and requirements quickly become apparent, causing challenges in the implementation of digital solutions. For ensuring that the data formats and digital processes meet their respective requirements and are applicable globally in different applications, harmonization is needed, as the current requirements may vary by the domain or region. Otherwise, it could be impossible, or at least highly cost-inefficient, for the organizations participating in the maintenance of the infrastructures, such as calibration laboratories and service providers, to adapt their services to be interoperable with the varying systems used by individual customers.
In this paper, we focus on the industrial part of the MI, in which the determining of the measurement uncertainty of individual instruments and traceability of measurements to the measurement unit system are established through calibrations. As calibrations are a confirmative part of the quality management of processes that are based on measurements, they are not directly profitable operations but instead are necessary for avoiding prohibitive expenses and delays or fulfilling compliance requirements, allowing access to markets. For this reason, estimating the total economic benefits of the digital transformation of the MI for all the involved organizations individually is complex.
The same also applies to other similar processes in different parts of the QI. As the purpose of the QI, and thus also the MI, is mainly to oversee and support the industry in providing reliable and safe products and services to customers, most economical benefits from the digital transformation are formed in the customer end of the infrastructure. In the MI, this includes, e.g., manufacturing companies that can have several thousands of measurement instruments monitoring and controlling production lines and processes. For these companies, a significant part of the value of digitalization comes from the improvements in efficiency and reduced need for manual work through automation, which leads to savings from reduced human resources tied to quality management. In the case of calibrations, where the costs and possible savings from managing instrument data and information are relative to the number of the instruments that need to be regularly calibrated, these benefits will not be as significant for smaller service providers. This means that any investments in the digital transformation become difficult to justify without support and demand from the customers.
A good example of an industry that is reliant on MI is the pharmaceutical industry, where measurements provide the means for controlling the drug manufacturing processes. Thus, the quality and safety of pharmaceutical products are directly dependent on the reliability of process measurements. For this reason, ensuring the trustworthiness of the process instruments is an essential part of quality management in the pharmaceutical industry, which is why the measurement systems and their maintenance and calibration procedures are highly regulated. Ensuring the data integrity of the measurements and any calibration-related data in particular is of the utmost importance for this purpose, which is why detailed records and audit trails for the processes are required. Consequently, any processes including human operations, e.g., related to the documentation and handling of calibration data, will typically require several inspection and approval procedures to prevent the occurrence of human errors.
Since harmonization of data formats is essential for the interoperability and overall efficiency of digital systems, important areas requiring in-depth examination are the established standards, regulations, and guidelines that provide the framework for the current data formats and processes. Thus, a key question for the success of digitalization efforts becomes, "how compatible and applicable are the requirements defined for the different parties in QI in a fully digital environment?"
In general, the success of industry-wide transition in the digital environment relies on organizational capabilities to adapt to and uptake new technologies and systems when necessary. Thus, an important aspect of the digital transformation is ensuring the ease of adaptation, inexpensiveness, and sufficient scalability of the processes and systems so that the requirements of various types and sizes of organizations can be fulfilled. One possible solution for arranging these kinds of systems and services in a scalable manner is the utilization of a common cloud platform. An example of such a concept in the domain of metrology is the European Metrology Cloud project, aiming to advance the digitalization of legal metrology in Europe.
Optimal digitalization of calibration data management requires a thorough understanding of the underlying processes and workflows. In this paper, we investigate the current practices and general requirements for calibration data management as a part of an overall quality management system (QMS) in the pharmaceutical industry. We analyze the significance and feasibility of the requirements for the harmonization of digital processes. The paper presents ways to optimize calibration data management processes, allowing improved efficiency by reduced manual work and removing the possibility of human errors in calibration data management. Thus, traceability and data integrity can be preserved between the organizations in the calibration chain. Furthermore, the proposed approach enables data analysis of the calibration data on a completely different scale compared to the printed A4/paper on glass (e.g., PDF) approach. A proof of concept (POC) of the method was developed by prioritizing the ease of adaptation to digital processes, and as such, the POC used systems already widely used in the industry, e.g., for user and access management. Thus, the set-up and operation of the platform required a less complex infrastructure compared to a fully proprietary infrastructure, as used by Thiel and Bruns et al.
To summarize, we present the following contributions beyond the state-of-the-art:
- Investigation of calibration procedures in the context of digital calibration certificates (DCC), including the core standards and regulations affecting the pharmaceutical industry;
- Analysis of the applicability of the standards and regulations for the implementation of machine-proof data formats and application programming interfaces (APIs), acknowledging that the existing procedures, standards, and regulations have been prepared for human interpretation;
- Optimized digital data management processes for preserving traceability and data integrity in calibration chains;
- A concept for a multitenant platform for establishing ecosystems for collaborating organizations within the metrology infrastructure; and
- A proof of concept (POC) realization of points 3 and 4.
he paper is organized as follows. The next section provides the relevant background on the digitalization of metrology infrastructure, current developments in the use of internet of things (IoT) in manufacturing and quality management, and requirements for calibration data management in the process and pharmaceutical industries. Next, the harmonization of digitalized calibration data management based on the requirements for different organizations is discussed, followed by the proposed approach for the exchange of digital calibration data within the metrology infrastructure. The possibilities of the digital calibration data management, remaining challenges, and proposed research topics are discussed in the penultimate section, followed by conclusions.
Metrology infrastructure as a part of the quality infrastructure
The worldwide MI is built upon standards, mutual trust, and recognition among organizations operating around the world. To make the infrastructure as comprehensive as possible, it consists of organizations with different hierarchical roles. At the top of the hierarchy are the national metrology institutes (NMIs) and designated institutes (DI), which maintain the SI system and metrological standards jointly defined by the International Bureau of Weights and Measures (BIPM). Figure 1 illustrates how the MI is established as a part of the QI. The hierarchy of the MI is often depicted in the form of a triangle or pyramid as the amount of measurement instruments and references increases when moving from the NMIs towards the industrial measurement application. (A few examples of how the MI has been implemented nationally in European countries are given in the work of Nikander et al.)
The measurement standards maintained by the NMIs or DIs are referred to as primary standards, as their purpose is to provide as accurate and precise physical representations of the unit definitions as possible. Between the industrial applications and primary standards are secondary measurement standards, which are maintained by accredited laboratories and calibrated by the NMIs or DIs. Calibrations are essential to maintain the traceability to SI unit definitions, which ensures the comparability and trustworthiness of measurement results given by individual instruments. Depending on the applications, additional levels of references may also be used in the industry. For example, in the pharmaceutical industry, where the number of instruments used for monitoring and controlling the manufacturing processes can be several thousand per manufacturing site, the use of working standards and travelling standards is common as they provide efficiency in terms of time and costs, e.g., by allowing the calibrations of the process instruments to be carried out on-site. Figure 2 shows an example of a calibration chain from the primary standard of an NMI, all the way through to a process instrument of a pharmaceutical manufacturing company.
Due to the complexity of the MI and diversity of the organizations involved, the progress of digitalization has been slow, and in many cases data are still being managed in paper or paper-on-glass formats, requiring human interpretation, e.g., in the case of documenting calibrations in calibration certificates. For field calibrations, digitalized solutions have become more common, as the costs of calibrations are proportional to the efficiency of the processes. However, these solutions tend to be instrument- and system-provider-specific, and due to competition, the willingness of the system providers to collaborate by harmonizing their systems and making them more open has been low. In addition, it is common that larger industrial companies cooperate with several instrument manufacturers and service providers. Consequently, in the worst case for the instrument owner, the data exchange from different organizations has been scattered in several systems using different formats. Because of this, human resources relative to the number of annual calibrations have been tied up in calibration management alone to ensure compliance with quality management regulations, meaning that it has been a significant expense.
As digitalization in the industry and other sectors is progressing, and the collection and use of data is growing rapidly, the topics concerning data quality and trustworthiness become increasingly important, which is driving the research and development efforts in digitalization within the metrology community. The ongoing efforts include the definition and development of digital formats for presenting metrological information, e.g., data models for presenting metrologically relevant data as semantic metadata, or formats for digital calibration certificates. For this work, the exchange of DCCs was tested using the DCC format originally defined by the Physikalisch-Technische Bundesanstalt (PTB) and further developed in EMPIR SmartCom and Gemimeg research projects.
Digitization of the data formats sets new requirements for the development of the infrastructure, as its operation has been based on mutual recognition and trust between organizations. Transition to a digital environment also means that this trust needs to be established digitally to enable its representation in a machine-understandable format. For this purpose, data security and cryptographic solutions, e.g., the use of digital signatures, are relevant.
IoT in pharmaceutical and process industries
The digital transformation of the manufacturing industry is typically referred to as Industry 4.0, based on the significance and potential of digitalization in industrial settings. The key technologies of Industry 4.0 include smart IoT devices and sensors that enable the collection of vast amounts of data from different manufacturing processes and operations. Combined with the technological advancements in computing power, analyzing methods for big data and machine learning, Industry 4.0 enables significant improvements in the efficiency of manufacturing processes.
In addition to automation, digitalization also provides new possibilities to enhance the work where automation is not feasible or needs to be supported by manual operations, i.e., smart working. Smart-working-enabling technologies include, e.g., wearable IoT devices and augmented reality, which aid the interaction between the operators and manufacturing systems. IoT technologies can also be used to improve worker safety in the working environment.
One of the main technologies that is being studied and developed based on Industry-4.0-enabling technologies is cyber-physical systems, i.e., digital twins, in which, e.g., data, simulation models, and predictive analytics are used to form as accurate a digital representation of a physical entity as possible to further improve the possibilities to analyze their performance and behavior.
In terms of quality management, the effects of Industry 4.0 are prominent in the digitalization of methodologies such as total quality management (TQM) and the use of IoT technologies in quality management operations and processes. In some cases, the advancements in IoT technologies have also led to situations where traditional quality management is not in alignment with the requirements of IoT devices.
In the process and pharmaceutical industries specifically, the benefits of IoT and Industry 4.0 can be seen as being the same as in other manufacturing industries. However, in the pharmaceutical industry, digitalization has been slowed due to the interdependence of processes and strict regulations requiring, e.g., comprehensive validation of newly developed systems and software, due to which the investments in Industry 4.0 solutions can be both risky and expensive. In a survey by Reinhardt et al., the main areas of focus for Industry 4.0 adoption in the pharmaceutical industry were optimizing processes, monitoring production performance, maintaining regulatory compliance, and minimizing production downtime.
Requirements for the calibration data management and data formats
Section 7.6 of the ISO 9001 standard for QMSs contains the general requirements for the handling of measuring and monitoring equipment. The standard requires that the equipment used for quantitative measurements must be calibrated periodically, for which the instrument owner must define and document an operational procedure.
For the process and pharmaceutical industry, there are legislative requirements for the federal legislations, e.g., pharmacopeias and good practices (GxP) such as good manufacturing practice (GMP) and good laboratory practices (GLP).
The main standard defining the requirements for calibration data management and certificate formats for accredited laboratories is the ISO/IEC 17025 standard on general requirements for the competence of testing and calibration laboratories. This standard, accompanied by other more general standards such as ISO 9001, is used as the basis for the accreditation of calibration laboratories. The requirements for the contents of a calibration certificate are defined in Section 7.8 of the ISO/IEC 17025 standard.
General guidelines and good practices
In addition to international standards and regional legislations, there also exist several non-obligatory recommendations in the form of good practices or guidelines by different organizations, which have become widely used and unofficially agreed de facto standards for the industry. These de facto standards can be guidelines by NMIs or regional metrology organizations (RMO) on calibration procedures for specific instrument types, guidelines by societies and associations of the experts in the field, or guides and whitepapers by companies, e.g., instrument manufacturers, which are typically based on the relevant legislative requirements.
Mass as an example measurand
The differences that the standards, regulations, and guidelines of the different organizations may have can be pointed out by examining a singular physical quantity and the respective measurement methodologies. Harmonization is beneficial, especially given that differences in the calibration procedures can result in differences, e.g., in uncertainty evaluations. Good examples of the harmonization of practices on a global level are the most recent revisions of the regulations and guidelines for mass and weighing, e.g., non-automatic weighing instruments.
Standards covering the use of non-automatic weighing instruments include the European EN 45501:2015 Metrological aspects of non-automatic weighing instruments and the American NIST Handbook 44 Specifications, Tolerances, and Other Technical Requirements for Weighing and Measuring Devices.
Regional guidelines for the calibration procedures for weighing instruments are published by the RMOs. In the case of non-automatic weighing instruments, these guidelines include, for example, Euramet Calibration Guideline No. 18 Guidelines on the Calibration of Non-Automatic Weighing Instruments, ASTM E898 Standard practice for calibration of non-automatic weighing instruments, SIM MWG7 Guidelines on the calibration of non-automatic weighing instruments, and JJF 1847 Calibration Specifications for Electronic Balances. Harmonization of these guidelines has been based on the other organizations adapting their guidelines to be commensurate with the EURAMET CG-18, which has also become better known in the Asian region, allowing the requirements to become a global de facto standard.
At the industrial level, the requirements for the weighing instruments and other measurement instruments are defined in the regional or national regulations for pharmaceutical products. In the US Pharmacopeia (USP), the requirements for balances are covered in General Chapter 41 “Balances,” with supporting guidance given in General Chapter 1251 “Weighing on an Analytical Balance.” Respectively, in the European Pharmacopoeia, the requirements for balances are covered in General Chapter 2.1.7 “Balances for analytical purposes.” After the latest revision of the European Pharmacopoeia, the requirements are commensurate with the USP, which has been a welcome advancement towards the harmonization of regulations.
Requirements for data integrity
One of the topics where improvements are pursued by investing in digitalization in the pharmaceutical industry is data integrity. The basis for data integrity regulation compliance is typically referred to as the ALCOA+ principle, the definition of which is presented in Table 1.
Regulatory requirements for data integrity in manufacturing are either defined as a part of the GMPs and GLPs or in separate regulations such as the US Food and Drug Administration's (FDA's) Title 21 Code of Federal Regulations Part 11 (21 CFR Part 11) or the Medicines & Healthcare Products Regulatory Agency's (MHRA's) Data Integrity Guidance and Definitions. An example of a de facto standard on data integrity is the ISPE GAMP Guide: Records & Data Integrity, where data integrity requirements defined in the regulations are approached in a more practical way.
Data integrity is also covered, although to a lesser extent, in Section 7.11 of ISO/IEC 17025, where requirements for data management systems used by calibration laboratories for collecting, processing, recording, reporting, storing, or retrieving data are defined. However, as the requirements for the calibration certificates focus on the content that must be included, regardless of whether the certificate is a physical or electronic document, the standard does not specify detailed requirements or methods for securing a digital file.
Harmonization challenges for developing digital solutions
The challenge in developing harmonized and interoperable formats and data exchange between the calibration service providers and the customers is that the requirements for the process and its documentation depend on the roles of the organizations in a calibration chain. For the NMI's and accredited laboratories, the calibration practices are mostly defined in the ISO/IEC 17025 standard and other requirements that are mandated as a part of the accreditation.
However, a significant amount of the calibrations performed in the industry are not done according to the ISO/IEC 17025 standard, mostly due to costs. This includes field calibrations that are performed on-site, as this is far more cost-efficient in comparison to having all the process-controlling instruments calibrated at a separate laboratory. For these calibrations, the requirements are defined in the quality management regulations and legislation of the national and regional authorities. Compared to ISO/IEC 17025, which specifically focuses on calibrations, legislative requirements focus on the QMS as a whole. The main principles regarding the calibrations of the instruments are no different, but the context and reasoning behind them is.
For NMIs and accredited laboratories, calibrations are a service that they provide for their customers, and they are regulated and audited to ensure that their processes are up to the level that is required; this is why the requirements are very specific. On the other hand, in the industry’s end of the calibration chain, the regulations are not as specifically defined in all cases. For example, instead of requiring, e.g., calibration processes to follow a specifically defined procedure, companies are mandated to have written instructions for the processes, meaning that some of the details are left open for companies to define themselves. Due to this openness, there are some guidelines and good practices that have been adopted as de facto standards for calibration procedures. To shortly summarize the differences between the regulations for accredited calibrations and field calibrations, accredited calibrations are focused on an individual instrument, whereas field calibrations consider the instrument a part of a process line.
The effects of these differences in the requirements can be noticed when comparing a typical format for a field calibration certificate to a certificate issued by an accredited laboratory. The field calibration certificates must include information such as the identification of the position or location where the instrument has been used in a particular process line—which is relevant information for the quality assurance of that process line—and a calibration due date based on the calibration interval specified in the manufacturing company’s requirements.
For the calibration data, the main difference is that the calibration data have been presented in a different way. The standards for the accredited laboratories require that the calibration results are presented, including the measurement uncertainties of the measurement points, whereas in the regulations for the industry, it is relatively common that the deviations or measurement errors are required. Another example of these kinds of differences includes requirements for the use of different units in different parts of the world.
In addition to the variances in requirements, some challenges for the digitalization of the processes arise from the way in which the requirements have been defined in the standards and legislation, as the current formats are written for human interpretation. The International Electrotechnical Commission (IEC) has presented a utility model for SMART Standards. The model defines the following levels for the digitization degree of a standard:
- Level 0: Paper format - This format is not suitable for direct automatic processing or usage.
- Level 1: Digital format (e.g., PDF) - This format allows automatic management and display of the document.
- Level 2: Machine-readable format - The structure of the document can be digitized, and certain granular content can be exported (chapters, graphics, definitions, etc.). Content and presentation are separated.
- Level 3: Machine-readable and -executable content - All essential granular information units can be clearly identified, and their reciprocal relationships can be recorded and made available for further processing or partial implementation.
- Level 4: Machine-interpretable content - The information in a standard is linked with implementation and usage information in such a way that it is implemented by machines directly or interpreted and combined with other information sources so that complex actions and decision-making processes take place automatically.
The German Institute for Standardisation (DIN) and German Association for Electrical, Electronic & Information Technologies (DKE) have presented an additional level that goes beyond Level 4 in the IEC model:
- Level 5: Machine-controllable content - The content of a standard can be amended by machines working unassisted and adopted by automated (distributed) decision-making processes. The content adopted in this way is automatically reviewed and published via the publication channels of the standardization organizations.
The model aptly emphasizes the difference among machine-readable, -executable, -interpretable, and -controllable formats. Ideally, for the development of digital calibration management processes, the requirements would have to be machine-interpretable, enabling requirement-specific data and criteria such as allowance limits to be directly interpreted by the management systems as process parameters, making further automation of the processes management possible. Currently, when the requirements are not yet available in such a format, this kind of specific information must be included in the systems through human interpretation, which means that revisions of the requirements will require updating of the systems accordingly.
Materials and methods
Studied cases for the proof of concept
In the proof of concept (POC), the focus of the work was to harmonize the data exchange between the parties of a calibration chain to enable automation of the data management processes in the receiving end, thus eliminating the need for manual work to interpret and input the data into calibration management systems (CMS) or enterprise resource planning (ERP) systems, making the processes more efficient and improving data integrity by reducing the chance of input errors. To achieve these goals, use of a multitenant platform was chosen as the basis for defining the data exchange processes and methods.
The DCC formats used in the POCs were the DCC Extensible Markup Language (XML) schema versions 2.4.0 in Case 1 and 3.0.0-rc2 in Case 2. In both cases, the means of using the DCC schema were agreed upon based on the requirements and needs of the participating organizations representing the calibration customer. The DCCs used in the testing were developed based on examples of existing calibration certificates.
In Case 1, the exchange and handling of DCCs was tested by mimicking a realistic calibration chain from a national measurement standard of an NMI to a working standard of a pharmaceutical manufacturing company. The main objectives for the POC were achieving an efficient and easy-to-use method for managing calibrations and the related data exchange between organizations. The physical quantities for which the harmonized file formats were agreed upon within the consortium were temperature and pressure, as the working standard in question is capable of measuring both quantities. In addition to Aalto University, the organizations involved were:
- VTT MIKES, the Finnish NMI;
- Vaisala Oyj, an instrument manufacturer and a provider of calibration services, with an accredited laboratory;
- Orion Oyj, a pharmaceutical company; and
- Beamex Oy Ab, a field calibration instrument and CMS provider.
Figure 3 shows the POC setting, where different colors highlight the systems and operations of the organizations involved.
In Case 1, a separate sub-schema was developed based on the DCC schema version 2.4.0 to limit the options for presenting the data only to those deemed necessary for the DCCs that were created as examples for testing.
The main principles and objectives of the second studied case were mostly similar to Case 1, as Case 2 focused on the management of the DCCs of temperature measurement instruments and their transfer between the instrument owner and calibration provider. This work was carried out in alignment with the German Gemimeg II research project, in which the DCC-related development work has been led by PTB. In addition to Aalto University, the organizations involved were:
- PTB, the German NMI and developer of the DCC concept and XML schema;
- Testo Industrial Services GmbH, a calibration service provider with an accredited laboratory;
- Boehringer Ingelheim, a pharmaceutical company; and
- Beamex Oy Ab, a field calibration instrument and CMS provider.
Based on the requirements defined in Case 2, PTB developed a DCC good practice for temperature certificates based on the DCC XML schema version 3.0.0-RC2. The DCC good practice for temperature certificates has been further developed in the Gemimeg II project.
Calibration and data management processes
The starting point for the development work was to assess the current practices and processes of the industry partners to observe the general requirements. Although there can be slight variations in the processes of different organizations, e.g., due to their size and the number of calibrated instruments, the main principles remain the same. Based on the processes of the industry partners, generalized overviews of the processes and workflows were then formed and utilized to define the requirements for the DCC-based processes and the POC platform implementation.
In general, calibrations are based on bilateral relationships between a service provider and an instrument owner, i.e., the customer. The process begins with a calibration order or request issued by the instrument owner. If the calibration is to be performed in an external laboratory, the instrument needs to be delivered as well. Upon receipt of the request and instrument, the calibration is performed and results are analyzed by the service provider based on the requirements defined by the customer. If an instrument does not perform within its tolerance, it will need to be adjusted or repaired, after which it must be recalibrated. The calibrations before and after any significant changes made to the calibrated instrument are commonly referred as “as-found” and “as-left” calibrations, respectively. To report the results of the calibration and a statement on the conformity of the instrument as required by the customer, the calibration vendor issues a calibration certificate, sends it to the instrument owner, and returns the instrument. At the receiving end, the calibration certificate is inspected to make sure it is correct and that it meets the requirements that have been defined for the calibration process. Once the certificate is deemed to fulfill the requirements, the data are imported into a CMS.
The calibration process should be considered as a closed-loop process between the instrument owner, i.e., calibration customer, and the calibration service provider. Thus, fully standardizing the communication and ensuring interoperability in wider industrial use requires also a format for the digital calibration request (DCR) in addition to the DCC. However, in the POC, the implementation concentrated only on the use of the DCC as its development has progressed further. Figure 4 shows a simple illustration of the communication between the instrument owner and calibration provider.
It is currently common that several different systems are used for the exchange and handling of the information in different parts of the process. An example of the systems used in the process is illustrated in Figure 5.
Digitalizing calibration data exchange and management
Digitalizing calibration data management will enable the closed-loop process and communication without any manual conversions or loss of data between the systems, making the process robust in terms of data integrity. For the industrial companies with high numbers of annual calibrations, the benefits of the digitalization can be significant merely based on the reductions in the calibration data processing time and need of labor alone.
To digitalize the processes while maintaining the bilateral relations as close to their current state as possible, several organization-specific integrations would be needed between all the cooperators. To simplify data exchange and management and improve the efficiency of development by concentrating the development load, an alternative approach was taken by focusing on the combination of several bilateral relations more as an ecosystem formed by the companies and their subcontractors and service providers. This does not affect the confidentiality of the information by default, but it enables the sharing of anonymized data confidentially with the partners involved, potentially providing shared benefits within the ecosystem.
As the sizes and calibration volumes of calibration service providers vary greatly, there are large differences in the systems used for creating calibration certificates. Currently, it is still common that especially the smaller calibration service providers’ laboratories may still use quite simple solutions for creating their calibration certificates, such as basic text editor tools and templates where the information is filled in manually. For the larger organizations, the investments in more advanced IT systems and automation are easier to justify and more beneficial.
For the organizations using more advanced systems, adapting to a fully digital and machine-readable format is typically a lot less challenging as the data formats that have already been in use can, in most cases, be converted into a different format with ease; for them, it is mostly simply a data-mapping task. For the organizations using more simple tools, the transition can be more challenging.
For data integrity reasons, the calibration certificates need to be signed when data are exchanged in a digital format, even if a signature is not considered to be a mandatory part of a calibration certificate, as is the case with ISO/IEC 17025. The reason behind this is that a digital signature can be validated to prove the authenticity and integrity of a document or file, i.e., ensure that it is authentic, originally created, and sent by the correct person or organization, and that its contents are unaltered.
The sharing of digital documents can be implemented in various ways; they can be sent via email, uploaded to a cloud server, or even delivered on a USB memory stick, to name a few possibilities. The differences in these methods are in the ways in which they can be integrated into the systems used to manage the documents.
On the industrial level, the calibration orders are typically based on a service contract that includes all the calibrations and services that the parties have agreed to be carried out over a defined period, e.g., a calendar year. On the manufacturing industry’s end of the calibration chain, especially in non-accredited field calibrations, the calibration request has a significant role in the calibration processes as the requests or contracts include the instructions for the calibration procedures as mandated by the quality management regulations. For accredited laboratories, this is not a significant issue, as their procedures are audited as a part of the accreditation process, based on which they are also compliant with the quality system regulations. In the field calibration cases, it is mostly up to the manufacturing companies to audit the service providers that they use and ensure that the calibrations are performed correctly.
The correct interpretation of the exchanged data in this kind of setting is heavily dependent on the format in which the data are exchanged and their harmonized use by the parties involved. This is why it is necessary for the receiver to have the means to validate that the received certificate uses the correct format in a correct manner, to make sure that the data can be interpreted and used correctly when they are uploaded to the CMS. Due to the use of digital signatures, validation also includes authenticating the source of the certificate and checking that it is unaltered to ensure its data integrity.
Long-term storing and archiving of calibration certificates is required for investigating any measurement-related quality issues and for auditing purposes. For electronic records and documents, the storage can be implemented with relative ease with conventional cloud storage applications. In addition to maintaining the records over the required period, they also need to be easily accessible and viewable, especially for auditing purposes, which can be implemented in the calibration management systems with relative ease.
The availability of calibration data in a machine-interpretable and processable format is one of the cornerstones for the use of the data in Industry 4.0 applications. For example, the data could be used to analyze the performance of individual devices, device models, or types over time to predict the optimal calibration intervals and select the devices used for controlling and processing the manufacturing processes. These kinds of features have already been available in commercial calibration management systems that have been developed for use in the industry, where field calibrations are common. However, the use of the features is limited to certificate formats used by collaborating instrument and system providers, meaning that a significant amount of the data is not usable without manually importing the data, which is an issue in terms of data integrity.
Aims and requirements for the POC DCC platform
Ideally, the POC DCC platform could be used through APIs, allowing the processes to be highly automated. However, due to the stage of the development work and the variances in the uptake capabilities of organizations, it is likely that there will be a lengthy transition period in which digitalization within the calibration ecosystem will progress step-by-step. This is why some of the features of the platform were developed to be used both manually and with an API. The aims of the POC were to make the different steps of the DCC exchange easy and efficient while also fulfilling the requirements of data integrity applied in the current calibration management procedures. Table 2 represents features that were considered necessary for the testing of the platform based on the defined aims and requirements.
Data transfer utilizing the POC platform
The features that were deemed necessary for the exchange of DCCs were implemented using five application programming interfaces (APIs):
- A signing API for creating digital seals for DCCs;
- An import API for uploading DCCs to the DCC platform;
- A sharing API for granting access rights to DCCs;
- An export API for retrieving DCCs from the DCC platform; and
- A validation API for validating the digital seals of received DCCs.
The data exchange process based on the APIs is presented in Figure 6.
To keep the platform simple, easy-to-use and -deploy, and cost-efficient, existing and commonly used technologies were intended to be utilized. Consequently, the file management and storage of DCCs on the platform was implemented based on Nextcloud and the supporting features such as the UI and access management were developed to enable better compatibility for the requirements of the use cases.
Respectively, the securing of the confidentiality of the data exchange on the platform through user management and authorization was implemented using Azure Active Directory (AD), which is a widely utilized user identity management system also used by the participating organizations.
Features for a calibration provider
The creation tool for DCCs was implemented in two ways. The first one was a conversion tool that was added to an existing in-house-developed calibration certificate creation software that allowed the creation of a DCC in a similar way to the conventional certificate. The second method was by using a simple data input form that was developed for manually creating DCCs.
The signing of the documents was implemented using organizational seals, which are digital signatures that identify the organization authenticating the document instead of the person creating it. In the POCs, the identifications of the persons who performed the calibration were stated in the DCC. The organizational seals are well suited for organizations that already have established personnel authentication methods that can be applied to authorize the rights to seal DCCs. As the rights are managed based on the existing authentication system, the issuing or removing of the signing rights happens automatically as a part of organization’s normal onboarding and offboarding processes. Depending on the system used by the calibration provider, two versions for sealing were implemented:
- A physical signing server using an Intel NUC mini PC, allowing the sealing to be done locally and separated from the cloud platform; and
- A cloud-based sealing tool as a part of the platform, enabling the sealing and sharing of the document with the same user interface.
The seals for the DCCs were created and validated using public and private key pairs designated to each organization issuing the DCCs in the POCs. For the authorization of the keys, a simple prototype PKI was set up at Aalto University, with the root key being securely stored in a safe. The keys issued for the testing were safely either stored in the NUCs or in the cloud depending on which system the organization was using. In either case, the authorization of key use was managed by the corresponding organization. For testing purposes, also separate, universally usable test users were created to enable using the platform without individual AD credentials.
To ensure that the format of the DCC was following the agreed upon structure, the DCC was validated against the DCC schema or good practice schema before a seal was added to the file. To further ensure that the information was interpretable, the files were canonicalized as a part of the seal creation process to remove non-meaningful variances in the documents. The signing service also included a function for validating the seals.
Features for a calibration customer
In the studied cases, the validation of the data format was performed when the calibration provider uploaded the DCC to the DCC platform. For this reason, one key benefit of the platform is that it ensures the interoperability between the platform users as all the data are validated when imported to the platform. Using the signing service, the instrument owner can validate the digital signatures to the received DCCs to ensure that the received data are authorized by the expected organization and that the document has not been manipulated.
After the data integrity of the document is validated, the calibration data need to be imported to the calibration customer’s CMS. For this purpose, an API was developed to enable integration between the DCC platform and receiving CMS. In the receiving CMS, a parser was used to convert the calibration data within the DCC to the data format used in the CMS. This allowed the imported calibration results to be reviewed and approved in the system using existing features to return the instrument for production use.
Alternatively, the human-readable format of the DCCs also allowed the data to be imported manually, similarly to how it is performed with paper or PDF certificates. However, this would also entail data integrity issues caused by the human interaction, making the approach far from ideal in comparison to the automated process.
Testing and validation of the platform-based calibration data exchange and management processes
The operation of the features of the platform, defined in Table 2, was examined based on the sufficiency of the implemented functionalities per feature and the technological readiness of the features. The results of the testing and validation are presented in Table 3. The testing indicated that the platform met its requirements and was mostly working as intended. Some of the features would benefit from further development and refinement. This was expected since the platform was developed as a POC and not a finalized system for industrial use as such.
In the testing and validation, the POC implementation of the DCC-based calibration data exchange and management platform proved to work as intended, providing an efficient way to exchange calibration data digitally, as all the features fulfilled the requirements. As the implementation was only developed as a POC, some areas for additional refinements were also found, which was as expected.
Due to the work being the first collaboration initiative for POCs on the utilization of DCCs for the participating organizations, the scope of the work was kept relatively narrow in terms of the whole calibration management process. Consequently, the results of the POC do not fully cover all parts of the processes that are necessary in the industrial calibration management. Still, the work provided valuable knowledge and a basis for further collaborative development in digitalization.
The chosen approach for implementing the platform was developed based on a relatively simple infrastructure utilizing already widely used services, such as the Azure AD. In addition, the signing and validation service was integrated into the platform. Thus, the number of new and separate systems required for enabling the secure use of DCCs was minimized. This approach is well-suited to the requirements of the industry, especially when smaller organizations are considered, as simplicity and ease of deployment were given relatively high priority. For example, in the testing and deployment of the platform, the cloud-based implementation of the signing service was found to be the preferred option compared to the server being deployed at the corresponding organization. In the tested cases, the data exchange was limited to only DCCs, but the same approach will be applicable also for other similar data formats or documents, e.g., calibration orders or invoices.
However, there are limitations for implementing the infrastructure in such a way for the NMIs and other organizations essential for maintaining national and regional infrastructures, as, e.g., dependencies on services provided by foreign companies may be considered as a threat to the security of the infrastructure. For these situations, approaches based on the proprietary infrastructures, such as the approach used on the Metrology Cloud, could be the better alternative, despite the potential complexity or significantly higher operation costs. Naturally, different types of solutions will be ideal for different organizations; thus, the availability of both simple and robust implementations would be beneficial provided that the compatibility and interoperability of the different systems can be ensured.
Challenges in advancing the digitalization of the metrology infrastructure
Although there are several ongoing efforts for advancing digitalization in metrology, from a broader view, the work is still at an early stage. Effectively, it will take several years before the transition of the whole MI into a digital environment is complete. Until then, the benefits of the digitalization will only be partial. However, even these partial benefits can be significant for the organizations, if closely related organizations working in the same domains are willing to collaborate.
The limitations for the uptake of digital solutions vary between the organizations within the metrology infrastructure based on their roles. The direct benefits of the digitalization are more significant the larger the amount of measurement devices and instruments used by the organization, which of course leads to the manufacturing companies obtaining the most benefits. From their point of view, the main concerns are justifying investments in digital solutions in terms of the stance of the regulation bodies towards the developed technologies. This is why active collaboration between the solution developers and regulators is necessary.
For the smaller organizations within the metrology infrastructure, e.g., calibration laboratories and service providers specialized in a specific domain, the direct benefits of the digitalization may not be significant enough that the investments for new systems could be justified by improvements, e.g., in data management efficiency, alone. For these organizations, the motivation for the digital transition is based on the needs and demands of the customers and their willingness to contribute the additional costs from the required investments.
At the moment, the requirements defined for the QMSs are defined in such a way that there are some flexibilities for implementing the systems digitally. This is mostly because the standards have been developed to be interpreted by humans, and the organizations pursuing compliance with the requirements are obliged to prove that the systems developed based on their interpretation are fulfilling these requirements. This raises a challenge when the QMSs are being digitalized, as this kind of flexibility is not suitable for machine interpretation. Transition to digital systems means that the interoperability of the data formats and harmonization of the data management procedures should be given higher priority in the standardization and regulation work. A good example of how the collaboration of the technology developers and regulation bodies can be established is the participation of the German authorities in the Gemimeg II project, which is a national project where solutions for the use of the DCC are being developed based on the needs of the German industry. Ideally, the future development initiatives and involvement of regulation bodies should be expanded towards a global scale.
Fortunately, the slow progress of the digitalization of the metrology infrastructure also means that many of these challenges have been already solved in other applications, so there is already plenty of expertise and tools available. What is needed is that the experts of the areas of metrology and information technology come together to share this expertise and discuss the metrology-specific challenges that remain to be solved. In other applications, this kind of work has been made possible by foundations such as the Open Platform Communications (OPC) Foundation, where organizations have been able to actively participate and contribute to the development of standards and tools on behalf of the community based on their own specific areas of interest and expertise. This way, the development load can be efficiently shared within the community, and the situations in which different consortia would be working on the same topics totally unaware of each other can be minimized.
Future work and research topics
The strict regulations on data integrity in the pharmaceutical industry have lead to significant interest in digitalization. Thus, eagerness for the early adoption of DCCs has also been high. Consequently, advancing DCC development requires extending the work and tests to other domains. Ideally, the testing should result in the development of good practices similar to the one developed for temperature calibrations in the presented Case 2.
As calibration, similarly to other maintenance operations, is a necessity for manufacturing companies for compliance reasons, it cannot truly be considered as a major area of competition for the companies but more of a common obstacle. This makes the joint development of calibration management-related technologies and sharing of knowledge a perfect example of anti-rival good, which is where the concept of a joint platform for the calibration ecosystem as an enabler for, e.g., big data, becomes prominent. For example, sharing some of the information about, e.g., instrument performance and calibration history in certain process conditions allows the companies to better understand the behavior of different instrument models or types, enabling more precise and optimized selections for the monitoring and controlling of processes. Respectively, the instrument manufacturers and calibration service providers could also receive a lot of valuable information about the usage and instrumentation needs in their current and potential customer bases, allowing them to produce and offer products and services more suitable for the needs of the customers.
In addition to the possibilities in improving the calibration data management of the instrument owners, the digital transition could be harnessed to make the audition and accreditation procedures more streamlined. For example, instead of inspecting the calibration information of randomly selected individual devices of a process line, the information could be made available for separate auditor interfaces, where the collective information of the measuring equipment of an entire process line could be made available for the auditors to access and view so that the audition could, at least by those parts, be possible to perform remotely in real time. Investments in the auditing infrastructure could be beneficial in lowering the costs of these procedures and help to increase the number of parties covered by accreditation, thus improving the overall quality and reliability of the operations within the metrology infrastructure.
Calibrations are an essential part of quality management in industries where measurements are required for monitoring and controlling processes. The recent developments in the digitalization of the global QI and metrology are enabling a transition towards automated calibration management processes. In addition to improved efficiency, another key benefit of digitalized calibration data management is improved data integrity, which has been one of the main interests in the development of IoT systems in the pharmaceutical industry. However, the digitalization of processes involving data exchange between several organizations can only be managed through standardization of the used data formats and interfaces. Due to the bilateral nature of calibration services, the service and system providers have been conservative in relation to collaboration, which has made further standardization difficult. To effectively advance the digitalization of calibration management, this kind of mindset would need to change from strictly bilateral relations towards a joint ecosystem.
In this paper, we presented a concept of a multitenant platform providing the necessary interfaces and functionalities for the secure exchange of DCCs as a method for ensuring interoperability between organizations. For large companies in the industry that use merely the improvements in working efficiency, they can see significant cost savings, which would justify the necessary investments in digitalization. However, the full benefits of the digitalization can only be achieved if even the smallest service providers are able to adapt to the necessary changes. For them, justifying any investments may not be possible in terms of efficiency alone. This is an issue where the platform approach for organizing the collaboration between the organizations can help, as the necessary digital operations could be provided by a central service provider, collectively reducing the development work within the ecosystem.
The testing performed based on real use cases in the pharmaceutical industry proved that the multitenant platform is an efficient means of organizing the exchange of DCCs. Considering the strict regulations of the pharmaceutical industry, the results indicate that the system would suit the requirements of other industries as well, providing cost-efficiency through economies of scale. However, the optimal means of implementing such a system can vary greatly with each organization. As stronger security solutions are more costly to develop and operate, a thorough assessment of potential risks and threats is essential for determining the level of security required. For example, the requirements of the industry are typically not as high as the requirements of the national infrastructures essential for society, in which case more advanced and robust solutions are ideal.
Abbreviations, acronyms, and initialisms
- AD: Active Directory
- API: application programming interface
- CMS: calibration management system
- DCC: digital calibration certificate
- DI: designated institute
- DIN: German Institute for Standardisation
- DKE: German Association for Electrical, Electronic & Information Technologies
- ERP: enterprise resource planning
- EMPIR: European Metrology Programme for Innovation and Research
- FDA: US Food and Drug Administration
- GAMP: good advanced manufacturing practice
- GLP: good laboratory practice
- GMP: good manufacturing practice
- GxP: good practices
- IEC: International Electrotechnical Commission
- INetQI: International Network on Quality Infrastructure
- ISA: International Society of Automation
- ISPE: International Society of Pharmaceutical Engineering
- MHRA: Medicines and Healthcare Products Regulatory Agency
- MI: metrology infrastructure
- NMI: national metrology institute
- OPC: Open Platform Communications
- PDF: Portable Document Format
- PTB: Physikalisch-Technische Bundesanstalt
- QI: quality infrastructure
- RMO: regional metrology organization
- SIM: Inter-American Metrology System
- TQM: total quality management
- UI: user interface
- XML: Extensible Markup Language
The authors would like to thank all the team members and participating organizations in the POC for their help and contributions.
Conceptualization, T.M., J.N. and R.V.; methodology, T.M. and J.N.; validation, T.M. and J.N.; investigation, T.M.; writing—original draft preparation, T.M.; writing—review and editing, T.M., J.N. and R.V.; visualization, T.M.; supervision, R.V. All authors have read and agreed to the published version of the manuscript.
This work was funded by Business Finland, grant number 1811/31/2019, in association with Aalto University.
Conflicts of interest
The authors declare no conflict of interest.
- "INetQI - International Network on Quality Infrastructure" (PDF). International Network on Quality Infrastructure. 2021. https://www.inetqi.net/media/uploads/inetqi_slides_2021.pdf. Retrieved 28 February 2022.
- "About the INetQI". International Network on Quality Infrastructure. 2022. https://www.inetqi.net/about/inetqi/. Retrieved 14 April 2022.
- "Quality Infrastructure Definition". INetQI Documentation. International Network on Quality Infrastructure. 2022. https://www.inetqi.net/documentation/quality-infrastructure-definition/. Retrieved 28 February 2022.
- Widenhoefer, T.; Physikalisch-Technische Bundesanstalt (January 2020). "Publishable Summary for 17IND02 SmartCom Communication and validation of smart data in IoT-networks" (PDF). EMPIR. https://www.ptb.de/empir2018/fileadmin/documents/empir/SmartCom/documents_for_download/SmartCom_17IND02_PublishableSummary.pdf. Retrieved 13 April 2020.
- "GEMIMEG". Physikalisch-Technische Bundesanstalt. Archived from the original on 14 August 2020. https://web.archive.org/web/20200814160819/https://www.gemimeg.ptb.de/startseite/. Retrieved 14 August 2020.
- Oppermann, Alexander; Eickelberg, Samuel; Exner, John (2021), Ziemba, Ewa; Chmielarz, Witold, eds., "Digital Transformation in Legal Metrology: An Approach to a Distributed Architecture for Consolidating Metrological Services and Data" (in en), Information Technology for Management: Towards Business Excellence (Cham: Springer International Publishing) 413: 146–164, doi:10.1007/978-3-030-71846-6_8, ISBN 978-3-030-71845-9, http://link.springer.com/10.1007/978-3-030-71846-6_8. Retrieved 2023-03-01
- Eichstädt, Sascha; Keidel, Anke; Tesch, Julia (1 December 2021). "Metrology for the digital age" (in en). Measurement: Sensors 18: 100232. doi:10.1016/j.measen.2021.100232. https://linkinghub.elsevier.com/retrieve/pii/S2665917421001951.
- Howarth, Preben; EUROMET, European Collaboration in Measurement Standards, eds. (2008). Metrology - in short: EURAMET project no. 1011 (3. ed ed.). Albertslund: Schultz Grafisk. ISBN 978-87-988154-5-7.
- "General Chapter 41 "Balances"". United States Pharmacopeia–National Formulary (USP-NF). United States Pharmacopeia. 2021.
- "Balances for analytical purposes (2.1.7)". European Pharmacopoeia (10.6 ed.). European Pharmacopoeia Commission. 2021.
- Fritsch, Klaus (1 December 2021). "Metrology of weighing in the pharmaceutical industry" (in en). Measurement: Sensors 18: 100262. doi:10.1016/j.measen.2021.100262. https://linkinghub.elsevier.com/retrieve/pii/S2665917421002257.
- ISPE GAMP Guide: Records and Data Integrity. International Society for Pharmaceutical Engineering. 2017. pp. 152. https://ispe.org/publications/guidance-documents/gamp-records-pharmaceutical-data-integrity.
- Thiel, F. (January 2018). "Digital Transformation of Legal Metrology - The European Metrology Cloud". OIML Bulletin LIX (1): 10–21. https://www.oiml.org/en/publications/bulletin/pdf/oiml_bulletin_jan_2018.pdf.
- Bruns, Thomas; Nordholz, Jan; Röske, Dirk; Schrader, Thorsten (1 December 2021). "A demonstrator for measurement workflows using digital calibration certificates (DCCs)" (in en). Measurement: Sensors 18: 100208. doi:10.1016/j.measen.2021.100208. https://linkinghub.elsevier.com/retrieve/pii/S2665917421001719.
- Hackel, Siegfried; Härtig, Frank; Schrader, Thorsten; Scheibner, Alexander; Loewe, Jan; Doering, Lutz; Gloger, Benjamin; Jagieniak, Justin et al. (1 December 2021). "The fundamental architecture of the DCC" (in en). Measurement: Sensors 18: 100354. doi:10.1016/j.measen.2021.100354. https://linkinghub.elsevier.com/retrieve/pii/S2665917421003172.
- "The ILAC Mutual Recognition Arrangement" (PDF). International Laboratory Accreditation Cooperation. October 2015. https://ilac.org/?ddownload=891. Retrieved 14 April 2022.
- Nikander, Pekka; Elo, Tommi; Mustapää, Tuukka; Kuosmanen, Petri; Hovhannisyan, Kristine; Maennel, Olaf; Brown, Clifford; Dawkins, John et al. (12 February 2020) (in en). Document specifying rules for the secure use of DCC covering legal aspects of metrology. doi:10.5281/ZENODO.3664211. https://zenodo.org/record/3664211.
- Kuster, Mark (1 June 2020). "A Measurement Information Infrastructure's Benefits for Industrial Metrology and IoT". 2020 IEEE International Workshop on Metrology for Industry 4.0 & IoT (Roma, Italy: IEEE): 479–484. doi:10.1109/MetroInd4.0IoT48571.2020.9138200. ISBN 978-1-7281-4892-2. https://ieeexplore.ieee.org/document/9138200/.
- Hutzschenreuter, Daniel; Härtig, Frank; Heeren, Wiebke; Wiedenhöfer, Thomas; Forbes, Alistair; Brown, Clifford; Smith, Ian; Rhodes, Susan et al. (4 November 2019). "SmartCom Digital System of Units (D-SI) Guide for the use of the metadata-format used in metrology for the easy-to-use, safe, harmonised and unambiguous digital transfer of metrological data" (in en). Zenodo. doi:10.5281/ZENODO.3522631. https://zenodo.org/record/3522631.
- "QUDT Ontologies Overview". QUDT.org. 1 March 2022. https://www.qudt.org/pages/QUDToverviewPage.html. Retrieved 23 March 2022.
- Eichstädt, Sascha; Gruber, Maximilian; Vedurmudi, Anupam Prasad; Seeger, Benedikt; Bruns, Thomas; Kok, Gertjan (12 March 2021). "Toward Smart Traceability for Digital Sensors and the Industrial Internet of Things" (in en). Sensors 21 (6): 2019. doi:10.3390/s21062019. ISSN 1424-8220. PMC PMC7998976. PMID 33809296. https://www.mdpi.com/1424-8220/21/6/2019.
- Primary Standards Lab; Delker, Colin; Sandia National Laboratories; Roberts, Mike; Robinson, Amaru; Solomon, Otis (2020). "Exploration of a Data-Enhanced Calibration Certificate as Part of a Complete Measurement Information Infrastructure". NCSL International Workshop & Symposium Conference Proceedings 2020 (NCSL International). doi:10.51843/wsproceedings.2020.05. https://ncsli.org/store/viewproduct.aspx?id=17886213.
- Boschung, Gregor; Wollensack, Michael; Zeier, Markus; Blaser, Cédric; Hof, Christian; Stathis, Manuel; Blattner, Peter; Stuker, Florian et al. (1 December 2021). "PDF/A-3 solution for digital calibration certificates" (in en). Measurement: Sensors 18: 100282. doi:10.1016/j.measen.2021.100282. https://linkinghub.elsevier.com/retrieve/pii/S2665917421002452.
- "VDI/VDE 2623: Format für den Austausch von Daten im Prüfmittelmanagement - Definition des Calibration-Data-Exchange-Datenformats (CDE-Datenformat)". VDI-Richtlinien. VDI Verein. March 2022. https://www.vdi.de/richtlinien/details/vdivde-2623-format-fuer-den-austausch-von-daten-im-pruefmittelmanagement-definition-des-calibration-data-exchange-datenformats-cde-datenformat. Retrieved March 2022.
- Heeren, Wiebke; Muller, Bernd; Miele, Gianfranco; Mustapaa, Tuukka; Hutzschenreuter, Daniel; Brown, Clifford; Baer, Oksana (7 June 2021). "SmartCom — Key Findings for Digitalisation in Metrology". 2021 IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4.0&IoT) (Rome, Italy: IEEE): 364–369. doi:10.1109/MetroInd4.0IoT51437.2021.9488450. ISBN 978-1-6654-1980-2. https://ieeexplore.ieee.org/document/9488450/.
- Mustapaa, Tuukka; Autiosalo, Juuso; Nikander, Pekka; Siegel, Joshua E.; Viitala, Raine (1 June 2020). "Digital Metrology for the Internet of Things". 2020 Global Internet of Things Summit (GIoTS) (Dublin, Ireland: IEEE): 1–6. doi:10.1109/GIOTS49054.2020.9119603. ISBN 978-1-7281-6728-2. https://ieeexplore.ieee.org/document/9119603/.
- Mustapää, Tuukka; Nikander, Pekka; Hutzschenreuter, Daniel; Viitala, Raine (21 August 2020). "Metrological Challenges in Collaborative Sensing: Applicability of Digital Calibration Certificates" (in en). Sensors 20 (17): 4730. doi:10.3390/s20174730. ISSN 1424-8220. PMC PMC7506767. PMID 32825715. https://www.mdpi.com/1424-8220/20/17/4730.
- Laiz, Héctor; Hackel, Siegfried; Härtig, Frank; Schrader, Thorsten; Shanna, Shanna; Loewe, Jan; Doering, Lutz; Gloger, Benjamin et al. (2022) (in en). 2nd international DCC-Conference 01 - 03 March 2022 Proceedings. pp. 89287184 bytes. doi:10.7795/820.20220411. https://oar.ptb.de/resources/show/10.7795/820.20220411.
- Brown, Clifford; Elo, Tommi; Hovhannisyan, Kristine; Hutzschenreuter, Daniel; Kuosmanen, Petri; Maennel, Olaf; Mustapaa, Tuukka; Nikander, Pekka et al. (1 June 2020). "Infrastructure for Digital Calibration Certificates". 2020 IEEE International Workshop on Metrology for Industry 4.0 & IoT (Roma, Italy: IEEE): 485–489. doi:10.1109/MetroInd4.0IoT48571.2020.9138220. ISBN 978-1-7281-4892-2. https://ieeexplore.ieee.org/document/9138220/.
- Neyezhmakov, Pavel; Zub, Stanislav; Pivnenko, Stanislav (1 December 2021). "Preliminary E-infrastructure for Digital Calibration Certificate" (in en). Measurement: Sensors 18: 100332. doi:10.1016/j.measen.2021.100332. https://linkinghub.elsevier.com/retrieve/pii/S2665917421002956.
- Xiong, XingChuang; Zhu, Yiwei; Li, Jinyuan; Duan, Yuning; Fang, Xiang (1 December 2021). "A digital framework for metrological information" (in en). Measurement: Sensors 18: 100122. doi:10.1016/j.measen.2021.100122. https://linkinghub.elsevier.com/retrieve/pii/S2665917421000854.
- Arden, N. Sarah; Fisher, Adam C.; Tyner, Katherine; Yu, Lawrence X.; Lee, Sau L.; Kopcha, Michael (1 June 2021). "Industry 4.0 for pharmaceutical manufacturing: Preparing for the smart factories of the future" (in en). International Journal of Pharmaceutics 602: 120554. doi:10.1016/j.ijpharm.2021.120554. https://linkinghub.elsevier.com/retrieve/pii/S0378517321003598.
- Dornelles, Jéssica de Assis; Ayala, Néstor F.; Frank, Alejandro G. (1 January 2022). "Smart Working in Industry 4.0: How digital technologies enhance manufacturing workers' activities" (in en). Computers & Industrial Engineering 163: 107804. doi:10.1016/j.cie.2021.107804. https://linkinghub.elsevier.com/retrieve/pii/S0360835221007087.
- Forcina, Antonio; Falcone, Domenico (2021). "The role of Industry 4.0 enabling technologies for safety management: A systematic literature review" (in en). Procedia Computer Science 180: 436–445. doi:10.1016/j.procs.2021.01.260. https://linkinghub.elsevier.com/retrieve/pii/S1877050921003033.
- Leng, Jiewu; Wang, Dewen; Shen, Weiming; Li, Xinyu; Liu, Qiang; Chen, Xin (1 July 2021). "Digital twins-based smart manufacturing system design in Industry 4.0: A review" (in en). Journal of Manufacturing Systems 60: 119–137. doi:10.1016/j.jmsy.2021.05.011. https://linkinghub.elsevier.com/retrieve/pii/S0278612521001151.
- Autiosalo, Juuso; Ala-Laurinaho, Riku; Mattila, Joel; Valtonen, Miika; Peltoranta, Valtteri; Tammi, Kari (12 January 2021). "Towards Integrated Digital Twins for Industrial Products: Case Study on an Overhead Crane" (in en). Applied Sciences 11 (2): 683. doi:10.3390/app11020683. ISSN 2076-3417. https://www.mdpi.com/2076-3417/11/2/683.
- Li, Lianhui; Lei, Bingbing; Mao, Chunlei (1 March 2022). "Digital twin in smart manufacturing" (in en). Journal of Industrial Information Integration 26: 100289. doi:10.1016/j.jii.2021.100289. https://linkinghub.elsevier.com/retrieve/pii/S2452414X21000868.
- Perno, Matteo; Hvam, Lars; Haug, Anders (1 January 2022). "Implementation of digital twins in the process industry: A systematic literature review of enablers and barriers" (in en). Computers in Industry 134: 103558. doi:10.1016/j.compind.2021.103558. https://linkinghub.elsevier.com/retrieve/pii/S0166361521001652.
- Carvalho, Adriana Ventura; Enrique, Daisy Valle; Chouchene, Amal; Charrua-Santos, Fernando (2021). "Quality 4.0: An Overview" (in en). Procedia Computer Science 181: 341–346. doi:10.1016/j.procs.2021.01.176. https://linkinghub.elsevier.com/retrieve/pii/S1877050921002167.
- Asif, Muhammad (1 June 2020). "Are QM models aligned with Industry 4.0? A perspective on current practices" (in en). Journal of Cleaner Production 258: 120820. doi:10.1016/j.jclepro.2020.120820. https://linkinghub.elsevier.com/retrieve/pii/S0959652620308672.
- Blichfeldt, Henrik; Faullant, Rita (1 July 2021). "Performance effects of digital technology adoption and product & service innovation – A process-industry perspective" (in en). Technovation 105: 102275. doi:10.1016/j.technovation.2021.102275. https://linkinghub.elsevier.com/retrieve/pii/S0166497221000560.
- Coito, Tiago; Martins, Miguel S.E.; Firme, Bernardo; Figueiredo, João; Vieira, Susana M.; Sousa, João M.C. (1 January 2022). "Assessing the impact of automation in pharmaceutical quality control labs using a digital twin" (in en). Journal of Manufacturing Systems 62: 270–285. doi:10.1016/j.jmsy.2021.11.014. https://linkinghub.elsevier.com/retrieve/pii/S0278612521002442.
- Hole, Glenn; Hole, Anastasia S.; McFalone-Shaw, Ian (1 December 2021). "Digitalization in pharmaceutical industry: What to focus on under the digital implementation process?" (in en). International Journal of Pharmaceutics: X 3: 100095. doi:10.1016/j.ijpx.2021.100095. PMC PMC8528719. PMID 34712948. https://linkinghub.elsevier.com/retrieve/pii/S2590156721000244.
- Kulkov, Ignat (1 August 2021). "The role of artificial intelligence in business transformation: A case of pharmaceutical companies" (in en). Technology in Society 66: 101629. doi:10.1016/j.techsoc.2021.101629. https://linkinghub.elsevier.com/retrieve/pii/S0160791X21001044.
- Chirumalla, Koteshwar (1 July 2021). "Building digitally-enabled process innovation in the process industries: A dynamic capabilities approach" (in en). Technovation 105: 102256. doi:10.1016/j.technovation.2021.102256. https://linkinghub.elsevier.com/retrieve/pii/S0166497221000377.
- Reinhardt, Ingrid Carla; Oliveira, Dr Jorge C.; Ring, Dr Denis T. (1 June 2020). "Current Perspectives on the Development of Industry 4.0 in the Pharmaceutical Sector" (in en). Journal of Industrial Information Integration 18: 100131. doi:10.1016/j.jii.2020.100131. https://linkinghub.elsevier.com/retrieve/pii/S2452414X20300066.
- "ISO 9001:2015 Quality management systems — Requirements". International Organization for Standardization. September 2015. https://www.iso.org/standard/62085.html.
- "ISO/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories". International Organization for Standardization. March 2018. https://www.iso.org/standard/66912.html.
- "Why use an Accredited Laboratory?". International Laboratory Accreditation Cooperation. October 2015. https://ilac.org/?ddownload=902. Retrieved 14 April 2022.
- Cable, Mike (2005). Calibration: a technician's guide. ISA technician series. Research Triangle Park, NC: ISA. ISBN 978-1-55617-912-9.
- GAMP Good Practice Guide: A Risk-Based Approach to Calibration Management (2nd ed.). International Society for Pharmaceutical Engineering. 2010. pp. 124. https://ispe.org/publications/guidance-documents/gamp-good-practice-guide-calibration-management.
- Hernandez-Vasquez, Jose Daniel; Pedraza-Yepes, Cristian; Higuera-Cobos, Oscar Fabian; Caballero, Javier Pedroza; Reyes Montes, Wilmer de Los (2018). "Calibration of analog scales based on a metrological comparison between the SIM Guide and OIML R 76-1" (in en). Contemporary Engineering Sciences 11 (82): 4049–4057. doi:10.12988/ces.2018.88447. http://www.m-hikari.com/ces/ces2018/ces81-84-2018/88447.html.
- "CEN/CLC/JTC 18 - Weighing instruments - EN 45501:2015: Metrological aspects of non-automatic weighing instruments". European Committee for Standardization. 13 August 2015. https://standards.cencenelec.eu/dyn/www/f?p=CEN:110:0::::FSP_PROJECT,FSP_ORG_ID:37556,2629687&cs=139FCCEE111F032B9BC3C0F462B0B3B88. Retrieved 15 April 2022.
- Barton, John W.; Bowers, Shelby L.; Butcher, Tina G.; Harshman, Richard A.; Lee, G. Diane; Warfield, Lisa (29 November 2021). Specifications, Tolerances, and Other Technical Requirements for Weighing and Measuring Devices. doi:10.6028/nist.hb.44-2022. https://nvlpubs.nist.gov/nistpubs/hb/2022/NIST.HB.44-2022.pdf.
- "EURAMET Calibration Guide No. 18—Guidelines on the Calibration of Non-Automatic Weighing Instruments—Version 4.0" (PDF). EURAMET e.V.. November 2015. ISBN 978-3-942992-40-4. https://www.euramet.org/index.php?eID=tx_securedownloads&p=40&u=0&g=0&t=1681679021&hash=a18b351a14fb3c111931f5fedfb71da1fba4186f&file=Media/docs/Publications/calguides/I-CAL-GUI-018_Calibration_Guide_No._18_web.pdf. Retrieved 16 April 2022.
- "ASTM E898-20 Standard Practice for Calibration of Non-Automatic Weighing Instruments". ASTM International. 22 June 2020. https://www.astm.org/e0898-20.html. Retrieved 15 April 2022.
- "SIM MWG7 Guidelines on the Calibration of Non-Automatic Weighing Instruments". Sistema Interamericano de Metrología. 2009. https://sim-metrologia.org/wp-content/uploads/2019/10/SIM-Guidelines-on-the-calibration-of-non-automatic-weighing-instruments-2009.pdf. Retrieved 16 April 2022.
- "JJF 1847-2020 Calibration Specification for Electronic balance (English Version)". Code of China. January 2021. https://www.codeofchina.com/standard/JJF1847-2020.html. Retrieved 15 April 2022.
- "General Information 1251 “Weighing on an Analytical Balance”". United States Pharmacopeia–National Formulary (USP-NF). United States Pharmacopeia. 2021.
- "ALCOA+ - what does it mean?". ECA Foundation, Mannheim. 19 May 2021. https://www.gmp-compliance.org/gmp-news/alcoa-what-does-it-mean. Retrieved 23 February 2022.
- Rattan, Anil K. (2018). "Data Integrity: History, Issues, and Remediation of Issues" (in en). PDA Journal of Pharmaceutical Science and Technology 72 (2): 105–116. doi:10.5731/pdajpst.2017.007765. ISSN 1079-7440. http://journal.pda.org/lookup/doi/10.5731/pdajpst.2017.007765.
- "CFR - Code of Federal Regulations Title 21". U.S. Food and Drug Administration. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=211.68. Retrieved 28 January 2022.
- "'GXP' Data Integrity Guidance and Definitions" (PDF). Medicines & Healthcare products Regulatory Agency (MHRA). March 2018. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/687246/MHRA_GxP_data_integrity_guide_March_edited_Final.pdf. Retrieved 15 April 2022.
- Czarny, D.A.; Diemer, J.; Schacht, M. et al. (June 2021). "Scenarios for Digitizing Standardization and Standards" (PDF). DIN e.V.; Deutsche Kommission Elektrotechnik. https://www.dke.de/resource/blob/2076816/facc9bde1806e2194a3d26a60c79bf77/idis-whitepaper-en---download-data.pdf. Retrieved 23 March 2022.
- Physikalisch-Technische Bundesanstalt (2019). "xmlns:dcc schema version 2.4.0". https://www.ptb.de/dcc/v2.4.0/dcc.xsd. Retrieved 06 June 2022.
- Physikalisch-Technische Bundesanstalt (2019). "xmlns:dcc schema version 3.0.0-rc.2". https://www.ptb.de/dcc/v3.0.0-rc.2/dcc.xsd. Retrieved 06 June 2022.
- Gloger, B. (25 November 2022). "Good Practice". Digital Calibration Certificate Wiki. Physikalisch-Technische Bundesanstalt. https://dccwiki.ptb.de/de/gp_home.
- "About us". Nextcloud GmbH. https://nextcloud.com/about. Retrieved 06 June 2022.
- "Azure Active Directory (Azure AD)". Microsoft Corporation. https://azure.microsoft.com/en-us/products/active-directory/. Retrieved 06 June 2022.
This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was tweaked significantly to improve readability. The PMCID and DOI were also added when they were missing from the original reference. The links to the DCC XML schemas, Nextcloud, Azure Active Directory, etc. of the original article were turned into proper references for this version. The authors fail to fully define "quality infrastructure" in their introduction; this version adds a definition, pulled from the authors' first reference, for clarity.