Journal:Fostering reproducibility, reusability, and technology transfer in health informatics

Full article title	Fostering reproducibility, reusability, and technology transfer in health informatics
Journal	iScience
Author(s)	Hauschild, Anne-Christin; Eick, Lisa; Wienbeck, Joachim; Heider, Dominik
Author affiliation(s)	Philipps University of Marburg
Primary contact	Email: dominik dot heider at uni-marburg dot de
Year published	2021
Volume and issue	24(7)
Article #	102803
DOI	10.1016/j.isci.2021.102803
ISSN	2589-0042
Distribution license	Creative Commons Attribution 4.0 International
Website	https://www.sciencedirect.com/science/article/pii/S2589004221007719
Download	https://www.sciencedirect.com/science/article/pii/S2589004221007719/pdfft (PDF)

Abstract

Computational methods can transform healthcare. In particular, health informatics combined with artificial intelligence (AI) has shown tremendous potential when applied in various fields of medical research and has opened a new era for precision medicine. The development of reusable biomedical software for research or clinical practice is time-consuming and requires rigorous compliance with quality requirements as defined by international standards.

However, research projects rarely implement such measures, hindering smooth technology transfer to the research community or manufacturers, as well as reproducibility and reusability.

Here, we present a guideline for quality management systems (QMS) for academic organizations incorporating the essential components, while confining the requirements to an easily manageable effort. It provides a starting point to effortlessly implement a QMS tailored to specific needs and greatly facilitates technology transfer in a controlled manner, thereby supporting reproducibility and reusability.

Ultimately, the emerging standardized workflows can pave the way for an accelerated deployment in clinical practice.

Keywords: health informatics, bioinformatics, software engineering, software robustness

Graphical abstract

Introduction

Computational approaches offer new opportunities to transform health care. In particular, modern artificial intelligence (AI) and machine learning (ML) techniques have shown substantial potential when applied in various fields of medical research and therefore have opened up a new era for precision medicine. (Hawgood et al., 2015) In the last decade, universities and other research organizations supported and encouraged by public funding agencies have allocated tremendous efforts to develop and enhance such predictive software models, algorithms, and “systems as medical devices” (SaMD) for clinical research and application.

A manifold of studies have proven AI to be advantageous for disease diagnosis, prognosis, and disease monitoring. In cancer research, for instance, ML is used on omics data to gain deeper insights and understanding of the genetic and metabolic alterations that determine disease progression and enable tailored prognoses and monitoring. Additionally, computational models on clinical information are used to assess individualized health risks, for instance, to identify high-risk patients for sepsis in intensive care units, the analysis of longitudinal data for the early detection of heart failure, or applications in infectious diseases.

Once the outcome of this development is sufficiently mature and robust to be published and used in routine treatment or diagnosis, there is a need to transfer this knowledge to other research groups and, ultimately, clinical practice. Therefore, a straightforward technology transfer to a manufacturer of medical devices would be beneficial.

Challenges of scientific software development for health

While the knowledge and technologies to develop effective and efficacious AI-driven clinical decision support systems exist, a manifold of pressing issues hinders further reuse in research and transfer to clinical practice. In software development projects in the industry, specialized developers typically work in large teams with considerable resources and are able to focus on usability and reusability. In contrast, academic teams often consist of small groups of researchers (e.g., graduate students and postdoctoral scholars) who are typically not trained software engineers and only have temporary contracts and frequent turnover. Thus, individuals often develop software on a one-person-one-project basis.

Moreover, funders and the academic hiring and promotion processes incentivize the pressure to publish and focus on “novelty” rather than software quality. Thus, it entices researchers to focus on theoretical aspects and proof of concept development. Therefore, most researchers implement software in a prototype-centered manner, which often lacks quality checks such as systematic testing and may be published quickly. However, these implementations often lack crucial qualities required for long-term reuse, such as documentation, usability, appropriate performance for real-life application, user-friendly interfaces, reusability, or minimized risk for potential users.

Recently, scientific journals such as GigaScience or Biostatistics have promoted reproducibility and reusability by mandating FAIR principles (findability, accessibility, interoperability, and reusability). The concept of FAIR establishes a guideline for scientific data management and documentation.

However, as recently surveyed by Pinto et al., documentation of scientific software is one of the most significant “pain points.” Journal publications are typically the primary source of documentation for scientific software and are quickly outdated by the agile software development style in academia. Ideally, documentation should be detailed enough so that a developer with no prior knowledge of the project should make productive use of the software and use it for further development without biases or limitations.

Another critical factor is accessibility. A lack of strict enforcement by journals, organizations, and funders has resulted in a loss of crucial data and software code. According to an extensive analysis by Mangul et al., almost 28% of all resources linked in publications were not accessible, indicating poor maintenance. Moreover, a large proportion of software tested by Mangul et al. was not usable due to non-installability or lack of portability. The main inhibitors were storage locations outside of the journal’s directories or public versioning systems.

Reproducibility and traceability are two of the most important aspects of biomedical and health informatics. The lack of publicly available and comprehensible source code undermines the auditing of published methods and results. Additionally, the traceability of changes via version control is critical for reproducibility and reuse of research and code that replicators get to use. These aspects ultimately undermine scientific rigor, transparency, and reproducibility. The previously described accessibility, documentation, portability, and reusability factors are essential to ensure reproducibility and underpin trust in the scientific record of scientific software.

Additionally, modern systems medicine approaches integrate all facets of private data such as electronic health records (EHR), laboratory results, medical imaging, omics resources such as the cancer genome atlas (TCGA) or the gene expression omnibus, or pathway information. However, sensitive patient data that enables an association of confidential personal information to single individuals underlies strict regulations such as the European General Data Protection Regulation (GDPR). Therefore, the exchange within and among institutes is perceived as insurmountable, posing a roadblock hampering significant data-based medical innovations.

Quality management

The design and development of medical devices used in clinical practice, including software as a medical device, have to comply with regulations and standards that ensure the safety and quality of medical devices worldwide. These regulations include standards concerning quality management (QM), the software development life cycle (including agile methods such as Scrum or Kanban), and risk management. However, such measures are not yet a standard in academic research work.

The challenges in academic development procedures, as described before, affect technology transfer as well as reproducibility and reusability among researchers, as well as to the industry. It is usually restricted to transferring an intellectual method and know-how while discarding all created artifacts (e.g., documents for development planning and software code). In particular, manufacturers will need to understand the principles of the new method, train their staff members, and maintain design and development records when creating the final product. The record will include design documents, meeting minutes, and other documents produced during the development. This effort causes a significant overhead in the time and work they have to invest before the product is ready for the market.

Introducing quality management systems (QMS) by industry has proven to guarantee continuous high quality among projects and procedures within an organization. Our goal is to facilitate this experience in academic environments. Establishing an academia-tailored QMS for research organizations can address the described challenges of software quality, documentation, accessibility, traceability, and reproducibility and help software development in research organizations reach higher quality and standards. These quality standards are particularly critical for software that is intended to be used in the medical context. We can significantly reduce software transfer barriers among researchers in academia and industry if these standards are established from the start.

Moreover, it has the potential to facilitate and speed up the transfer to clinical practice and thereby improve medical diagnosis or treatment of patients rapidly. However, our recommendations are based on standards that are common for medical device development but do not have the intention to shortcut any steps of the regular medical device development; this will always have to be done according to the strict rules and regulations that apply. Other benefits of a QMS for medical device development are also beneficial for research organizations when developing such software, including making responsibilities clearer, phasing in new team members quickly, and ensuring everyone knows how their work contributes to project and organizational goals.

Requirements for QM are defined in ISO 9001, in ISO 13485, and the Code of Federal Regulation (CFR) Title 21. However, while ISO 9001 is generic and can be used in different fields, ISO 13485 is specific for QM in medical device development. The main differences are that ISO 9001 requires the organization to demonstrate continual improvement. In contrast, ISO 13485 only requires the certified organization to verify that the QMS is effectively implemented and maintained. Additionally, ISO 9001 includes customer satisfaction, which is not relevant to ISO 13485. There are also some other differences in ISO 13485, e.g., that the promotion and awareness of regulatory requirements is a management responsibility, that risk assessment and management need to be carried out (according to ISO 14971), and some others. Unfortunately, ISO 13485 is most likely outside of what is feasible for research groups in universities and other research organizations.

Our goal: Academia-tailored QMS

Until now, there has been no guidance on how to establish an academia-tailored QMS for research organizations that balances between not having any QMS at all and a wholly formulated QMS that complies with official regulatory standards. Here, we present a synopsis of relevant requirements in such standards and adjust these toward software development requirements in academia. We aim to embrace the benefits and facilitate the technology transfer as much as possible while keeping the overhead for researchers and developers in a well manageable range. Thus, we tailor the content for use in universities or similar research organizations that focus on software in the medical setting.

However, the ideas presented here do not provide a shortcut when developing medical software. Any development of medical software must strictly follow the relevant regulations in the specific country or region. Therefore, our suggestions provide a starting point intended to be adapted to specific requirements as needed and we refer to it as MDx-ready (i.e., ready for transferring it into molecular diagnostics SaMD implementations). This flexibility is well in the mindset of most QMS: If there is a profound reason for the change, change it, but consider all relevant aspects and document the reasons and the changes.

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, though grammar and word usage was substantially updated for improved readability. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version—by design—lists them in order of appearance.