Quality management for academic laboratories: burden or boon? Professional quality management could be very beneficial for academic research but needs to overcome specific caveats.

Abstract

Science & Society19 October 2018free access Quality management for academic laboratories: burden or boon? Professional quality management could be very beneficial for academic research but needs to overcome specific caveats Ulrich Dirnagl [email protected] orcid.org/0000-0003-0755-6119 Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany QUEST Center for Transforming Biomedical Research, Berlin Institute of Health (BIH), Berlin, Germany Search for more papers by this author Claudia Kurreck orcid.org/0000-0002-9567-626X Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany Search for more papers by this author Esmeralda Castaños-Vélez orcid.org/0000-0002-2670-1118 Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany Search for more papers by this author René Bernard orcid.org/0000-0003-3265-2372 Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany Search for more papers by this author Ulrich Dirnagl [email protected] orcid.org/0000-0003-0755-6119 Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany QUEST Center for Transforming Biomedical Research, Berlin Institute of Health (BIH), Berlin, Germany Search for more papers by this author Claudia Kurreck orcid.org/0000-0002-9567-626X Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany Search for more papers by this author Esmeralda Castaños-Vélez orcid.org/0000-0002-2670-1118 Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany Search for more papers by this author René Bernard orcid.org/0000-0003-3265-2372 Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany Search for more papers by this author Author Information Ulrich Dirnagl1,2, Claudia Kurreck1, Esmeralda Castaños-Vélez1 and René Bernard1 1Department of Experimental Neurology and Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany 2QUEST Center for Transforming Biomedical Research, Berlin Institute of Health (BIH), Berlin, Germany EMBO Rep (2018)19:e47143https://doi.org/10.15252/embr.201847143 PDFDownload PDF of article text and main figures. ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinked InMendeleyWechatReddit Figures & Info Basic and translational biomedical research explores biological and pathophysiological mechanisms with the aim of developing novel therapies, preventive measures, and diagnostics to improve human health. Disappointingly, however, most new therapies fail when they are tested in clinical trials. Although the causes of this “translational attrition” are diverse and often rooted in the complexity of the underlying biology, it has also become clear that methodology is a major issue. The “translational roadblock”, along with what has been dubbed a “reproducibility crisis” 1, has fueled discussions about the reliability and reproducibility of biomedical research in general. There is strong evidence that weaknesses in planning, conducting, analyzing, and (non)reporting of research 2, as well as misidentification or contamination of reagents, biologicals, and cell lines 3, are prevalent factors. Meta-research has shown that these problems can lead to an inflation of effect size and false positives and consequently decrease the reproducibility and predictiveness of research results. At the same time, the increasing methodological complexity combined with the immense proliferation in research outputs greatly complicates the production and evaluation of reliable evidence. Pressure to publish and hypercompetition for resources further compromise the robustness and rigor of research. Arguably, basic and translational biomedical research has a quality problem. Quality management to the rescue? In the 1970s, US cars had major quality problems. By comparison, Japanese cars were much more reliable. The introduction of rigorous quality management in the production process was largely credited for this competitive advantage, which helped the Japanese car industry to dominate the market for decades to come. By now, most industries, including US car manufacturers, the health and pharmaceutical industries as well as clinical medicine, have established sophisticated quality management systems (QMS) on which they spend several percent of their budget. Clearly, these investments pay off as companies and institutions with managed quality report higher revenues, earnings, stock market value, and better performance indicators 4. Could the introduction of structured QM, which has a proven record to increase value and reduce waste and which has transformed entire industries that previously suffered from inferior quality, be effective in academia too? Arguably, basic and translational biomedical research has a quality problem. In principle, a QMS is as structured, evidence-based approach to improve quality. Most QMS follow a so-called PDCA (plan–do–check–act or plan–do–check–adjust) cycle to constantly analyze processes and adapt to changes, which fosters continuous improvement. Many QMS are modular in the sense that they consist of key processes, such as institutional policy, organizational structure and responsibilities, data management, and so on, as well as support processes like training/education, guidelines/regulations, etc. QMS require centrally organized documentation and control, which is periodically reviewed, records of meetings (agenda, participant list, entries in action plan with follow-up checks, protocols), and measures to comply with internal and external rules and regulations. Most QMS also describe provisions to ensure that the system follows specific requirements: Internal or external auditors regularly revise the processes and results. In addition, specific QMS, like those following the ISO-9001 norm, can be certified by accredited organizations to ensure and demonstrate compliance with customer and regulatory requirements. Why structured QM never made it into academia In contrast to clinical practice and R&D in the pharmaceutical industry, structured QM is virtually unknown in basic and preclinical biomedical research even though it is plagued by methodological complexities, proneness to errors, a plethora of laws and regulations, and high fluctuation in personnel. Scientists, who usually have no practical knowledge of QM, find its normative language, nomenclature, and processes aversive. Moreover, most QMS have been developed for companies or service providers and have therefore limited applicability to academic research, which makes it hard to motivate scientists to work with the QMS on a daily basis. For example, the notion that QMS helps to deliver a quality “product” to a “customer” defies most scientists. Yet, are fellow scientists, funders, and the public not our customers? And is evidence, usually delivered as publications, not our product? … most QMS have been developed for companies or service providers and have therefore limited applicability to academic research. Researchers also fear that QM would curtail their creativity and the inquisitiveness of the research process. In particular, auditing, which is a central element of QMS, is often perceived as a dystopian surveillance scheme or prone to leak laboratory secrets to competitors. Vice versa, external auditors from certifying agencies are often not familiar with the academic research settings and quality problems in biomedical research. This subjective aversion to QMS is further aggravated by objective impediments. Resources in academia are tight, and universities and funders cover only costs that are directly related to research. Importantly, the quality or robustness of research results are generally not important factors that determine academic success in terms of grant applications, publications, or career options. The primary “products” of academic research are publications, preferably in prestigious journals, and not useful evidence. A low-quality academic “product” may also go unnoticed, as only a minority of findings are replicated, and novelty, not robustness of results, is the main criterion for publication. This is in stark contrast to the pharmaceutical industry 5, where the utility and reliability of research are a major criterion, as investment in questionable results leads to financial losses. Also, other than industry, academic institutions do not provide infrastructure to set up and maintain QM; academic research has existed for centuries without internal or external quality assurance—if it ain't broke, don't fix it! Why basic and translational biomedical research benefits from QMS Arguably there are few professional environments that are more complex and error-prone, depend more heavily on exchange of knowledge, are more regulated (safety, animal welfare, genetic engineering, etc.), have more diverse staff (PhDs, MDs, students, technicians, and so on), more turnaround of personnel, and depend more on the robustness—or quality—of their results than biological and biomedical research. This is a fertile ground for a QMS, and it would help scientists to avoid many of the potential pitfalls and biases of the research process, as it prescribes clearly assigned responsibilities, helps to learn from errors (error management), and provides a broad knowledge base that could otherwise be lost (standard operating procedures). In addition, QMS support data management, and systematic and comprehensive training and education. Unbeknownst and potentially surprising to most academic researchers, structured measures to promote quality were introduced to biomedical research as early as in the 1970s. After serious safety problems, the US Food and Drug Administration introduced Good Laboratory Practice (GLP) regulations for non-clinical studies in 1976. A few years later, the Organization for Economic Co-operation and Development (OECD) drafted similar regulations for its member states, which are still in place today. However, the GLP principles are generally not applied to laboratories involved in “discovery research”, probably owing to their rigidity and sanctions in case of non-compliance. There have been several other attempts to introduce structured preclinical quality systems 6. In 2005, the World Health Organization (WHO) published a Handbook on Quality Practices in Basic Biomedical Research. It contains useful instructions on documentation and monitoring, but lacks necessary design concepts, such as blinding, randomization, sample size calculations, or measures to safeguard data integrity. The Research Quality Association and the American Society for Quality have each released quality systems for biomedical research. However, both are mainly geared for laboratories focused on drug development and are wanting in categories such as study design, data analysis, or data integrity. The pioneering Quality Central Program at the University of Minnesota 7 has provided a Quality Assurance Toolkit, facilitating best practices for management of resources, documents, data, and method validation 8. These forms have a clear focus on improving laboratory documentation in an academic setting, but are self-assessment tools and not intended as QMS. … the quality or robustness of research results are generally not important factors that determine academic success in terms of grant applications, publications or career options. Our ISO-9001 experience During the past decade, many neuroscientists have realized that their research has an exceedingly high translational attrition rate and is suffering from bias, low statistical power, selective use of data, and non-reporting of “unfitting” results. In response, the Department of Experimental Neurology at the Charité Universitätsmedizin Berlin implemented the highly flexible and globally adopted QMS standard ISO-9001 in 2014. At our institute, about 100 researchers and technicians study basic mechanisms of brain physiology and pathophysiology, using a plethora of state-of-the-art technologies and approaches, including molecular biology, cell biology, biochemistry, ultra-highfield magnetic resonance imaging, histology, multiphoton microscopy, in vivo modeling of disease in rodents, and so on. Despite initial skepticism and mild resistance, most staff members understood the benefits and adopted the QMS measures wholeheartedly. Through anonymous surveys and meetings, we learned that they particularly appreciated that QMS resulted in better record keeping, better exchange of resources, knowledge and devices, and a more sophisticated culture of dealing with errors. They also acknowledged that it improved the distribution of responsibilities within their work groups. Despite initial skepticism and mild resistance, most staff members understood the benefits and adopted the QMS measures wholeheartedly. However, in 2017, we decided to discontinue QMS based on the ISO-9001 standard. It appeared too rigid and its terminology was alien to many of our scientists. Maintaining ISO-9001 was resource intense, and, importantly, its overall design did not match with the idiosyncrasies of research processes, such as experimental design, data storage or validity problems, such as performance, detection, or attrition bias. Moreover, the introduction of a rigid normative regulatory system applied additional pressure on scientists who already work in a hypercompetitive system. Even those, who recognized the need to improve the quality of their work and understood the potential of a quality-controlled environment, were ultimately not convinced of the cost–benefit ratio of working under ISO-9001. Auditing was experienced as an examination, especially when done by the external auditors of the certification organization. Critically, the auditing did not cover essential elements of the scientific method, mostly because the auditors were not familiar with the research process. The disappointment was not with QM in general, but rather with the fact that some of the QM activities had only little impact on research and outcomes, but at the same time drained too much time and resources. However, our experience with ISO-9001 left us convinced that a structured approach to QM has enormous potential to improve the quality of research without stifling creativity if done properly. As no QMS tailored to the needs of academic research is currently available, we set out, with the help of the Volks-wagen Foundation and an expert advisory board, to design and implement a researcher and science-driven QMS. Informed by our experience with ISO-9001, we framed the following desirable features: It shall consist of mandatory core elements and optional supplement modules and therefore be scalable and adjustable to smaller or larger research environments; it must be financeable and sustainable; it shall be open source; it should foster innovation; it must support common daily laboratory practices and address prevalent biases and validity threats; it must incorporate various regulations on occupational safety, animal welfare, or genetic manufacturing; and it should lead to a more transparent and trustworthy research process. Academic QMS beyond ISO—The PREMIER project The ultimate goal of our PREMIER (Productiveness and Robustness through Modular Improvement of Experimental Research) QMS is to provide a biomedically oriented, structured but flexible quality assurance system that is acceptable for academic researchers. Compliance with its quality standards shall be assessable through novel forms of auditing. The QMS will be made freely available, including tools and training modules. PREMIER has been preregistered at the Open Science Framework 9, where our approach is described in greater detail. In brief (Fig 1), PREMIER will consist of four modules for management responsibilities, key processes, support processes, and a module that assesses and improves the QM process (indicators, audits, etc.). Figure 1.Modular structure of thePREMIERQMS (for details see text). Download figure Download PowerPoint Productiveness and Robustness through Modular Improvement of Experimental Research aspires to be lean, and it uses the language and nomenclature of the laboratory. The key processes of the QMS mirror the scientific research life cycle: experimental design, execution, evaluation, and reporting. Audits may come in the form of expert peer auditing, where two research groups using similar approaches and methods exchange their protocols and review each other to compare methodologies, check for adherence to protocol, trade best practice details, and discuss potential problems. In addition, this process fosters scientific collaboration, the development of common protocols, and transparency of the scientific process. Such peer auditing could become an important element of open science. Another form of auditing might be peer-paper auditing. An external expert group is given a selection of three recent published articles from the laboratory, of which they select one and request specific information to verify key methodology, results, and conclusions. This might include raw data availability, analysis records, study plan, power calculations, randomization table, methods to reduce bias, approval for animal experiments, and so on. Productiveness and Robustness through Modular Improvement of Experimental Research will use performance indicators such as fraction of articles published with open access and open data, number of publications and meetings with students and post docs, graduations of students, citations to publications, corrigenda or retractions, loss of data, number of reported errors, prizes and awards, citations to articles, number of trainings, and continuing education, among others. … our experience with ISO-9001 left us convinced that a structured approach to QM has enormous potential to improve the quality of research without stifling creativity if done properly. To eliminate the hurdle of having to set up a full-blown QMS before the staff has any experience with it, and to make it adjustable to group, departmental or institutional size, PREMIER will be designed for stepwise implementation. Figure 2 illustrates the process of implementation as a board game. It starts with initiating the process, defining a policy and governance structure, and evaluating the available resources. Step 2 concerns setting up a proper document management system, followed by a communication structure. Next group, departmental and institutional resources have to be assessed (3): What scientific devices need to be managed (PCR machines, microscopes, etc.), how is the laboratory structured (who does what?), and what kind of protocols, standard operating procedures exist already. This is followed by an assessment of the legal requirements (4) of the institutional environment (animal regulations, work with genetically modified organisms, etc.), and finally, the key processes are defined (5), which are specific to the particular research environment. Figure 2.Setting up ofPREMIER in a stepwise fashion (for details see text). Download figure Download PowerPoint As any QMS, PREMIER requires infra-structure and resources. For example, a central data/document storage solution will be mandatory, and staff members will need to devote time to specific tasks. An electronic lab book, which is not mandatory, would help to facilitate many QM-related activities, such as data storage, device management, working with standard operating procedures and protocols, but it may involve licensing fees. Designing PREMIER and making its elements available to interested laboratories is in full swing. We hope to be able to provide a first version by the end of 2019. The current need for QMS fit for preclinical research is further exemplified by the EQIPD (European Quality in Preclinical Data) consortium. EQIPD (www.eqipd.org) is funded by the EU's Innovative Medicines Initiative and brings together 29 partners from industry and academia. Like PREMIER, its goal is to establish a science-driven, flexible quality system that will support a set of research guidelines for industry and academia. There is intense cooperation between both initiatives. We anticipate that the resulting QMS will be complementary and increase the variety of QMS available and useful to academic researchers. Conclusions A structured approach to quality management has great potential to improve the rigor and reproducibility, and ultimately value, of basic and translational biomedical research, without curtailing academic freedom. QMS have already been successful in a number of different settings, including clinical medicine, where quality problems were prevalent. Currently, no QMS tailored to the idiosyncrasies of the research process, work organization, and personnel structure of academic biomedical research are available, and the skepticism of scientists toward managing quality is based on prejudice rather than on personal experience. Importantly, once science-driven systems will become available, they need to demonstrate that they can live up to the promise and improve the robustness and value of research with a favorable cost–benefit ratio. We strongly believe that it is worth a try and that universities, research organizations, and funders are well advised to spend a small percentage of their budget on quality management, as well as incentivize quality of research on the individual level 10. It will pay off. … once science-driven systems will become available, they need to demonstrate that they can live up to the promise and improve the robustness and value of research… Acknowledgements This study was funded by the Volkswagen Foundation and the Innovative Medicines Initiative 2 Joint Undertaking (IMI) under grant agreement No 777364. Conflict of interest The authors declare that they have no conflict of interest.