Scientific Standards Of Evidence Research Articles

Relevance of learning health systems to physiatrists and its synergy with implementation science: A commentary.

As health care attempts to bridge the gap between evidence and practice, the concept of the learning health system (LHS) is becoming increasingly relevant. LHS integrates evidence with health systems data, driving health care quality and outcomes through updates in policy, practice, and care delivery. In addition, LHS research is becoming critically important as there are several initiatives underway to increase research capacity, expertise, and implementation, including attempts to stimulate increasing numbers of LHS researchers. Physical Medicine & Rehabilitation (PM&R) physicians (physiatrists), nurses, therapists (physical therapists, occupational therapists, speech therapists, clinical psychologists), and scientists are affiliated with LHSs. As LHS research expands in health care systems, better awareness and understanding of LHSs and LHS research competencies are key for rehabilitation professionals including physiatrists. To address this need, the Agency of Healthcare Research and Quality (AHRQ) identified 33 core competencies, grouped into eight domains, for training LHS researchers. The domains are: (1) Systems Science; (2) Research Questions and Standards of Scientific Evidence; (3) Research Methods; (4) Informatics; (5) Ethics of Research and Implementation in Health Systems; (6) Improvement and Implementation Science; (7) Engagement, Leadership, and Research Management; and the recently added (8) Health and Healthcare Equity and Justice. The purpose of this commentary is to define LHS and its relevance to physiatrists, present the role of implementation science (IS) in LHSs and application of IS principles to design LHSs, illustrate current LHS research in rehabilitation, and discuss potential solutions to improve awareness and to stimulate interest in LHS research and IS among physiatrists in LHSs.

Development of a learning health system science competency assessment to guide training and proficiency assessment.

IntroductionLearning health systems (LHS) science is fundamentally a transdisciplinary field. To capture the breadth of the competencies of an LHS scientist, AHRQ and national experts defined a series of 42 competencies across seven domains that support success. Clinicians, researchers, and leaders who are new to the LHS field can identify and prioritize proficiency development among these domains. In addition, existing leaders and researchers will assemble teams of experts who together represent the LHS science domains. To serve LHS workforce development and proficiency assessment, the AHRQ‐funded ACCELERAT K12 training program recruited domain experts and trainees to define and operationalize items to include in an LHS Competency Assessment to support emerging and existing LHS scientists in prioritizing and monitoring proficiency development.MethodsSequential interviews with 18 experts iteratively defined skills and tasks to illustrate stage in proficiency, and its progression, for each of 42 competencies in the seven LHS expertise domains: systems science; research questions and standards of scientific evidence; research methods; informatics; ethics of research and implementation in health systems; improvement and implementation science; and engagement, leadership, and research management. An educational assessment expert and LHS scientist refined the assessment criteria at each stage to use parallel language across domains. Last, current trainees reviewed and pilot tested the assessment and the LHS Competency Assessment was further refined using their feedback. The assessment framework was informed by Bloom's revised taxonomy of educational objectives (Anderson and Krathwohl, A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives, 2001) where learning progresses from recalling, defining, understanding, and awareness at the lower levels of the taxonomy, to applying and adopting and finally to creating, designing, and critiquing at the upper levels of the taxonomy. We also developed assessment criteria that could be used for longer term assessment of direct performance. Van der Vleuten et al. (Best Pract Res Clin Obstetr Gynaecol. 2010;24(6):703‐719) define longer term direct assessment methods as assessment that occurs over a period ranging from weeks to even years and involves multiple assessment methods and exposure to the learner's work over an extended period.ResultsThis experience report describes the content of the LHS Competency Assessment. For each domain and competency, the assessment lists examples of evidence to support expertise at each level of proficiency: no exposure; foundational (awareness/understanding); emerging (early application); and proficient (application with a high level of skill). Trainees begin with baseline standard assessment tables, where they can indicate no exposure or mark the foundational and emerging skills with which they have competence. For domains where foundational and emerging skills have been achieved, users can move on to assessment tables that list evidence of proficiency.ConclusionThe LHS Competency Assessment offers consistent, graded criteria across the seven LHS domains to guide trainees and mentors to evaluate progress from no experience to foundational knowledge, emerging proficiency, and proficiency. The assessment can also be used to design training and mentoring for those newly exposed to LHS science and for those with key expertise who wish to expand LHS expertise.

Proposal for Life Course Intervention Researcher Core Competencies.

Proposal for Life Course Intervention Researcher Core Competencies.

Standards of Scientific Evidence in Preferential Trade Agreements

Standards of Scientific Evidence in Preferential Trade Agreements

Development of the Learning Health System Researcher Core Competencies.

To develop core competencies for learning health system (LHS) researchers to guide the development of training programs. Data were obtained from literature review, expert interviews, a modified Delphi process, and consensus development meetings. The competencies were developed from August to December 2016 using qualitative methods. The literature review formed the basis for the initial draft of a competency domain framework. Key informant semi-structured interviews, a modified Delphi survey, and three expert panel (n=19 members) consensus development meetings produced the final set of competencies. The iterative development process yielded seven competency domains: (1) systems science; (2) research questions and standards of scientific evidence; (3) research methods; (4) informatics; (5) ethics of research and implementation in health systems; (6) improvement and implementation science; and (7) engagement, leadership, and research management. A total of 33 core competencies were prioritized across these seven domains. The real-world milieu of LHS research, the embeddedness of the researcher within the health system, and engagement of stakeholders are distinguishing characteristics of this emerging field. The LHS researcher core competencies can be used to guide the development of learning objectives, evaluation methods, and curricula for training programs.

Outcomes-based drug coverage in British Columbia.

For the past decade the provincial drug plan in British Columbia has based the allocation of public subsidy on scientific standards of evidence. Coverage policies under B.C. PharmaCare are marked by the restriction of public subsidy until manufacturers provide valid evidence of a comparative health outcome advantage versus therapeutic alternatives. Implementing and maintaining such outcomes-based coverage policies has required a system of evidentiary review and support. Since 1994 the Therapeutics Initiative at the University of British Columbia has provided provincial decisionmakers with such support. We describe lessons from the B.C. experience for jurisdictions interested in basing coverage on evidence of proven comparative advantage for patients' health.

UK: precautionary commercialization?

As genetically modified (GM) products approach the market stage, the UK government and agro-food industry have faced a suspicious or hostile public. Since 1998 many retail chains have undertaken to exclude any GM-derived ingredients from their own-brand lines. This commercial blockage has intensified pressures for greater precaution, even for a moratorium on cultivating GM crops. Political protest has led to strategies for precautionary commercialization. Government and industry have cooperated to plan a ‘managed development’ of GM crops. Across the agricultural supply chain, industry has devised voluntary guidelines to ensure segregation of GM crops and to limit the spread of GM herbicide-tolerance. In particular UK regulators seek to test the risk that broad-spectrum herbicide sprays could damage wildlife habitats; they have broadened the advisory expertise accordingly. These measures open up the precautionary content to further debate, at both national and EU levels. Market-stage precautions establish a means to test claims that GM crops are environmentally-friendly products. By translating public concerns into broader risk-assessment criteria, the UK procedure involves critics in potentially influencing standards of scientific evidence and environmental harm. This social process has become a prerequisite for legitimizing commercial use.

RISK MANAGEMENT AND EXPERTISE: UK: Strategies for Precautionary Commercialization of GM Crops

As genetically modified (GM) products approach the market stage, the UK government and agro-food industry have faced a suspicious or hostile public. Since 1998 many retail chains have undertaken to exclude any GM-derived ingredients from their own-brand lines. This commercial blockage has intensified pressures for greater precaution, even for a moratorium on cultivating GM crops. Political protest has led to strategies for precautionary commercialization. Government and industry have cooperated to plan a “managed development” of GM crops. Across the agricultural supply chain, industry has devised voluntary guidelines to ensure segregation of GM crops and to limit the spread of GM herbicide-tolerance. In particular UK regulators seek to test the risk that broad-spectrum herbicide sprays could damage wildlife habitats; they have broadened the advisory expertise accordingly. These measures open up the precautionary content to further debate, at both national and EU levels. Market-stage precautions establish a means to test claims that GM crops are environmentally-friendly products. By translating public concerns into broader risk-assessment criteria, the UK procedure involves critics in potentially influencing standards of scientific evidence and environmental harm. This social process has become a prerequisite for legitimizing commercial use.

Interactionism, flexibility, and inferences about the past.

chology. Clearly, such connections haveserved psychology very well, helping us, forexample, to understand and treat broadspectrum of mental diseases, as well as re-ducing the stigma previously associated withthese diseases. Although the contributionsthat the physical sciences can make to thesocial sciences should be investigated, Busset al. went step beyond science and present-ed philosophically committed view of theworld, promoting an agenda that is not withinthe realm of the scientific method. Specifical-ly, Buss et al. put the cart before the horsewhen they said that exaptations and span-drels provide a promising theoretical per-spective when of the exaptations andspandrels (borrowed from Gould, 1991) thatthey cited have met scientific standards ofevidence. The consequences of their philo-sophical presuppositions are reductionisticand call into question much of what th e Amer-ican Psychological Association (APA, 1992)advocates in its code of ethics.Any introductory text that treats the sci-entific method present s it as characterized bya cycle of theory building, hypothesis forma-tion, data collection, empirical hypothesis test-ing, an d theoretical modification (cf. Wagner& Hollenbeck, 1998). As such, the scientificmethod requires that factual statements beverified by observation or by repeatable ex-periments (Johnson, 1995, p. 200). This isin contrast to philosophical assumptions thatcannot be proved empirically and that fre-quently cannot be disproved either. Buss etal.'s (1998) article seems to be arguing for aphilosophical perspective, rather than sci-entific one. For example, they advocated us-ing the concepts of exaptations and span-drels, along with the concept of adaptation, tocreate hypotheses that may help explain psy-chological phenomenon. Yet, some ofGould's (1991) statements, which Buss et al.rely on to build their case, explain everythingfrom art to religion as exaptations; theseshould clearly be categorized as metaphysicalstatements and are not subject to disproofthrough the use of scientific methodology.Although Buss et al. (1998) presentedfive criteria for testing (p. 543) hypothesesabout exaptations and spandrels, these crite-ria cannot be used to determine whether aparticular phenomenon is actually an exapta-tion or spandrel; that is, if such hypothesisis supported, it demonstrates only that thehypothesis is internally consistent, not that itcorresponds with how particular behavioractually came about. Further, Buss et al. sug-gested that when particular hypothesisabout an evolved mechanism fails to besupported empirically, then number of op-tions are available to researchers (p. 543),and yet of these alternatives involverejecting the notion that exaptations and span-drels are useful theoretical heuristic. This isparticularly disturbing because, as the au-thors acknowledge, none of the items onGould's (1991) list of proposed spandrelsand exaptations... have met these standardsof evidence (p. 546). In other words, we arebeing asked to accept that an idea that doesnot yet have any support in biology might beuseful in the psychological sciences. Howev-er, if and when hypothesis based on such anidea proves false, we should not question theidea on which it is based. This is clearlyspeculative philosophy of questionable qual-ity—not science.A second primary concern with thisarticle relates to the consequences of using anunproved evolutionary psychology perspec-tive in explaining human behavior. Viewingall human actions as genetically programmedexpressions of self-interest has important ram-ifications for individual and organizationalethics. Many fundamental moral concepts,such as those promoted by the APA (1992)code of ethics, would be rendered meaning-less. Moral standard s and judgment rest onconcept such freedom of action and re-sponsibility for choices made. When all ac-tion s ar e viewed th product of un-consciou s an unguide force of naturalselection, moral culpabilit y is clearl jetti-soned becaus e all actions are, to larg de-gree, preprogrammed into the brain.In addition, the implications of explain-ing everything from religion to art throughthe lens of exaptations and spandrels under-mines morality in an even more encompass-ing way. Specifically, such view of theworld has no basis on which to offer norma-tive definitions of good (vs. evil) actionsbecause these are concepts requiring inputand authority from sources transcending na-ture. In other words, evolutionary psycholo-gy offers no basis for the judgment that thestandards stated in the APA (1992) code ofethics, such as honesty, respect for others,and professional responsibility, are superiorto duplicity, treachery, and shiftlessness. Canthe adoption of standards be adequately ex-plained through organizational survival? Al-though it is true that principles such as integritycan contribute to an organization's flourish-ing, that codes of ethics are necessary makesit clear that dishonesty can also pay. If thelatter is proven to be the case, should mem-bers of the APA adopt such code as merespandrel?We do believe that investigations of theintersections of the natural sciences and so-cial sciences are valuable and should contin-ue. However, our basic concerns with Busset al.'s (1998) article relate to their lack ofacknowledgment that they are advocating aworldview (rather than truly empirical sci-ence) and the fact that they do not address thepotential implications of their metaphysicalassumptions.

Interactionism, flexibility, and inferences about the past.

chology. Clearly, such connections haveserved psychology very well, helping us, forexample, to understand and treat a broadspectrum of mental diseases, as well as re-ducing the stigma previously associated withthese diseases. Although the contributionsthat the physical sciences can make to thesocial sciences should be investigated, Busset al. went a step beyond science and present-ed a philosophically committed view of theworld, promoting an agenda that is not withinthe realm of the scientific method. Specifical-ly, Buss et al. put the cart before the horsewhen they said that exaptations and span-drels provide "a promising theoretical per-spective" when none of the exaptations andspandrels (borrowed from Gould, 1991) thatthey cited have met scientific standards ofevidence. The consequences of their philo-sophical presuppositions are reductionisticand call into question much of what th e Amer-ican Psychological Association (APA, 1992)advocates in its code of ethics.Any introductory text that treats the sci-entific method present s it as characterized bya cycle of theory building, hypothesis forma-tion, data collection, empirical hypothesis test-ing, an d theoretical modification (cf. Wagner& Hollenbeck, 1998). As such, the scientificmethod requires that "factual statements beverified by observation or by repeatable ex-periments" (Johnson, 1995, p. 200). This isin contrast to philosophical assumptions thatcannot be proved empirically and that fre-quently cannot be disproved either. Buss etal.'s (1998) article seems to be arguing for aphilosophical perspective, rather than a sci-entific one. For example, they advocated us-ing the concepts of exaptations and span-drels, along with the concept of adaptation, tocreate hypotheses that may help explain psy-chological phenomenon. Yet, some ofGould's (1991) statements, which Buss et al.rely on to build their case, explain everythingfrom art to religion as exaptations; theseshould clearly be categorized as metaphysicalstatements and are not subject to disproofthrough the use of scientific methodology.Although Buss et al. (1998) presentedfive criteria for "testing" (p. 543) hypothesesabout exaptations and spandrels, these crite-ria cannot be used to determine whether aparticular phenomenon is actually an exapta-tion or a spandrel; that is, if such a hypothesisis supported, it demonstrates only that thehypothesis is internally consistent, not that itcorresponds with how a particular behavioractually came about. Further, Buss et al. sug-gested that when a particular hypothesis"about an evolved mechanism fails to besupported empirically, then a number of op-tions are available to researchers" (p. 543),and yet none of these alternatives involverejecting the notion that exaptations and span-drels are a useful theoretical heuristic. This isparticularly disturbing because, as the au-thors acknowledge, "none of the items onGould's (1991) list of proposed spandrelsand exaptations... have met these standardsof evidence" (p. 546). In other words, we arebeing asked to accept that an idea that doesnot yet have any support in biology might beuseful in the psychological sciences. Howev-er, if and when a hypothesis based on such anidea proves false, we should not question theidea on which it is based. This is clearlyspeculative philosophy of questionable qual-ity—not science.A second primary concern with thisarticle relates to the consequences of using anunproved evolutionary psychology perspec-tive in explaining human behavior. Viewingall human actions as genetically programmedexpressions of self-interest has important ram-ifications for individual and organizationalethics. Many fundamental moral concepts,such as those promoted by the APA (1992)code of ethics, would be rendered meaning-less. Moral standard s and judgment rest onconcept such a freedom of action and re-sponsibility for choices made. When all ac-tion s ar e viewed a th product of un-consciou s an unguide force of naturalselection, moral culpabilit y is clearl jetti-soned becaus e all actions are, to a larg de-gree, preprogrammed into the brain.In addition, the implications of explain-ing everything from religion to art throughthe lens of exaptations and spandrels under-mines morality in an even more encompass-ing way. Specifically, such a view of theworld has no basis on which to offer norma-tive definitions of good (vs. evil) actionsbecause these are concepts requiring inputand authority from sources transcending na-ture. In other words, evolutionary psycholo-gy offers no basis for the judgment that thestandards stated in the APA (1992) code ofethics, such as honesty, respect for others,and professional responsibility, are superiorto duplicity, treachery, and shiftlessness. Canthe adoption of standards be adequately ex-plained through organizational survival? Al-though it is true that principles such as integritycan contribute to an organization's flourish-ing, that codes of ethics are necessary makesit clear that dishonesty can also pay. If thelatter is proven to be the case, should mem-bers of the APA adopt such a code as a merespandrel?We do believe that investigations of theintersections of the natural sciences and so-cial sciences are valuable and should contin-ue. However, our basic concerns with Busset al.'s (1998) article relate to their lack ofacknowledgment that they are advocating aworldview (rather than truly empirical sci-ence) and the fact that they do not address thepotential implications of their metaphysicalassumptions.

Questions and Concerns about the Miller Assessment for Preschoolers

The Miller Assessment for Preschoolers (MAP) is a developmental test, a stated purpose of which is to identify children at risk for future school-related problems. In this article we call attention to ambiguities and omissions in previous reviews of three longitudinal studies that evaluated the MAP's predictive accuracy. A closer look at the original studies reveals that the test displays unacceptably high error rates in predicting members of the target population. We conclude that the MAP has not been validated for its intended purpose. Professional practitioners are urged to evaluate whether claims of test validity meet the standards of scientific evidence.

The right stuff: advanced methods in neuropsychology today.

Abstract It is highly unlikely that neuropsychologist researchers and clinicians are in much better shape today than their nonneuropsychological scientific contemporaries in terms of the constraints on their progress. That is, merely having the brain and the mind as an investigational focus does not provide us with any particular freedom from the need to gauge our advances carefully against realistic scientific standards of evidence. It would be comforting, but thoroughly misleading, to actually believe Public Television documentaries that tell us how wonderful we have become.

POLICY PROBLEMS ASSOCIATED WITH WATERBORNE ASBESTOS1

ABSTRACT: This article discusses the implications of accepting various standards of scientific evidence in disputes arising over the dumping of potentially hazardous materials into public waterways and drinking water supplies. Standards based on the types of risks and levels of benefits suffered and received by adjacent populations are evaluated within the context of the several pieces of federal legislation which relate to water quality. A case study of Reserve Mining Corporation's dumping of taconite wastes into Lake Superior is presented to illustrate the ramifications of decisions based on differing perspectives of the problem and its impacts by the courts, the affected public, RMC, and the regulatory agencies involved.