Expert Disagreement Research Articles

Psychometric properties of the tool to measure parenting self-efficacy (TOPSE) for Iranian parents of children with autism spectrum disorder

Objectives This study aims to evaluate the psychometric properties of the Persian version of the Tool to Measure Parenting Self-Efficacy (TOPSE) for Iranian parents of children with autism spectrum disorder (ASD). Methods The TOPSE was translated into Persian, and face validity was assessed through experts’ and parents’ feedback. Content validity was evaluated using the Content Validity Index (CVI) and Content Validity Ratio (CVR). Reliability was examined among 100 (n = 100) parents by assessing internal consistency with Cronbach’s alpha and test-retest reliability using Intraclass Correlation Coefficients (ICC). Results The Persian TOPSE demonstrated strong face validity, with a scale-level face validity index (S-FVI/AVE) of 0.98. Overall content validity was strong, with a scale-level content validity index (S-CVI) of 0.98, while CVR values of 0.60 for eight items indicated some expert disagreement. Reliability analysis revealed high internal consistency, with Cronbach’s alpha ranging from 0.862 to 0.938 for subscales and 0.964 for the total scale. Test-retest reliability indicated excellent stability, with ICC values ranging from 0.950 to 0.990. Conclusions The Persian TOPSE is a valid and reliable instrument for assessing parental self-efficacy (PSE) among Iranian parents of children with ASD.

Experts—Part II: The Sources of Epistemic Authority

AbstractThis paper investigates the topic of epistemic authority from the perspective of the ordinary people facing expert testimony. In particular, two central questions are discussed: how one should respond to expert testimony; and what should one do before expert disagreement.

Updating on the evidence of others

One often learns the opinions of others without getting to hear the evidence behind them. How should you revise your own opinions in such cases? Dietrich (2010) shows that, for opinions about objective chance, the method known as upco effectively adds your interlocutor’s evidence to your own. We provide a simple way of viewing upco that makes properties like Dietrich’s easy to appreciate, and we do three things with it. First, we unify Dietrich’s motivation for upco with another motivation due to Easwaran et al. (2016). Second, we show that laypeople can sometimes use upco to resolve expert disagreements. And third, we use it to cricitize the social argument for the uniqueness thesis.

Delphi method applicability in drug foresight

BackgroundThe aim of the current study was to assess the accuracy of expert predictions, which were derived using a Delphi panel foresight study between 2009 and 2011, on a variety of drug-related topics in Finland in 2020.MethodsThe material used to evaluate the accuracy of the predictions consists of published reports on statistics, survey results, official register data, wastewater analyses and official documents. Whenever possible, we used multiple information sources to ascertain possible changes related to the predictions.ResultsBetween 2009 and 2011, the majority – but not all – of the experts accurately predicted an increase in drug use. Indeed, more people experimented with or used drugs, and more drug residues were found in wastewater monitoring. The experts also correctly predicted an increase in population-level approval of drug use, but this development has been rather slow. Contrary to predictions, there was no marked increase in the use of new synthetic drugs. However, the misuse of buprenorphine increased during the 2010s. In the drug market, unit prices were surprisingly stable over the ten-year period. There were no changes in legislation related to the legal status of drugs, as was foreseen by the experts. However, enforcement moved in the direction foreseen by the experts: more lenient measures have been taken against users. Drug care system reforms favored a combination of mental health and addiction care units between 2009 and 2011, and 2020, as foreseen by the experts.ConclusionsIt seems to have been easier for the experts to foresee the continuation of existing trends, e.g., increasing use of drugs or widening approval of drugs, than to predict possible changes in the popularity of distinct groups of drugs such as new psychoactive substances (NPS). Even armed with the prediction that drug imports and wholesale would increasingly fall into the domain of organized crime, this undesirable development could not be stopped. Expert disagreement can also be seen as a valuable indication of uncertainty regarding the future. Foresight related to drug-related issues can produce relatively accurate and realistic views of the future at least up to ten years ahead.

Algorithmic Decision-Making in Difficult Scenarios

We present an approach to algorithmic decision-making that emulates key facets of human decision-making, particularly in scenarios marked by expert disagreement and ambiguity. Our system employs a case-based reasoning framework, integrating learned experiences, contextual factors, probabilistic reasoning, domain-specific knowledge, and the personal traits of decision-makers. A primary aim of the system is to articulate algorithmic decision-making as a human-comprehensible reasoning process, complete with justifications for selected actions.

Faculty’s perception of their role as a tutor during Problem-Based Learning activity in undergraduate medical education

Introduction: The study aimed to ascertain how the faculty at the Faculty of Medicine, Hasanuddin University perceived their role as a tutor during a problem-based learning activity during the academic phase of medical education, based on the length of time they acted as a tutor. Methods: This was prospective observational research with an explanatory sequential mixed-method design, which was performed at the Undergraduate Medical Study Program, Faculty of Medicine, Hasanuddin University, from January 2023 until May 2023. Research subjects were divided into two groups: a) the Novice group and b) the Expert group. Quantitative data were collected by giving a questionnaire containing six categories with 35 questions and distributed by Google form. An independent t-test was used to compare the faculty’s perception, with a p-value <.05 significant. Followed by Focus Group Discussion (FGD) for qualitative data, which then were analysed by thematic analysis. The last stage is integrating quantitative and qualitative data. Results: There were statistically significant differences in seven issues between the two groups. Most of the tutors in both groups had favorable opinions, except for the expert group's disagreement with the passive role of the tutor in the tutorial group. Eight positive and twelve negative perceptions were found in the FGD. Conclusion: Most tutors positively perceived their role in PBL, with the expert group having more dependable opinions and well-reasoned suggestions. Keywords: Problem-Based Learning, Undergraduate Medical Education, Focus Group Discussion

EXPERTISE, DISAGREEMENT, and TRUST IN VACCINE SCIENCE AND POLICY. The importance of transparency in a world of experts.

We discuss the relationship between expertise, expert authority, and trust in the case of vaccine research and policy, with a particular focus on COVID-19 vaccines. We argue that expert authority is not merely an epistemic notion, but entails being trusted by the relevant public and is valuable if it is accompanied by expert trustworthiness. Trustworthiness requires, among other things, being transparent, acknowledging uncertainty and expert disagreement (e.g., around vaccines' effectiveness and safety), being willing to revise views in response to new evidence, and being clear about the values that underpin expert recommendations. We explore how failure to acknowledge expert disagreement and uncertainty can undermine trust in vaccination and public health experts, using expert recommendations around COVID-19 vaccines as a case study.

Robust detection of marine life with label-free image feature learning and probability calibration

Advances in in situ marine life imaging have significantly increased the size and quality of available datasets, but automatic image analysis has not kept pace. Machine learning has shown promise for image processing, but its effectiveness is limited by several open challenges: the requirement for large expert-labeled training datasets, disagreement among experts, under-representation of various species and unreliable or overconfident predictions. To overcome these obstacles for automated underwater imaging, we combine and test recent developments in deep classifier networks and self-supervised feature learning. We use unlabeled images for pretraining deep neural networks to extract task-relevant image features, allowing learning algorithms to cope with scarcity in expert labels, and carefully evaluate performance in subsequent label-based tasks. Performance on rare classes is improved by applying data rebalancing together with a Bayesian correction to avoid biasing inferred in situ class frequencies. A divergence-based loss allows training on multiple, conflicting labels for the same image, leading to better estimates of uncertainty which we quantify with a novel accuracy measure. Together, these techniques can reduce the required label counts ∼100-fold while maintaining the accuracy of standard supervised training, shorten training time, cope with expert disagreement and reduce overconfidence.

Why the Heck Would You Do Philosophy?

Philosophy plausibly aims at knowledge; it would thus be tempting to hold that much of the value of doing philosophy turns on securing knowledge. Enter the agnostic challenge: suppose that a philosophical agnostic (named 'Betsy‘) wants to discover only fundamental philosophical truths. However, the intractable disagreement among philosophical experts gives her pause. After reflecting on expert disagreement, she decides that doing philosophy, for her truth-seeking error-avoiding purposes, is irrational. In this paper, I argue that the agnostic challenge isn‘t easily overcome. Although there are many reasons to do philosophy, the agnostic challenge implies there is less value to doing philosophy than many philosophers may have believed.

Information and deliberation in the Covid-19 crisis and in the climate crisis: how expertocratic practices undermine self-government and compliance

At the beginning of the Covid-19 pandemic, democracy’s promise to enable well-informed, responsible decisions gained almost unprecedented appeal. At this stage, many European governments mainly deferred to expert judgment. This is what some experts and activist groups occasionally call for in the case of an even more severe global crisis: the climate crisis. But where citizens are asked to more or less blindly follow the lead of expert judgments, politics takes what Lafont (Democracy without shortcuts: a participatory conception of deliberative democracy. Oxford University Press, Oxford, https://doi.org/10.1093/oso/9780198848189.001.0001, 2020) calls an ‘expertocratic shortcut’. In the first part of this paper, we delineate the perceptions of threat that characterize these two cases and that can lead to expertocratic temptations. We point out that shortcuts to democratic decisions not only constitute dead ends, but can also be used to reinforce existing power structures. In the second part, we show how and why such shortcuts are sociologically likely to cause alienation and reactance, as accountability is lost and the rationale for decisions cannot be retraced. We conclude that if a democratic system is to live up to its promise of rationality, legitimate expert involvement has to meet three requirements: political mandate and control, transparency of uncertainty and expert disagreement, linkage to inclusive and effective citizen deliberation.

The limits of support for differentiated integration in the European Union as perceived by academic experts

ABSTRACT Differentiated integration (DI) in the European Union has recently attracted considerable scholarly and political attention. Yet, we know rather little about where scholars’ normative support of DI begins and where it ends, and whether there is scholarly consensus about which type of DI warrants support. This contribution addresses which type of DI scholars support, and which policy areas should be exempt. It explores these questions by means of a novel expert survey (n = 95). Three broad observations can be made. First, whilst support for DI is strong in the abstract, it becomes much weaker when empirically applied. Second, the high levels of support are not necessarily in tune with the perceived risks of DI. Third, there is a fair amount of expert disagreement around DI. We defend the view that the type of disagreement we see in the findings is valid and substantively relevant because it highlights genuine diffusion (as opposed to conceptual confusion) in the distribution of preferences among experts that has previously been largely obscured. The article thereby also makes a contribution to the literature on expert surveys, discussing the distinction between benchmarking and non-benchmarking expert surveys, and the legitimacy of expert disagreement.

Characterisation and analysis of key studies used to restrict substances under REACH

BackgroundUnderstanding how scientific studies are used in regulatory risk assessments is important since it influences the outcome of an assessment, and thus the level of protection of human health and the environment. Within the REACH legislation (Registration, Evaluation, Authorisation and Restriction of Chemicals, EC Nr. 1907/2006) hazard information on clearly defined (eco-)toxicological endpoints are submitted for the registration of substances, and this information is intended for concluding on potential hazards and risk as well as subsequent risk management measures such as restrictions. The present study aimed to (1) characterise key studies used by the European Chemicals Agency’s Committee for Risk Assessment to restrict hazardous substances; (2) analyse if the REACH registration database provided the key studies used in these restrictions, and (3) investigate potential expert disagreements related to the use of non-standard studies in the restrictions.ResultsOur analysis showed that 58% of the 53 scrutinised key studies were non-standard studies, all available by paying a fee or through open access. Sixteen (30%) of the key studies were consulted from external sources outside the REACH registration database by the Committee for Risk Assessment. Only one study of the 16 external key studies was a standard study. Further, 9% (5/53) of the key studies used by the Committee for Risk Assessment were inaccessible to third parties, all were standard studies. The uses of non-standard studies were (unsuccessfully) challenged for five substances during the public consultation.ConclusionsThese results suggest that non-standard studies contributed to the identification and management of substances of concern, that the REACH registration database may not be sufficient for the identification and management of uncontrolled hazards and risks, and that the transparency of the decisions made by the Committee for Risk Assessment was partially hampered due to the use of standard studies inaccessible to third parties.

Searching for Similarity

Throughout the twentieth and twenty-first centuries regulations on expert testimony have sought to minimize the impact of disagreeing experts. Yet, disagreements between forensic musicologists still play a large role in contemporary music copyright decisions. This paper suggests that the disagreement between partisan experts is due, in part, to confirmation bias rather than ethical or financial allegiance. An expert hired by a plaintiff, or the party alleging copyright infringement, may start their analysis by searching for similarities between two works. On the other hand, an expert retained by the defendant, or the party denying infringement, may start their analysis by searching for differences. Given the multiple musical components present in even the “simplest” musical work, both starting points will lead to valid observations about the work, allowing for expert disagreement. This article uses a hypothetical case study to demonstrate the risk of confirmation bias in forensic musicology and concludes by proposing that appointing a panel of third-party musicologists to conduct forensic analyses from a neutral starting point could minimize the effect of confirmation bias in such cases.

Deep Quantile Regression for Uncertainty Estimation in Unsupervised and Supervised Lesion Detection

Despite impressive state-of-the-art performance on a wide variety of machine learning tasks in multiple applications, deep learning methods can produce over-confident predictions, particularly with limited training data. Therefore, quantifying uncertainty is particularly important in critical applications such as lesion detection and clinical diagnosis, where a realistic assessment of uncertainty is essential in determining surgical margins, disease status and appropriate treatment. In this work, we propose a novel approach that uses quantile regression for quantifying aleatoric uncertainty in both supervised and unsupervised lesion detection problems. The resulting confidence intervals can be used for lesion detection and segmentation. In the unsupervised settings, we combine quantile regression with the Variational AutoEncoder (VAE). The VAE is trained on lesion-free data, so when presented with an image with a lesion, it tends to reconstruct a lesion-free version of the image. To detect the lesion, we then compare the input (lesion) and output (lesion-free) images. Here we address the problem of quantifying uncertainty in the images that are reconstructed by the VAE as the basis for principled outlier or lesion detection. The VAE models the output as a conditionally independent Gaussian characterized by its mean and variance. Unfortunately, joint optimization of both mean and variance in the VAE leads to the well-known problem of shrinkage or underestimation of variance. Here we describe an alternative Quantile-Regression VAE (QR-VAE) that avoids this variance shrinkage problem by directly estimating conditional quantiles for the input image. Using the estimated quantiles, we compute the conditional mean and variance for input image from which we then detect outliers by thresholding at a false-discovery-rate corrected p-value. In the supervised setting, we develop binary quantile regression (BQR) for the supervised lesion segmentation task. We show how BQR can be used to capture uncertainty in lesion boundaries in a manner that characterizes expert disagreement.

Public justification and expert disagreement over non-pharmaceutical interventions for the COVID-19 pandemic

A wide range of non-pharmaceutical interventions (NPIs) have been introduced to stop or slow down the COVID-19 pandemic. Examples include school closures, environmental cleaning and disinfection, mask mandates, restrictions on...

The Precautionary Principle and Expert Disagreement

The Precautionary Principle is typically construed as a conservative decision rule aimed at preventing harm. But Martin Peterson (JME 33: 5–10, 2007; The ethics of technology: A geometric analysis of five moral principles, Oxford University Press, Oxford, 2017) has argued that the principle is better understood as an epistemic rule, guiding decision-makers in forming beliefs rather than choosing among possible acts. On the epistemic view, he claims there is a principle concerning expert disagreement underlying precautionary-based reasoning called the ecumenical principle: all expert views should be considered in a precautionary appraisal, not just those that are the most prominent or influential. In articulating the doxastic commitments of decision-makers under this constraint, Peterson precludes any probabilistic rule that might result in combining expert opinions. For combined or consensus probabilities are likely to provide decision-makers with information that is more precise than warranted. Contra Peterson, I argue that upon adopting a broader conception of probability, there is a probabilistic rule, under which expert opinions are combined, that is immune to his criticism and better represents the ecumenical principle.

Improve performance and robustness of knowledge-based FUZZY LOGIC habitat models

Previous criticisms of knowledge-based fuzzy logic modelling have identified some of its limitations and revealed weaknesses regarding the development of fuzzy sets, the integration of expert knowledge, and the outcomes of different defuzzification processes.We show here how expert disagreement and fuzzy logic mechanisms associated with the rule development and combinations can positively or adversely affect model performance and the interpretation of results. We highlight how expert disagreement can induce uncertainty into model outputs when defining fuzzy sets and selecting a defuzzification method. We present a framework to account for sources of error and bias and improve the performance and robustness of knowledge-based fuzzy logic models. We recommend to 1) provide clear/unambiguous instructions on model development, processes and objectives, including the definition of input variables and fuzzy sets, 2) incorporate the disagreement among experts into the analysis, 3) increase the use of short rules and the OR operator to reduce complexity, and 4) improve model performance and robustness by using narrow fuzzy sets for extreme values of input variables to expand the universe of discourse adequately. Our framework is focused on fuzzy logic models but can be applied to all knowledge-based models that require expert judgment, including expert systems, decision trees and (fuzzy) Bayesian inference systems.

Deep limitations? Examining expert disagreement over deep learning

We investigate expert disagreement over the potential and limitations of deep learning. We conducted 25 expert interviews to reveal the reasons and arguments that underlie the disagreement about the limitations of deep learning, here evaluated in respect to high-level machine intelligence. Experts in our sample named 40 limitations of deep learning. Using interview data, we identify and explore five crucial, unresolved research subjects that underpin this scholarly disagreement: abstraction, generalisation, explanatory models, emergence of planning and intervention. We suggest that such origins of disagreement can be used to form a research road map to guide efforts towards overcoming the limitations of deep learning.

Expert Panel Survey to Update the American Congress of Rehabilitation Medicine Definition of Mild Traumatic Brain Injury

ObjectiveAs part of an initiative led by the Brain Injury Special Interest Group Mild Traumatic Brain Injury (TBI) Task Force of the American Congress of Rehabilitation Medicine (ACRM) to update the 1993 ACRM definition of mild TBI, the present study aimed to characterize current expert opinion on diagnostic considerations. DesignCross-sectional web-based survey. SettingNot applicable. ParticipantsAn international, interdisciplinary group of clinician-scientists (N=31) with expertise in mild TBI completed the survey by invitation between May and July 2019 (100% completion rate). InterventionsNot applicable. Main Outcome MeasuresRatings of agreement with statements related to the diagnosis of mild TBI and ratings of the importance of various clinical signs, symptoms, test findings, and contextual factors for increasing the likelihood that the individual sustained a mild TBI, on a scale ranging from 1 (“not at all important”) to 10 (“extremely important”). ResultsMen (n=25; 81%) and Americans (n=21; 68%) were over-represented in the sample. The survey revealed areas of expert agreement (eg, acute symptoms are diagnostically useful) and disagreement (eg, whether mild TBI with abnormal structural neuroimaging should be considered the same diagnostic entity as “concussion”). Observable signs were generally rated as more diagnostically important than subjective symptoms (Wilcoxon signed ranks test, Z=3.77; P<.001; r=0.68). Diagnostic importance ratings for individual symptoms varied widely, with some common postconcussion symptoms (eg, fatigue) rated as unhelpful (<75% of respondents indicated at least 5 out of 10 importance). Certain acute test findings (eg, cognitive and balance impairments) and contextual factors (eg, absence of confounds) were consistently rated as highly important for increasing the likelihood of a mild TBI diagnosis (≥75% of respondents indicated at least 7 out of 10). ConclusionsThe expert survey findings identified several potential revisions to consider when updating the ACRM mild TBI definition, including preferentially weighing observable signs in a probabilistic framework, incorporating symptoms and test findings, and adding differential diagnosis considerations.

Dealing with disagreement: The roles of topic familiarity and disagreement explanation in evaluation of conflicting expert claims and sources

Learners may increasingly encounter conflicting expert reports. However, little is known about how they deal with this challenge. We examined how learners' familiarity with a controversial historical topic affects their epistemic judgments of conflicting expert claims and sources, the interplay of their claim and source evaluation strategies, and their meta-epistemic understanding of the legitimacy of the disagreement (absolutist, multiplist, and evaluativist perspectives). In two studies, topic familiarity increased agreement with belief-consistent expert claims and the perceived trustworthiness of the expert who presented these claims. Topic familiarity also impacted the coordination of evaluation strategies and led to greater reliance on knowledge-based validation. However, topic familiarity did not affect meta-epistemic understanding of the legitimacy of the controversy. In the second study, reading an explanation about reasons for disagreements between historians resulted in higher evaluativism. Teaching about expert disagreement may be a productive approach for promoting appreciation of the diversity of knowledge.