Expert Ranking Research Articles

The role of expert judgments in wine quality assessment: the mismatch between chemical, sensorial and extrinsic cues

Purpose The complexity in determining the quality of a credence good like wine increases due to the lack of mandatory ingredient labeling. This has generated a significant information asymmetry in the wine market, leading consumers to delegate their purchase decisions to expert rankings and wine guides. This paper explores whether expert assessments reduce the information asymmetry caused by the absence of ingredient labeling in the wine market.Design/methodology/approach By employing analysis of variance (ANOVA) in a sample of 304 wines included in the Wine Guide of the Spanish Consumers Organization (OCU), this paper assesses the extent to which expert assessments based on sensory evaluations converge with the objective cues provided by laboratory analysis in wine quality evaluations.Findings Results reveal a mismatch between expert assessments and laboratory analyses. Chemical aspects such as SO2 levels or volatile acidity, sensorial factors such as intensity and persistence, and extrinsic variables such as the region of origin or wine type play an important role in the quality ranking of wines.Originality/value These findings call for the inclusion of objective intrinsic cues in expert sensory assessments to provide consumers reliable information about wines and to resolve the apparent dissonances in wine quality assessments.

Development and psychometric evaluation of nurses' perception towards the gap between knowledge and practice.

AimThe aim of the study was to develop a survey instrument and psychometrically evaluate nurses’ perceptions of the gap between knowledge and practice. Although the nursing literature has widely documented a gap between knowledge and practice, no instrument has been developed to measure this gap.DesignPsychometric analysis was done on 513 nurses working in different positions at two large hospitals in Jeddah City.MethodsCronbach's alpha was used to determine the internal consistency and reliability of the research instrument, and a panel of experts evaluated the validity of the tool's content.ResultThe content validity index computed from expert rankings was 0.89. Factor analysis yielded four major components: knowledge, practice, environment and learning. Cronbach's alpha indicated a high level of internal consistency and reliability for the component items. The newly developed scale will facilitate measuring nurses’ perceptions of the knowledge–practice gap in clinical settings.

A novel maturity index for assessing medical device startups.

Startup companies in the healthcare sector often fail because they lack sufficient entrepreneurial, regulatory, and business development expertise. Maturity models provide useful frameworks to assess the state of business elements more systematically than heuristic assessments. However, previous models were developed primarily to characterize the business state of larger nonmedical companies. A maturity index designed specifically for startup companies in the medical product sector could help to identify areas in which targeted interventions could assist business development. A novel MedTech Startup Maturity Index (SMI) was developed by a collaborative team of academic and industry experts and refined through feedback from external stakeholders. Pediatric medical device startups associated with the West Coast Consortium for Technology & Innovation in Pediatrics (CTIP) were scored and ranked according to the SMI following semi-structured interviews. The CTIP executive team independently ranked the maturity of each company based on their extensive experiences with the same companies. SMI scores for 16 companies ranged from 1.2 to 3.8 out of 4. These scores were well aligned with heuristic CTIP rankings for 14 out of 16 companies, reflected by strong correlations between the two datasets (Spearman's rho = 0.721, P = 0.002, and Kendall's tau-b = 0.526, P = 0.006). The SMI yields maturity scores that correlate well with expert rankings but can be assessed without prior company knowledge and can identify specific areas of concern more systematically. Further research is required to generalize and validate the SMI as a pre-/post-evaluation tool.

ОЦЕНКА ВЕСОМОСТИ ПОКАЗАТЕЛЕЙ ТЕХНИЧЕСКОГО СОСТОЯНИЯ ЗДАНИЙ И СООРУЖЕНИЙ

The article presents the author's approach to determining the weight of indicators of the technical condition of buildings and structures, which allows solving the problem of increasing the reliability and effectiveness of assessing their real readiness for use for their intended purpose. The implementation of this approach is based on the experience of solving practical problems in assessing the technical condition of buildings and structures, when the weight of their indicators cannot be directly measured. In this case, it is proposed to use expert ranking methods to determine it and direct evaluation. The effectiveness of their use largely depends on it depends on the competence of evaluating experts and the consistency of their opinions on the issues under study, which are established by determining the coefficients of awareness and argumentation of experts and step-by-step solution of research tasks based on the use of the rank correlation method. A distinctive aspect of the described approach to assessing the weight of indicators of the technical condition of buildings and structures is the establishment of a criterion for determining their importance for solving the tasks set, as well as uniform rules for measuring them. The proposed procedure for assessing the weight of the investigated problems has been successfully tested during the survey of protective structures of the executive authorities of the Russian Federation. It can be recommended to determine a reliable assessment of the technical condition of buildings and structures for various purposes.

Follow-up Interactive Long-Term Expert Ranking (FILTER): a crowdsourcing platform to adjudicate risk for survivorship care.

ObjectivesTo develop an online crowdsourcing platform where oncologists and other survivorship experts can adjudicate risk for complications in follow-up.Materials and MethodsThis platform, called Follow-up Interactive Long-Term Expert Ranking (FILTER), prompts participants to adjudicate risk between each of a series of pairs of synthetic cases. The Elo ranking algorithm is used to assign relative risk to each synthetic case.ResultsThe FILTER application is currently live and implemented as a web application deployed on the cloud.DiscussionWhile guidelines for following cancer survivors exist, refinement of survivorship care based on risk for complications after active treatment could improve both allocation of resources and individual outcomes in long-term follow-up.ConclusionFILTER provides a means for a large number of experts to adjudicate risk for survivorship complications with a low barrier of entry.

Flood depth-damage and fragility functions derived with structured expert judgment

Depth-damage relationships of residential typologies have an important bearing on the reliability of flood risk assessments. Yet, the information for creating adequate functions is frequently missing in places where they are needed the most: developing countries where floods inflict severe losses. To overcome the lack of site-specific functions, modelers usually adopt functions from foreign typologies, extrapolate from limited flood damage data, and conduct expert-elicitation surveys. The latter, although not exempt from challenges, are a helpful resource in situations in which there are sufficient experts available. The classical model is a structured expert judgment technique which is one of the most used because it minimizes the number of subjective elements in the ranking of experts’ experience and its results can be compared with similar studies. This paper presents depth-damage and fragility functions developed with the classical model conducted with experts with extensive knowledge in the flood damage mechanism of typologies in northeast Argentina—a region under a severe flood hazard. The depth-damage functions were partially validated with damage data and exhibited good agreement, also they are consistent with the fragility functions. Hence, these functions can be helpful for probabilistic flood risk assessments, provided that the underlying assumptions are deemed acceptable.

A collaborative experiential problem-solving approach to develop shams for complex physical interventions: a case study of dry needling

ObjectiveComplex physical interventions are widely used in physiotherapy, despite doubts over the validity of clinical trial results due to lack of credible shams. Credible shams are critically needed, so too, therefore, is a process by which they can be developed. The authors used a novel methodology to develop and test blinding protocols for dry needling, a complex physical intervention for which blinding is particularly difficult. DesignThe research design was a practical three-day workshop influenced by Participatory Action Research, which uses iteration and reflection to solve a problem. ParticipantsFive multidisciplinary experts (researchers, clinicians, technician, magician) were invited. Healthy volunteers (‘recipients’, n=17) and accredited physiotherapists (n=6) were recruited to enable testing of blinding strategies. Main outcome measuresPrimary outcomes were expert opinion on the potential to blind recipients/therapists for (1) individual blinding strategies, and (2) entire blinding protocols. Secondary outcomes included recipient/therapist blinding effectiveness and acceptability. ResultsExperts iteratively developed 11 blinding protocols involving 22 blinding strategies. Experts rated 18 of the blinding strategies to ‘definitely have potential’ and identified four categories: knowledge of the sham, clinical interaction, disinformation, and sensation. Recipient and therapist blinding became more successful as the protocols evolved. ConclusionsCredible shams capable of simultaneous recipient and therapist blinding have been regarded to be impossible in dry needling. The preliminary success of the devised protocols suggest that our novel approach may be a crucial step in sham development. Improvements in expert rankings and blinding effectiveness as the protocols progressed support the value of this workshop approach.

Development and Evaluation of Technology Enhanced Interaction Framework Method for Designing Accessible Technologies for Visually Impaired People

This research developed and evaluated a software development support method to help non-expert developers evaluating or gathering requirements and designing or evaluating digital technology solutions to accessibility barriers people with visual impairment encounter. The Technology Enhanced Interaction Framework (TEIF) Visual Impairment (VI) Method was developed through literature review and interviews with 20 students with visual impairment, 10 adults with visual impairment and five accessibility experts. It is an extension of the Technology Enhanced Interaction Framework (TEIF) and its “HI-Method” that had been developed and validated and evaluated for hearing impairment and supports other methods by providing multiple-choice questions to help identify requirements, the answers to which help provide technology suggestions that support the design stage. Four accessibility experts and three developer experts reviewed and validated the TEIF VI-Method. It was experimentally evaluated by 18 developers using the TEIF VI-Method and another 18 developers using their preferred “Other Methods” to identify the requirements and solution to a scenario involving barriers for people with visual impairment. The “Other Methods” group were then shown the TEIF VI-Method and both groups were asked their opinions of its ease of use. The mean number of correctly selected requirements was significantly higher (p < 0.001) for developers using the TEIF VI-Method (X̄ = 8.83) than the Other Method (X̄ = 6.22). Developers using the TEIF VI-Method ranked technology solutions closer to the expert rankings than developers using Other Methods (p < 0.05). All developers found the TEIF VI-Method easy to follow. Developers could evaluate requirements and technology solutions to interaction problems involving people with visual impairment using the TEIF VI-Method better than existing Other Methods. Developers could benefit from using the TEIF VI-Method when developing technology solutions to interaction problems faced by people with visual impairment.

Experts’ multiple criteria evaluations of fuel management options to reduce wildfire susceptibility. The role of closer knowledge of the local socioeconomic context

Expert opinion can be a valuable tool for informed decision making. Concerning wildfire susceptibility reduction at the landscape scale, forest ecosystem experts play a key role in offering advice about appropriate fuel management practices to be applied by forest owners or their organizations, and in shaping public policies. A literature review aimed at identifying fuel management interventions and techniques found multiple and even opposing strategies. Recognizing the interdisciplinary and multi-dimensional nature of fuel management, we go beyond existing studies on forest experts’ opinions by comparing evaluations across forest experts with diverse training and experience, and by considering different evaluation criteria such as technical effectiveness, impact on soil or biodiversity, socioeconomic impact, and preference. Following an online survey to a sample of Portuguese experts, distinct socio-professional clusters were established and experts’ evaluations associated with their views on fire, forests, owners’ coordination, and rural development. Results show that experts rank their preferences by weighing effectiveness and impacts in different ways. Closer knowledge of the local context distinguishes expert preference, favouring more active fuels reduction strategies. Since experts with a closer knowledge of socioeconomic context tend to be further from policy-making processes, we urge their more balanced participation in those processes.

Consensus statements on the information to deliver after a febrile seizure.

Febrile seizures (FS) are usually self-limiting and cause no morbidity. Nevertheless they represent very traumatic events for families. There is a need to identify key messages that reassure carers and help to prevent inappropriate, anxiety-driven behaviors associated with "fever phobia." No recommendations have been proposed to date regarding the content of such messages. Using a Delphi process, we have established a consensus regarding the information to be shared with families following a FS. Twenty physicians (child neurologists and pediatricians) from five European countries participated in a three-step Delphi process between May 2018 and October 2019. In the first step, each expert was asked to give 10 to 15 free statements about FS. In the second and third steps, statements were scored and selected according to the expert ranking of importance. A list of key messages for families has emerged from this process, which offer reassurance about FS based on epidemiology, underlying mechanisms, and the emergency management of FS should they recur. Interestingly, there was a high level of agreement between child neurologists and general pediatricians.Conclusion: We propose key messages to be communicated with families in the post-FS clinic setting. What is Known: • Febrile seizures (FS) are traumatic events for families. • No guidelines exist on what information to share with parents following a FS. What is New: • A Delphi process involving child neurologists and pediatricians provides consensual statement about information to deliver after a febrile seizure. • We propose key messages to be communicated with families in the post-FS clinic setting.

Method of using the transitive graph of a Markovian process as part of ranking of heterogeneous items

Hierarchy analysis developed by Thomas Saaty is a closed logical structure that uses simple and well-substantiated rules that allow solving multicriterial problems that include both quantitative, and qualitative factors, whereby the quantitative factors can differ in terms of their dimensionality. The method is based on problem decomposition and its representation as a hierarchical arrangement, which allows including into such hierarchy all the knowledge the decision-maker has regarding the problem at hand and subsequent processing of decision-makers’ judgements. As the result, the relative degree of the interaction between the elements of such hierarchy can be identified and later quantified. Hierarchy analysis includes the procedure of multiple judgement synthesis, criteria priority definition and rating of the compared alternatives. The method’s significant limitation consists in the requirement of coherence of pairwise comparison matrices for correct definition of the weights of compared alternatives. The Aim of the paper is to examine a non-conventional method of solving the problem of alternative ratings estimation based on their pairwise comparisons that arises in the process of expert preference analysis in various fields of research. Approaches are discussed to the generation of pairwise comparison matrices taking into consideration the problem of coherence of such matrices and expert competence estimation. Method. The methods of hierarchy analysis, models and methods of the Markovian process theory were used. Result. The paper suggested a method of using the transitive graph of a Markovian process as part of expert ranking of items of a certain parent entity subject to the competence and qualification of the experts involved in the pairwise comparison. It is proposed to use steady-state probabilities of a Markovian process as the correlation of priorities (weights) of the compared items. The paper sets forth an algorithm for constructing the final scale of comparison taking into consideration the experts’ level of competence. Conclusion. The decision procedures, in which the experts are expected to choose the best alternatives out of the allowable set, are quite frequently used in a variety of fields for the purpose of estimation and objective priority definition, etc. The described method can be applied not only for comparing items, but also for solving more complicated problems of expert group estimation, i.e., planning and management, prediction, etc. The use of the method contributes to the objectivity of analysis, when comparing alternatives, taking into consideration various aspects of their consequences, as well as the decision-maker’s attitude to such consequences. The suggested model-based approach allows the decision-maker identifying and adjusting his/her preferences and, consequently, choosing the decisions according to such preferences, avoiding logical errors in long and complex reasoning chains. This approach can be used in group decision-making, description of the procedures that compensate a specific expert’s insufficient knowledge by using information provided by the other experts.

Abwägung von Nutzen und Schädlichkeit von berauschenden und schmerzlindernden Substanzen aus der Perspektive von deutschen Suchtmedizinern

In Europe, there have been several addiction-expert rankings of harms related to the use of psychotropic substances in the last 15 years. Among them, only one expert ranking took into account the potential benefits of these drugs. Non-Opioidergic Analgesics (NOAs), such as gabapentinoids and NSAIDs, which have been increasingly the subject of abuse / misuse reports, have not been considered in such expert rankings. Likewise, there is currently no multi-substance comparison as to whether the valuation rank of the harmfulness of an illegal drug may change along with an imagined change in legal status in Germany. Using a questionnaire, 101 experienced addiction physicians (first cohort) evaluated 33 psychoactive substances including analgesics with regard to their health and social harms as well as potential usefulness for the consumer and their environment / society ('others'). In addition, this cohort investigated whether the harmfulness assessment of an illegal substance changes if it would be legalized. In order to obtain the average overall harmfulness (overall risk) of a substance, the percentage contribution of each dimension to the overall harmfulness was determined in a second survey (second cohort, 36 experienced addiction medicine experts). Finally, the average benefit and overall risk ratings of each substance were related to each other. Prescription psychoactive substances such as analgesics, NOAs (including gabapentinoids) and opioidergic maintenance medications to treat opiate dependence were judged to have a favorable benefit-harm profile. Cannabis and ketamine were placed in the midfield of both, the harm and benefit rankings. Together with most illicit narcotic drugs, alcohol and nicotine, have been ranked among the most harmful and least useful substances, whereby alcohol was judged on average to be more harmful but also more useful than nicotine. In the event of potential legalization, the overall harm of the traditional illegal drugs methamphetamine, heroin, cocaine and cannabis was estimated to be reduced. This was mainly due to a more favorable valuation of the harm to others under these virtual conditions. Prescription substances including opioidergic and non-opioidergic analgesics as well as opioid maintenance therapy medications (methadone and buprenorphine) were assigned a favorable benefit-harm profile. Alcohol, nicotine and traditional illicit drugs (with the exception of cannabis and ketamine) were determined to have an unfavorable profile. The overall harm of traditional illicit drugs was assessed to decrease along with legalization, mainly by decreasing the harm to others in this virtual event.

A review of ranking systems for new plant threats in EU Member States and Third Countries: concepts and applications

A review of existing ranking systems for plant threats developed by EU Member States and Third Countries is presented in this report. The analysis of these ranking systems is done according to six main topics: 1) purpose of the ranking, 2) characteristics of pests, 3) patterns and features of the systems, 4) structure of the systems in terms of point scales and score aggregation, 5) treatment of missing data, expression of uncertainty and validation of the system via comparison with expert ranking or judgments, and 6) possibility of running several scenarios and application of different weightings. Common features between these systems as well as their main differences are highlighted. General recommendations are put forward, especially with a view to the development of a future ranking system for new non-regulated pests.

Leveraging crowd knowledge to curate documentation for agile software industry using deep learning and expert ranking

Leveraging crowd knowledge to curate documentation for agile software industry using deep learning and expert ranking

Expert assessment of the important professional competencies of aerospace engineers

This article is dedicated to examination of the important professional competencies of aerospace engineers and specificity of their professional activity. The study represents an expert assessment of the activity and important professional competencies of aerospace engineers. The research program consisted of several stages. The first stage includes the formation of the relevant expert group and assessment of the characteristics of activity and important professional competencies based on the methodology of expert assessment of I. L. Solomin. The second stage implies the expert ranking of the important psychological characteristics in the activity of aerospace, and determination of psychological factors that affect the solution of professional tasks. Based on the acquired results, the author reveals the socioeconomic, industrial-technical, and psychological characteristics of the activity of aerospace engineers. The most important professional competencies of aerospace engineers include: attention span, ability to predict, psychic stability, memory, imagination, and distress tolerance. The effective solution of professional tasks by aerospace engineers requires psychological resilience to external influences, ability to maintain attention control, etc. The acquired data can be valuable in assessment of the important professional competencies of young specialists, as well as carrying out a set of measures aimed at developing attention span, ability to predict, and psychic stability.

Who Is a Better Decision Maker? Data‐Driven Expert Ranking Under Unobserved Quality

The capacity to rank expert workers by their decision quality is a key managerial task of substantial significance to business operations. However, when no ground truth information is available on experts’ decisions, the evaluation of expert workers typically requires enlisting peer‐experts, and this form of evaluation is prohibitively costly in many important settings. In this work, we develop a data‐driven approach for producing effective rankings based on the decision quality of expert workers; our approach leverages historical data on past decisions, which are commonly available in organizational information systems. Specifically, we first formulate a new business data science problem: Ranking Expert decision makers’ unobserved decision Quality (REQ) using only historical decision data and excluding evaluation by peer experts. The REQ problem is challenging because the correct decisions in our settings are unknown (unobserved) and because some of the information used by decision makers might not be available for retrospective evaluation. To address the REQ problem, we develop a machine‐learning–based approach and analytically and empirically explore conditions under which our approach is advantageous. Our empirical results over diverse settings and datasets show that our method yields robust performance: Its rankings of expert workers are consistently either superior or at least comparable to those obtained by the best alternative approach. Accordingly, our method constitutes a de facto benchmark for future research on the REQ problem.

On obtaining a stable vote ranking methodology for implementing e-government strategies

Recently, a lot of studies have been conducted on what factors and strategies affect the successful implementation of e-government programs around the world. Most of these studies lack a clear plan for selecting strategies and setting their priorities for implementation and thus ensuring the success of such programs. In a previous study conducted by the same researchers, these strategies and their implementation priorities were defined based on the opinions of 20 government experts who were interviewed and asked to rank the strategies based on 23 factors resulting from the SWOT analysis that summarizes the context under study, specifically Jordan as an example of a developing country. However, there is a problem that still exists facing many of these countries, which is the divergence of opinions of government decision-makers in ranking their priorities for implementing various government strategies, including e-government programs, and thus affecting the quality of the strategic decision taken. Therefore, this study will address the problem by finding a new methodology that enables to obtain a stable ranking of strategies in order to make the best strategic decision to implement the e-government program in the context under study or other similar contexts in the region. This study assumes that we can obtain a stable ranking of strategies related to implementing e-government programs when we only use the SWOT factors that produce stable ranking votes. To validate this, three main experiments with a number of proposed scenarios were developed to aggregate expert rankings on the implementation of e-government strategies using some common vote ranking methods, namely, parametric methods such as Borda, Copeland and Maximin, and algorithmic methods such as Stuart and Robust Rank Aggregation (RRA). Then, to evaluate the results of the three experimental scenarios, two new measures were proposed, namely the Similarity Index measure and the Matching Index measure. The results confirmed the validity of the assumption regarding the importance of obtaining a stable vote ranking in selecting the best strategies to ensure success in implementing e-government program in the context under study or other similar contexts in the region.

Modeling design information systems with multi-criteria

This work is devoted to the development of a new method for modeling information systems of multi-criteria design, based on expert ranking of criteria. The possibilities of the fuzzy-set theory and the possibility theory allow for a comprehensive analysis of criteria by reducing numerical indicators to a qualitative form, passing to the corresponding fuzzy relations of preference. Some of the information may be lost, therefore conclusions should be applied, both on the basis of “quantitative” processing of the results of measuring the criteria, and on the basis of “qualitative” processing. If these conclusions coincide, then their adequacy will be confirmed based on the initial data. Since the modeling of complex design systems is determined by several criteria defined on different base sets, and the problem of determining the multi-criteria efficiency in conditions of incompleteness of the initial data remains relevant, within the framework of this article, an optimization method is proposed simultaneously according to many criteria with incompleteness of the initial data. The difference between the described method lies in the fact that the expert ranking of criteria enables to formally set fuzzy preference graphs that make it possible to obtain comprehensive information about estimates in the rank scale during pairwise comparisons, with the aim of subsequent search for Pareto-optimal solutions in multi-criteria problems.

Aggregating multiple ordinal rankings in engineering design: the best model according to the Kendall\u2019s coefficient of concordance

Aggregating the preferences of a group of experts is a recurring problem in several fields, including engineering design; in a nutshell, each expert formulates an ordinal ranking of a set of alternatives and the resulting rankings should be aggregated into a collective one. Many aggregation models have been proposed in the literature, showing strengths and weaknesses, in line with the implications of Arrow's impossibility theorem. Furthermore, the coherence of the collective ranking with respect to the expert rankings may change depending on: (i) the expert rankings themselves and (ii) the aggregation model adopted. This paper assesses this coherence for a variety of aggregation models, through a recent test based on the Kendall's coefficient of concordance (W), and studies the characteristics of those models that are most likely to achieve higher coherence. Interestingly, the so-called Borda count model often provides best coherence, with some exceptions in the case of collective rankings with ties. The description is supported by practical examples.

A Novel Expert Finding System for Community Question Answering

With the popularity of community question answering (CQA) sites, the research on identifying the expert users in online communities attracted increasing attention. We present a novel expert ranking algorithm based on the quality of user posts and the authority of user in community, and the similarity between the knowledge tags of users and questions in CQA sites is adopted in our scheme. Experimental results show that our scheme has better performance and accuracy under the same background with an amount of data samples.