Expert Ratings Research Articles

Selection of Optimal Segmentation Algorithm for Satellite Images by Intuitionistic Fuzzy PROMETHEE Method

The combination of MCDM and fuzzy sets offers new potential ways to solve the challenges posed by complex image contents, such as selecting the optimal segmentation algorithm or evaluating the segmentation quality based on various parameters. Since no single segmentation algorithm can achieve the best results on satellite image datasets, it is essential to determine the most appropriate segmentation algorithm for each satellite image, the content of which can be characterized by relevant visual features. In this research, we proposed a set of visual criteria representing the fundamental aspects of satellite image segmentation. The segmentation algorithms chosen for testing were evaluated for their performance against each criterion. We introduced a new method to create a decision matrix for each image using fuzzy fusion, which combines the image content vector and the evaluation matrix of the studied segmentation algorithms. An extension of the Preference Ranking Organization Method Enrichment Evaluation (PROMETHEE) using intuitive fuzzy sets (IFSs) was applied to solve this problem. The results acquired by the proposed methodology were validated by comparing them with those obtained in expert ratings and by performing a sensitivity analysis.

Intra- and inter-rater reliability in ultrasonographic measurements of coracohumeral distance

BackgroundUltrasonography (US) is a widely used diagnostic tool in physical therapy. One of the US variables often utilized to quantify the dimensions of the subacromial space and its association with shoulder pathology is the coracohumeral distance (CHD), however, this measurement presents diverse evidence in terms of reliability. ObjectivesTo assess the intra- and inter-rater reliability of both expert and non-expert raters when measuring CHD through US in asymptomatic subjects. Additionally, we determined the effect of rater experience and measurement conditions on the recording of CHD. MethodsCHD of 15 individuals were recorded from US images of the glenohumeral joint of both upper extremities in three different positions. An expert and a non-expert rater in US usage recorded three CHD measurements, after a randomization procedure. To determine intra- and inter-rater reliability, the interclass correlation coefficient (ICC) and a multivariate variance model for the effects of rater experience, joint position and time of measure were used. Standard Error of Measurement and Minimal Detectable Change was also estimated for CHD measurements. ResultsIntra-rater reliability ranged 0.970 to 0.998) and Inter-rater reliability ranged 0.48 to 0.876). Joint position (F2;55 = 38.308; p < 0.001; ηp2 = 0.582) and measurement time (F2;55 = 6.019; p = 0.004; ηp2 = 0.180) effect was observed on CHD. ConclusionExcellent intra- and poor to moderate inter-rater reliability between expert and non-expert clinicians was determined, the latter being influenced by the position of the glenohumeral joint position at the time of US recording and the time of recording the measurement.

Investigating expert and lay judgments of pathogen transmission risk in urban and architectural environments

ABSTRACT The threat of infection posed by the COVID-19 virus forced the general public to use the visible cues within urban and architectural environments as a ‘lens’ through which pathogen transmission risk could be inferred. This study presents a model that quantifies associations between (1) the visible cues of urban and architectural environments and lay ratings of pathogen transmission risk, (2) the same relationship for experts (i.e. Infection Control Practitioners), and (3) the association between the lay ratings and the expert ratings. A series of urban and architectural environments were rated on twenty visible architectural cues and for their perceived pathogen transmission risk by lay and expert raters. Correlational analyses between the two groups yielded considerable consensus between risk ratings, as well as between which cues were significantly associated with risk ratings, which included the space’s crowdedness, the potential for crowds, and cleanliness. Expert risk ratings were also significantly associated with corridor size, and marginally significantly associated with the number of touchable surfaces, the number of furniture/seating, and access to fresh air. In this way, expert cue utilization is more complex than lay assessments. Implications for public health policy makers and designers of the built environment are discussed.

Incobotulinum Toxin-A in Professional Musicians with Focal Task-Specific Dystonia: A Double Blind, Placebo Controlled, Cross-Over Study.

Musician's focal task-specific dystonia is a complex disorder of fine motor control, with incomplete understanding of its etiology. There have been relatively few trials of botulinum toxin in upper limb task-specific dystonia, and prior studies have yielded variable results, leading to skepticism regarding the utility of this approach in elite performers. We conducted a double-blind, placebo-controlled, randomized, cross-over study of incobotulinum toxin-A in 21 professional musicians with focal upper extremity task-specific dystonia affecting performance on their instrument, using a novel paradigm of initial injections followed by booster injections at two- and four-week intervals. The primary outcome measure was the change in blinded dystonia rating of the active arm by two expert raters using a Clinical Global Impression numeric scale at week 8 compared to enrollment. 19 men and 2 women with musicians' dystonia were enrolled over a six-year period. Nineteen patients completed the study. Analysis of the primary outcome measure in comparison to baseline revealed a change in dystonia severity of P = 0.04 and an improvement in overall musical performance of P = 0.027. No clinically significant weakness was observed, and neutralizing antibodies to toxin were not found. Despite its small sample size, our study demonstrated a statistically significant benefit of incobotulinum toxin-A injections as a treatment for musicians' task-specific dystonia. Tailoring the use of toxin with booster injections allowed refinement of dosing strategy and outcomes, with benefits that were meaningful to patients clearly visible on videotaped evaluations. In addition to its application to musicians' dystonia, this approach may have relevance to optimize application of botulinum toxin in other forms of focal dystonia such as blepharospasm, cervical dystonia, writer's cramp, and spasmodic dysphonia.

Enhancing fruit orchard establishment: A multicriteria approach for plum variety selection

This research is focused to identifying the most suitable plum variety for establishing a new orchard, aiming to achieve optimal outcomes. To accomplish this goal, multi-criteria decision-making model has been developed striving to support decision-making process. The gained results are based on experience of experts engaged in assessment of certain plum varieties. The analysis of collected expert ratings has involved mutual use of 3 methods, where the FUll COnsistency Method (FUCOM) and CRiteria Importance Through Intercriteria Correlation (CRITIC) methods were applied to determine the weights of selected criteria. The results of applying these methods demonstrated that the Maretability criterion is of highest importance. Besides, the fuzzy MARCOS method (Measurement Alternatives and Ranking according to Compromise Solution) has been applied to rank the plum varieties according to expert ratings. Derived results revealed that the ?Cacanska rodna? and ?Stanley? varieties were ranked as the most favorable, while the ?Prezident? variety was assessed as the most unfavorable. Further, gained research outcomes were corroborated by the sensitivity analysis and results validation. This research contributes to improvement of fruit growing in BiH by previous adequate selection of available plum varieties towards the establishing of new orchards and yielding optimal outcomes.

Finite-element analysis case retrieval based on an ontology semantic tree

Finite-element analysis case retrieval based on an ontology semantic tree

Automatic Scoring of Verbal Divergent Thinking Tests: From Lexical Databases to Large Language Models

The article explores the evolution of methods for automatically assessing verbal divergent thinking tests. Resear - chers have increasingly focused on the ability to evaluate the originality of respondents' answers by calculating their semantic distance from the stimulus task. From 2009 to 2019, latent semantic analysis became the primary method for assessing semantic distances. Overall, in terms of internal consistency and correlation with expert ratings, its application yielded satisfactory results, maintaining an acceptable balance of quality and effort expended. However, issues emerged (dependence on the corpus used, result instability, systematic distortions related to the length of analyzed responses), prompting researchers to transition to more advanced models of distributional semantics (GloVe, Word2Vec etc.), large language models, and supervised learning. Large language models, especially those fine-tuned on creativity test materials, demonstrated higher effectiveness compared to models assessing semantic distances and approached expert evaluations. In addition to evaluating originality, the article considers works proposing methods for automatic assessment of elaboration, flexibility, associative flow, and divergent semantic integration. References to online platforms that allow for automatic assessments of originality in responses to divergent tests are provided. The issue of interpreting results obtained through large language models is discussed. A drawback of using these models is the lack of understanding of the basis on which judgments of the originality of creative products are made. The perspectives of applying explainable artificial intelligence for evaluating results of verbal and non-verbal tests of creative thinking are being discussed.

Assessing the utility of artificial intelligence throughout the triage outpatients: a prospective randomized controlled clinical study.

Currently, there are still many patients who require outpatient triage assistance. ChatGPT, a natural language processing tool powered by artificial intelligence technology, is increasingly utilized in medicine. To facilitate and expedite patients' navigation to the appropriate department, we conducted an outpatient triage evaluation of ChatGPT. For this evaluation, we posed 30 highly representative and common outpatient questions to ChatGPT and scored its responses using a panel of five experienced doctors. The consistency of manual triage and ChatGPT triage was assessed by five experienced doctors, and statistical analysis was performed using the Chi-square test. The expert ratings of ChatGPT's answers to these 30 frequently asked questions revealed 17 responses earning very high scores (10 and 9.5 points), 7 earning high scores (9 points), and 6 receiving low scores (8 and 7 points). Additionally, we conducted a prospective cohort study in which 45 patients completed forms detailing gender, age, and symptoms. Triage was then performed by outpatient triage staff and ChatGPT. Among the 45 patients, we found a high level of agreement between manual triage and ChatGPT triage (consistency: 93.3-100%, p<0.0001). We were pleasantly surprised to observe that ChatGPT's responses were highly professional, comprehensive, and humanized. This innovation can help patients win more treatment time, improve patient diagnosis and cure rates, and alleviate the pressure of medical staff shortage.

LLM Plus Machine Learning Outperform Expert Rating to Predict Life Satisfaction From Self-Statement Text

LLM Plus Machine Learning Outperform Expert Rating to Predict Life Satisfaction From Self-Statement Text

영어 쓰기 찬반 토론 평가의 타당도 탐색

The present study aims to explore the validity of EFL written debate assessment. One hundred EFL students whose major was English education took an argumentative essay test and participated in an EFL written debate contest. They also conducted self-and peer-assessment in the essay test as well as in the written debate. Five students took part in interviews. Four expert raters rated the students’ essays and postings they had uploaded on a debate forum. Later, three experts presented their ideas in in-depth interviews. The result showed that scores of the written debate weakly correlated with those of the argumentative essay test. From the interview, the expert raters responded that they had had difficulty in rating the written debate because of the unwanted variables embedded in the assessment. On the other hand, students showed a positive outlook regarding EFL written debate assessment.

러시아-우크라이나 전쟁이후 러시아 부패 동향 분석

This study is significant in that it analyzes the impact of corruption issues on war while the war continues through analysis of corruption crime trends after the Russian-Ukranian war. Therefore, analyzing Russia's corruption trends and anti-corruption policies has important significance in solving Russia's corruption problem and researching corruption problems in existing former socialist countries. The problem of corruption has an extremely negative impact on the entire state apparatus and is resulting in corruption crimes being committed systematically. Another important characteristic of corruption is that this phenomenon involves enormous cash flows, which ultimately results in the greatest material damage from corruption crimes. Experts rate corruption as the country's most profitable systematic business. Ultimately, the war between Russia and Ukraine, which started in 2014 and escalated into a full-scale war in 2022, has a lot of meaning in terms of war. Contrary to expectations that the war would end quickly with overwhelming power, the war is being prolonged, and there is a possibility that Russian corruption may be a major factor in this. The issue of corruption related to war will be worth studying in the future as an alternative task.

Simulation and analysis of hydraulic transition process based on mechanical hydraulics.

With the scarcity of water resources in various regions, the pressure on water supply projects is also increasing, which has led to a sharp increase in the water hammer effect in water supply pump projects. In response to this issue, the study proposes to apply a unidirectional pressure regulating tower based on mechanical hydraulic technology to water hammer protection in water supply pumps. In addition, the study also optimizes the calculation method of water hammer and designs one simulation software, which is used to simulate and analyze the proposed water hammer protection measures. The study first determines through simulation software that the optimal initial water level of the unidirectional pressure regulating tower is 2m and the optimal diameter of the make-up water pipe is 600 mm. Afterwards, simulation analysis is conducted on different water hammer protection measures, and it is found that the maximum pressure of the water hammer under the proposed protection measures is the lowest, at 14.8m, which was lower than the comparison measures. In addition, according to expert ratings, the average protective effect rating of protective measure 1 by relevant technical personnel is 9.4 points, which is better than the comparative measure. The above results indicate that through the simulation analysis of hydraulic transition process based on mechanical hydraulics, it can be found that the water hammer protection measures proposed in the study have good protective functions and can effectively reduce the water hammer effect in water supply engineering.

Set that record straight! Cardinal line orientations in music album artwork boost market performance and music consumption

AbstractIn three studies, we examine the effect of music album artwork on album market performance and music consumption. Based on the perceptual preference for visual stimuli with cardinal (orthogonal) over oblique (tilted) line/edge orientations (a phenomenon known as the “oblique effect”), albums with a predominance of cardinal line/edge orientations in their artwork should perform better than albums with more oblique artwork, as indicated by the albums' market performance and consumers' listening behavior. Study 1, using secondary data, shows that the cardinality of album artwork is a positive predictor of the cover's esthetic appeal and of market performance as evinced by its position in the US charts, expert ratings of the album, and the number of weeks the album remained in the US charts. Studies 2 and 3 use experiments to show that consumers listen longer to music when album artwork is relatively more cardinal than oblique. These studies also explore whether the effect is mediated by higher esthetic appreciation of cardinal (vs. oblique) album artwork in turn affecting song liking. Overall, these findings underscore the importance of visual preferences in packaging design for music consumption.

Wessex Head Injury Matrix in Patients with Prolonged Disorders of Consciousness: A Reliability Study.

Introduction: The Wessex Head Injury Matrix (WHIM) was developed to assess patients with disorders of consciousness (DOC) and was tested in terms of inter-rater reliability (IRR) and test-retest reliability (TRR) in the year 2000. The American Congress of Rehabilitation and Medicine reported that IRR and TRR were unproven. We aim to assess the reliability of the WHIM in prolonged DOC patients (PDOC). Methods: A total of 51 PDOC patients (32 unresponsive wakefulness syndrome (UWS/VS) and 19 minimally conscious state (MCS)) who were hosted in a dedicated unit for long-term brain injury care were enrolled. The time from injury ranged from 182 to 3325 days. Two raters administered the Coma Recovery Scale-Revised (CRS-R) and the WHIM to test the IRR and TRR. The TRR was administered two weeks after the first assessment. Results: For the CRS-R, the agreement in IRR and TRR was perfect between the two raters. The agreement for the WHIM ranged from substantial to almost perfect for IRR and from fair to substantial for the TRR. Conclusions: The WHIM showed a strong IRR when administered by expert raters and strongly correlated with the CRS-R. This study provides further evidence of the psychometric qualities of the WHIM and the importance of its use in PDOC patients.

Investigating the reliability and validity of subacromial space measurements using ultrasound and MRI

BackgroundWhile ultrasound (US) measures of the subacromial space (SAS) have demonstrated excellent reliability, measurements are typically captured by experts with extensive ultrasound experience. Further, the agreement between US measured SAS width and other imaging modalities has not been explored. This research evaluated the agreement between SAS measures captured by novice and expert raters and between US and magnetic resonance imaging (MRI). This study also evaluated the effect of US transducer tilt on measured SAS.MethodsNine men and nine women participated in this study. US images were captured by a novice and expert with the participant in both seated and supine positions. An inclinometer was fixed to the US probe to measure transducer tilt. SAS width was measured in real time from freeze framed images. MRI images were captured, and the humerus and acromion manually segmented. The SAS width was measured using a custom algorithm.ResultsIntraclass correlation coefficients (ICCs) between novice and expert raters were 0.74 and 0.63 for seated and supine positions, respectively. Intra-rater agreement was high for both novice (ICC = 0.83–0.84) and expert (ICC ≥ 0.94) raters. Agreement between US and MRI was poor (ICC = 0.21–0.49) but linearly related.ConclusionsModerate agreement between novice and expert raters was demonstrated, while the agreement between US and MRI was poor. High intra-rater reliability within each rater suggests that US measures of the SAS may be completed by a novice with introductory training.

Smart grading: A generative AI-based tool for knowledge-grounded answer evaluation in educational assessments

Evaluating text-based answers obtained in educational settings or behavioral studies is time-consuming and resource-intensive. Applying novel artificial intelligence tools such as ChatGPT might support the process. Still, currently available implementations do not allow for automated and case-specific evaluations of large numbers of student answers. To counter this limitation, we developed a flexible software and user-friendly web application that enables researchers and educators to use cutting-edge artificial intelligence technologies by providing an interface that combines large language models with options to specify questions of interest, sample solutions, and evaluation instructions for automated answer scoring. We validated the method in an empirical study and found the software with expert ratings to have high reliability. Hence, the present software constitutes a valuable tool to facilitate and enhance text-based answer evaluation.•Generative AI-enhanced software for customizable, case-specific, and automized grading of large amounts of text-based answers.•Open-source software and web application for direct implementation and adaptation.

Assessment of Quality of Life in Gestational Diabetes Mellitus Care-Study Protocol of the GDM-QOL Project.

In recent years, the concept of quality of life (QoL) has gained significant importance within health care and clinical research, e.g., as in patient-reported outcomes. In gestational diabetes mellitus (GDM) care, enhancing QoL through reasonable interventions is considered equally important as achieving metabolic control and preventing complications in the treatment process, leading to the suggestion that QoL assessment should be implemented as a clinical standard in GDM care. Although a considerable number of questionnaires for the measurement of general as well as health-related and diabetes-specific QoL are frequently used in GDM research, a validated QoL questionnaire tailored to women with GDM does not exist in German-speaking countries. To develop and test such an instrument, we plan to conduct the following steps: (a) translate the Persian questionnaire GDMQ-36, the only GDM-specific questionnaire to date; (b) conduct expert ratings as well as pretests featuring cognitive debriefings and structured interviews with women suffering from GDM for evaluating comprehensibility, face and content validity; (c) pilot and validate the preliminary questionnaire in terms of testing its psychometric performance (e.g., via confirmatory factor analysis). The resulting GDM-specific questionnaire will facilitate a broader perspective of the pregnant women's expectations, needs, impairments, and burdens related to their disease, and its treatment. This enables physicians and other health professionals to establish an individualized treatment plan and to provide customized information, support, and psychological counseling, which helps to optimize the provided care.

Toward a generalizable machine learning workflow for neurodegenerative disease staging with focus on neurofibrillary tangles

Machine learning (ML) has increasingly been used to assist and expand current practices in neuropathology. However, generating large imaging datasets with quality labels is challenging in fields which demand high levels of expertise. Further complicating matters is the often seen disagreement between experts in neuropathology-related tasks, both at the case level and at a more granular level. Neurofibrillary tangles (NFTs) are a hallmark pathological feature of Alzheimer disease, and are associated with disease progression which warrants further investigation and granular quantification at a scale not currently accessible in routine human assessment. In this work, we first provide a baseline of annotator/rater agreement for the tasks of Braak NFT staging between experts and NFT detection using both experts and novices in neuropathology. We use a whole-slide-image (WSI) cohort of neuropathology cases from Emory University Hospital immunohistochemically stained for Tau. We develop a workflow for gathering annotations of the early stage formation of NFTs (Pre-NFTs) and mature intracellular (iNFTs) and show ML models can be trained to learn annotator nuances for the task of NFT detection in WSIs. We utilize a model-assisted-labeling approach and demonstrate ML models can be used to aid in labeling large datasets efficiently. We also show these models can be used to extract case-level features, which predict Braak NFT stages comparable to expert human raters, and do so at scale. This study provides a generalizable workflow for various pathology and related fields, and also provides a technique for accomplishing a high-level neuropathology task with limited human annotations.

Arterial spin‐labeling spatial coefficient of variation is associated with cerebral small vessel disease

AbstractBackgroundCerebral small vessel disease (SVD) is the most common pathology to co‐occur with Alzheimer’s disease but pathophysiologic mechanisms leading to SVD are poorly understood. The arterial spin‐labeling (ASL) spatial coefficient of variation (sCoV), a proxy marker of arterial transit time (ATT), may better reflect SVD‐related hemodynamic disturbances than ASL‐derived cerebral blood flow (CBF). We investigated whether grey matter CBF and grey matter sCoV each related to neuroimaging markers of SVD, including white matter hyperintensities (WMHs), basal ganglia enlarged perivascular spaces (ePVS), cerebral microbleeds (CMBs), and lacunes.MethodsVanderbilt Memory and Aging Project participants free of clinical dementia and stroke at study entry (n = 312, 73±7 years, 39% mild cognitive impairment, 41% female) underwent multimodal 3T brain magnetic resonance imaging to quantify ASL parameters and SVD burden. Pseudo‐continuous ASL assessed CBF in total grey matter. Grey matter sCoV was calculated as σ(CBFGrey Matter)/μ(CBFGrey Matter). WMHs and ePVS volumes were quantified using automated methods and ePVS volumes were standardized to basal ganglia volume. CMBs and lacune counts were quantified by an expert rater. Ordinary least squares regression models cross‐sectionally related grey matter CBF and sCoV individually to log‐transformed burden of each SVD marker. Models were adjusted for age, sex, race/ethnicity, education, cognitive status, Framingham Stroke Risk Profile (minus age), intracranial volume, grey matter volume, and apolipoprotein E‐ε4 status.ResultsGrey matter CBF was unrelated to all SVD outcomes (p‐values&gt;0.25). Higher grey matter sCoV was associated with greater WMHs (β = 2.1, p = 0.001), ePVS (β = 0.004, p = 0.0003), and lacunar infarcts (β = 1.4, p = 0.0006) but not CMBs (β = 1.0, p = 0.06).ConclusionsOur findings align with previous results showing that grey matter sCoV is a valuable marker of SVD. In contrast to previous work, we found an association between higher grey matter sCoV and higher basal ganglia ePVS burden. Potentially, our use of volumetric ePVS quantification rather than qualitative ePVS burden increased our ability to detect this association. As sCoV is a proxy marker of ATT, future work should consider quantifying sCoV or ATT in addition to CBF to better characterize SVD‐related hemodynamic disturbances. Funding: R01‐AG034962, K24‐AG046373, F31‐AG079640, P20‐AG068082

Development and initial evaluation of a treatment integrity measure for low-intensity group psychoeducational interventions.

Despite the importance of assessing the quality with which low-intensity (LI) group psychoeducational interventions are delivered, no measure of treatment integrity (TI) has been developed. To develop a psychometrically robust TI measure for LI psychoeducational group interventions. This study had two phases. Firstly, the group psychoeducation treatment integrity measure-expert rater (GPTIM-ER) and a detailed scoring manual were developed. This was piloted by n=5 expert raters rating the same LI group session; n=6 expert raters then assessed content validity. Secondly, 10 group psychoeducational sessions drawn from routine practice were then rated by n=8 expert raters using the GPTIM-ER; n=9 patients also rated the quality of the group sessions using a sister version (i.e. GPTIM-P) and clinical and service outcome data were drawn from the LI groups assessed. The GPTIM-ER had excellent internal reliability, good test-retest reliability, but poor inter-rater reliability. The GPTIM-ER had excellent content validity, construct validity, formed a single factor scale and had reasonable predictive validity. The GPTIM-ER has promising, but not complete, psychometric properties. The low inter-rater reliability scores between expert raters are the main ongoing concern and so further development and testing is required in future well-constructed studies.