Non-expert Ratings Research Articles

Leveraging Artificial Intelligence to Improve the Diversity of Dermatological Skin Color Pathology: Protocol for an Algorithm Development and Validation Study

BackgroundThe paucity of dark skin images in dermatological textbooks and atlases is a reflection of racial injustice in medicine. The underrepresentation of dark skin images makes diagnosing skin pathology in people of color challenging. For conditions such as skin cancer, in which early diagnosis makes a difference between life and death, people of color have worse prognoses and lower survival rates than people with lighter skin tones as a result of delayed or incorrect diagnoses. Recent advances in artificial intelligence, such as deep learning, offer a potential solution that can be achieved by diversifying the mostly light-skin image repositories through generating images for darker skin tones. Thus, facilitating the development of inclusive cancer early diagnosis systems that are trained and tested on diverse images that truly represent human skin tones.ObjectiveWe aim to develop and evaluate an artificial intelligence–based skin cancer early detection system for all skin tones using clinical images.MethodsThis study consists of four phases: (1) Publicly available skin image repositories will be analyzed to quantify the underrepresentation of darker skin tones, (2) Images will be generated for the underrepresented skin tones, (3) Generated images will be extensively evaluated for realism and disease presentation with quantitative image quality assessment as well as qualitative human expert and nonexpert ratings, and (4) The images will be utilized with available light-skin images to develop a robust skin cancer early detection model.ResultsThis study started in September 2020. The first phase of quantifying the underrepresentation of darker skin tones was completed in March 2021. The second phase of generating the images is in progress and will be completed by March 2022. The third phase is expected to be completed by May 2022, and the final phase is expected to be completed by September 2022.ConclusionsThis work is the first step toward expanding skin tone diversity in existing image databases to address the current gap in the underrepresentation of darker skin tones. Once validated, the image bank will be a valuable resource that can potentially be utilized in physician education and in research applications. Furthermore, generated images are expected to improve the generalizability of skin cancer detection. When completed, the model will assist family physicians and general practitioners in evaluating skin lesion severity and in efficient triaging for referral to expert dermatologists. In addition, the model can assist dermatologists in diagnosing skin lesions.International Registered Report Identifier (IRRID)DERR1-10.2196/34896

EVALUATION OF THE ACCURACY OF PERSONALITY JUDGMENTS BASED ON WRITTEN VERBAL PRODUCTION

The process of personality judgment occurs in daily social interactions and represents an attempt to identify characteristics of someone else's personality, in the way to explain past and predict future behaviors. The results of this process have implications on future decisions and actions of people. This research aimed to examine the accuracy of non-expert ratings of Extraversion and Conscientiousness based on short written texts. The sample consisted of 215 participants (Mage = 28.58, SD = 10.30; 80.5% females). The exclusion criterion was that participants were psychologists or psychology students, i.e., individuals familiar with personality research and taxonomies. Participants rated Extraversion and Conscientiousness, based on the texts written by five different individuals. Criteria used to estimate the accuracy of judgments were the agreement between self-report measures on HEXACO PI-R from people who wrote the texts and ratings from participants, as well as the agreement between multiple raters. The results showed that there was a moderate self-other agreement for Extraversion and Conscientiousness. Also, the results showed that there was a high between-raters agreement for Extraversion and Conscientiousness. This study indicates that it is possible to judge one’s personality based on written verbal production, as well that raters tend to form similar impressions about the personality from written texts.

Beauty and the reef: Evaluating the use of non-expert ratings for monitoring aesthetic values of coral reefs

Aesthetic values are a key driver of tourist and recreational visitation to natural areas and are listed among the selection criteria for World Heritage properties. However, assessment and monitoring of aesthetic values in natural areas, and coral reefs in particular, have proven to be challenging. In our study we explored the value and limitations of a rapid assessment approach involving non-expert ratings of aesthetic beauty as a potential tool for long-term monitoring of aesthetic values in the Great Barrier Reef World Heritage Area, Australia. We investigated the sensitivity of a rating scale for detecting change and sampling requirements for monitoring, as well as observer biases, using an online survey of 1417 Australians in which respondents rated the aesthetic beauty of 181 coral reef images on a ten-point scale. Our results show average aesthetic rating scores ranged from 4.35 to 8.34 on a scale from 1 (ugly) to 10 (beautiful), with potential to detect differences of statistical significance within one point, indicating sufficient sensitivity to change for monitoring purposes. We found that a sample size of c.100 ratings per image provided a reasonable balance between cost (i.e. sample size) and accuracy (i.e. error). Older respondents (>65 years) with higher levels of coral reef visitation, experience and interest were more likely to give extreme ratings, however, there was no apparent predictor for this bias to be positive or negative (high or low ratings). Based on these results we provide recommendations to assist coral reef managers in their use and interpretation of non-expert aesthetic ratings in coral reef monitoring.

Concordance Between Expert and Nonexpert Ratings of Condensed Video-Based Trainee Operative Performance Assessment

We examined the impact of video editing and rater expertise in surgical resident evaluation on operative performance ratings of surgical trainees. Randomized independent review of intraoperative video. Operative video was captured at a single, tertiary hospital in Boston, MA. Six common general surgery procedures were video recorded of 6 attending-trainee dyads. Full-length and condensed versions (n = 12 videos) were then reviewed by 13 independent surgeon raters (5 evaluation experts, 8 nonexperts) using a crossed design. Trainee performance was rated using the Operative Performance Rating Scale, System for Improving and Measuring Procedural Learning (SIMPL) Performance scale, the Zwisch scale, and ten Cate scale. These ratings were then standardized before being compared using Bayesian mixed models with raters and videos treated as random effects. Editing had no effect on the Operative Performance Rating Scale Overall Performance (-0.10, p = 0.30), SIMPL Performance (0.13, p = 0.71), Zwisch (-0.12, p = 0.27), and ten Cate scale (-0.13, p = 0.29). Additionally, rater expertise (evaluation expert vs. nonexpert) had no effect on the same scales (-0.16 (p = 0.32), 0.18 (p = 0.74), 0.25 (p = 0.81), and 0.25 (p = 0.17). There is little difference in operative performance assessment scores when raters use condensed videos or when raters who are not experts in surgical resident evaluation are used. Future validation studies of operative performance assessment scales may be facilitated by using nonexpert surgeon raters viewing videos condensed using a standardized protocol.

Art expertise in construing meaning of representational and abstract artworks

Aesthetic appraisal of artwork can present the observer with visual problems to solve in the process of grasping its meaning and ‘visual rightness’ (i.e. “good” structure; Locher, 2003), with an elaboration on perceptual, semantic and affective dimensions (e.g. Marković, 2011). Thus observer's expertise is a factor in aesthetic appraisal. To examine the influence of art training on the aesthetic response, and to clarify the nature of the Representational/Abstract distinction, 30 Experts and 33 Non-experts (Art and Psychology students, respectively) were asked to rate 24 paintings on six affective and affective-evaluative semantic differential scales. Stimuli were images of paintings from the period 1900–1935, 12 broadly Representational and 12 broadly Abstract. Relative to Non-experts, Experts rated Abstract artworks as more Interesting, Beautiful, Informative and Sophisticated, distinguishing them less markedly from Representational artworks. Aggregate Expert and Non-expert ratings, processed by factor analysis, resulted in a two-factor solution. The first factor, contrasting Abstract and Representational artworks, appeared more salient for Non-experts. The second factor, Cool–Warm, separating vibrantly-colored paintings from those with a blue-dominated/dull palette, was more salient for Experts. While Non-experts exaggerated differences between Abstract and Representational paintings, Experts appraised these two types of art similarly, attending more to artwork collative properties. We conclude that appreciation of art by Experts involves ‘cognitive mastery’ (Leder, Belke, Oeberst, & Augustin, 2004), i.e. more complex, cues-based visual schemata which equip them with more sophisticated strategies for analysing collative properties and semantics of an artwork while parsing ‘visual rightness’ to unfold its visual meaning.

Development and validation of a classification and scoring system for the diagnosis of oral squamous cell carcinomas through confocal laser endomicroscopy.

BackgroundConfocal laser endomicroscopy (CLE) is an optical biopsy method allowing in vivo microscopic imaging at 1000-fold magnification. It was the aim to evaluate CLE in the human oral cavity for the differentiation of physiological/carcinomatous mucosa and to establish and validate, for the first time, a scoring system to facilitate CLE assessment.MethodsThe study consisted of 4 phases: (1) CLE-imaging (in vivo) was performed after the intravenous injection of fluorescein in patients with histologically confirmed carcinomatous oral mucosa; (2) CLE-experts (n = 3) verified the applicability of CLE in the oral cavity for the differentiation between physiological and cancerous tissue compared to the gold standard of histopathological assessment; (3) based on specific patterns of tissue changes, CLE-experts (n = 3) developed a classification and scoring system (DOC-Score) to simplify the diagnosis of oral squamous cell carcinomas; (4) validation of the newly developed DOC-Score by non-CLE-experts (n = 3); final statistical evaluation of their classification performance (comparison to the results of CLE-experts and the histopathological analyses).ResultsExperts acquired and edited 45 sequences (260 s) of physiological and 50 sequences (518 s) of carcinomatous mucosa (total: 95 sequences/778 s). All sequences were evaluated independently by experts and non-experts (based on the newly proposed classification system). Sensitivity (0.953) and specificity (0.889) of the diagnoses by experts as well as sensitivity (0.973) and specificity (0.881) of the non-expert ratings correlated well with the results of the present gold standard of tissue histopathology. Experts had a positive predictive value (PPV) of 0.905 and a negative predictive value (NPV) of 0.945. Non-experts reached a PPV of 0.901 and a NPV of 0.967 with the help of the DOC-Score. Inter-rater reliability (Fleiss` kappa) was 0.73 for experts and 0.814 for non-experts. The intra-rater reliability (Cronbach’s alpha) of the experts was 0.989 and 0.884 for non-experts.ConclusionsCLE is a suitable and valid method for experts to diagnose oral cancer. Using the DOC-Score system, an accurate chair-side diagnosis of oral cancer is feasible with comparable results to the gold standard of histopathology—even in daily clinical practice for non-experienced raters.

Complexity of Judgment: What Makes Possible the Convergence of Expert and Nonexpert Ratings in Assessing Creativity

This work is part of the debate regarding the possibility to judge the creativity of a particular object (an idea, a painting, an industrial product, etc.), by expert or nonexpert raters, with the same results or not. The study is focused on the concept complexity of judgment, considered fundamental to fully understand the problem. The objective is to discover the main reason of the match between expert and nonexpert ratings in assessing creativity. Results suggest the situations in which such ratings tend to converge or diverge, underlining the fundamental role played by the complexity of judgment. In addition, some suggestions for further studies are presented, aimed at assessing the generalizability of the proposed considerations in other context or in adopting other creativity assessment techniques.

Validity and reliability assessment of detailed scoring checklists for use during perioperative emergency simulation training.

Few valid and reliable grading checklists have been published for the evaluation of performance during simulated high-stakes perioperative event management. As such, the purposes of this study were to construct valid scoring checklists for a variety of perioperative emergencies and to determine the reliability of scores produced by these checklists during continuous video review. A group of anesthesiologists, intensivists, and educators created a set of simulation grading checklists for the assessment of the following scenarios: severe anaphylaxis, cerebrovascular accident, hyperkalemic arrest, malignant hyperthermia, and acute coronary syndrome. Checklist items were coded as critical or noncritical. Nonexpert raters evaluated 10 simulation videos in a random order, with each video being graded 4 times. A group of faculty experts also graded the videos to create a reference standard to which nonexpert ratings were compared. P < 0.05 was considered significant. Team leaders in the simulation videos were scored by the expert panel as having performed 56.5% of all items on the checklist (range, 43.8%-84.0%), and 67.2% of the critical items (range, 30.0%-100%). Nonexpert raters agreed with the expert assessment 89.6% of the time (95% confidence interval, 87.2%-91.6%). No learning curve development was found with repetitive video assessment or checklist use. The κ values comparing nonexpert rater assessments to the reference standard averaged 0.76 (95% confidence interval, 0.71-0.81). The findings indicate that the grading checklists described are valid, are reliable, and could be used in perioperative crisis management assessment.

Do you see what I see? Can non-experts with minimal training reproduce expert ratings in behavioral assessments of working dogs?

Working-dog organizations often use behavioral ratings by experts to evaluate a dog's likelihood of success. However, these experts are frequently under severe time constraints. One way to alleviate the pressure on limited organizational resources would be to use non-experts to assess dog behavior. Here, in populations of military working dogs (Study 1) and explosive-detection dogs (Study 2), we evaluated the reliability and validity of behavioral ratings assessed by minimally trained non-experts from videotapes. Analyses yielded evidence for generally good levels of inter-observer reliability and criterion validity (indexed by convergence between the non-expert ratings and ratings made previously by experts). We found some variation across items in Study 2 such that reliability and validity was significantly lower for three out of the 18 items, and one item had reliability and validity estimates that were impacted heavily by the behavioral test environment. There were no differences in reliability and validity based on the age of the dog. Overall the results suggest that ratings made by minimally trained non-experts for most items can serve as a viable alternative to expert ratings freeing limited resources of highly trained staff.This article is part of a Special Issue entitled: Canine Behavior.

Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges

Evaluating individual creativity is an important challenge in creativity research. We developed a training module for non-expert judges in which participants learned the definitions of components of creativity and received expert feedback in an interactive creativity judgment exercise. We aimed to test whether and how the training module would increase the reliability and validity of non-expert ratings. Study 1 (N=79) showed that the training had a positive effect on the test–retest reliability and validity of creativity ratings. Study 2 (N=126) replicated the results on test–retest reliability and validity but with low absolute values, indicating that trained participants cannot substitute experts. In addition, Study 2 showed that the effect of the training module on the validity of creativity ratings was mediated by increased validity of ratings of novelty and elaboration. The results are discussed in terms of theoretical and practical relevance.

Validierung einer automatischen Analyse der Sprechproben von Kindern mit isolierter Gaumenspalte

Following surgical repair of cleft lip and palate, hearing and speech and language development are important issues for the continued care of affected childhood and adolescent patients. Therefore, PEAKS (Program for Evaluation and Analysis of all Kinds of Speech Disorders) was developed in order to rate speech intelligibility automatically and reduce the time required for diagnostics. PEAKS is based on a speech recognition system and was extended to incorporate a speaker model. This investigation validated PEAKS for isolated cleft palate. From each of the 39 children with isolated cleft palate (3.1-14.5 years), 99 word productions were recorded digitally and analyzed-once "subjectively" by five experts and five nonexperts; once "objectively" using PEAKS. The automatic speech recognition system and the experts arrive at similar results with regard to speech intelligibility. The expert and nonexpert ratings differ significantly from each other. Within the group of nonexperts, a weak interrater reliability demonstrates the uncertainty associated with their ratings. PEAKS delivers reliable and representative results with regard to speech intelligibility among children and adolescents with isolated cleft palate. The automatic measurement of speech quality in children and adolescents with isolated cleft palate is possible.

Effect of Ergonomics Training on Agreement Between Expert and Nonexpert Ratings of the Potential for Musculoskeletal Harm in Manufacturing Tasks

To evaluate the effect of ergonomics training on non-ergonomists' ability to recognize and characterize the potential for musculoskeletal harm in manufacturing tasks. Ergonomics training was delivered to members of a participatory ergonomics team in a manufacturing facility. Before and after training, participatory ergonomics team members and the research team rated the potential for musculoskeletal harm for each of 30 tasks. Measures of agreement included Pearson, concordance, and intraclass correlation coefficients. Measures of agreement generally improved after training. The greatest agreement was observed for ratings of the potential for musculoskeletal harm to the low back. The greatest improvement in agreement was observed for ratings of the potential for musculoskeletal harm to the neck/shoulder. The training seemed to improve non-experts' ability to identify the potential for musculoskeletal harm.

Nonexpert ratings of family and parent–child interaction.

Observational methods benefit the study of family process, but many expert rating systems are costly and time-consuming. This study examined the utility of using small groups of eight to ten nonexperts to rate family conflict and maternal sensitivity. Videotaped triadic interactions of 39 families were drawn from Lindahl (1998), and 22 mother-toddler free-play interactions were drawn from Baker, Messinger, Lyons, and Grantz (2010). Sixty undergraduates rated interactions from these samples in real time using computer-assisted technology. Nonexpert ratings of family conflict were reliable, demonstrated high concordance with expert ratings, and replicated a key finding from Lindahl (1998). Nonexpert ratings of maternal sensitivity replicated a relevant finding from Baker, Messinger et al. (2010). Concordance was lower for maternal sensitivity, however, because of the tendency of nonexperts to overattend to sensitive structuring compared with emotional supportiveness. A second study indicated that as few as six nonexperts could effectively rate maternal sensitive structuring, but that nonexperts were unable to accurately rate emotional supportiveness. Implications for research methods and for our understanding of these important family constructs are discussed.

The Elusive Nature of Expertise in Spatial Interpolation

This study examines expert knowledge in the context of spatial interpolation through an analysis of surfaces manually interpolated by experts and non-experts. A comparison of expert and non-expert evaluations of computer-interpolated surfaces is provided as well. No significant differences were found in metric accuracy between expert and non-expert manual interpolations. No significant differences were found either in the number or the types of mapping problems identified by experts and non-experts on computer or manual maps, although the data tend to suggest differences consistently. Similarly, no significant differences were found between expert and non-expert ratings of computer maps on a linear scale relative to manual maps, although the data tend to suggest that experts had judged the computer maps more harshly than non-experts. These results suggest that subject-matter expertise alone may not truly qualify one as an expert in the context of precipitation or bedrock surface interpolation, and that in...