Subjective Performance Evaluation Research Articles

Measuring aviator workload using EEG: an individualized approach to workload manipulation.

Measuring an operator's physiological state and using that data to predict future performance decrements has been an ongoing goal in many areas of transportation. Regarding Army aviation, the realization of such an endeavor could lead to the development of an adaptive automation system which adapts to the needs of the operator. However, reaching this end state requires the use of experimental scenarios similar to real-life settings in order to induce the state of interest that are able to account for individual differences in experience, exposure, and perception to workload manipulations. In the present study, we used an individualized approach to manipulating workload in order to account for individual differences in response to workload manipulations, while still providing an operationally relevant flight experience. Eight Army aviators participated in the study, where they completed two visits to the laboratory. The first visit served the purpose of identifying individual workload thresholds, with the second visit resulting in flights with individualized workload manipulations. EEG data was collected throughout both flights, along with subjective ratings of workload and flight performance. Both EEG data and workload ratings suggested a high workload. Subjective ratings were higher during the high workload flight compared to the low workload flight (p < 0.001). Regarding EEG, frontal alpha (p = 0.04) and theta (p = 0.01) values were lower and a ratio of beta/(alpha+theta) (p = 0.02) were higher in the baseline flight scenario compared to the high workload scenario. Furthermore, the data were compared to that collected in previous studies which used a group-based approach to manipulating workload. The individualized method demonstrated higher effect sizes in both EEG and subjective ratings, suggesting the use of this method may provide a more reliable way of producing high workload in aviators.

HD-Former: A hierarchical dependency Transformer for medical image segmentation

Medical image segmentation is a compelling fundamental problem and an important auxiliary tool for clinical applications. Recently, the Transformer model has emerged as a valuable tool for addressing the limitations of convolutional neural networks by effectively capturing global relationships and numerous hybrid architectures combining convolutional neural networks (CNNs) and Transformer have been devised to enhance segmentation performance. However, they suffer from multilevel semantic feature gaps and fail to account for multilevel dependencies between space and channel. In this paper, we propose a hierarchical dependency Transformer for medical image segmentation, named HD-Former. First, we utilize a Compressed Bottleneck (CB) module to enrich shallow features and localize the target region. We then introduce the Dual Cross Attention Transformer (DCAT) module to fuse multilevel features and bridge the feature gap. In addition, we design the broad exploration network (BEN) that cascades convolution and self-attention from different percepts to capture hierarchical dense contextual semantic features locally and globally. Finally, we exploit uncertain multitask edge loss to adaptively map predictions to a consistent feature space, which can optimize segmentation edges. The extensive experiments conducted on medical image segmentation from ISIC, LiTS, Kvasir-SEG, and CVC-ClinicDB datasets demonstrate that our HD-Former surpasses the state-of-the-art methods in terms of both subjective visual performance and objective evaluation. Code: https://github.com/barcelonacontrol/HD-Former.

Effects of hearing acuity on psychophysiological responses to effortful speech perception

In recent studies, psychophysiological measures have been used as markers of listening effort, but there is limited research on the effect of hearing loss on such measures. The aim of the current study was to investigate the effect of hearing acuity on physiological responses and subjective measures acquired during different levels of listening demand, and to investigate the relationship between these measures. A total of 125 participants (37 males and 88 females, age range 37–72 years, pure-tone average hearing thresholds at the best ear between -5.0 to 68.8 dB HL and asymmetry between ears between 0.0 and 87.5 dB) completed a listening task. A speech reception threshold (SRT) test was used with target sentences spoken by a female voice masked by male speech. Listening demand was manipulated using three levels of intelligibility: 20 % correct speech recognition, 50 %, and 80 % (IL20 %/IL50 %/IL80 %, respectively). During the task, peak pupil dilation (PPD), heart rate (HR), pre-ejection period (PEP), respiratory sinus arrhythmia (RSA), and skin conductance level (SCL) were measured. For each condition, subjective ratings of effort, performance, difficulty, and tendency to give up were also collected. Linear mixed effects models tested the effect of intelligibility level, hearing acuity, hearing asymmetry, and tinnitus complaints on the physiological reactivity (compared to baseline) and subjective measures. PPD and PEP reactivity showed a non-monotonic relationship with intelligibility level, but no such effects were found for HR, RSA, or SCL reactivity. Participants with worse hearing acuity had lower PPD at all intelligibility levels and showed lower PEP baseline levels. Additionally, PPD and SCL reactivity were lower for participants who reported suffering from tinnitus complaints. For IL80 %, but not IL50 % or IL20 %, participants with worse hearing acuity rated their listening effort to be relatively high compared to participants with better hearing. The reactivity of the different physiological measures were not or only weakly correlated with each other. Together, the results suggest that hearing acuity may be associated with altered sympathetic nervous system (re)activity. Research using psychophysiological measures as markers of listening effort to study the effect of hearing acuity on such measures are best served by the use of the PPD and PEP.

How did you perform? Investigating football players' perception of self-regulated passing performances under auditory noise environments.

This paper deals with the question on how sport performances may be influenced by internal, emotional processes, which stem from outside feedback. In terms of methods, players' subjective performance ratings for four experimental auditory cue conditions were examined; these included both 'positive' and 'negative' stadium noise, 'no (auditory) conditions,' and a control/'baseline' condition. This resulted in a qualitative-analytic data set that was obtained succeeding each auditory cue condition using a unique football training machine (i.e., known as 'Footbonaut'). Without having received any coaching/performance feedback, players were asked to rate and individually comment on their perceived performance ratings for each experimental auditory condition. Findings indicate stronger and more significant correlations between auditory conditions and subjective ratings compared to the non-auditory condition and its subjective rating. Furthermore, data provides initial insight into players' emotional experiences during each of the practice conditions. These noteworthy findings on players' abilities to accurately judge their performances based on selfmonitoring and intrinsic feedback are discussed from an Ecological Dynamics perspective, linked to a Nonlinear Pedagogy for coaching. Here, representative and affective learning designs for skill learning and performance preparation are presented. Finally, a hypothetical catalyst effect of auditory stadium noise on subjective performance rating is proposed.

Comparisons of Self-Report With Objective Measurements Suggest Faster Responding but Little Change in Response Quality Over Time in Ecological Momentary Assessment Studies.

Response times (RTs) to ecological momentary assessment (EMA) items often decrease after repeated EMA administration, but whether this is accompanied by lower response quality requires investigation. We examined the relationship between EMA item RTs and EMA response quality. In one data set, declining response quality was operationalized as decreasing correspondence over time between subjective and objective measures of blood glucose taken at the same time. In a second EMA study data set, declining response quality was operationalized as decreasing correspondence between subjective ratings of memory test performance and objective memory test scores. We assumed that measurement error in the objective measures did not increase across time, meaning that decreasing correspondence across days within a person could be attributed to lower response quality. RTs to EMA items decreased across study days, while no decrements in the mean response quality were observed. Decreasing EMA item RTs across study days did not appear problematic overall.

Understanding the role of gaze behaviour and coaching experience in the assessment of youth soccer teams

ABSTRACT This study aimed to explore the perceptual-cognitive characteristics of coaches as they assessed team performance in youth soccer. The primary focus was to investigate the alignment between coaches’ subjective analyses of team behaviour and objective analyses, while also examining the relationship between coaches’ gaze behaviour and their levels of coaching experience, particularly considering the potential differences that may exist among coaches with varying levels of experience. Sixty-five male and female adults with various soccer coaching experience (experienced, novice, other team sport experience, and non-team sport experience) watched five 4-minute videos and assessed team behaviour. These subjective evaluations were compared to objective data obtained from video analysis and GPS, which included measures of completed skills, spatiotemporal characteristics, and passing networks. The participants’ fixation duration and frequency were measured for each video clip, and the area around the ball specifically. The study found no significant differences between groups regarding the number of times participants’ subjective analysis aligned with the objective data (p = 0.059, ɳ2 = 0.07). However, coaches with soccer coaching experience demonstrated a higher fixation frequency and more revisits to the ball area when compared to participants without soccer experience (p = <.001, ɳ2 = 0.09). The current study offers a unique approach to uncovering soccer coaching expertise by combining objective and subjective evaluations of team performance. In summary, the study reveals that coaching experience did not impact how often coaches subjective analyses matched objective data. However, soccer coaches had distinctive gaze behaviour patterns where they revisited the area around the ball more often.

Photorealistic style transfer fusing frequency separation channel attention mechanism and mirror flow pyramid integration

Photorealistic image style transfer aims to transfer style information while preserving the realistic details of the content image. However, an existing limitation is the inability to effectively balance the relationship between image realism and stylization intensity, resulting in poor image transfer performance. To address this issue, we propose an photorealistic style transfer method that fusing Frequency Separation Channel Attention Mechanism (FSCAM) and Mirror Fluid Pyramid Integration (MFPI). This method achieves superior stylization intensity while improves image realism. Firstly, we propose an improved channel attention mechanism called FSCAM. This mechanism utilizes Discrete Cosine Transform (DCT) to decompose features into different frequency components and screens out high-valued texture and color features, thereby enhancing the stylization intensity of the generated images. In addition, we designed a MFPI module. The module is able to integrate information from different scales, enhance the preservation of low-level detail features in high-level features, and thus improve the realism of the images. Experimental results demonstrate that our method not only enhances the stylization intensity but also improves the image realism. It achieves satisfactory performance in terms of subjective visual performance and objective evaluation metrics.

Academic performance through multiple lenses: Intelligence, conscientiousness, and achievement striving motivation as differential predictors of objective and subjective measures of academic achievement in two studies of adolescents

In a two-sample investigation of 766 adolescents, we examined the associations and incremental validity of test-based intelligence, parent-rated conscientiousness, and self-rated achievement striving motivation with objective (grades) and subjective (parent-reported and self-reported academic performance) measures of academic achievement. The two studies yielded largely similar results. Intelligence was related to objective and subjective performance ratings. Conscientiousness showed associations and explained variance beyond intelligence in grades and parent-reported but mostly not in self-reported academic performance. Achievement striving motivation was largely related to and explained variance beyond intelligence and conscientiousness in grades and subjective performance ratings. Findings indicate that traits and motives predict objective and subjective academic achievement measures incrementally to abilities. Differential relationships for conscientiousness depended on the informant of academic performance.

Ten weeks of Capsicum annuum L. extract supplementation did not change adipose tissue-derived hormones, appetite, body composition, and muscle strength when combined with resistance training in healthy untrained men: A clinical trial study

Capsiate (CAP) is a nonpungent capsaicin analog (Capsicum annuum L. extract) that has been studied as a potential antiobesity agent. However, the interaction between chronic CAP supplementation and resistance training is not clear. The purpose of this study was to examine the changes in adipose tissue–derived hormones, body composition, appetite, and muscle strength after 10 weeks of resistance training, combined with chronic CAP supplementation in healthy untrained men. We hypothesized that CAP could induce higher benefits when combined with resistance training after 10 weeks of intervention compared to resistance training alone. Twenty-four young men (age, 22.0 ± 2.9) were randomized to either capsiate supplementation (CAP = 12 mg/day) or placebo (PL), and both groups were assigned to resistance training. Body composition, leptin and adiponectin concentrations, subjective ratings of appetite, energy intake, and exercise performance were assessed at before and after 10 weeks of progressive resistance training. There was a significant increase in body mass (P < .001), fat-free mass (CAP: 58.0 ± 7.1 vs. post, 59.7 ± 7.1 kg; PL: pre, 58.4 ± 7.3 vs. post, 59.8 ± 7.1 kg; P < .001), resting metabolic rate (CAP: pre, 1782.9 ± 160.6 vs. post, 1796.3 ± 162.0 kcal; PL: pre, 1733.0 ± 148.9 vs. post, 1750.5 ± 149.8 kcal; P < .001), maximal strength at 45 leg press (P < .001) and bench press (P < .001) in both groups, but no significant (P > .05) supplementation by training period interaction nor fat mass was observed. For subjective ratings of appetite, energy intake, leptin, and adiponectin, no significant effect of supplementation by training period interaction was observed (P > .05). In conclusion, 10 weeks of resistance training increased total body weight, muscle mass, and maximum strength in healthy untrained men; however, CAP supplementation (12 mg, 7 days per week) failed to change adipose tissue–derived hormones, appetite, body composition and muscle strength in this population.Registered under Brazilian Registry of Clinical Trials (RBR-8cz9kfq).

Pupillometry reveals differences in cognitive demands of listening to face mask-attenuated speech.

Face masks offer essential protection but also interfere with speech communication. Here, audio-only sentences spoken through four types of masks were presented in noise to young adult listeners. Pupil dilation (an index of cognitive demand), intelligibility, and subjective effort and performance ratings were collected. Dilation increased in response to each mask relative to the no-mask condition and differed significantly where acoustic attenuation was most prominent. These results suggest that the acoustic impact of the mask drives not only the intelligibility of speech, but also the cognitive demands of listening. Subjective effort ratings reflected the same trends as the pupil data.

The bonding cost in subjective performance evaluation

This paper studies the optimal contract under conditions where the principal modifies the level of subjectivity in performance evaluation, thereby augmenting the existing literature that assumes subjective evaluations as fully unverifiable. The level of subjectivity represents the ease with which the principal can modify evaluations, and this choice impacts the costs of implementation and alteration. We show that in the optimal contract, the principal invariably engages in downward adjustments of evaluation and that the principal determines the level of subjectivity through a trade-off between providing incentives to the agent and making these downward revisions.

PSVIII-17 In-Home Validation of a Canine Supplement Containing a Saccharomyces Cerevisiae Fermentation Product

Abstract Controlled, evidence-based research studies are essential to ensure functional pet food, treat and supplement ingredients are efficacious. However, understanding how nutritional technologies influence pet health and well-being in the pet's home environment is an additional way to derive further insight on performance and effectiveness. Therefore, the objective of this study was to assess the efficacy of supplemental Saccharomyces cerevisiae fermentation product (SCFP) using subjective pet owner insights coupled with an analysis of fecal metagenomic characteristics in dogs using an in-home testing approach. Healthy adult dogs (n = 348; 170 males and 178 females; mean age = 6.0 y; mean BW = 18.2 kg) were recruited from across the U.S. and randomly allotted to one of two treatment groups blocked by breed, gender, age and BW. Owners were asked to feed their dogs their usual food, treats and supplements throughout the 60-d study. The SCFP was delivered in a daily chew. The control was a placebo chew void of SCFP (CON), and the treatment chew (EPI) was formulated to deliver 7 mg· kg BW-1· d-1 SCFP (EpiCor postbiotic, Ankeny, IA). Surveys were fielded on d 0, 15, 30, 45 and 60 to obtain pet owner subjective ratings of supplement performance across key health attributes. Fecal sample collection kits were provided to respondents on d 0, 30 and 60 for shotgun metagenomic analysis. All statistical analyses were conducted in R v.4.1.2. Ordinal logistic regression using the package ordinal in R was conducted on the survey data to understand relationships between the predictor variables and the ranked outcomes. Metagenomic data were analyzed using phyloseq, vegan, lme4, LinDA and emmeans packages in R. Differences P &amp;lt; 0.05 were considered significant and P ≤ 0.10 as trends after FDR correction. Dogs receiving the EPI chew received numerically higher immunity ratings throughout the study and overall tended to be 95% more likely (P = 0.10) to receive a higher immune health rating versus CON. By d 60, EPI dogs were more likely (211% greater odds; P = 0.007) to exhibit less breath odor and 165% more likely to be less anxious (P = 0.02) than CON dogs. An increased relative abundance in butyrate producing taxa Anaerostipes hadrus (P = 0.002) and Megamonas (P = 0.001) and Fusobacteria (P = 0.004) were observed in EPI dogs emerging at d 30. Under the conditions of this study, daily supplementation of a SCFP supplement provided a spectrum of positive health outcomes in healthy adult dogs. Moreover, there was consistency in physiological benefits noted in previous controlled studies in healthy dogs fed SCFP to the owner-perceived changes in several health attributes in this study. Our data also suggest that SCFP may beneficially affect key indicators of breath odor and anxiety in dogs.

RLaMs-Dehazing: Optimized Depth Map Improvement Single Colour Image Dehazing

Visibility Degradation is a classical problem owing to the presence of Atmospheric Particulate Matter (APM). There are different image dehazing algorithms. Any one method cannot be relied upon as each haze condition is unique. An innovative algorithm has been proposed inverting the image formation atmospheric scattering model [2, 32]. The model has been improvised by one key factor. This is Regularized Lagrangian multiplier (RLaM) based Depth Map (DM) refinement. The algorithm has low time complexity which intrigues real-time efficient applications. Different state-of-the-art visibility algorithms have been studied and their subjective and objective performance evaluations have been evaluated. Extensive investigation shows remarkable improvement with the proposed algorithm. This method is equally applicable to different atmospheric conditions. Time complexity experimented with execution time and Big (O) for real-time effectiveness. Extensive experiment results show the potential of the proposed algorithm independent of the influence of atmospheric conditions and capturing devices adaptive to computer vision applications. Time complexity and quality output trade-off achieved with the removal of ringing artifacts efficiently.

Objective and subjective neurocognitive functioning in functional motor symptoms and functional seizures: preliminary findings

ABSTRACT Introduction This study aimed to provide a preliminary assessment of objective and subjective neurocognitive functioning in individuals with functional motor symptoms (FMS) and/or functional seizures (FS). We tested the hypotheses that the FMS/FS group would display poorer objective attentional and executive functioning, altered social cognition, and reduced metacognitive accuracy. Method Individuals with FMS/FS (n = 16) and healthy controls (HCs, n = 17) completed an abbreviated CANTAB battery, and measures of intellectual functioning, subjective cognitive complaints, performance validity, and comorbid symptoms. Subjective performance ratings were obtained to assess local metacognitive accuracy. Results The groups were comparable in age (p = 0.45), sex (p = 0.62), IQ (p = 0.57), and performance validity (p-values = 0.10–0.91). We observed no impairment on any CANTAB test in this FMS/FS sample compared to HCs, although the FMS/FS group displayed shorter reaction times on the Emotional Bias task (anger) (p = 0.01, np2 = 0.20). The groups did not differ in subjective performance ratings (p-values 0.15). Whilst CANTAB attentional set-shifting performance (total trials/errors) correlated with subjective performance ratings in HCs (p-values<0.005, rs = −0.85), these correlations were non-significant in the FMS/FS sample (p-values = 0.10–0.13, rs-values = −0.46–0.50). The FMS/FS group reported more daily cognitive complaints than HCs (p = 0.006, g = 0.92), which were associated with subjective performance ratings on CANTAB sustained attention (p = 0.001, rs = −0.74) and working memory tests (p < 0.001, rs = −0.75), and with depression (p = 0.003, rs = 0.70), and somatoform (p = 0.003, rs = 0.70) and psychological dissociation (p-values<0.005, rs-values = 0.67–0.85). Conclusions These results suggest a discordance between objective and subjective neurocognitive functioning in this FMS/FS sample, reflecting intact test performance alongside poorer subjective cognitive functioning. Further investigation of neurocognitive functioning in FND subgroups is necessary.

Public speaking training in front of a supportive audience in Virtual Reality improves performance in real-life

Public speaking is a challenging task that requires practice. Virtual Reality allows to present realistic public speaking scenarios in this regard, however, the role of the virtual audience during practice remains unknown. In the present study, 73 participants completed a Virtual Reality practice session while audience was manipulated to be supportive or unsupportive or presentations were practiced without audience. Importantly, following the virtual practice, participants held the presentation during a real university course via Zoom. We measured emotional experience, self-efficacy, and the subjective evaluation of performance at baseline, after VR practice, and after the real presentation. Additionally, participants’ performance in the real presentation was evaluated by instructors (blinded to condition). Supportive in contrast to unsupportive audiences led to more positive believes about one’s own performance, while there were no changes in beliefs in the group without audience. Importantly, practice in front of a supportive compared to unsupportive audience resulted in a more positive evaluation of speaker confidence in real-life public speaking as rated by the instructors. These results demonstrate an impact of virtual social feedback during public speaking on subsequent subjective performance evaluation. This may increase self-confidence resulting in actual improved public speaking performance in real-life.

Do Birds of a Feather Flock Together? The Joint Effects of Manager and Subordinate Narcissism on Performance Evaluation

This study investigates the interaction of superior and subordinate manager narcissism on subjective performance evaluation. We build on extant research in the psychology literature and propose that superiors evaluate narcissistic subordinates less favorably than non-narcissistic subordinates. However, based on the homophily effect (i.e., the tendency of like to associate with like), we also hypothesize that narcissistic superiors show greater tolerance toward narcissistic subordinates than non-narcissistic superiors do. We conduct two online experiments and find that superiors, both high and low in narcissism, evaluate narcissistic subordinates less favorably than non-narcissistic subordinates. Our results further show that this effect is mediated by the superiors’ weaker feelings of closeness to narcissistic subordinates. Finally, we find that superiors high in narcissism show greater feelings of closeness to narcissistic subordinates and, hence, evaluate them more favorably than non-narcissistic superiors do. Our results provide novel insight into the pivotal role of individual traits in subjective performance evaluation.

Unwanted Recommendation

Abstract: In performance appraisal processes, corporations face the problem that the subjective ratings of employee performance tend to be largely biased toward being too favorable and too homogeneous. Drawing on prospect theory, we conceptualize below-average employee ratings as perceived losses and suggest that the proportion of such ratings increases under a recommended rating distribution because of a higher rating dispersion. The perceived losses may decrease employees’ organizational commitment, job satisfaction, and cooperation while increasing their turnover intentions. Using a large-scale, dual-source proprietary panel dataset from the German Federal Employment Agency, we show that a recommended distribution rating system is negatively related to organizational commitment, while job satisfaction, turnover intentions, and cooperation are unaffected when considering the whole sample. Post-hoc analyses reveal that the adverse effects appear in employees without managerial responsibility, while employees with managerial responsibility remain unaffected by the recommended distribution rating system.

Peer effects in subjective performance evaluation

AbstractWe investigate the influence of peer quality on subjective performance evaluation using 75,413 ratings of 130 employees from 6,908 raters in a business school setting. We find that subjective performance ratings are lower for employees with higher quality peer groups in both randomized and nonrandomized settings. Using a novel long‐window setting, we observe peer effects persisting, but slowly decaying, for several months even when priming raters with the employees' previous performance information. We find that the strength of the peer effects is greater for focal employees with weaker performance, for the peers with higher attribute similarity, and when the performance of peers is more extreme. Overall, we find strong and persistent peer effects in subjective performance evaluation.

Performance: Differences in measuring performance

An important issue in human resources is the procedure of the employees’ performance evaluation. The appraisal is essential in the sense of employee appreciation and motivation. Most employers use a subjective performance evaluation of a single superior or a group of persons involved in the employee’s working processes. The subjective evaluation of a group or one person is often questioned about being appropriate. An often-named solution for a more objective criteria could be data driven performance measures. Professional sport provides a unique opportunity to compare objective and subjective performance evaluation measures. A data set of the German Bundesliga was used to test if the two different performance measure come to equal results. It is shown that differences in means exist but equivalence tests support the hypothesis that both measures could be treated as equal. In toto, it seems that in an environment where performance is relatively good to measure objective and subjective performance evaluations lead to equivalent results.

Narratives in Subjective Performance Evaluations: Do Ratings Change the Narrative?

ABSTRACT Several high-profile companies have removed numerical subjective performance ratings from their performance-management processes in favor of only using narrative evaluations. Using an experiment, I examine whether requiring supervisors to provide a numerical subjective performance rating in addition to a narrative evaluation moderates the effects of supervisors’ directional evaluation incentives (i.e., reasons to evaluate an employee more or less favorably) on the favorability of their narrative evaluations. As predicted, I find the favorability of supervisors’ narrative evaluations reflects their directional evaluation incentives, but to a lesser degree when they also provide a numerical performance rating. Theory suggests the moderating effects of providing a numerical rating occur because (1) numerical ratings are biased toward the middle of the scale, and (2) supervisors strive for “consistency” between their numerical ratings and narrative evaluations. I demonstrate that the rating process affects narratives, and removing subjective numerical ratings may have unintended consequences in practice.