Objective Performance Evaluation Research Articles

객관적 성과평가와 주관적 시민만족도의 관계 : 서울시 자치구 민원행정서비스를 대상으로

본 연구는 기존연구의 한계를 보완하여 객관적 성과평가와 주관적 시민만족도의 관계에 대한 보다 타당한 결과를 도출하고자 하였다. 객관적 성과는 DEA 효율성 점수로, 그리고 주관적 시민만족도는 서울시 자치구 민원행정서비스 시민평가의 종합만족도 지수와 효율성 차원만족도 지수로 측정하였다. 양자 간의 관계는 서울시 25개 자치구의 민원행정서비스를 대상으로 상관관계분석, OLS, HLM을 이용하여 검증하였다. 상관관계분석 결과, 양자 간의 관계는 자치구 수준에서는 통계적으로 무의미하나, 개인 수준에서는 실질적인 관계의 정도는 약하지만 통계적으로 유의미한 것으로 나타났다. 반면, 회귀분석에서는 OLS와 HLM 모두에서 객관적 성과가 종합만족도 지수와 효율성 차원만족도 지수 모두에 통계적으로 유의미한 양(+)의 영향을 미치는 것으로 나타났다. 본 연구의 결과는 개념적 부합성 부족, 자료의 집계화, 모형설정 오류 등이 양자간 불일치를 유발한 원인이라는 이론적 주장이 타당하며, 양자를 결합한 종합성과평가 모형을 적용하여 공공부문의 성과를 평가하는 것이 올바른 방향이라는 것을 시사한다.

Not Minding the Gap: How Hostile Sexism Encourages Choice Explanations for the Gender Income Gap

Researchers have shown that prejudice encourages explanations for inequality that attribute stigmatized groups’ negative outcomes to internal-controllable causes. We extended this research by investigating how ambivalent sexism affects attributions for gender income inequality. Hostile sexism should facilitate acceptance of gender income inequality through attributions that emphasize individual choice. We tested this hypothesis in two web-based samples of predominately White American men and women, ranging in age from 18 to 82 years ( Mage = 33.8). In Study 1 ( N = 650), hostile sexism, but not benevolent sexism, positively predicted acceptance of gender income inequality. Attributions of choice and societal unfairness mediated this effect. In Study 2 ( N = 242), following exposure to hostile sexism, participants increased acceptance of gender income inequality; choice explanations mediated this relation, although these effects occurred for political conservatives only. Consistent with prior work on attributions, hostile sexism was linked to victim-blaming attributions for gender income inequality. Overall, hostile sexism creates an attitudinal barrier—especially for conservatives—to supporting equal pay for women. To overcome this barrier, organizations could implement strategies aimed at ensuring more objective performance evaluations and pay decisions. Further, policy makers and communicators should be careful in choosing how they frame the gender pay gap. Additional online materials for this article are available on PWQ’s website at http://journals.sagepub.com/doi/suppl/10.1177/0361684318815468

A combined mono- and multi-turbine approach for fault indicator synthesis and wind turbine monitoring using SCADA data

The monitoring of wind turbines using SCADA data has received lately a growing interest from the fault diagnosis community because of the very low cost of these data, which are available in number without the need for any additional sensor. Yet, these data are highly variable due to the turbine constantly changing its operating conditions and to the rapid fluctuations of the environmental conditions (wind speed and direction, air density, turbulence, …). This makes the occurrence of a fault difficult to detect. To address this problem, we propose a multi-level (turbine and farm level) strategy combining a mono- and a multi-turbine approach to create fault indicators insensitive to both operating and environmental conditions. At the turbine level, mono-turbine residuals (i.e. a difference between an actual monitored value and the predicted one) obtained with a normal behavior model expressing the causal relations between variables from the same single turbine and learnt during a normal condition period are calculated for each turbine, so as to get rid of the influence of the operating conditions. At the farm level, the residuals are then compared to a wind farm reference in a multi-turbine approach to obtain fault indicators insensitive to environmental conditions. Indicators for the objective performance evaluation are also proposed to compare wind turbine fault detection methods, which aim at evaluating the cost/benefit of the methods from a production manager’s point of view. The performance of the proposed combined mono- and multi-turbine method is evaluated and compared to more classical methods proposed in the literature on a large real data set made of SCADA data recorded on a French wind farm during four years : it is shown than it can improve the fault detection performance when compared to a residual analysis limited at the turbine level only.

The Effect of Ground Truth on Performance Evaluation of Hyperspectral Image Classification

In the field of hyperspectral image classification, a widely used way for objective performance evaluation of different classification methods is calculating three accuracy indexes, i.e., the overall accuracy, the average accuracy, and the Kappa coefficient. These accuracy indexes are obtained by comparing the classification results with the ground truth, i.e., a reference classification map labeled by human experts. In this paper, the effect of ground truths on the objective performance evaluation of hyperspectral image classification is studied. The purpose is to investigate, if the ground truth is insufficient, whether the above accuracy indexes can be completely responsible. Furthermore, in order to measure the robustness of different classification methods to those insufficient ground truths, four evaluation metrics, i.e., the Pearson linear correlation coefficient, root-mean-square error, Spearman's rank correlation coefficient, and Kendall's rank correlation coefficient have been adopted for further analysis. Based on these experiments, an interesting conclusion can be obtained that insufficient ground truths may limit the assessment capability of existing accuracy indexes. This underlines that overoptimistic performance evaluations may exist and stresses the demand of designing more appropriate accuracy indexes for objective performance evaluation with insufficient ground truths.

An objective and subjective quality assessment study of passive gaming video streaming

SummaryPassive gaming video‐streaming applications have recently gained much attention as evident with the rising popularity of many Over The Top (OTT) providers such as Twitch.tv and YouTube Gaming. For the continued success of such services, it is imperative that the user Quality of Experience (QoE) remains high, which is usually assessed using subjective and objective video quality assessment methods. Recent years have seen tremendous advancement in the field of objective video quality assessment (VQA) metrics, with the development of models that can predict the quality of the videos streamed over the Internet. A study on the performance of objective VQA on gaming videos, which are artificial and synthetic and have different streaming requirements than traditionally streamed videos, is still missing. Towards this end, we present in this paper an objective and subjective quality assessment study on gaming videos considering passive streaming applications. Subjective ratings are obtained for 90 stimuli generated by encoding six different video games in multiple resolution‐bitrate pairs. Objective quality performance evaluation considering eight widely used VQA metrics is performed using the subjective test results and on a data set of 24 reference videos and 576 compressed sequences obtained by encoding them in 24 resolution‐bitrate pairs. Our results indicate that Video Multimethod Assessment Fusion (VMAF) predicts subjective video quality ratings the best, while Naturalness Image Quality Evaluator (NIQE) turns out to be a promising alternative as a no‐reference metric in some scenarios.

A computer vision technique for automated assessment of surgical performance using surgeons' console-feed videos.

To develop and validate an automated assessment of surgical performance (AASP) system for objective and computerized assessment of pelvic lymph node dissection (PLND) as an integral part of robot-assisted radical cystectomy (RARC) using console-feed videos recorded during live surgery. Video recordings of 20 PLNDs were included. The quality of lymph node clearance was assessed based on the features derived from the computer vision process which include: the number and cleared area of the vessels/nerve (N-Vs); image median color map; and mean entropy (measures the level of disorganization) in the video frame. The automated scores were compared to the validated pelvic lymphadenectomy appropriateness and completion evaluation (PLACE) scoring rated by a panel of expert surgeons. Logistic regression analysis was employed to compare automated scores versus PLACE scores. Fourteen procedures were used to develop the AASP algorithm. A logistic regression model was trained and validated using the aforementioned features with 30% holdout cross-validation. The model was tested on the remaining six procedures, and the accuracy of predicting the expert-based PLACE scores was 83.3%. To our knowledge, this is the first automated surgical skill assessment tool that provides an objective evaluation of surgical performance with high accuracy compared to expert surgeons' assessment that can be extended to any endoscopic or robotic video-enabled surgical procedure.

Performance Appraisal and Employee Commitment in Abia State Civil Service: A Focus on Ministries of Education and Works

This work focused on performance appraisal and employee commitment in Abia State Civil Service. The dynamic nature of employees’ commitment to their Ministries made this investigation very necessary. The objectives of the study were; to find out whether objective evaluation of employee performance has a positive relationship with employee affective commitment, to ascertain whether subjective evaluation of employee performance has relationship with employee continuance commitment. To achieve this objective, a survey research design was adopted. The techniques employed in analyzing the data were descriptive statistics and Pearson Correlation Coefficient. The results indicated that objective evaluation of employee performance has positive relationship with employee affective commitment. It was also found that subjective evaluation of employee performance has a weak and in significant relationship with employee continuance commitment. The study concluded that objective evaluation of employee performance has a positive relationship with employee affective commitment and subjective evaluation of employee performance has a weak relationship with employee continuance commitment in Ministries of Education and Works Umuahia. It was recommended that Directors/managers of public sector should ensure that performance criteria are clearly spelt against which employees at any level of management is appraised to ensure objectivity in the assessment.

Evaluating model performance: towards a non-parametric variant of the Kling-Gupta efficiency

ABSTRACTGoodness-of-fit measures are important for an objective evaluation of runoff model performance. The Kling-Gupta efficiency (RKG), which has been introduced as an improvement of the widely used Nash-Sutcliffe efficiency, considers different types of model errors, namely the error in the mean, the variability, and the dynamics. The calculation of RKG is implicitly based on the assumptions of data linearity, data normality, and the absence of outliers. In this study, we propose a modification of RKG as an efficiency measure comprising non-parametric components, i.e. the Spearman rank correlation and the normalized flow–duration curve. The performances of model simulations for 100 catchments using the new measure were compared to those obtained using RKG based on a number of statistical metrics and hydrological signatures. The new measure resulted overall in better or comparable model performances, and thus it was concluded that efficiency measures with non-parametric components provide a suitable alternative to commonly used measures.

Application of the steering law to virtual reality walking navigation interfaces

Navigation through immersive virtual environments is a key concept for virtual reality as it allows users to explore those environments. Therefore, it is important to understand virtual reality navigation interfaces and their impact on the users’ experience. This paper presents an objective performance evaluation of two types of navigation: natural (real walking and walk-in-place) vs. unnatural (gamepad). Steering Law was the objective performance metric chosen since it captures the relationship between the time to travel a path and the difficulty of that path. In addition to performance, subjective metrics were also considered, namely the feeling of presence, cybersickness and user satisfaction. The experiments consisted of having participants complete a series of paths with different indexes of difficulty and the time that a participant took to walk each path was measured. Overall results show that the navigation through real walking yielded better results when it comes to performance, cybersickness, and user satisfaction than the walk-in-place and gamepad navigation interfaces.

Simulation of wind turbine faulty production profiles and performance assessment of fault monitoring methods

Wind turbines being one of the fastest growing sources of renewable energy have garnered significant scientific interest for the monitoring and fault analysis using SCADA data. Various monitoring approaches using power curves, i.e. industry wide characteristic curves expressing produced power as a function of wind speed, have been proposed in the literature. However, an objective comparison of the performance of these methods is difficult. The difficulty comes from (i) the variability in operational and environmental conditions taken into account; (ii) the nature, size and type of data-sets used and (iii) the type and signatures of faults considered for validation. To solve this problem, an approach with a twofold contribution is proposed in this work: 1) an original procedure to generate realistic and controlled simulations of 10 minutes SCADA data, simulating situations when the wind turbine is operating in normal or faulty conditions, is presented; 2) a framework for objective performance assessment of the fault detection methods, based on the proposed controlled and standardized simulation scheme is presented. Objective performance evaluation metrics, such as detection probability and false alarm rates are computed and represented as characteristic Receiver Operating Curves (ROC). The proposed simulation approach is shown to provide a useful global framework for objective performance analysis. A number of realistically simulated and controlled data streams are used to compare and discuss the performances of two fault detection methods referenced in the literature.

Development and validation of surgical training tool: cystectomy assessment and surgical evaluation (CASE) for robot-assisted radical cystectomy for men.

We aimed to develop a structured scoring tool: cystectomy assessment and surgical evaluation (CASE) that objectively measures and quantifies performance during robot-assisted radical cystectomy (RARC) for men. A multinational 10-surgeon expert panel collaborated towards development and validation of CASE. The critical steps of RARC in men were deconstructed into nine key domains, each assessed by five anchors. Content validation was done utilizing the Delphi methodology. Each anchor was assessed in terms of context, score concordance, and clarity. The content validity index (CVI) was calculated for each aspect. A CVI ≥ 0.75 represented consensus, and this statement was removed from the next round. This process was repeated until consensus was achieved for all statements. CASE was used to assess de-identified videos of RARC to determine reliability and construct validity. Linearly weighted percent agreement was used to assess inter-rater reliability (IRR). A logit model for odds ratio (OR) was used to assess construct validation. The expert panel reached consensus on CASE after four rounds. The final eight domains of the CASE included: pelvic lymph node dissection, development of the peri-ureteral space, lateral pelvic space, anterior rectal space, control of the vascular pedicle, anterior vesical space, control of the dorsal venous complex, and apical dissection. IRR > 0.6 was achieved for all eight domains. Experts outperformed trainees across all domains. We developed and validated a reliable structured, procedure-specific tool for objective evaluation of surgical performance during RARC. CASE may help differentiate novice from expert performances.

PD58-04 MODELING AUTOMATED ASSESSMENT OF SURGICAL PERFORMANCE UTILIZING COMPUTER VISION: PROOF OF CONCEPT

You have accessJournal of UrologySurgical Technology & Simulation: Training & Skills Assessment II1 Apr 2018PD58-04 MODELING AUTOMATED ASSESSMENT OF SURGICAL PERFORMANCE UTILIZING COMPUTER VISION: PROOF OF CONCEPT Amir Baghdadi, Lora Cavuoto, Ahmed Aly Hussein, Youssef Ahmed, and Khurshid Guru Amir BaghdadiAmir Baghdadi More articles by this author , Lora CavuotoLora Cavuoto More articles by this author , Ahmed Aly HusseinAhmed Aly Hussein More articles by this author , Youssef AhmedYoussef Ahmed More articles by this author , and Khurshid GuruKhurshid Guru More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2018.02.2792AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES Thorough lymph node dissection (LND) is an integral part of robot-assisted radical cystectomy (RARC). There is a lack of consensus about what constitutes adequate LND. Current methods are subject to inter-rater variability. In this context, we sought to use computer vision methods to identify and extract valid measures to develop and validate an automated scoring system for LND. METHODS 20 recorded LNDs were included with a total of 200 frames/case from the console feed before and after LND with near-equivalent view and zoom. The quality of lymph node clearance was assessed based on the features derived from a computer vision algorithm: the number and area of the nerve/vessels (N-Vs) detected using the proposed Automated Structure Detection (ASD) method; image median Color Map by the assumption of decrease in yellow color after lymphatic and fatty tissue removal; and mean entropy, which measures the level of disorganization in the image. Each video frame was pre-processed for N-Vs detection using a series of image processing operations including binary conversion, edge and line detection, and object identification while considering geometrical characteristics of target objects, e.g. aspect ratio, width, height, and orientation. The N-Vs were labeled by fusing the information from both line and object detection processes (Figure 1). The automated scores (AS) were compared to Pelvic Lymphadenectomy Appropriateness and Completion Evaluation (PLACE), which is a subjective evaluation based on objective measures scored by a panel of expert surgeons. Logistic regression analysis was employed to compare AS and PLACE scores. RESULTS 14 cases were used to develop the automated scoring algorithm. A logistic regression model was trained and validated using the aforementioned features with 30% holdout cross validation. This model was applied to the remaining 6 previously unseen cases for testing and the accuracy of predicting the PLACE scores was 83.3% (5 correct score allocation across the 6 test cases). CONCLUSIONS To our knowledge,this is the first automated surgical skill assessment tool that provides objective evaluation of surgical performance with high accuracy compared to expert surgeon assessment. © 2018FiguresReferencesRelatedDetails Volume 199Issue 4SApril 2018Page: e1134-e1135 Advertisement Copyright & Permissions© 2018MetricsAuthor Information Amir Baghdadi More articles by this author Lora Cavuoto More articles by this author Ahmed Aly Hussein More articles by this author Youssef Ahmed More articles by this author Khurshid Guru More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...

Performance analysis and computational cost evaluation of high-resolution time-frequency distributions derived from compact support time-lag kernels

This paper considers the objective performance evaluation of kernel-based time-frequency distributions (TFDs) using several concentration performance measures and resolution examination through a deep analysis of time slice plots. On the other hand, the numerical complexity of each TFD is evaluated; a parameter that is particularly critical when real-time implementation is intended. The performance of TFDs based on time-lag kernels with compact support (KCS) namely the Cheriet–Belouchrani (CB), the separable (CB) (SCB) and the polynomial CB (PCB) TFDs is compared to the well-known kernel-based TFDs using several tests on real-life and multicomponent signals with linear and nonlinear frequency modulation (FM) components including the noise effects and the influence of the kernel length. In all presented examples, the time-lag KCS TFDs, and particularly the PCB TFD, provide the best compromise between highest autoterm resolution and interference rejection while still requiring moderate computational costs thanks to the compact support nature of their kernels that reduces the number of points needing computation. On the other hand, the derived distributions do not require any smoothing window neither in time nor frequency in order to achieve the best time-frequency resolution. Furthermore, they have an extremely interesting practical advantage since their adjustment is performed by simply changing a single parameter which is integer for the PCB TFD.

Image smoothing via truncated gradient regularisation

Edge-preserving image smoothing aims at maintaining the fundamental constituents, i.e. salient edges of a given image, while removing the noise and insignificant details in the meantime. It is often employed at the pre-processing step of many image processing tasks, including reconstruction, segmentation, recognition and three-dimensional content generation, to name just a few. Recently, a sparse gradient counting scheme in an optimisation framework has attracted much attention, and it confines the discrete number of intensity changes among neighbouring pixels. This l 0 -regularised sparsity pursuit scheme performs favourably in a global optimisation manner. However, it often achieves unsatisfactory performance at diminishing trivial details and at smoothing discrete regions. In this study, a new image smoothing scheme with truncated l 0 regularisation is proposed, which is especially effective for sharpening critical edges. For objective evaluation of the smoothing performance, images are linearly quantised into several layers to generate the experimental images, then these quantised images are smoothed using several methods for reconstructing the smoothly changed shape and intensity of the original images. Compared with the l 0 smoothing scheme, extensive experimental results demonstrate that the proposed method performs much better at preserving main structures and removing trivial details.

Objective visual performance evaluation with visual evoked potential measurements based on an adaptive optics system

An objective visual performance evaluation with visual evoked potential (VEP) measurements was first integrated into an adaptive optics (AO) system. The optical and neural limits to vision can be bypassed through this system. Visual performance can be measured electrophysiologically with VEP, which reflects the objective function from the retina to the primary visual cortex. The VEP measurements without and with AO correction were preliminarily carried out using this system, demonstrating the great potential of this system in the objective visual performance evaluation. The new system will provide the necessary technique and equipment support for the further study of human visual function.

Upgrading service quality of mobile banking

This study combines the opinions of consumers and experts (managers and scholars) and employs the four-phase research methods (IPGA, DEMATEL, QFD and MBO) in order to determine the key factors for the superior service of mobile banking. The results of IPGA and DEMATEL show a total of 7 critical service quality factors that need to be improved for mobile banking. According to the QFD results, 6 service quality improvement strategies have been developed, and the sequence of improvements is adequately discussed herein. Lastly, this study establishes a set of mobile banking objective management progress control and performance evaluation sheets, based on the concept of MBO, in order to manage and control the improvement progress and effect of mobile banking service quality. The proposed mechanism can achieve the objective of superior service quality in an expedited manner with limited resources.

Analiza performansi objektivnih mera procene kvaliteta slika i videa sa potpunim referenciranjem

This paper presents the performance evaluation of image and video quality assessment metrics on two publicly available datasets with subjective quality ratings. In addition to the performance analysis at the global level - at the level of complete datasets, the paper presents the objective measures performance evaluation on subsets of signals inside them. The image dataset contains five subsets created by using different types of JPEG compression, while the video dataset contains six subsets of sequences - four created by compression of original sequences, and two subsets are with video signal transmission characteristic degradations. To determine the success of objective measures, i.e. comparison of subjective and objective quality scores, there were used measures accepted by the International Telecommunication Union - ITU (linear correlation coefficient, rank-order correlation coefficient, mean absolute error, root mean squared error and outlier ratio). It was shown that objective quality measures can reach a high level of agreement with the results of subjective tests on subsets of datasets. Objective measures performances depend on the type of degradation which significantly affects the performance at the complete dataset level. The difference in performances is more pronounced on video sequences due to considerable visual differences in sequences created by using compression, packet losses and additive Gaussian noise. Therefore, we can say that a universal objective measure, i.e. measure that is useful for different types of signal degradation, for different degradation levels, and for different applications currently does not exist.

Relationship Between Surgeon’s Brain Functional Network Reconfiguration and Performance Level During Robot-Assisted Surgery

The current methods of assessment of surgical performance for robot-assisted surgery are subjective. In this paper, we propose a cognitive-based method for objective evaluation of performance. Changes in brain functional networks were extracted and their relationship with performance level was investigated. We used electroencephalogram data recorded from a mentor surgeon’s brain while supervising and performing surgical tasks of varying complexity [urethrovesical anastomosis (UVA) and lymph-node dissection (LND)]. Multilayer community detection techniques were used to extract functional network communities at frequency bands of $\theta$ , $\alpha$ , lower $\beta$ , upper $\beta$ , and $\gamma$ . Results showed different detected communities while supervising and performing LND (more complex). However, for UVA (less complex), the majority of functional communities were similar. This is likely because, in less complicated tasks, the trainee’s performance more closely matched the mentor’s expectation. Entropy and power distribution through frequency bands showed minimum thermodynamic stability during $\alpha$ and the maximum during $\gamma$ . The relaxation time for channels with high entropy level was also extracted as a brain functional metric at thermodynamic stability state. These metrics may be used to quantify changes of brain functional network as performance improves.

Instance spaces for machine learning classification

This paper tackles the issue of objective performance evaluation of machine learning classifiers, and the impact of the choice of test instances. Given that statistical properties or features of a dataset affect the difficulty of an instance for particular classification algorithms, we examine the diversity and quality of the UCI repository of test instances used by most machine learning researchers. We show how an instance space can be visualized, with each classification dataset represented as a point in the space. The instance space is constructed to reveal pockets of hard and easy instances, and enables the strengths and weaknesses of individual classifiers to be identified. Finally, we propose a methodology to generate new test instances with the aim of enriching the diversity of the instance space, enabling potentially greater insights than can be afforded by the current UCI repository.

Multi-focus image fusion algorithm based on pulse coupled neural networks and modified decision map

In this paper, we present a new approach for the multi-focus image fusion, which utilizes Orientation Information(OI) Motivated Pulse Coupled Neural Networks(PCNN) and modified decision map. First, the source multi-focus images are fused using the Orientation Information Motivated Pulse Coupled Neural Networks (OI-PCNN), and the initial decision map is obtained. Second, after analyzing the decision map, we modified the decision map by employing a mathematical morphology post-processing technique. Finally, based on the modified decision map, the final fused image is obtained by selecting the pixels in the focus areas. Objective performance evaluation criteria and visual observation demonstrate that the proposed method is better than various existing spatial domain and transform domain fusion methods, including NSCT, PCNN-NSCT, SF-PCNN-NSCT, and EOE-PCNN-NSCT method.