Fine-grained Evaluation Research Articles

Hierarchical attention for aspect extraction using LSTM in fine-grained sentiment analysis and evaluation

In the current digital world, aspect-based sentiment analysis (ABSA) from a review sentence is a big challenge. In recent works, the aspect term detection from the reviews of the discourse is performed independently, even for contextually interdependent reviews. Further, the baseline techniques could not correlate valid multi-word aspect extraction from different review sentences, which further degrades the aspect extraction performance. Therefore, this paper aims to improve the performance of the aspect term detection subtask using a supervised hierarchical attention-based model. A sentence alignment step is added as preprocessing before ABSA on multi-sentence reviews. This step removes deficient knowledge, captures aspect alignment from different review sentences, and properly relates the aspect to the context domain. Next, the context information is used to extract contextual aspects using word-vector representation. The input sentences and labeled data using sequence tagging are supplied to the proposed supervised hierarchical attention network. The proposed model can capture the semantics of words for different domains, enhancing the extraction of domain-specific relevant aspect terms. The experimental results shown in the SemEval-16 reviews present the effectiveness of the proposed attention-based approach for aspect term detection. The F-score is improved by approximately 3 % as compared to the recent hybrid unsupervised approach for both the laptop and restaurant domains when coreference resolution is not performed. On the other side, F-score is improved by approximately 4 % and 3 % as compared to the recent hybrid unsupervised approach for the laptop and restaurant domains, respectively, when coreference resolution is not performed. When the results of the proposed approach are compared with those of recent supervised approaches, the F-score of the proposed model is improved by 5 % and 9 % for laptops and restaurants, respectively, when coreference resolution is not performed. Whereas the F-score of the proposed model is improved by 9 % and 12 % for laptops and restaurants, respectively, when coreference resolution is performed before the aspect extraction subtask.

An Improved Instance Segmentation Method for Fast Assessment of Damaged Buildings Based on Post-Earthquake UAV Images.

Quickly and accurately assessing the damage level of buildings is a challenging task for post-disaster emergency response. Most of the existing research mainly adopts semantic segmentation and object detection methods, which have yielded good results. However, for high-resolution Unmanned Aerial Vehicle (UAV) imagery, these methods may result in the problem of various damage categories within a building and fail to accurately extract building edges, thus hindering post-disaster rescue and fine-grained assessment. To address this issue, we proposed an improved instance segmentation model that enhances classification accuracy by incorporating a Mixed Local Channel Attention (MLCA) mechanism in the backbone and improving small object segmentation accuracy by refining the Neck part. The method was tested on the Yangbi earthquake UVA images. The experimental results indicated that the modified model outperformed the original model by 1.07% and 1.11% in the two mean Average Precision (mAP) evaluation metrics, mAPbbox50 and mAPseg50, respectively. Importantly, the classification accuracy of the intact category was improved by 2.73% and 2.73%, respectively, while the collapse category saw an improvement of 2.58% and 2.14%. In addition, the proposed method was also compared with state-of-the-art instance segmentation models, e.g., Mask-R-CNN and YOLO V9-Seg. The results demonstrated that the proposed model exhibits advantages in both accuracy and efficiency. Specifically, the efficiency of the proposed model is three times faster than other models with similar accuracy. The proposed method can provide a valuable solution for fine-grained building damage evaluation.

Fully Automatic Fine-Grained Grading of Lumbar Intervertebral Disc Degeneration Using Regional Feature Recalibration Network.

Accurate fine-grained grading of lumbar intervertebral disc (LIVD) degeneration is essential for the diagnosis and treatment design of high-incidence low back pain. However, the grading accuracy is still challenged by lacking the fine-grained degenerative details, which is mainly due to the existing grading methods are easily dominated by the salient nucleus pulposus regions in LIVD, overlooking the inconspicuous degeneration changes of the surrounding structures. In this study, a novel regional feature recalibration network (RFRecNet) is proposed to achieve accurate and reliable LIVD degeneration grading. Detection transformer (DETR) is first utilized to detect all LIVDs and then input to the proposed RFRecNet for the fine-grained grading. To obtain sufficient features from both the salient nucleus pulposus and the surrounding regions, a regional cube-based feature boosting and suppression (RC-FBS) module is designed to adaptively recalibrate the feature extraction and utilization from the various regions in LIVD, and a feature diversification (FD) module is proposed to capture the complementary semantic information from the multi-scale features for the comprehensive fine-grained degeneration grading. Extensive experiments were conducted on a clinically collected dataset, which consists of 500 MR scans with a total of 10225 LIVDs. An average grading accuracy of 90.5%, specificity of 97.5%, sensitivity of 90.8%, and Cohen's kappa correlation coefficient of 0.876 are obtained, which indicate that the proposed framework is promising to provide doctors with reliable and consistent fine-grained quantitative evaluation results of the LIVD degeneration conditions for the optimal surgical plan design.

Can ChatGPT evaluate research quality?

Abstract Purpose Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Design/methodology/approach Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements. Findings ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author’s significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations. Research limitations The data is self-evaluations of a convenience sample of articles from one academic in one field. Practical implications Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use. Originality/value This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.

Self-Supervised Antipodal Grasp Learning With Fine-Grained Grasp Quality Feedback in Clutter

It is a challenging goal in robotics to make a robot grasp like a human being in a cluttered environment. Self-supervised grasp learning is one of the most promising approaches to human-like robotic grasp. However, due to inadequate feedback on grasp quality, almost existing self-supervised grasp learning methods are coarse-grained. This paper presents a fine-grained antipodal grasp learning (FAGL) method with augmented learning feedback. First, an indicator called antipodal degree of a grasp (ADG) is designed as a non-increasing monotonous function for the fine-grained evaluation of grasp quality according to the disturbance incurred by a grasp to the surroundings in the image space. Next, we design a restorative sampling strategy to collect antipodal grasp samples and propose a refined affordance network to generate grasp affordance maps for FAGL's decision of grasp policies. Finally, in grasping actual metal workpieces, FAGL outperforms its peers in grasp success rate and ADG in cluttered scenarios by reducing the grasp effects on the surroundings.

DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation

Instruction-following is particularly crucial for large language models (LLMs) to support diverse user requests. While existing work has made progress in aligning LLMs with human preferences, evaluating their capabilities on instruction-following remains a challenge due to complexity and diversity of real-world user instructions. While existing evaluation methods focus on general skills, they suffer from two main shortcomings, i.e., lack of fine-grained task-level evaluation and reliance on singular instruction expression. To address these problems, this paper introduces DINGO, a fine-grained and diverse instruction-following evaluation dataset that has two main advantages: (1) DINGO is based on a manual annotated, fine-grained and multi-level category tree with 130 nodes derived from real-world user requests; (2) DINGO includes diverse instructions, generated by both GPT-4 and human experts. Through extensive experiments, we demonstrate that DINGO can not only provide more challenging and comprehensive evaluation for LLMs, but also provide task-level fine-grained directions to further improve LLMs.

GFBE: A Generalized and Fine-Grained Blockchain Evaluation Framework

GFBE: A Generalized and Fine-Grained Blockchain Evaluation Framework

Generative AI-based Data Completeness Augmentation Algorithm for Data-driven Smart Healthcare.

In the decade, artificial intelligence has achieved great popularity and applications in medicine and healthcare. Various AI-based algorithms have shown astonishing performance. However, in various data-driven smart healthcare algorithms, the problem of incomplete dataset remains a huge challenge. In this paper, we propose a data completeness enhancement algorithm based on generative AI (i.e., GenAI-DAA) to solve the problems of the in-sufficient data for model training, the data imbalance, and the biases of the training samples. We first construct the cognitive field of the generative models and effectively understand the state of incomplete cognition in generative models. Secondly, on this basis, we propose a quest algorithm for abnormal samples in the cognitive field based on local outlier factor. By fine-grained value evaluation, abnormal samples are given more refined attention. Finally, integrating the above process through multiple cognitive adjustments, GenAI-DAA gradually improves the cognitive ability. GenAI-DAA can be summarized as "Quest-→Estimate-→Tune-up". We have conducted extensive experiments to demonstrate the effectiveness of our proposed algorithm, and shown widely applications to some typical data-driven smart healthcare algorithms.

AdaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds

The advent of Multilingual Language Models (MLLMs) and Large Language Models (LLMs) has spawned innovation in many areas of natural language processing. Despite the exciting potential of this technology, its impact on developing high-quality Machine Translation (MT) outputs for low-resource languages remains relatively under-explored. Furthermore, an open-source application, dedicated to both fine-tuning MLLMs and managing the complete MT workflow for low-resources languages, remains unavailable. We aim to address these imbalances through the development of adaptMLLM, which streamlines all processes involved in the fine-tuning of MLLMs for MT. This open-source application is tailored for developers, translators, and users who are engaged in MT. It is particularly useful for newcomers to the field, as it significantly streamlines the configuration of the development environment. An intuitive interface allows for easy customisation of hyperparameters, and the application offers a range of metrics for model evaluation and the capability to deploy models as a translation service directly within the application. As a multilingual tool, we used adaptMLLM to fine-tune models for two low-resource language pairs: English to Irish (EN↔ GA) and English to Marathi (EN↔MR). Compared with baselines from the LoResMT2021 Shared Task, the adaptMLLM system demonstrated significant improvements. In the EN→GA direction, an improvement of 5.2 BLEU points was observed and an increase of 40.5 BLEU points was recorded in the GA→EN direction representing relative improvements of 14% and 117%, respectively. Significant improvements in the translation performance of the EN↔MR pair were also observed notably in the MR→EN direction with an increase of 21.3 BLEU points which corresponds to a relative improvement of 68%. Finally, a fine-grained human evaluation of the MLLM output on the EN→GA pair was conducted using the Multidimensional Quality Metrics and Scalar Quality Metrics error taxonomies. The application and models are freely available.

Person-action Instance Search in Story Videos: An Experimental Study

Person-Action instance search (P-A INS) aims to retrieve the instances of a specific person doing a specific action, which appears in the 2019–2021 INS tasks of the world-famous TREC Video Retrieval Evaluation (TRECVID). Most of the top-ranking solutions can be summarized with a Division-Fusion-Optimization (DFO) framework, in which person and action recognition scores are obtained separately, then fused, and, optionally, further optimized to generate the final ranking. However, TRECVID only evaluates the final ranking results, ignoring the effects of intermediate steps and their implementation methods. We argue that conducting the fine-grained evaluations of intermediate steps of DFO framework will (1) provide a quantitative analysis of the different methods’ performance in intermediate steps; (2) find out better design choices that contribute to improving retrieval performance; and (3) inspire new ideas for future research from the limitation analysis of current techniques. Particularly, we propose an indirect evaluation method motivated by the leave-one-out strategy, which finds an optimal solution surpassing the champion teams in 2020–2021 INS tasks. Moreover, to validate the generalizability and robustness of the proposed solution under various scenarios, we specifically construct a new large-scale P-A INS dataset and conduct comparative experiments with both the leading NIST TRECVID INS solution and the state-of-the-art P-A INS method. Finally, we discuss the limitations of our evaluation work and suggest future research directions.

A Dynamic and Fine-Grained User Trust Evaluation Model for Micro-Segmentation Cloud Computing Environment

<p>With the diversity and complexity of user access behaviors in the “micro-segmentation” cloud computing environment, it is no longer possible to control unauthorized access of authorized users by only relying on user identity login authentication to control user access to cloud resources. The existing trust evaluation methods can not cope with the characteristics of “micro-isolated” cloud environment, which is characterized by high granularity of resources, increasing number of users’ access requests and rapid changes. Based on the zero-trust principle of “Never trust, al-ways verify”, we propose a dynamic, fine-grained user trust evaluation model for micro-segmentation cloud computing environment, which combines multiple user trust attributes and leverages the subjective-objective approach to assign weights to trust attribute indicators to achieve dynamic scoring of users’ real-time behaviors. To capture the characteristics of users’ intrinsic behaviors, we use correlation analysis to identify the correlation between users’ current and historical behaviors, and combine sliding windows and penalty functions to optimize the model. The massive simulation experiments demonstrate the effectiveness of the proposed dynamic and fine-grained method, which can effectively combine the intrinsic correlation of users’ own access behavior and the difference of access behavior among different users.</p> <p> </p>

A spatiotemporal graph wavelet neural network for traffic flow prediction

The traffic flow prediction is fast becoming a key instrument in the transportation system, which has achieved impressive performance for traffic management. The graph neural network plays a critical role in the development of the traffic network management. However, it is worthwhile mentioning that the complexity of road networks and traffic conditions makes it unable to obtain sufficient spatiotemporal information. In view of capturing precise environment characteristics, the context could have a precise effect on the prediction results while previous methods rarely took this into account. Besides, the nonlinear characteristics of the graph neural network are hard to quantify with fine granularity and to eliminate overfitting. To stack these challenges, in this paper, we present a spatiotemporal graph wavelet neural network to improve the ability of representations. Specifically, we introduce the wavelet transforms into the deep learning model according to the strong nonlinear optimization ability. Furthermore, we dig the location and time patterns to evaluate the temporal dependence and the spatial proximity correlation. In addition, we introduce a historical context attention mechanism giving fine-grained historical context grade evaluation to ease the phenomenon of over-smoothing. The experimental results on real-world datasets show that our work gets considerable results compared with the baseline and start-of-the-art models. Moreover, our work has better learning performance by employing the connection and interaction of graphs.

A Survey on Evaluating the Quality of Autonomic Internet of Things Applications

The rapid evolution of the Internet of Things (IoT) facilitates the development of IoT applications in domains such as manufacturing, smart cities, retail, agriculture, etc. Such IoT applications collect data, analyze, and extract insightful information to enable decision-making and actuation. There is an unprecedented growth of IoT applications that automate decision-making and actuation without requiring human intervention, which we term autonomic IoT applications. The increasing scale of such applications necessitates holistic measurement and evaluation of application quality. Existing literature has evaluated quality from an end-user perspective, which may be unsuitable when dealing with the complexity of modern IoT applications, especially when they are autonomic. In this paper, we refer to IoT application quality as the aggregate quantitative value of various IoT quality metrics measured at each stage of the autonomic IoT application life cycle. We present an in-depth survey of current state-of-the-art techniques and approaches for evaluating quality of IoT applications. In particular, we survey various definitions to identify the factors that contribute to understanding and evaluating quality in IoT. Furthermore, we present open issues and identify future research directions towards realizing fine-grained quality evaluation of IoT applications. We envision that the identified research directions will, in turn, enable real-time diagnostics of IoT applications and make them quality-aware. This survey can serve as the basis for designing and developing modern, resilient quality-aware autonomic IoT applications.

A novel approach for fine-grained traffic risk characterization and evaluation of urban road intersections

An urban road intersection environment with low traffic risk is an essential component of sustainable cities. Various risk analysis and simulation techniques have been developed to measure and evaluate traffic risks. However, flexible assessment of intersection risk at a fine resolution remains a great challenge. This study proposes a novel approach for characterizing the field distribution of traffic risks at an urban road intersection and assessing traffic risks in multiple dimensions to increase the flexibility of risk assessment. First, we define a new traffic risk metric to quantify the risk of intersection traffic conflict points by combining their frequency and severity. Second, the plume diffusion model is introduced to model the non-linear relationship between traffic conflict points by considering the propagation of the traffic risk in the intersection. In this way, the entire intersection risk distribution map is derived for real-time risk analysis. Finally, a multidimensional hierarchical evaluation system of traffic risk at intersections is designed, including the traffic risk indicators at the intersection, path, and turn mode levels. The results of a case study in Wuhan, China, demonstrate the continuous characterization results of the risk field and provide multidimensional risk assessment index results, which are highly consistent with the cognitive results of manual risk scoring (up to 90% similarity). It also allows for risk assessment in a single path and turn modes through the risk field characterization and reveals the spatiotemporal dynamic risk distribution, thus providing an alternative tool for risk decision-making at urban road intersections.

Social inequity of park accessibility in Taiyuan: highlighting the unfair layout of parks in second-tier cities of China and the relative role of contributors

ABSTRACT The scarcity of public resources and environmental pollution caused by rapid urbanization highlight the practical significance of parks in ensuring the sustainable development of a city. Therefore, the social equity of parks warrants further study. This paper proposes a fine-grained comprehensive evaluation framework that combines geographic accessibility models, geo-statistical analysis, and machine learning algorithms to explore social inequity in Taiyuan, China. In this framework, gini coefficient and lorentz curve express spatial equality, accessibility shows spatial equity, and ridge regression model handles the interdependence of variables with different dimensions to quantify the relative effects of local participants on changes in park accessibility. On this basis, the imbalance between vulnerable groups and park supply is analyzed to further understand the core concept of social equity. Results highlight serious spatial inequality in all three types of parks allocation of six urban areas, especially in community parks. The actual access level of people to parks is also stratified by their demographic and socioeconomic characteristics, revealing the social inequity in access to parks. Park distribution is indeed not conducive to some social vulnerable groups, whose contradiction between supply and demand is highly prominent in urban – rural junctions and new urban areas. This paper also confirms the unfair layout of public facilities can be observed in second-tier cities of China by highlighting the social inequity of parks in Taiyuan. The findings of this work have profound implications for urban planning and sustainable development.

Effect of different sport environments on proactive and reactive motor inhibition: A study on open- and closed-skilled athletes via mouse-tracking procedure.

This study aimed to investigate the effect of different sport environments (open-and closed-skill sports) on proactive and reactive inhibitory processes as two distinct components of motor inhibition. A mouse-tracking procedure was employed to compare behavioral performance among three groups of participants (tennis players, swimmers and non-athletes) in non-sport-specific cued Go/No-Go (GNG) and Stop Signal Task (SST), which mainly engage proactive and reactive inhibitory control, respectively. Reaction times (RTs), inhibitory failures, and Stop Signal Reaction Times (SSRTs) were measured. To investigate dynamic aspects of inhibitory control, movement trajectories classified as one-shot (absence of trajectory alteration reflected in a steep slope) or non-one-shot (non-linear/multipeaked trajectory, with one or multiple corrections) were analyzed and compared among groups. Results showed no group differences in RTs in Go/No-Go and Stop conditions. SSRTs were significant shorter for the athletes than non-athletes in SST, but no differences emerged for inhibitory failures in cued GNG. During inhibitory failures athletes showed higher proportion of non-one-shot movements than non-athletes. Higher proportion of non-one-shot profiles was observed in cued GNG compared to SST. Finally, no differences between open-and closed-skilled athletes were found in both tasks. Our findings suggest that both proactive and reactive inhibitory controls do benefit from sport practice, but open-and closed-skill sports do not differ in influencing inhibitory processes. Movement profile analysis could be a promising, complementary behavioral analysis to integrate for more fine-grained evaluation and differentiation of inhibitory motor control in athletes, specifically when using GNG tasks.

Comprehensive Analysis of Knowledge Graph Embedding Techniques Benchmarked on Link Prediction

In knowledge graph representation learning, link prediction is among the most popular and influential tasks. Its surge in popularity has resulted in a panoply of orthogonal embedding-based methods projecting entities and relations into low-dimensional continuous vectors. To further enrich the research space, the community witnessed a prolific development of evaluation benchmarks with a variety of structures and domains. Therefore, researchers and practitioners face an unprecedented challenge in effectively identifying the best solution to their needs. To this end, we propose the most comprehensive and up-to-date study to systematically assess the effectiveness and efficiency of embedding models for knowledge graph completion. We compare 13 models on six datasets with different sizes, domains, and relational properties, covering translational, semantic matching, and neural network-based encoders. A fine-grained evaluation is conducted to compare each technique head-to-head in terms of standard metrics, training and evaluation times, memory consumption, carbon footprint, and space geometry. Our results demonstrate the high dependence between performance and graph types, identifying the best options for each scenario. Among all the encoding strategies, the new generation of translational models emerges as the most promising, bringing out the best and most consistent results across all the datasets and evaluation criteria.

A fine-grained evaluation of social vulnerability to COVID-19, healthcare access and built environmental disparities in Tokyo, Japan

ABSTRACT Measuring vulnerability to COVID-19 and healthcare accessibility at the fine-grained level serves as the foundation for spatially explicit health planning and policy making in response to future public health crises. However, the evaluation of vulnerability and healthcare accessibility is insufficient in Japan – a nation with high population density and super-aging challenges. Drawing on the 2022 census data, transport network, medical and digital cadastral data, land use maps, and points of interest data, our study extends the concept of vulnerability in the context of COVID-19 and constructs the first fine-grained measure of vulnerability and healthcare accessibility in Tokyo Metropolis, Japan – the most populated metropolitan region in the world. We delineate the vulnerable neighbourhoods with low healthcare access and further evaluate the disparity in healthcare access and built environment of areas at different levels of vulnerability. Our outcome datasets and findings provide nuanced and timely evidence to government and health authorities to have a holistic and latest understanding of social vulnerability to COVID-19 and healthcare access at a fine-grained level. Our analytical framework can be employed in different geographic contexts, guiding through place-based health planning and policy making in the post-COVID era and beyond.

How higher order mutant testing performs for deep learning models: A fine-grained evaluation of test effectiveness and efficiency improved from second-order mutant-classification tuples

Given the prevalence of Deep Learning (DL) models in daily life, it is crucial to guarantee their reliability by DL testing. Recently, researchers have adapted mutation testing into DL testing to measure the test power of test sets. The bottleneck of DL mutation testing is the expensive costs of generating a large number of mutants.

Informatics in Undergraduate Medical Education: Analysis of Competency Frameworks and Practices Across North America.

BackgroundWith the advent of competency-based medical education, as well as Canadian efforts to include clinical informatics within undergraduate medical education, competency frameworks in the United States have not emphasized the skills associated with clinical informatics pertinent to the broader practice of medicine.ObjectiveBy examining the competency frameworks with which undergraduate medical education in clinical informatics has been developed in Canada and the United States, we hypothesized that there is a gap: the lack of a unified competency set and frame for clinical informatics education across North America.MethodsWe performed directional competency mapping between Canadian and American graduate clinical informatics competencies and general graduate medical education competencies. Directional competency mapping was performed between Canadian roles and American common program requirements using keyword matching at the subcompetency and enabling competency levels. In addition, for general graduate medical education competencies, the Physician Competency Reference Set developed for the Liaison Committee on Medical Education was used as a direct means of computing the ontological overlap between competency frameworks.ResultsUpon mapping Canadian roles to American competencies via both undergraduate and graduate medical education competency frameworks, the difference in focus between the 2 countries can be thematically described as a difference between the concepts of clinical and management reasoning.ConclusionsWe suggest that the development or deployment of informatics competencies in undergraduate medical education should focus on 3 items: the teaching of diagnostic reasoning, such that the information tasks that comprise both clinical and management reasoning can be discussed; precision medical education, where informatics can provide for more fine-grained evaluation; and assessment methods to support traditional pedagogical efforts (both at the bedside and beyond). Assessment using cases or structured assessments (eg, Objective Structured Clinical Examinations) would help students draw parallels between clinical informatics and fundamental clinical subjects and would better emphasize the cognitive techniques taught through informatics.