F1 Metrics Research Articles

Detection of Risk Predictors of COVID-19 Mortality with Classifier Machine Learning Models Operated with Routine Laboratory Biomarkers

Early evaluation of patients who require special care and who have high death-expectancy in COVID-19, and the effective determination of relevant biomarkers on large sample-groups are important to reduce mortality. This study aimed to reveal the routine blood-value predictors of COVID-19 mortality and to determine the lethal-risk levels of these predictors during the disease process. The dataset of the study consists of 38 routine blood-values of 2597 patients who died (n = 233) and those who recovered (n = 2364) from COVID-19 in August–December, 2021. In this study, the histogram-based gradient-boosting (HGB) model was the most successful machine-learning classifier in detecting living and deceased COVID-19 patients (with squared F1 metrics F12 = 1). The most efficient binary combinations with procalcitonin were obtained with D-dimer, ESR, D-Bil and ferritin. The HGB model operated with these feature pairs correctly detected almost all of the patients who survived and those who died (precision > 0.98, recall > 0.98, F12 > 0.98). Furthermore, in the HGB model operated with a single feature, the most efficient features were procalcitonin (F12 = 0.96) and ferritin (F12 = 0.91). In addition, according to the two-threshold approach, ferritin values between 376.2 μg/L and 396.0 μg/L (F12 = 0.91) and procalcitonin values between 0.2 μg/L and 5.2 μg/L (F12 = 0.95) were found to be fatal risk levels for COVID-19. Considering all the results, we suggest that many features combined with these features, especially procalcitonin and ferritin, operated with the HGB model, can be used to achieve very successful results in the classification of those who live, and those who die from COVID-19. Moreover, we strongly recommend that clinicians consider the critical levels we have found for procalcitonin and ferritin properties, to reduce the lethality of the COVID-19 disease.

Emotion Analysis of Literary Works Based on Attentional Mechanisms and the Fusion of Two-Channel Features

In order to analyze sentiment data of foreign literary works, this paper proposes an algorithm for sentiment classification of literary works. We do this by fusing different features of literary works, which in turn captures more feature information for the classifier. As traditional word embedding models are difficult to achieve fusion with the sentiment features of literary works, we consider a multifeature fusion approach of word embedding features and lexical features of literary works. A two-channel and single-channel comparison is also used to analyze the classification accuracy based on the two feature fusion methods, and a parallel CNN and BiLSTM-attention two-channel neural network model proposed. Finally, the proposed model was evaluated using a real dataset of sentiment reviews of literary works and compared with different classification algorithms in the experiments. The experimental results show that the new hybrid approach has better classification accuracy, recall, and F1 metrics. The proposed methodology is an important guide for the creation of literary works and their screenplays, as it can be used to judge whether a work appeals to readers and, importantly, can also be considered as one of the criteria for the success of a film adaptation of a literary work.

Evaluating semantic representations for extended association rules

In this work, we evaluate the impact of changing the semantic text representation on the performance of the AR-SVS (extended association rules in semantic vector spaces) algorithm on the sentiment polarity classification task on a paper reviews dataset. To do this, we use natural language processing techniques in conjunction with machine learning classifiers. In particular, we report the classification performance using the F1 and accuracy metrics. The semantic representations that we used in our evaluation were chosen based on a systematic literature review, leading to an evaluation of AR-SVS with FastText, GloVe, and LDA2vec representations, with word2vec providing the baseline performance. The results of the experiments indicate that the choice of semantic text representation does not have major effects on the performance of AR-SVS for polarity classification. Furthermore, the results resemble those obtained in the original AR-SVS article, both in quantitative and qualitative terms. Thus, while direct improvements in classification performance were not found, we discuss other aspects and advantages of using different semantic representations.

Intelligent IoT security monitoring based on fuzzy optimum-path forest classifier

Detection of intrusions in Internet of Things networks is essential to maintain the availability and integrity of the data generated and transmitted by connected devices. Such a procedure is paramount when the data originate from critical activities, such as military, financial, industrial, and health sectors. In the last decades, machine learning (ML)-based approaches have become one of the most suitable and adopted procedures for the task, providing automatic, fast, and accurate results. Despite such success, the literature still presents a gap regarding valid applications of intrusion detection in the IoT environments, which usually stands for a challenging task composed of different types of attacks. In this context, this work applies a recent technique based on graphs and logic fuzzy, namely Fuzzy Optimum-Path Forest (Fuzzy OPF), to detect threats that escape an IoT network’s regular traffic. We evaluate our model against five well-known ML algorithms, i.e., Linear Discriminant Analysis, Support Vector Machine, Naive Bayes, K-Nearest Neighbors, and the standard Optimum-Path Forest. Experimental results show that Fuzzy OPF outperforms the baselines considering accuracy, recall, and F1 metrics. As a result, the Fuzzy OPF proposal for intrusion detection had a hit rate of 98 and 99%.

Anomaly Detection For Time Series Data Based on Multi-granularity Neighbor Residual Network

Anomaly detection of time series data is one of the most important problems in industrial applications. Most of methods are difficult to obtain rich features of samples due to the lack of dimension. In this paper, we develop an anomaly detection method for time series data based on multi-granularity neighbor residual network (MGNRN). First, we construct a neighbor input vector with a sliding time window for each data sample, and define neighbor-based input matrix by considering multi-granularity neighborhood features. Second, we compute the linear and non-linear neighbor features of multi-granularity time windows for the sample. Finally, by combining the linear neighborhood residual with nonlinear residual, we predict the abnormal probability of the sample. Experiments verify the multi-granularity neighbor residual network improves the accuracy of abnormal detection, and show good performance on precision and F1 metrics.

Experimental Study of Language Models of "Transformer" in the Problem of Finding the Answer to a Question in a Russian-Language Text

The aim of the study is to obtain a more lightweight language model that is comparable in terms of EM and F1 with the best modern language models in the task of finding the answer to a question in a text in Russian. The results of the work can be used in various question-and-answer systems for which response time is important. Since the lighter model has fewer parameters than the original one, it can be used on less powerful computing devices, including mobile devices. In this paper, methods of natural language processing, machine learning, and the theory of artificial neural networks are used. The neural network is configured and trained using the Torch and Hugging face machine learning libraries. In the work, the DistilBERT model was trained on the SberQUAD dataset with and without distillation. The work of the received models is compared. The distilled DistilBERT model (EM 58,57 and F1 78,42) was able to outperform the results of the larger ruGPT-3-medium generative network (EM 57,60 and F1 77,73), despite the fact that ruGPT-3-medium had 6,5 times more parameters. The model also showed better EM and F1 metrics than the same model, but to which only conventional training without distillation was applied (EM 55,65, F1 76,51). Unfortunately, the resulting model lags further behind the larger robert discriminative model (EM 66,83, F1 84,95), which has 3,2 times more parameters. The application of the DistilBERT model in question-and-answer systems in Russian is substantiated. Directions for further research are proposed.

Machine Learning Approach to Facilitate Knowledge Synthesis at the Intersection of Liver Cancer, Epidemiology, and Health Disparities Research.

Liver cancer is a global challenge, and disparities exist across multiple domains and throughout the disease continuum. However, liver cancer's global epidemiology and etiology are shifting, and the literature is rapidly evolving, presenting a challenge to the synthesis of knowledge needed to identify areas of research needs and to develop research agendas focusing on disparities. Machine learning (ML) techniques can be used to semiautomate the literature review process and improve efficiency. In this study, we detail our approach and provide practical benchmarks for the development of a ML approach to classify literature and extract data at the intersection of three fields: liver cancer, health disparities, and epidemiology. We performed a six-phase process including: training (I), validating (II), confirming (III), and performing error analysis (IV) for a ML classifier. We then developed an extraction model (V) and applied it (VI) to the liver cancer literature identified through PubMed. We present precision, recall, F1, and accuracy metrics for the classifier and extraction models as appropriate for each phase of the process. We also provide the results for the application of our extraction model. With limited training data, we achieved a high degree of accuracy for both our classifier and for the extraction model for liver cancer disparities research literature performed using epidemiologic methods. The disparities concept was the most challenging to accurately classify, and concepts that appeared infrequently in our data set were the most difficult to extract. We provide a roadmap for using ML to classify and extract comprehensive information on multidisciplinary literature. Our technique can be adapted and modified for other cancers or diseases where disparities persist.

Visualization and Analysis Model of Industrial Economy Status and Development Based on Knowledge Graph and Deep Neural Network.

This paper adopts knowledge mapping combined with a deep neural network algorithm to conduct in-depth research and analysis on the current situation and development of the industrial economy and designs a visual analysis model of economic development based on knowledge mapping combined with a deep neural network algorithm. Cultivate the concept of coordinated development and legal system of the subject, improve the awareness of network security and integrity self-discipline of the subject, improve the level of network hardware equipment manufacturing, improve the level of network platform construction, build a network security technology prevention system, improve the repair system of network information alienation, set up a specialized agency setting for the coordinated development of network ecology and industrial economy, and increase the capital investment in network infrastructure and network information technology research and development. A framework of breadth and depth recommendation ranking based on a knowledge graph is proposed and implemented. This paper provides a visual analysis method to sort and classify multivariate data. The method first determines users' preferences through their interactive operations, calculates the weights of each attribute according to the users' preference model, then uses the obtained attribute weight sets to sort the whole data set, and finally completes the category classification according to the sorting results and the users' markings on some data. The visual display allows users to intuitively perform data sorting and classification operations and quickly understand the characteristics and category features of the data. The framework achieves modeling and integration of knowledge graph neighborhood information from breadth dimension and depth dimension to realize personalized recommendation sorting and improves the F1 metrics by 8.59%, 14.36%, and 15.22% on the public datasets Amazon-book, Yelp2018, and ILast-FM compared with the previous optimal model.

Revisiting Gradient Boosting-Based Approaches for Learning Imbalanced Data: A Case of Anomaly Detection on Power Grids

Gradient boosting ensembles have been used in the cyber-security area for many years; nonetheless, their efficacy and accuracy for intrusion detection systems (IDSs) remain questionable, particularly when dealing with problems involving imbalanced data. This article fills the void in the existing body of knowledge by evaluating the performance of gradient boosting-based ensembles, including gradient boosting machine (GBM), extreme gradient boosting (XGBoost), LightGBM, and CatBoost. This paper assesses the performance of various imbalanced data sets using the Matthew correlation coefficient (MCC), area under the receiver operating characteristic curve (AUC), and F1 metrics. The article discusses an example of anomaly detection in an industrial control network and, more specifically, threat detection in a cyber-physical smart power grid. The tests’ results indicate that CatBoost surpassed its competitors, regardless of the imbalance ratio of the data sets. Moreover, LightGBM showed a much lower performance value and had more variability across the data sets.

Pair-wise aspect and opinion terms extraction as graph parsing via a novel mutually-aware interaction mechanism

The pair-wise aspect and opinion term extraction (PAOTE) task aims to extract aspect terms and opinion terms from reviews in the form of opinion pairs, which provides a global profile for reviews of goods or users. Up-to-date studies ignore the interaction between term detection and term pairing, which may be crucial for the PAOTE task. Other studies use syntactic dependency structures to enhance their models, which cannot better provide task-specific structural information. In this work, we design an aspect-to-opinion graph and transform PAOTE into a graph parsing task. To exploit the interaction between term detection and pairing, we propose a novel mutually-aware interaction network (MAIN), which interactively updates the representations for term detection and pairing via graph sampling and convolution. Further, the word-word graph learned during training can be iteratively refined and gradually approaches the aspect-to-opinion graph. Experimental results on four benchmark datasets show that our proposed method significantly outperforms strong baselines with state-of-the-art performance and achieves a maximum increase of 2.01 points on the F1 metric. Further analysis demonstrates the advance of the aspect-to-opinion graph and the effectiveness of the mutually-aware interaction mechanism.

Extracting associations and meanings of objects depicted in artworks through bi-modal deep networks

We present a novel bi-modal system based on deep networks to address the problem of learning associations and simple meanings of objects depicted in "authored" images, such as ne art paintings and drawings. Our overall system processes both the images and associated texts in order to learn associations between images of individual objects, their identities and the abstract meanings they signify. Unlike past deep net that describe depicted objects and infer predicates, our system identies meaning-bearing objects ("signifiers") and their associations ("signifieds") as well as basic overall meanings for target artworks. Our system had precision of 48% and recall of 78% with an F1 metric of 0.6 on a curated set of Dutch vanitas paintings, a genre celebrated for its concentration on conveying a meaning of great import at the time of their execution. We developed and tested our system on ne art paintings but our general methods can be applied to other authored images.

Deep learning applied to breast imaging classification and segmentation with human expert intervention.

Automatic classification and segmentation of tumors in breast ultrasound images enables better diagnosis and planning treatment strategies for breast cancer patients. We collected 953 breast ultrasound images from two open-source datasets and classified them with help of an expert radiologist according to BI-RADS criteria. The data was split into normal, benign and malignant classes. We then used machine learning to develop classification and segmentation algorithms. We found 3.92% of the images across the open-source datasets had erroneous classifications. Post-radiologist intervention, three algorithms were developed based on the classification categories. Classification algorithms distinguished images with healthy breast tissue from those with abnormal tissue with 96% accuracy, and distinguished benign from malignant images with 85% accuracy. Both algorithms generated robust F1 and AUROC metrics. Finally, the masses within images were segmented with an 80.31% DICE score. Our work illustrates the potential of deep learning algorithms to improve the accuracy of breast ultrasound assessments and to facilitate automated assessments.

Validation of an automatic tool for the rapid measurement of brain atrophy and white matter hyperintensity: QyScore\xae

ObjectivesQyScore® is an imaging analysis tool certified in Europe (CE marked) and the US (FDA cleared) for the automatic volumetry of grey and white matter (GM and WM respectively), hippocampus (HP), amygdala (AM), and white matter hyperintensity (WMH). Here we compare QyScore® performances with the consensus of expert neuroradiologists.MethodsDice similarity coefficient (DSC) and the relative volume difference (RVD) for GM, WM volumes were calculated on 50 3DT1 images. DSC and the F1 metrics were calculated for WMH on 130 3DT1 and FLAIR images. For each index, we identified thresholds of reliability based on current literature review results. We hypothesized that DSC/F1 scores obtained using QyScore® markers would be higher than the threshold. In contrast, RVD scores would be lower. Regression analysis and Bland–Altman plots were obtained to evaluate QyScore® performance in comparison to the consensus of three expert neuroradiologists.ResultsThe lower bound of the DSC/F1 confidence intervals was higher than the threshold for the GM, WM, HP, AM, and WMH, and the higher bounds of the RVD confidence interval were below the threshold for the WM, GM, HP, and AM. QyScore®, compared with the consensus of three expert neuroradiologists, provides reliable performance for the automatic segmentation of the GM and WM volumes, and HP and AM volumes, as well as WMH volumes.ConclusionsQyScore® represents a reliable medical device in comparison with the consensus of expert neuroradiologists. Therefore, QyScore® could be implemented in clinical trials and clinical routine to support the diagnosis and longitudinal monitoring of neurological diseases.Key Points• QyScore® provides reliable automatic segmentation of brain structures in comparison with the consensus of three expert neuroradiologists.• QyScore® automatic segmentation could be performed on MRI images using different vendors and protocols of acquisition. In addition, the fast segmentation process saves time over manual and semi-automatic methods.• QyScore® could be implemented in clinical trials and clinical routine to support the diagnosis and longitudinal monitoring of neurological diseases.

Multifeature Transformation and Fusion-Based Ship Detection With Small Targets and Complex Backgrounds

With the development of deep learning, synthetic aperture radar (SAR) image ship detection based on the convolutional neural network has made significant progress. However, there are two problems. 1) The false alarm detection rate is high due to complex background and coherent speckle noise interference. 2) For smaller ship targets, missed detection is prone to occur. In this letter, a novel ship detection model (MFTF-Net) based on multi-feature transformation and fusion is proposed to address the issues. First, to avoid the randomness of initial point selection and the influence of outlier points, the anchor frame clustering approach based on the K-medians++ algorithm is presented to cluster the object candidate frames. Second, the low-level feature information is passed to the high level by constructing a local enhancement network; then, an improved Transformer structure is introduced to replace the last convolutional block of the backbone network to obtain rich contextual information. Finally, a four-scale residual feature fusion network is designed, which fully fuses the object’s detailed and semantic information. In addition, improved convolutional block attention module (CBAM) and squeeze and excitation (SE) attention mechanisms are applied in the lower two layers and upper two layers of the network output to reduce the interference of confusing information, respectively. The experimental results demonstrate that the proposed method is superior to the state-of-the-art thirteen baseline models on SAR ship detection dataset (SSDD), high-resolution SAR images dataset (HRSID), and SAR-ship-dataset public datasets in terms of the mAP, recall, accuracy, and F1 metrics.

Machine Learning for Securing Traffic in Computer Networks

Computer network attacks are among the most significant and common threats against computer-wired and wireless communications. Intrusion detection technology is used to secure computer networks by monitoring network traffic and identifying attacks. In this paper, we investigate and evaluate the application of four machine learning classification algorithms for identifying attacks that target computer networks: DDoS, Brute Force Web, and SQL Injection attacks, in addition to Benign Traffic. A public dataset of 80 features was used to build four machine learning models using Random Forest, Logistic Regression, CN2, and Neural Networks. The constructed models were evaluated based on 10-fold cross-validation using Classification Accuracy (CA), Area under the Curve (AUC), F1, Recall, Specificity, and Sensitivity metrics in addition to Confusion Matrix, Calibration, Lift, and ROC plots. The Random Forest model achieved 98% in the CA score and 99% in the AUC score, while the Logistic regression achieved 90% in the CA score and 98% in the AUC score.

Multi-Keyword Classification: A Case Study in Finnish Social Sciences Data Archive

In this paper, we consider the task of assigning relevant labels to studies in the social science domain. Manual labelling is an expensive process and prone to human error. Various multi-label text classification machine learning approaches have been proposed to resolve this problem. We introduce a dataset obtained from the Finnish Social Science Archive and comprised of 2968 research studies’ metadata. The metadata of each study includes attributes, such as the “abstract” and the “set of labels”. We used the Bag of Words (BoW), TF-IDF term weighting and pretrained word embeddings obtained from FastText and BERT models to generate the text representations for each study’s abstract field. Our selection of multi-label classification methods includes a Naive approach, Multi-label k Nearest Neighbours (ML-kNN), Multi-Label Random Forest (ML-RF), X-BERT and Parabel. The methods were combined with the text representation techniques and their performance was evaluated on our dataset. We measured the classification accuracy of the combinations using Precision, Recall and F1 metrics. In addition, we used the Normalized Discounted Cumulative Gain to measure the label ranking performance of the selected methods combined with the text representation techniques. The results showed that the ML-RF model achieved a higher classification accuracy with the TF-IDF features and, based on the ranking score, the Parabel model outperformed the other methods.

Ceasing hate with MoH: Hate Speech Detection in Hindi–English code-switched language

Warning: This manuscript may contain upsetting language.Social media has become a bedrock for people to voice their opinions worldwide. Due to the greater sense of freedom with the anonymity feature, it is possible to disregard social etiquette online and attack others without facing severe consequences, inevitably propagating hate speech. The current measures to sift the online content and offset the hatred spread do not go far enough. One factor contributing to this is the prevalence of regional languages in social media and the paucity of language flexible hate speech detectors. The proposed work focuses on analyzing hate speech in Hindi–English code-switched language. Our method explores transformation techniques to capture precise text representation. To contain the structure of data and yet use it with existing algorithms, we developed ‘MoH’ or (Map Only Hindi), which means ‘Love’ in Hindi. ‘MoH’ pipeline which consists of language identification, Roman to Devanagari Hindi transliteration using a knowledge base of Roman Hindi words, and finally employs the fine-tuned Multilingual Bert, and MuRIL language models. We conducted several quantitative experiment studies on three datasets, and evaluated performance using Precision, Recall and F1 metrics. The first experiment studies ‘MoH’ mapped text’s performance with classical machine learning models and shows an average increase of 13% in F1 scores. The second compares the proposed work’s scores with those of the baseline models and shows a rise in performance by 6%. Finally, the third compares the proposed ‘MoH’ technique with various data simulations using the existing transliteration library. Here, ‘MoH’ outperforms the rest by 15%. Our results demonstrate a significant improvement in the state-of-the-art scores on all three datasets.

The impact of using biased performance metrics on software defect prediction research

Software engineering researchers have undertaken many experiments investigating the potential of software defect prediction algorithms. Unfortunately some widely used performance metrics are known to be problematic, most notably F1, but nevertheless F1 is widely used. To investigate the potential impact of using F1 on the validity of this large body of research. We undertook a systematic review to locate relevant experiments and then extract all pairwise comparisons of defect prediction performance using F1 and the unbiased Matthews correlation coefficient (MCC). We found a total of 38 primary studies. These contain 12,471 pairs of results. Of these comparisons, 21.95% changed direction when the MCC metric is used instead of the biased F1 metric. Unfortunately, we also found evidence suggesting that F1 remains widely used in software defect prediction research. We reiterate the concerns of statisticians that the F1 is a problematic metric outside of an information retrieval context, since we are concerned about both classes (defect-prone and not defect-prone units). This inappropriate usage has led to a substantial number (more than one fifth) of erroneous (in terms of direction) results. Therefore we urge researchers to (i) use an unbiased metric and (ii) publish detailed results including confusion matrices such that alternative analyses become possible.

Relation classification via BERT with piecewise convolution and focal loss.

Recent relation extraction models’ architecture are evolved from the shallow neural networks to natural language model, such as convolutional neural networks or recurrent neural networks to Bert. However, these methods did not consider the semantic information in the sequence or the distance dependence problem, the internal semantic information may contain the useful knowledge which can help relation classification. Focus on these problems, this paper proposed a BERT-based relation classification method. Compare with the existing Bert-based architecture, the proposed model can obtain the internal semantic information between entity pair and solve the distance semantic dependence better. The pre-trained BERT model after fine tuning is used in this paper to abstract the semantic representation of sequence, then adopt the piecewise convolution to obtain semantic information which influence the extraction results. Compare with the existing methods, the proposed method can achieve a better accuracy on relational extraction task because of the internal semantic information extracted in the sequence. While, the generalization ability is still a problem that cannot be ignored, and the numbers of the relationships are difference between different categories. In this paper, the focal loss function is adopted to solve this problem by assigning a heavy weight to less number or hard classify categories. Finally, comparing with the existing methods, the F1 metric of the proposed method can reach a superior result 89.95% on the SemEval-2010 Task 8 dataset.

Development of an AI system for accurately diagnose hepatocellular carcinoma from computed tomography imaging data.

Computed tomography (CT) scan is frequently used to detect hepatocellular carcinoma (HCC) in routine clinical practice. The aim of this study is to develop a deep-learning AI system to improve the diagnostic accuracy of HCC by analysing liver CT imaging data. We developed a deep-learning AI system by training on CT images from 7512 patients at Henan Provincial Peoples' Hospital. Its performance was validated on one internal test set (Henan Provincial Peoples' Hospital, n = 385) and one external test set (Henan Provincial Cancer Hospital, n = 556). The area under the receiver-operating characteristic curve (AUROC) was used as the primary classification metric. Accuracy, sensitivity, specificity, precision, negative predictive value and F1 metric were used to measure the performance of AI systems and radiologists. AI system achieved high performance in identifying HCC patients, with AUROC of 0.887 (95% CI 0.855-0.919) on the internal test set and 0.883 (95% CI 0.855-0.911) on the external test set. For internal test set, accuracy was 81.0% (76.8-84.8%), sensitivity was 78.4% (72.4-83.7%), specificity was 84.4% (78.0-89.6%) and F1 (harmonic average of precision and recall rate) was 0.824. For external test set, accuracy was 81.3% (77.8-84.5%), sensitivity was 89.4% (85.0-92.8%), specificity was 74.0% (68.5-78.9%) and F1 was 0.819. Compared with radiologists, AI system achieved comparable accuracy and F1 metric on internal test set (0.853 versus 0.818, P = 0.107; 0.863 vs. 0.824, P = 0.082) and external test set (0.805 vs. 0.793, P = 0.663; 0.810 vs. 0.814, P = 0.866). The predicted HCC risk scores by AI system in HCC patients with multiple tumours and high fibrosis stage were higher than those with solitary tumour and low fibrosis stage (tumour number: 0.197 vs. 0.138, P = 0.006; fibrosis stage: 0.183 vs. 0.127, P < 0.001). Radiologists' review showed that the accuracy of saliency heatmaps predicted by algorithms was 92.1% (95% CI: 89.2-95.0%). AI system achieved high performance in the detection of HCC compared with a group of specialised radiologists. Further investigation by prospective clinical trials was necessitated to verify this model.