Manual Labeling Research Articles

A Human-In-One-Loop Active Domain Adaptation Framework for Digit Recognition

ABSTRACT Domain adaptation can effectively enhance a model’s performance on target domain data with limited data. However, when some target domain labels are obtainable, training the model with both source and target domain data simultaneously can lead to worse performance due to the lower density of target domain data. If a large amount of target domain data is labeled without discrimination, it will necessitate a considerable expenditure of human resources. To address this issue, this paper proposes a human-in-one-loop active domain adaptation framework based on Target Domain Feature Generation to solve the problems. The oracle participates in only one iteration of data labeling, and a target domain classifier will take over the subsequent rest iterations. An image generator based on multiple CycleGANs forms an iterative co-training mechanism, which can continuously generate more high-quality labeled fake target domain data in iterations to improve the performance of the target domain classifier. The Top-N labeled data selection method with high confidence is devised to select the most accurately predicted data for labeling, reducing manual labeling workload. This framework can achieve an average accuracy of 0.8869 on six domain pairs, doubling the classical domain adaptation method DSN, requiring only a small amount of manual labeling.

GA-GGD: Improving semantic discriminability in graph contrastive learning via Generative Adversarial Network

Graph contrastive learning has garnered considerable research interest due to its ability to effectively embed graph data without manual labels. Among them, methods based on Deep Graph Infomax (DGI) have been widely studied and favored in the industry because of their fast training speed, applicability to large-scale data. DGI-based methods usually obtain a noise graph through node shuffling. The proxy task of these methods encourages the encoder to distinguish whether the nodes come from the original graph or the noise graph, thereby maximizing the mutual information between the node representation and the graph it belongs to, while also maximizing the Jenson–Shannon divergence between the nodes of original graph and noise graph. However, we argue that these approaches only enable the encoder to differentiate between semantically meaningful graphs and noise graphs, but not to effectively identify different semantic graphs. This leads to the inability of the encoder to effectively embed information between different semantics, significantly reducing the robustness and affecting the performance of downstream tasks. In addition, this training mode makes the model more sensitive to attacks. To improve their semantic discriminability, we take advantage of the natural ability of generative adversarial networks to generate semantic data, proposing a method called Generative Adversarial Graph Group Discrimination (GA-GGD). Specifically, it consists of a graph group discriminator and a semantic attack generator. The discriminator aims to encode the graph and identify whether nodes originate from the original graph. The goal of the generator is to use random features and graph structure to find vulnerabilities of discriminator and generate node representations with similar but wrong semantic to confuse the discriminator. GA-GGD can improve the model’s semantic information embedding without significantly increasing computational overhead and memory occupancy. We test the effectiveness of the proposed model on commonly used data sets and large-scale datasets, as well as in various downstream tasks such as classification, clustering, and adversarial attacks defence. A wealth of experimental results confirm the efficacy of the proposed model.

Catalyzing Precision Medicine: Artificial Intelligence Advancements in Prostate Cancer Diagnosis and Management.

The aim was to analyze the current state of deep learning (DL)-based prostate cancer (PCa) diagnosis with a focus on magnetic resonance (MR) prostate reconstruction; PCa detection/stratification/reconstruction; positron emission tomography/computed tomography (PET/CT); androgen deprivation therapy (ADT); prostate biopsy; associated challenges and their clinical implications. A search of the PubMed database was conducted based on the inclusion and exclusion criteria for the use of DL methods within the abovementioned areas. A total of 784 articles were found, of which, 64 were included. Reconstruction of the prostate, the detection and stratification of prostate cancer, the reconstruction of prostate cancer, and diagnosis on PET/CT, ADT, and biopsy were analyzed in 21, 22, 6, 7, 2, and 6 studies, respectively. Among studies describing DL use for MR-based purposes, datasets with magnetic field power of 3 T, 1.5 T, and 3/1.5 T were used in 18/19/5, 0/1/0, and 3/2/1 studies, respectively, of 6/7 studies analyzing DL for PET/CT diagnosis which used data from a single institution. Among the radiotracers, [68Ga]Ga-PSMA-11, [18F]DCFPyl, and [18F]PSMA-1007 were used in 5, 1, and 1 study, respectively. Only two studies that analyzed DL in the context of DT met the inclusion criteria. Both were performed with a single-institution dataset with only manual labeling of training data. Three studies, each analyzing DL for prostate biopsy, were performed with single- and multi-institutional datasets. TeUS, TRUS, and MRI were used as input modalities in two, three, and one study, respectively. DL models in prostate cancer diagnosis show promise but are not yet ready for clinical use due to variability in methods, labels, and evaluation criteria. Conducting additional research while acknowledging all the limitations outlined is crucial for reinforcing the utility and effectiveness of DL-based models in clinical settings.

Enhancing the automatic facies classification of Brazilian presalt acoustic image logs with SwinV2-Unet: Leveraging transfer learning and confident learning

Facies classification of image logs plays a vital role in reservoir characterization, especially in the heterogeneous and anisotropic carbonate formations of the Brazilian presalt region. Although manual classification remains the industry standard for handling the complexity and diversity of image logs, it has the notable disadvantages of being time consuming, labor intensive, subjective, and nonrepeatable. Recent advancements in machine learning offer promising solutions for automation and acceleration. However, previous attempts to train deep neural networks for facies identification have struggled to generalize to new data due to insufficient labeled data and the inherent intricacy of image logs. In addition, human errors in manual labels further hinder the performance of trained models. To overcome these challenges, we develop adopting the state-of-the-art SwinV2-Unet to provide depthwise facies classification for Brazilian presalt acoustic image logs. The training process incorporates transfer learning to mitigate overfitting and confident learning to address label errors. Through a k-fold cross-validation experiment, with each fold spanning more than 350 m, we achieve an impressive macro F1 score of 0.90 for out-of-sample predictions. This significantly surpasses the previous model modified from the widely recognized U-Net, which provides a macro F1 score of 0.68. These findings highlight the effectiveness of the used enhancements, including the adoption of an improved neural network and an enhanced training strategy. Moreover, our SwinV2-Unet enables a highly efficient and accurate facies analysis of the complex yet informative image logs, significantly advancing our understanding of hydrocarbon reservoirs, saving human effort, and improving productivity.

CABnet: A channel attention dual adversarial balancing network for multimodal image fusion

Infrared and visible image fusion aims to generate informative images by leveraging the distinctive strengths of infrared and visible modalities. These fused images play a crucial role in subsequent downstream tasks, including object detection, recognition, and segmentation. However, complementary information is often difficult to extract. Existing generative adversarial network-based methods generate fused images by modifying the distribution of source images' features to preserve instances and texture details in both infrared and visible images. Nevertheless, these approaches may result in a degradation of the fused image quality when the original image quality is low. Considering the balance of information from different modalities can improve the quality of the fused image. Hence, we introduce CABnet, a Channel Attention dual adversarial Balancing network. CABnet incorporates a channel attention mechanism to capture crucial channel features, thereby, enhancing complementary information. It also employs an adaptive factor to control the mixing distribution of infrared and visible images, which ensures the preservation of instances and texture details during the adversarial process. To enhance efficiency and reduce reliance on manual labeling, our training process adopts a semi-supervised learning strategy. Through qualitative and quantitative experiments across multiple datasets, CABnet surpasses existing state-of-the-art methods in fusion performance, notably achieving a 51.3% enhancement in signal to noise ratio and a 13.4% improvement in standard deviation.

VR.net: A Real-world Dataset for Virtual Reality Motion Sickness Research.

Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches would certainly benefit from an accurately labeled, real-world, diverse dataset that enables the development of generalizable ML models. We introduce 'VR.net', a dataset comprising 165-hour gameplay videos from 100 real-world games spanning ten diverse genres, evaluated by 500 participants. VR.net accurately assigns 24 motion sickness-related labels for each video frame, such as camera/object movement, depth of field, and motion flow. Building such a dataset is challenging since manual labeling would require an infeasible amount of time. Instead, we implement a tool to automatically and precisely extract ground truth data from 3D engines' rendering pipelines without accessing VR games' source code. We illustrate the utility of VR.net through several applications, such as risk factor detection and sickness level prediction. We believe that the scale, accuracy, and diversity of VR.net can offer unparalleled opportunities for VR motion sickness research and beyond.We also provide access to our data collection tool, enabling researchers to contribute to the expansion of VR.net.

Automated medication verification system (AMVS): System based on edge detection and CNN classification drug on embedded systems

A novel automated medication verification system (AMVS) aims to address the limitation of manual medication verification among healthcare professionals with a high workload, thereby reducing medication errors in hospitals. Specifically, the manual medication verification process is time-consuming and prone to errors, especially in healthcare settings with high workloads. The proposed system strategy is to streamline and automate this process, enhancing efficiency and reducing medication errors. The system employs deep learning models to swiftly and accurately classify multiple medications within a single image without requiring manual labeling during model construction. It comprises edge detection and classification to verify medication types. Unlike previous studies conducted in open spaces, our study takes place in a closed space to minimize the impact of optical changes on image capture. During the experimental process, the system individually identifies each drug within the image by edge detection method and utilizes a classification model to determine each drug type. Our research has successfully developed a fully automated drug recognition system, achieving an accuracy of over 95 % in identifying drug types and conducting segmentation analyses. Specifically, the system demonstrates an accuracy rate of approximately 96 % for drug sets containing fewer than ten types and 93 % for those with ten types. This verification system builds an image classification model quickly. It holds promising potential in assisting nursing staff during AMVS, thereby reducing the likelihood of medication errors and alleviating the burden on nursing staff.

Weakly Supervised Tracklet Association Learning With Video Labels for Person Re-Identification.

Supervised person re-identification (re-id) methods require expensive manual labeling costs. Although unsupervised re-id methods can reduce the requirement of the labeled datasets, the performance of these methods is lower than the supervised alternatives. Recently, some weakly supervised learning-based person re-id methods have been proposed, which is a balance between supervised and unsupervised learning. Nevertheless, most of these models require another auxiliary fully supervised datasets or ignore the interference of noisy tracklets. To address this problem, in this work, we formulate a weakly supervised tracklet association learning (WS-TAL) model only leveraging the video labels. Specifically, we first propose an intra-bag tracklet discrimination learning (ITDL) term. It can capture the associations between person identities and images by assigning pseudo labels to each person image in a bag. And then, the discriminative feature for each person is learned by utilizing the obtained associations after filtering the noisy tracklets. Based on that, a cross-bag tracklet association learning (CTAL) term is presented to explore the potential tracklet associations between bags by mining reliable positive tracklet pairs and hard negative pairs. Finally, these two complementary terms are jointly optimized to train our re-id model. Extensive experiments on the weakly labeled datasets demonstrate that WS-TAL achieves 88.1% and 90.3% rank-1 accuracy on the MARS and DukeMTMC-VideoReID datasets respectively. The performance of our model surpasses the state-of-the-art weakly supervised models by a large margin, even outperforms some fully supervised re-id models.

Unified weakly and semi-supervised crack segmentation framework using limited coarse labels

Obtaining extensive, high-quality datasets for crack segmentation with pixel-level labels is expensive and labor-intensive. The Unified Weakly and Semi-supervised Crack Segmentation (UWSCS) framework addresses this challenge by leveraging a limited number of images with coarse labels and a larger set of unlabeled images. Two label correction modules, based on super-pixel segmentation and a shrink module, are incorporated in the model training to improve crack label accuracy and optimize edge refinement. UWSCS employs a dual-encoder fusion network, combining transformers and convolutional neural networks, to enhance crack segmentation in complex backgrounds. An enhanced algorithm using the medial axis transform is proposed for accurately quantifying crack length and width. Extensive experiments were conducted on both synthetic and real crack datasets to validate the superior performance of UWSCS. The results underscore the significant impact of label quality and quantity used in training on model prediction accuracy. Trained on a concrete crack dataset with limited coarse labels, UWSCS achieves an Intersection of Union (IoU) of 77.53%, surpassing the fully supervised model using the same number of coarse labels by 28.64%. It closely approaches the performance of a fully supervised model with the same number of fine labels (IoU of 80.21%). UWSCS outperforms other advanced networks and semi-supervised/weakly supervised algorithms when trained with a limited set of more cost-effective manually labeled coarse labels. Integrated with the crack segmentation network, super-pixel segmentation, and shrink modules during training, UWSCS with limited coarse labels performs similarly to a fully supervised model using fine labels, thereby reducing manual labeling costs by over 90% and enhancing detection efficiency in practical engineering.

Synergizing Deep Learning-Enabled Preprocessing and Human-AI Integration for Efficient Automatic Ground Truth Generation.

The progress of incorporating deep learning in the field of medical image interpretation has been greatly hindered due to the tremendous cost and time associated with generating ground truth for supervised machine learning, alongside concerns about the inconsistent quality of images acquired. Active learning offers a potential solution to these problems of expanding dataset ground truth by algorithmically choosing the most informative samples for ground truth labeling. Still, this effort incurs the costs of human labeling, which needs minimization. Furthermore, automatic labeling approaches employing active learning often exhibit overfitting tendencies while selecting samples closely aligned with the training set distribution and excluding out-of-distribution samples, which could potentially improve the model's effectiveness. We propose that the majority of out-of-distribution instances can be attributed to inconsistent cross images. Since the FDA approved the first whole-slide image system for medical diagnosis in 2017, whole-slide images have provided enriched critical information to advance the field of automated histopathology. Here, we exemplify the benefits of a novel deep learning strategy that utilizes high-resolution whole-slide microscopic images. We quantitatively assess and visually highlight the inconsistencies within the whole-slide image dataset employed in this study. Accordingly, we introduce a deep learning-based preprocessing algorithm designed to normalize unknown samples to the training set distribution, effectively mitigating the overfitting issue. Consequently, our approach significantly increases the amount of automatic region-of-interest ground truth labeling on high-resolution whole-slide images using active deep learning. We accept 92% of the automatic labels generated for our unlabeled data cohort, expanding the labeled dataset by 845%. Additionally, we demonstrate expert time savings of 96% relative to manual expert ground-truth labeling.

Language model-based labeling of German thoracic radiology reports.

The aim of this study was to explore the potential of weak supervision in a deep learning-based label prediction model. The goal was to use this model to extract labels from German free-text thoracic radiology reports on chest X-ray images and for training chest X-ray classification models.The proposed label extraction model for German thoracic radiology reports uses a German BERT encoder as a backbone and classifies a report based on the CheXpert labels. For investigating the efficient use of manually annotated data, the model was trained using manual annotations, weak rule-based labels, and both. Rule-based labels were extracted from 66071 retrospectively collected radiology reports from 2017-2021 (DS 0), and 1091 reports from 2020-2021 (DS 1) were manually labeled according to the CheXpert classes. Label extraction performance was evaluated with respect to mention extraction, negation detection, and uncertainty detection by measuring F1 scores. The influence of the label extraction method on chest X-ray classification was evaluated on a pneumothorax data set (DS 2) containing 6434 chest radiographs with associated reports and expert diagnoses of pneumothorax. For this, DenseNet-121 models trained on manual annotations, rule-based and deep learning-based label predictions, and publicly available data were compared.The proposed deep learning-based labeler (DL) performed on average considerably stronger than the rule-based labeler (RB) for all three tasks on DS 1 with F1 scores of 0.938 vs. 0.844 for mention extraction, 0.891 vs. 0.821 for negation detection, and 0.624 vs. 0.518 for uncertainty detection. Pre-training on DS 0 and fine-tuning on DS 1 performed better than only training on either DS 0 or DS 1. Chest X-ray pneumothorax classification results (DS 2) were highest when trained with DL labels with an area under the receiver operating curve (AUC) of 0.939 compared to RB labels with an AUC of 0.858. Training with manual labels performed slightly worse than training with DL labels with an AUC of 0.934. In contrast, training with a public data set resulted in an AUC of 0.720.Our results show that leveraging a rule-based report labeler for weak supervision leads to improved labeling performance. The pneumothorax classification results demonstrate that our proposed deep learning-based labeler can serve as a substitute for manual labeling requiring only 1000 manually annotated reports for training. · The proposed deep learning-based label extraction model for German thoracic radiology reports performs better than the rule-based model.. · Training with limited supervision outperformed training with a small manually labeled data set.. · Using predicted labels for pneumothorax classification from chest radiographs performed equally to using manual annotations.. Wollek A, Haitzer P, Sedlmeyr T et al. Language modelbased labeling of German thoracic radiology reports. Fortschr Röntgenstr 2024; DOI 10.1055/a-2287-5054.

Opening the black box: explainable deep-learning classification of wood microscopic image of endangered tree species

BackgroundTraditional method of wood species identification involves the use of hand lens by wood anatomists, which is a time-consuming method that usually identifies only at the genetic level. Computer vision method can achieve "species" level identification but cannot provide an explanation on what features are used for the identification. Thus, in this study, we used computer vision methods coupled with deep learning to reveal interspecific differences between closely related tree species.ResultA total of 850 images were collected from the cross and tangential sections of 15 wood species. These images were used to construct a deep-learning model to discriminate wood species, and a classification accuracy of 99.3% was obtained. The key features between species in machine identification were targeted by feature visualization methods, mainly the axial parenchyma arrangements and vessel in cross section and the wood ray in tangential section. Moreover, the degree of importance of the vessels of different tree species in the cross-section images was determined by the manual feature labeling method. The results showed that vessels play an important role in the identification of Dalbergia, Pterocarpus, Swartzia, Carapa, and Cedrela, but exhibited limited resolutions on discriminating Swietenia species.ConclusionThe research results provide a computer-assisted tool for identifying endangered tree species in laboratory scenarios, which can be used to combat illegal logging and related trade and contribute to the implementation of CITES convention and the conservation of global biodiversity.

RotatedStomataNet: a deep rotated object detection network for directional stomata phenotype analysis.

Innovatively, we consider stomatal detection as rotated object detection and provide an end-to-end, batch, rotated, real-time stomatal density and aperture size intelligent detection and identification system, RotatedeStomataNet. Stomata acts as a pathway for air and water vapor in the course of respiration, transpiration, and other gas metabolism, so the stomata phenotype is important for plant growth and development. Intelligent detection of high-throughput stoma is a key issue. Nevertheless, currently available methods usually suffer from detection errors or cumbersome operations when facing densely and unevenly arranged stomata. The proposed RotatedStomataNet innovatively regards stomata detection as rotated object detection, enabling an end-to-end, real-time, and intelligent phenotype analysis of stomata and apertures. The system is constructed based on the Arabidopsis and maize stomatal data sets acquired destructively, and the maize stomatal data set acquired in a non-destructive way, enabling the one-stop automatic collection of phenotypic, such as the location, density, length, and width of stomata and apertures without step-by-step operations. The accuracy of this system to acquire stomata and apertures has been well demonstrated in monocotyledon and dicotyledon, such as Arabidopsis, soybean, wheat, and maize. The experimental results that the prediction results of the method are consistent with those of manual labeling. The test sets, the system code, and their usage are also given ( https://github.com/AITAhenu/RotatedStomataNet ).

Embedding Democratic Values into Social Media AIs via Societal Objective Functions

Mounting evidence indicates that the artificial intelligence (AI) systems that rank our social media feeds bear nontrivial responsibility for amplifying partisan animosity: negative thoughts, feelings, and behaviors toward political out-groups. Can we design these AIs to consider democratic values such as mitigating partisan animosity as part of their objective functions? We introduce a method for translating established, vetted social scientific constructs into AI objective functions, which we term societal objective functions, and demonstrate the method with application to the political science construct of anti-democratic attitudes. Traditionally, we have lacked observable outcomes to use to train such models-however, the social sciences have developed survey instruments and qualitative codebooks for these constructs, and their precision facilitates translation into detailed prompts for large language models. We apply this method to create a democratic attitude model that estimates the extent to which a social media post promotes anti-democratic attitudes, and test this democratic attitude model across three studies. In Study 1, we first test the attitudinal and behavioral effectiveness of the intervention among US partisans (N=1,380) by manually annotating (alpha=.895) social media posts with anti-democratic attitude scores and testing several feed ranking conditions based on these scores. Removal (d=.20) and downranking feeds (d=.25) reduced participants' partisan animosity without compromising their experience and engagement. In Study 2, we scale up the manual labels by creating the democratic attitude model, finding strong agreement with manual labels (rho=.75). Finally, in Study 3, we replicate Study 1 using the democratic attitude model instead of manual labels to test its attitudinal and behavioral impact (N=558), and again find that the feed downranking using the societal objective function reduced partisan animosity (d=.25). This method presents a novel strategy to draw on social science theory and methods to mitigate societal harms in social media AIs.

Enhanced YOLO v3 for precise detection of apparent damage on bridges amidst complex backgrounds

A bridge disease identification approach based on an enhanced YOLO v3 algorithm is suggested to increase the accuracy of apparent disease detection of concrete bridges under complex backgrounds. First, the YOLO v3 network structure is enhanced to better accommodate the dense distribution and large variation of disease scale characteristics, and the detection layer incorporates the squeeze and excitation (SE) networks attention mechanism module and spatial pyramid pooling module to strengthen the semantic feature extraction ability. Secondly, CIoU with better localization ability is selected as the loss function for training. Finally, the K-means algorithm is used for anchor frame clustering on the bridge surface disease defects dataset. 1363 datasets containing exposed reinforcement, spalling, and water erosion damage of bridges are produced, and network training is done after manual labelling and data improvement in order to test the efficacy of the algorithm described in this paper. According to the trial results, the YOLO v3 model has enhanced more than the original model in terms of precision rate, recall rate, Average Precision (AP), and other indicators. Its overall mean Average Precision (mAP) value has also grown by 5.5%. With the RTX2080Ti graphics card, the detection frame rate increases to 84 Frames Per Second, enabling more precise and real-time bridge illness detection.

Design and Test of Intelligent Farm Machinery Operation Control Platform for Unmanned Farms

The unmanned farm control platform is of great significance in promoting the supervision of farm production with less manpower or autonomous operation of farm machinery and the construction of farm informatization. Addressing the existing control platform for farm location information acquisition is time-consuming, labor-intensive, and lacks the whole process control of multiple types of farm machinery. In this paper, we propose an Internet of Things (IoT) control scheme for intelligent farm machinery operation of unmanned farms and design the access standards for multiple types of farm machinery, as well as realize the remote control of intelligent farm machinery operation by constructing a remote control model. A high-precision map construction method is designed to improve the DeepLabV3+ algorithm to identify fields and roads. The control models of path planning, remote task, remote control, and safety system are built to achieve the remote control of intelligent agricultural machinery operation. The proposed technology is implemented in the platform integration and application tests are carried out. The error of the constructed high-precision map is less than 3 cm, the completeness rate of the automatic boundary extraction rate is 96.71%, and the correctness rate is 95.63%, which can be used to obtain the boundary instead of manual labeling or on-site point picking. The use of the platform for the simultaneous control of three farm machinery operations reduces the number of people in operation and production and reduces the professional requirements of the personnel, which will promote the management of the entire farm by one person or even by no one in the future.

Self supervised learning based emotion recognition using physiological signals.

The significant role of emotional recognition in the field of human-machine interaction has garnered the attention of many researchers. Emotion recognition based on physiological signals can objectively reflect the most authentic emotional states of humans. However, existing labeled Electroencephalogram (EEG) datasets are often of small scale. In practical scenarios, a large number of unlabeled EEG signals are easier to obtain. Therefore, this paper adopts self-supervised learning methods to study emotion recognition based on EEG. Specifically, experiments employ three pre-defined tasks to define pseudo-labels and extract features from the inherent structure of the data. Experimental results indicate that self-supervised learning methods have the capability to learn effective feature representations for downstream tasks without any manual labels.

MA-MIL: Sampling point-level abnormal ECG location method via weakly supervised learning

Background and ObjectiveCurrent automatic electrocardiogram (ECG) diagnostic systems could provide classification outcomes but often lack explanations for these results. This limitation hampers their application in clinical diagnoses. Previous supervised learning could not highlight abnormal segmentation output accurately enough for clinical application without manual labeling of large ECG datasets. MethodIn this study, we present a multi-instance learning framework called MA-MIL, which has designed a multi-layer and multi-instance structure that is aggregated step by step at different scales. We evaluated our method using the public MIT-BIH dataset and our private dataset. ResultsThe results show that our model performed well in both ECG classification output and heartbeat level, sub-heartbeat level abnormal segment detection, with accuracy and F1 scores of 0.987 and 0.986 for ECG classification and 0.968 and 0.949 for heartbeat level abnormal detection, respectively. Compared to visualization methods, the IoU values of MA-MIL improved by at least 17 % and at most 31 % across all categories. ConclusionsMA-MIL could accurately locate the abnormal ECG segment, offering more trustworthy results for clinical application.

Data Sorting Influence on Short Text Manual Labeling Quality for Hierarchical Classification

The precise categorization of brief texts holds significant importance in various applications within the ever-changing realm of artificial intelligence (AI) and natural language processing (NLP). Short texts are everywhere in the digital world, from social media updates to customer reviews and feedback. Nevertheless, short texts’ limited length and context pose unique challenges for accurate classification. This research article delves into the influence of data sorting methods on the quality of manual labeling in hierarchical classification, with a particular focus on short texts. The study is set against the backdrop of the increasing reliance on manual labeling in AI and NLP, highlighting its significance in the accuracy of hierarchical text classification. Methodologically, the study integrates AI, notably zero-shot learning, with human annotation processes to examine the efficacy of various data-sorting strategies. The results demonstrate how different sorting approaches impact the accuracy and consistency of manual labeling, a critical aspect of creating high-quality datasets for NLP applications. The study’s findings reveal a significant time efficiency improvement in terms of labeling, where ordered manual labeling required 760 min per 1000 samples, compared to 800 min for traditional manual labeling, illustrating the practical benefits of optimized data sorting strategies. Comparatively, ordered manual labeling achieved the highest mean accuracy rates across all hierarchical levels, with figures reaching up to 99% for segments, 95% for families, 92% for classes, and 90% for bricks, underscoring the efficiency of structured data sorting. It offers valuable insights and practical guidelines for improving labeling quality in hierarchical classification tasks, thereby advancing the precision of text analysis in AI-driven research. This abstract encapsulates the article’s background, methods, results, and conclusions, providing a comprehensive yet succinct study overview.

Towards the development of an automated robotic storyteller: comparing approaches for emotional story annotation for non-verbal expression via body language

AbstractStorytelling is a long-established tradition and listening to stories is still a popular leisure activity. Caused by technization, storytelling media expands, e.g., to social robots acting as multi-modal storytellers, using different multimodal behaviours such as facial expressions or body postures. With the overarching goal to automate robotic storytelling, we have been annotating stories with emotion labels which the robot can use to automatically adapt its behavior. With it, three different approaches are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), and 3) a machine learning based approach (ML). In an online study showing videos of a storytelling robot, the annotations were validated, with LIWC and MA achieving the best, and ML the worst results. In a laboratory user study, the three versions of the story were compared regarding transportation and cognitive absorption, revealing no significant differences but a positive trend towards MA. On this empirical basis, the Automated Robotic Storyteller was implemented using manual annotations. Future iterations should include other robots and modalities, fewer emotion labels and their probabilities.