Ground Truth Annotations Research Articles

PreGLAM: A Predictive, Gameplay-based Layered Affect Model

We present the Predictive Gameplay-based Layered Affect Model (PreGLAM), an affective game spectator model that flexibly integrates into a game design process. PreGLAM combines elements of real-time Player Experience Models, and Affective Non-Player-Character models to output real-time estimated values for a spectator's valence, arousal, and tension during gameplay. Because tension is related to prospective events, PreGLAM attempts to predict future gameplay events. We implement and evaluate PreGLAM in a custom game <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Galactic Defense</i> , which we also describe. PreGLAM significantly outperforms a random walk time series in how accurately it matches ground-truth annotations, and has comparable accuracy to state of the art affect models.

Neural inverse procedural modeling of knitting yarns from images

We investigate the capabilities of neural inverse procedural modeling to infer high-quality procedural yarn models with fiber-level details from single images of depicted yarn samples. While directly inferring all parameters of the underlying yarn model based on a single neural network may seem an intuitive choice, we show that the complexity of yarn structures in terms of twisting and migration characteristics of the involved fibers can be better encountered in terms of ensembles of networks that focus on individual characteristics. We analyze the effect of different loss functions including a parameter loss to penalize the deviation of inferred parameters to ground truth annotations, a reconstruction loss to enforce similar statistics of the image generated for the estimated parameters in comparison to training images as well as an additional regularization term to explicitly penalize deviations between latent codes of synthetic images and the average latent code of real images in the encoder’s latent space. We demonstrate that the combination of a carefully designed parametric, procedural yarn model with respective network ensembles as well as loss functions even allows robust parameter inference when solely trained on synthetic data. Since our approach relies on the availability of a yarn database with parameter annotations and we are not aware of such a respectively available dataset, we additionally provide, to the best of our knowledge, the first dataset of yarn images with annotations regarding the respective yarn parameters. For this purpose, we use a novel yarn generator that improves the realism of the produced results over previous approaches.

Communities in C. elegans connectome through the prism of non-backtracking walks

The fundamental relationship between the mesoscopic structure of neuronal circuits and organismic functions they subserve is one of the major challenges in contemporary neuroscience. Formation of structurally connected modules of neurons enacts the conversion from single-cell firing to large-scale behaviour of an organism, highlighting the importance of their accurate profiling in the data. While connectomes are typically characterized by significant sparsity of neuronal connections, recent advances in network theory and machine learning have revealed fundamental limitations of traditionally used community detection approaches in cases where the network is sparse. Here we studied the optimal community structure in the structural connectome of Caenorhabditis elegans, for which we exploited a non-conventional approach that is based on non-backtracking random walks, virtually eliminating the sparsity issue. In full agreement with the previous asymptotic results, we demonstrated that non-backtracking walks resolve the ground truth annotation into clusters on stochastic block models (SBM) with the size and density of the connectome better than the spectral methods related to simple random walks. Based on the cluster detectability threshold, we determined that the optimal number of modules in a recently mapped connectome of C. elegans is 10, which precisely corresponds to the number of isolated eigenvalues in the spectrum of the non-backtracking flow matrix. The discovered communities have a clear interpretation in terms of their functional role, which allows one to discern three structural compartments in the worm: the Worm Brain (WB), the Worm Movement Controller (WMC), and the Worm Information Flow Connector (WIFC). Broadly, our work provides a robust network-based framework to reveal mesoscopic structures in sparse connectomic datasets, paving way to further investigation of connectome mechanisms for different functions.

Semiautomatic Assessment of Facet Tropism From Lumbar Spine MRI Using Deep Learning: A Northern Finland Birth Cohort Study.

This is a retrospective, cross-sectional, population-based study that automatically measured the facet joint (FJ) angles from T2-weighted axial magnetic resonance imagings (MRIs) of the lumbar spine using deep learning (DL). This work aimed to introduce a semiautomatic framework that measures the FJ angles using DL and study facet tropism (FT) in a large Finnish population-based cohort. T2-weighted axial MRIs of the lumbar spine (L3/4 through L5/S1) for (n=1288) in the NFBC1966 Finnish population-based cohort were used for this study. A DL model was developed and trained on 430 participants' MRI images. The authors computed FJ angles from the model's prediction for each level, that is, L3/4 through L5/S1, for the male and female subgroups. Inter-rater and intrarater reliability was analyzed for 60 participants using annotations made by two radiologists and a musculoskeletal researcher. With the developed method, we examined FT in the entire NFBC1966 cohort, adopting the literature definitions of FT thresholds at 7° and 10°. The rater agreement was evaluated both for the annotations and the FJ angles computed based on the annotations. FJ asymmetry ( - was used to evaluate the agreement and correlation between the raters. Bland-Altman analysis was used to assess the agreement and systemic bias in the FJ asymmetry. The authors used the Dice score as the metric to compare the annotations between the raters. The authors evaluated the model predictions on the independent test set and compared them against the ground truth annotations. This model scored Dice (92.7±0.1) and intersection over union (87.1±0.2) aggregated across all the regions of interest, that is, vertebral body (VB), FJs, and posterior arch (PA). The mean FJ angles measured for the male and female subgroups were in agreement with the literature findings. Intrarater reliability was high, with a Dice score of VB (97.3), FJ (82.5), and PA (90.3). The inter-rater reliability was better between the radiologists with a Dice score of VB (96.4), FJ (75.5), and PA (85.8) than between the radiologists and the musculoskeletal researcher. The prevalence of FT was higher in the male subgroup, with L4/5 found to be the most affected region. The authors developed a DL-based framework that enabled us to study FT in a large cohort. Using the proposed method, the authors present the prevalence of FT in a Finnish population-based cohort.

Unsupervised CNN-based DIC method for 2D displacement measurement

Digital image correlation (DIC) is a widely used technique for non-contact measurement of deformation. However, traditional DIC methods face challenges in balancing calculation efficiency and the quantity of seed points. Deep learning approaches, particularly supervised learning methods, have shown promise in improving DIC efficiency. However, these methods require high-quality training data, which can be time-consuming to generate ground truth annotations. To address these challenges, we propose an unsupervised convolutional neural network (CNN) based DIC method for 2D displacement measurement. Our approach leverages an encoder-decoder architecture with multi-level feature extraction, a dual-path correlation block, and an attention block to extract informative features from speckle images with varying characteristics. We utilize a speckle image warp model to transform the deformed speckle image to the predicted reference speckle image based on the predicted 2D displacement map. The unsupervised training is achieved by comparing the predicted and original reference speckle images. To optimize the network's parameters, we employ a composite loss function that takes into account both the Mean Squared Error (MSE) and Pearson correlation coefficient. By using unsupervised convolutional neural network (CNN) based DIC method, we eliminate the need for extensive training data annotation, which is a time-consuming process in supervised learning DIC methods. We have conducted several experiments to demonstrate the validity and robustness of our proposed method. The results show a significant reduction in Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) compared to a method proposed by Zhao et al. This indicates that our unsupervised CNN-based DIC approach can achieve accuracy comparable to supervised CNN-based DIC methods. For implementation and evaluation, we provide PyTorch code and datasets, which will be released at the following URL :https://github.com/fead1/DICNet-corr-unsupervised-learning-.

Evaluation of mediastinal lymph node segmentation of heterogeneous CT data with full and weak supervision

Accurate lymph node size estimation is critical for staging cancer patients, initial therapeutic management, and assessing response to therapy. Current standard practice for quantifying lymph node size is based on a variety of criteria that use uni-directional or bi-directional measurements. Segmentation in 3D can provide more accurate evaluations of the lymph node size. Fully convolutional neural networks (FCNs) have achieved state-of-the-art results in segmentation for numerous medical imaging applications, including lymph node segmentation. Adoption of deep learning segmentation models in clinical trials often faces numerous challenges. These include lack of pixel-level ground truth annotations for training, generalizability of the models on unseen test domains due to the heterogeneity of test cases and variation of imaging parameters. In this paper, we studied and evaluated the performance of lymph node segmentation models on a dataset that was completely independent of the one used to create the models. We analyzed the generalizability of the models in the face of a heterogeneous dataset and assessed the potential effects of different disease conditions and imaging parameters. Furthermore, we systematically compared fully-supervised and weakly-supervised methods in this context. We evaluated the proposed methods using an independent dataset comprising 806 mediastinal lymph nodes from 540 unique patients. The results show that performance achieved on the independent test set is comparable to that on the training set. Furthermore, neither the underlying disease nor the heterogeneous imaging parameters impacted the performance of the models. Finally, the results indicate that our weakly-supervised method attains 90%− 91% of the performance achieved by the fully supervised training.

Home recording of 3-Hz spike-wave discharges in adults with absence epilepsy using the wearable Sensor Dot.

Home monitoring of 3-Hz spike-wave discharges (SWDs) in patients with refractory absence epilepsy could improve clinical care by replacing the inaccurate seizure diary with objective counts. We investigated the use and performance of the Sensor Dot (Byteflies) wearable in persons with absence epilepsy in their home environment. Thirteen participants (median age = 22 years, 11 female) were enrolled at the university hospitals of Leuven and Freiburg. At home, participants had to attach the Sensor Dot and behind-the-ear electrodes to record two-channel electroencephalogram (EEG), accelerometry, and gyroscope data. Ground truth annotations were created during a visual review of the full Sensor Dot recording. Generalized SWDs were annotated if they were 3 Hz and at least 3 s on EEG. Potential 3-Hz SWDs were flagged by an automated seizure detection algorithm, (1) using only EEG and (2) with an additional postprocessing step using accelerometer and gyroscope to discard motion artifacts. Afterward, two readers (W.V.P. and L.S.) reviewed algorithm-labeled segments and annotated true positive detections. Sensitivity, precision, and F1 score were calculated. Patients had to keep a seizure diary and complete questionnaires about their experiences. Total recording time was 394 h 42 min. Overall, 234 SWDs were captured in 11 of 13 participants. Review of the unimodal algorithm-labeled recordings resulted in a mean sensitivity of .84, precision of .93, and F1 score of .89. Visual review of the multimodal algorithm-labeled segments resulted in a similar F1 score and shorter review time due to fewer false positive labels. Participants reported that the device was comfortable and that they would be willing to wear it on demand of their neurologist, for a maximum of 1 week or with intermediate breaks. The Sensor Dot improved seizure documentation at home, relative to patient self-reporting. Additional benefits were the short review time and the patients' device acceptance due to user-friendliness and comfortability.

Rethinking precision of pseudo label: Test-time adaptation via complementary learning

Test-time adaptation (TTA) has been proven that is capable of transferring the source model to a different data distribution. The main challenge of TTA is how to learn the target-domain representation without the help of ground truth annotation. Previous studies have employed unsupervised objectives, such as the entropy of model predictions, as optimization targets to effectively learn features for test-time samples. However, the performance of the model is easily compromised by the quality of pseudo labels. In this work, we propose a novel way to leverage complementary learning to facilitate target feature learning, along with a complementary-labeling scenario and dynamical refining of the decision boundary. The use of complementary label learning effectively prevents the model from collapsing to high-confidence instances with incorrect pseudo labels. The complementary labels are less informative than ordinary labels and thus a suitable risk estimator is derived to better learn from them. We highlight that this learning of complementary labels agrees with Vanilla learning with ordinary data distribution. Comprehensive experiments show that the proposed learning algorithm achieves state-of-the-art performance on different datasets and experiment settings.

Automated neuron tracking inside moving and deforming C. elegans using deep learning and targeted augmentation.

Reading out neuronal activity from three-dimensional (3D) functional imaging requires segmenting and tracking individual neurons. This is challenging in behaving animals if the brain moves and deforms. The traditional approach is to train a convolutional neural network with ground-truth (GT) annotations of images representing different brain postures. For 3D images, this is very labor intensive. We introduce 'targeted augmentation', a method to automatically synthesize artificial annotations from a few manual annotations. Our method ('Targettrack') learns the internal deformations of the brain to synthesize annotations for new postures by deforming GT annotations. This reduces the need for manual annotation and proofreading. A graphical user interface allows the application of the method end-to-end. We demonstrate Targettrack on recordings where neurons are labeled as key points or 3D volumes. Analyzing freely moving animals exposed to odor pulses, we uncover rich patterns in interneuron dynamics, including switching neuronal entrainment on and off.

Ultrasonic array tomography-oriented subsurface crack recognition and cross-section image reconstruction of reinforced concrete structure using deep neural networks

Manual interpretation from technicians with prior expertise knowledge is still needed for ultrasonic array tomography when detecting subsurface damages in reinforced concrete (RC) structures. Ultrasonic scans were usually utilized in qualitative manners that have not been leveraged for quantification purposes in prior studies and practical engineering. This study presented an ultrasonic array tomography-oriented subsurface crack recognition and pixel-wise cross-section image reconstruction method for RC structures based on deep neural networks (DNN). B-scans were acquired by testing entity RC components with preset artificial cracks using shear-wave low-frequency ultrasonic array. A DNN with the basic encoder-decoder architecture was developed, which was improved by introducing skip connections and residual modules to adapt to semantic structure of ultrasonic B-scans. A supervised generative adversarial DNN was proposed to generate fake but convincing B-scans for data augmentation. Results reported that F-scores were higher than 78 %, and satisfactory agreements were observed between ground truth annotations and reconstructed cross-section images, substantiating the effectiveness of the proposed method. Furthermore, with the physical information-assisted registration, individual local reconstructed cross-section images were combined to global representations indicating crack distribution in the entire component to facilitate convincing structural assessments.

Evaluating rhyme annotations for large corpora

Abstract Recent methods have been proposed to produce automatic rhyme annotators for large rhymed corpora. These methods, such as Baley (2022b) greatly reduce the cost of annotating rhymed material, allowing historical linguists to focus on the analysis of the rhyme patterns. However, evidence for the quality of those annotations has been anecdotal, consisting of a handful of individual poem case studies. This paper proposes to address the issue: first, we discuss previously proposed metrics that evaluate the quality of an annotator’s output against a ground-truth annotation (List, Hill, and Foster; 2019) and we propose an alternative metric that is better suited to the task. Then, sampling from Baley’s published annotated corpus and re-annotating it by hand, we use the sample to demonstrate the lacunae in the original approach and show how to fix them. Finally, the hand-annotated sample and source code are published as additional data, so that other researchers can compare the performance of their own annotators.

DGU-HAU: A Dataset for 3D Human Action Analysis on Utterances

Constructing diverse and complex multi-modal datasets is crucial for advancing human action analysis research, providing ground truth annotations for training deep learning networks, and enabling the development of robust models across real-world scenarios. Generating natural and contextually appropriate nonverbal gestures is essential for enhancing immersive and effective human–computer interactions in various applications. These applications include video games, embodied virtual assistants, and conversations within a metaverse. However, existing speech-related human datasets are focused on style transfer, so they have limitations that make them unsuitable for 3D human action analysis studies, such as human action recognition and generation. Therefore, we introduce a novel multi-modal dataset, DGU-HAU, a dataset for 3D human action on utterances that commonly occurs during daily life. We validate the dataset using a human action generation model, Action2Motion (A2M), a state-of-the-art 3D human action generation model.

The Segment Anything Model (SAM) for accelerating the smart farming revolution

Precision agriculture uses accurate identification and mapping of crop features by automated mechanisms. Using computer vision techniques implemented by supervised deep learning systems to solve many precision agricultural problems necessitates large-scale data collection and prolonged ground truth annotation by humans. The so-called foundation models in Artificial Intelligence (AI) are becoming increasingly significant. Meta AI Research is working on a project called Segment Anything to provide a base model for image segmentation. It can accomplish zero-shot generalisation to strange objects and images without additional training. This study evaluates the performance of the Segment Anything Model (SAM) for the problem of semantic segmentation of objects in the context of precision agriculture.

Assessing Acetabular Index Angle in Infants: A Deep Learning-Based Novel Approach.

Developmental dysplasia of the hip (DDH) is a disorder characterized by abnormal hip development that frequently manifests in infancy and early childhood. Preventing DDH from occurring relies on a timely and accurate diagnosis, which requires careful assessment by medical specialists during early X-ray scans. However, this process can be challenging for medical personnel to achieve without proper training. To address this challenge, we propose a computational framework to detect DDH in pelvic X-ray imaging of infants that utilizes a pipelined deep learning-based technique consisting of two stages: instance segmentation and keypoint detection models to measure acetabular index angle and assess DDH affliction in the presented case. The main aim of this process is to provide an objective and unified approach to DDH diagnosis. The model achieved an average pixel error of 2.862 ± 2.392 and an error range of 2.402 ± 1.963° for the acetabular angle measurement relative to the ground truth annotation. Ultimately, the deep-learning model will be integrated into the fully developed mobile application to make it easily accessible for medical specialists to test and evaluate. This will reduce the burden on medical specialists while providing an accurate and explainable DDH diagnosis for infants, thereby increasing their chances of successful treatment and recovery.

Pixel-level clustering network for unsupervised image segmentation

While image segmentation is crucial in various computer vision applications, such as autonomous driving, grasping, and robot navigation, annotating all objects at the pixel-level for training is nearly impossible. Therefore, the study of unsupervised image segmentation methods is essential. In this paper, we present a pixel-level clustering framework for segmenting images into regions without using ground truth annotations. The proposed framework includes feature embedding modules with an attention mechanism, a feature statistics computing module, image reconstruction, and superpixel segmentation to achieve accurate unsupervised segmentation. Additionally, we propose a training strategy that utilizes intra-consistency within each superpixel, inter-similarity/dissimilarity between neighboring superpixels, and structural similarity between images. To avoid potential over-segmentation caused by superpixel-based losses, we also propose a post-processing method. Furthermore, we present an extension of the proposed method for unsupervised semantic segmentation. We conducted experiments on three publicly available datasets (Berkeley segmentation dataset, PASCAL VOC 2012 dataset, and COCO-Stuff dataset) to demonstrate the effectiveness of the proposed framework. The experimental results show that the proposed framework outperforms previous state-of-the-art methods.

Automatic correction of performance drift under acquisition shift in medical image classification

Image-based prediction models for disease detection are sensitive to changes in data acquisition such as the replacement of scanner hardware or updates to the image processing software. The resulting differences in image characteristics may lead to drifts in clinically relevant performance metrics which could cause harm in clinical decision making, even for models that generalise in terms of area under the receiver-operating characteristic curve. We propose Unsupervised Prediction Alignment, a generic automatic recalibration method that requires no ground truth annotations and only limited amounts of unlabelled example images from the shifted data distribution. We illustrate the effectiveness of the proposed method to detect and correct performance drift in mammography-based breast cancer screening and on publicly available histopathology data. We show that the proposed method can preserve the expected performance in terms of sensitivity/specificity under various realistic scenarios of image acquisition shift, thus offering an important safeguard for clinical deployment.

Inter- and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation

Acquiring pixel-level annotations is often limited in applications such as histology studies that require domain expertise. Various semi-supervised learning approaches have been developed to work with limited ground truth annotations, such as the popular teacher-student models. However, hierarchical prediction uncertainty within the student model (intra-uncertainty) and image prediction uncertainty (inter-uncertainty) have not been fully utilized by existing methods. To address these issues, we first propose a novel inter- and intra-uncertainty regularization method to measure and constrain both inter- and intra-inconsistencies in the teacher-student architecture. We also propose a new two-stage network with pseudo-mask guided feature aggregation (PG-FANet) as the segmentation model. The two-stage structure complements with the uncertainty regularization strategy to avoid introducing extra modules in solving uncertainties and the aggregation mechanisms enable multi-scale and multi-stage feature integration. Comprehensive experimental results over the MoNuSeg and CRAG datasets show that our PG-FANet outperforms other state-of-the-art methods and our semi-supervised learning framework yields competitive performance with a limited amount of labeled data.

Predicting viral rumors and vulnerable users with graph-based neural multi-task learning for infodemic surveillance

In the age of the infodemic, it is crucial to have tools for effectively monitoring the spread of rampant rumors that can quickly go viral, as well as identifying vulnerable users who may be more susceptible to spreading such misinformation. This proactive approach allows for timely preventive measures to be taken, mitigating the negative impact of false information on society. We propose a novel approach to predict viral rumors and vulnerable users using a unified graph neural network model. We pre-train network-based user embeddings and leverage a cross-attention mechanism between users and posts, together with a community-enhanced vulnerability propagation (CVP) method to improve user and propagation graph representations. Furthermore, we employ two multi-task training strategies to mitigate negative transfer effects among tasks in different settings, enhancing the overall performance of our approach. We also construct two datasets with ground-truth annotations on information virality and user vulnerability in rumor and non-rumor events, which are automatically derived from existing rumor detection datasets. Extensive evaluation results of our joint learning model confirm its superiority over strong baselines in all three tasks: rumor detection, virality prediction, and user vulnerability scoring. For instance, compared to the best baselines based on the Weibo dataset, our model makes 3.8% and 3.0% improvements on Accuracy and MacF1 for rumor detection, and reduces mean squared error (MSE) by 23.9% and 16.5% for virality prediction and user vulnerability scoring, respectively. Our findings suggest that our approach effectively captures the correlation between rumor virality and user vulnerability, leveraging this information to improve prediction performance and provide a valuable tool for infodemic surveillance.

Detecting and Characterizing Inferior Vena Cava Filters on Abdominal Computed Tomography with Data-Driven Computational Frameworks

Two data-driven algorithms were developed for detecting and characterizing Inferior Vena Cava (IVC) filters on abdominal computed tomography to assist healthcare providers with the appropriate management of these devices to decrease complications: one based on 2-dimensional data and transfer learning (2D + TL) and an augmented version of the same algorithm which accounts for the 3-dimensional information leveraging recurrent convolutional neural networks (3D + RCNN). The study contains 2048 abdominal computed tomography studies obtained from 439 patients who underwent IVC filter placement during the 10-year period from January 1st, 2009, to January 1st, 2019. Among these, 399 patients had retrievable filters, and 40 had non-retrievable filter types. The reference annotations for the filter location were obtained through a custom-developed interface. The ground truth annotations for the filter types were determined based on the electronic medical record and physician review of imaging. The initial stage of the framework returns a list of locations containing metallic objects based on the density of the structure. The second stage processes the candidate locations and determines which one contains an IVC filter. The final stage of the pipeline classifies the filter types as retrievable vs. non-retrievable. The computational models are trained using Tensorflow Keras API on an Nvidia Quadro GV100 system. We utilized a fine-tuning supervised training strategy to conduct our experiments. We find that the system achieves high sensitivity on detecting the filter locations with a high confidence value. The 2D + TL model achieved a sensitivity of 0.911 and a precision of 0.804, and the 3D + RCNN model achieved a sensitivity of 0.923 and a precision of 0.853 for filter detection. The system confidence for the IVC location predictions is high: 0.993 for 2D + TL and 0.996 for 3D + RCNN. The filter type prediction component of the system achieved 0.945 sensitivity, 0.882 specificity, and 0.97 AUC score with 2D + TL and 0. 940 sensitivity, 0.927 specificity, and 0.975 AUC score with 3D + RCNN. With the intent to create tools to improve patient outcomes, this study describes the initial phase of a computational framework to support healthcare providers in detecting patients with retained IVC filters, so an individualized decision can be made to remove these devices when appropriate, to decrease complications. To our knowledge, this is the first study that curates abdominal computed tomography (CT) scans and presents an algorithm for automated detection and characterization of IVC filters.

Medimatrix: innovative pre-training of grayscale images for rheumatoid arthritis diagnosis revolutionises medical image classification.

Efficient and accurate medical image classification (MIC) methods face two major challenges: (1) high similarity between images of different disease classes; and (2) generating large medical image datasets for training deep neural networks is challenging due to privacy restrictions and the need for expert ground truth annotations. In this paper, we introduce a novel deep learning method called pre-training grayscale images with supervised learning for MIC (MediMatrix). Instead of pre-training on color ImageNet, our approach uses MediMatrix on grayscale ImageNet. To improve the performance of the network, we introduce ShuffleAttention (SA), a self-attention mechanism. By combining SA with the multiple residual structure (ResSA block) and replacing short-cut connections with dense residual connections between corresponding layers (densepath), our network can dynamically adjust channel attention weights and receive image inputs of different sizes, resulting in improved feature representation and better discrimination of similarities between different categories. MediMatrix effectively classifies X-ray images of rheumatoid arthritis (RA), enabling efficient screening without the need for expert analysis or invasive testing. Through extensive experiments, we demonstrate the superiority of MediMatrix over state-of-the-art methods and that color is not critical for rich natural image classification. Our results highlight the potential of computer-aided diagnosis combined with MediMatrix as a valuable screening tool for early detection and intervention in RA.