Manual Annotation Research Articles

CLIP-FSSC: A Transferable Visual Model for Fish and Shrimp Species Classification Based on Natural Language Supervision

Fish and shrimp species classification is a practical need in the field of aquaculture. Traditional classification method includes extracting modal features of the single image and training them on downstream datasets. However, this method has the disadvantage of requiring manual annotated image data and significant training time. To address these issues, this paper introduces a method named CLIP-FSSC (Contrastive Language–Image Pre-training for Fish and Shrimp Species Classification) for zero-shot prediction using a pre-trained model. The proposed method aims to classify fish and shrimp species in the field of aquaculture using a multimodal pre-trained model that utilizes semantic text description as an image supervision signal for transfer learning. In the downstream fish dataset, we use natural language labels for three types of fish - grass carp, common carp, and silver carp. We extract text category features using a transformer and compare the results obtained from three different CLIP-based backbones for the image modality - vision transformer, Resnet50, and Resnet101. We compare the performance of these models with previous methods that performed well. After performing zero-shot predictions on samples of the three types of fish, we achieve similar or even better classification accuracy than models trained on downstream fish datasets. Our experiment results show an accuracy of 98.77%, and no new training process is required. This proves that using the semantic text modality as the label for the image modality can effectively classify fish species. To demonstrate the effectiveness of this method on other species in the field of aquaculture, we collected two sets of shrimp data - prawn and cambarus. Through zero-shot prediction, we achieve the highest classification accuracy of 92.00% for these two types of shrimp datasets. Overall, our results demonstrate that using a multimodal pre-trained model with semantic text description as an image supervision signal for transfer learning can effectively classify fish and shrimp species with high accuracy, while reducing the need for manual annotation and training time.

Machine Learning Approach to Injury Monitoring in Children and Adolescents

Abstract Pediatric injuries are common and associated with individual suffering and costs for the healthcare systems and society. The incidence of injuries among children in Switzerland has so far been insufficiently monitored. Emergency department patient records can be a valuable database to assess medically relevant injuries and gain information for monitoring and evidence-based prevention. The talk will outline our stepwise development of a machine-learning program for a pediatric injury monitoring system and lessons learnt. In this feasibility study we draw on electronic case histories of children treated for injuries between 2018 and 2022 at the University Children’s Hospital Zürich emergency department (N= to 30’884). The ML- approach follows different steps: step 1. utility evaluation of electronic pediatric patient records regarding ML and injury monitoring and prevention requirements in a test-sample of 100 electronic pediatric case histories; step 2. choice/adaptation of coding tree; step 3. coder training and inter-coder reliability testing; step 4. manual annotation of a sub-sample (n 1000) of the electronic pediatric patient records, step 5. design and testing of the ML program in an iterative process, and step 6. performance testing in a random sample of pediatric patient records. First lessons learnt indicate that electronic case history data contain relevant data and fulfill requirements for monitoring and prevention purposes. The IDB coding system proved too complex and detailed for the pediatric data base, providing too few examples for the ML-learning process, and needed to be adapted. Initial data analyses indicate the majority of cases are classified as less severe (73%) and only few (<1%) as very severe but the later provide the most detailed data. ML is a promising approach and makes use of hospital data for a national monitoring of pediatric injuries in Switzerland.

A comparative study of LLMs and NLP approaches for supporting business process analysis

ABSTRACT This paper compares two approaches to support Business Process Analysis (BPA) for the construction of a Business Process Knowledge Base (BPKB). The methodology is based on the BPA Canvas metamodel, which starts from the preliminary output of the business analysis (an interview). We focus on the extraction of key elements from the text to build the BPKBcore. We experiment with approaches based on Natural Language Processing (NLP) and Large Language Models (LLMs), using a running example to compare the outcomes with manual annotations. The experiment shows that the LLM-based approach yields better performance, especially with enriched prompts.

CriteriaMapper: establishing the automatic identification of clinical trial cohorts from electronic health records by matching normalized eligibility criteria and patient clinical characteristics

The use of electronic health records (EHRs) holds the potential to enhance clinical trial activities. However, the identification of eligible patients within EHRs presents considerable challenges. We aimed to develop a CriteriaMapper system for phenotyping eligibility criteria, enabling the identification of patients from EHRs with clinical characteristics that match those criteria. We utilized clinical trial eligibility criteria and patient EHRs from the Mount Sinai Database. The CriteriaMapper system was developed to normalize the criteria using national standard terminologies and in-house databases, facilitating computability and queryability to bridge clinical trial criteria and EHRs. The system employed rule-based pattern recognition and manual annotation. Our system normalized 367 out of 640 unique eligibility criteria attributes, covering various medical conditions including non-small cell lung cancer, small cell lung cancer, prostate cancer, breast cancer, multiple myeloma, ulcerative colitis, Crohn’s disease, non-alcoholic steatohepatitis, and sickle cell anemia. About 174 criteria were encoded with standard terminologies and 193 were normalized using the in-house reference tables. The agreement between automated and manual normalization was high (Cohen’s Kappa = 0.82), and patient matching demonstrated a 0.94 F1 score. Our system has proven effective on EHRs from multiple institutions, showing broad applicability and promising improved clinical trial processes, leading to better patient selection, and enhanced clinical research outcomes.

Highlighted Diffusion Model as Plug-in Priors for Polyp Segmentation.

Automated polyp segmentation from colonoscopy images is crucial for colorectal cancer diagnosis. The accuracy of such segmentation, however, is challenged by two main factors. First, the variability in polyps' size, shape, and color, coupled with the scarcity of well-annotated data due to the need for specialized manual annotation, hampers the efficacy of existing deep learning methods. Second, concealed polyps often blend with adjacent intestinal tissues, leading to poor contrast that challenges segmentation models. Recently, diffusion models have been explored and adapted for polyp segmentation tasks. However, the significant domain gap between RGB-colonoscopy images and grayscale segmentation masks, along with the low efficiency of the diffusion generation process, hinders the practical implementation of these models. To mitigate these challenges, we introduce the Highlighted Diffusion Model Plus (HDM+), a two-stage polyp segmentation framework. This framework incorporates the Highlighted Diffusion Model (HDM) to provide explicit semantic guidance, thereby enhancing segmentation accuracy. In the initial stage, the HDM is trained using highlighted ground-truth data, which emphasizes polyp regions while suppressing the background in the images. This approach reduces the domain gap by focusing on the image itself rather than on the segmentation mask. In the subsequent second stage, we employ the highlighted features from the trained HDM's U-Net model as plug-in priors for polyp segmentation, rather than generating highlighted images, thereby increasing efficiency. Extensive experiments conducted on six polyp segmentation benchmarks demonstrate the effectiveness of our approach.

Deep Learning Segmentation of Chromogenic Dye RNAscope From Breast Cancer Tissue.

RNAscope staining of breast cancer tissue allows pathologists to deduce genetic characteristics of the cancer by inspection at the microscopic level, which can lead to better diagnosis and treatment. Chromogenic RNAscope staining is easy to fit into existing pathology workflows, but manually analyzing the resulting tissue samples is time consuming. There is also a lack of peer-reviewed, performant solutions for automated analysis of chromogenic RNAscope staining. This paper covers the development and optimization of a novel deep learning method focused on accurate segmentation of RNAscope dots (which signify gene expression) from breast cancer tissue. The deep learning network is convolutional and uses ConvNeXt as its backbone. The upscaling portions of the network use custom, heavily regularized blocks to prevent overfitting and early convergence on suboptimal solutions. The resulting network is modest in size for a segmentation network and able to function well with little training data. This deep learning network was also able to outperform manual expert annotation at finding the positions of RNAscope dots, having a final -score of 0.745. In comparison, the expert inter-rater -score was 0.596.

Efficient labeling of french mammogram reports with MammoBERT

Recent advances in deep learning and natural language processing (NLP) have broadened opportunities for automatic text processing in the medical field. However, the development of models for low-resource languages like French is challenged by limited datasets, often due to legal restrictions. Large-scale training of medical imaging models often requires extracting labels from radiology text reports. Current methods for report labeling primarily rely on sophisticated feature engineering based on medical domain knowledge or manual annotations by radiologists. These methods can be labor-intensive. In this work, we introduce a BERT-based approach for the efficient labeling of French mammogram image reports. Our method leverages both the expansive scale of existing rule-based systems and the precision of radiologist annotations. Our experimental results showcase the superiority of the proposed approach. It was initially fine-tuned on a limited dataset of radiologist annotations. Then, it underwent training on annotations generated by a rule-based labeler. Our findings reveal that our final model, MammoBERT, significantly outperforms the rule-based labeler while simultaneously reducing the necessity for radiologist annotations during training. This research not only advances the state of the art in medical image report labeling but also offers an efficient and effective solution for large-scale medical imaging model development.

Deep learning prediction of error and skill in robotic prostatectomy suturing.

Manual objective assessment of skill and errors in minimally invasive surgery have been validated with correlation to surgical expertise and patient outcomes. However, assessment and error annotation can be subjective and are time-consuming processes, often precluding their use. Recent years have seen the development of artificial intelligence models to work towards automating the process to allow reduction of errors and truly objective assessment. This study aimed to validate surgical skill rating and error annotations in suturing gestures to inform the development and evaluation of AI models. SAR-RARP50 open data set was blindly, independently annotated at the gesture level in Robotic-Assisted Radical Prostatectomy (RARP) suturing. Manual objective assessment tools and error annotation methodology, Objective Clinical Human Reliability Analysis (OCHRA), were used as ground truth to train and test vision-based deep learning methods to estimate skill and errors. Analysis included descriptive statistics plus tool validity and reliability. Fifty-four RARP videos (266min) were analysed. Strong/excellent inter-rater reliability (range r = 0.70-0.89, p < 0.001) and very strong correlation (r = 0.92, p < 0.001) between objective assessment tools was demonstrated. Skill estimation of OSATS and M-GEARS had a Spearman's Correlation Coefficient 0.37 and 0.36, respectively, with normalised mean absolute error representing a prediction error of 17.92% (inverted "accuracy" 82.08%) and 20.6% (inverted "accuracy" 79.4%) respectively. The best performing models in error prediction achieved mean absolute precision of 37.14%, area under the curve 65.10% and Macro-F1 58.97%. This is the first study to employ detailed error detection methodology and deep learning models within real robotic surgical video. This benchmark evaluation of AI models sets a foundation and promising approach for future advancements in automated technical skill assessment.

Decomposed Prototype Learning for Few-Shot Scene Graph Generation

Today's scene graph generation (SGG) models typically require abundant manual annotations to learn new predicate types. Therefore, it is difficult to apply them to real-world applications with massive uncommon predicate categories whose annotations are hard to collect. In this paper, we focus on Few-Shot SGG (FSSGG) , which encourages SGG models to be able to quickly transfer previous knowledge and recognize unseen predicates well with only a few examples. However, current methods for FSSGG are hindered by the high intra-class variance of predicate categories in SGG: On one hand, each predicate category commonly has multiple semantic meanings under different contexts. On the other hand, the visual appearance of relation triplets with the same predicate differs greatly under different subject-object compositions. Such great variance of inputs makes it hard to learn generalizable representation for each predicate category with current few-shot learning (FSL) methods. However, we found that this intra-class variance of predicates is highly related to the composed subjects and objects. To model the intra-class variance of predicates with subject-object context, we propose a novel Decomposed Prototype Learning (DPL) model for FSSGG. Specifically, we first construct a decomposable prototype space to capture diverse semantics and visual patterns of subjects and objects for predicates by decomposing them into multiple prototypes. Afterwards, we integrate these prototypes with different weights to generate query-adaptive predicate representation with more reliable semantics for each query sample. We conduct extensive experiments and compare with various baseline methods to show the effectiveness of our method.

Unsupervised anomaly detection in shearers via autoencoder networks and multi-scale correlation matrix reconstruction

As the main equipment of coal mining production, the anomaly detection of shearer is important to ensure production efficiency and coal mine safety. One key challenge lies in the limited or even absence of labeled monitoring data for the equipment, coupled with the high costs associated with manual annotation. Another challenge stems from the complex structure of the mining machines, making it difficult to reflect the overall operational state through local anomaly detection. Consequently, the application of decoupled local anomaly detection for mining machines in practical production remains challenging. This paper presents an unsupervised learning-based method for detecting anomalies in shearer. The method includes a module for constructing a Multi-scale Correlation Matrix (MSCM) of mining machine operating conditions, as well as the CNN-ConvLSTM Autoencoder (C-CLA) network. The module for constructing an MSCM enhances the representation of interrelationships between various features of the equipment from different perspectives using multiple correlation analysis methods. The C-CLA network integrates convolutional and convolutional recurrent neural networks, with the convolutional structure extracting local spatial features and the ConvLSTM structure further capturing information from different time scales and feature scales, thereby enhancing the model’s perceptual capabilities towards changes in equipment status. Finally, shearer anomaly detection is achieved through the analysis of reconstructed residual matrices. The rationality and practicality of the proposed method have been validated on our dataset, and the model’s generalization capability has been verified through repeated experiments in similar scenarios. However, due to variations in the working environment of different mining faces and differences in equipment models, implementing detection on other mining faces often requires retraining the model with new data. Furthermore, we compared our method with other anomaly detection techniques, and our detection efficiency was superior by approximately 3%. This method effectively detects anomalies in the shearer.

Segmentation of multi-view video speed extraction method that can cross lane dividing lines

Aiming at the problem that the existing video traffic parameter extraction methods are overly dependent on manual annotation and a single perspective cannot effectively correct the dynamic driving deviation of vehicles on site, a multi-perspective video traffic parameter extraction method that can segment and cross lane dividing lines is proposed. The method consists of an automatic annotation point generation module and a multi-perspective correction module. The automatic annotation point generation module realizes the process of automatically generating annotation points by constructing a reference block based on equal-length lane dividing lines. The multi-perspective correction module proposes multiple mapping methods between vehicles and lane dividing lines and a correction speed measurement method based on the average speed probability density function to correct the two types of deviations generated by vehicles during dynamic driving. Experimental results on public data sets and measured data sets show that the vehicle speed extraction accuracy of the proposed method is better than other speed measurement methods, and it has a certain degree of universality.

The Framework of Modeling Buddhist Head in 3D with Semantic Information

Abstract. Buddhist statues hold profound spiritual and cultural significance, yet their comprehensive analysis and identification remain largely reliant on manual annotation and expert scrutiny. Despite advancements in 3D modeling, the nuanced representation of these statues, essential for automated analysis, remains elusive. Integrating semantic information such as symbolic and cultural elements poses a further challenge. There currently exists no standardized method for modeling Buddhist heads in 3D with semantic information. This paper proposes a framework for the component-based analysis of Buddhist statues, particularly focusing on their heads, to bridge this gap. Drawing from diverse disciplines including archaeology, art history, and conservation, the framework leverages advancements in 3D reconstruction to provide a detailed depiction of intricate features. Building upon existing research outcomes, the framework facilitates the seamless integration of semantic information, capturing not only the physical attributes but also the cultural, historical, and spiritual dimensions of these sculptures. This approach aids in overcoming challenges in automated analysis and virtual restoration, providing a dynamic tool for exploration, preservation, and interpretation of cultural heritage. By enabling interdisciplinary synergy, the framework emerges as a dynamic tool for exploring, preserving, and interpreting the profound heritage of Buddhist art with unprecedented depth and clarity.

ExoJ - a Fiji/ImageJ2 plugin for automated spatiotemporal detection and analysis of exocytosis.

Exocytosis is a dynamic physiological process that enables the release of biomolecules to the surrounding environment via the fusion of membrane compartments to the plasma membrane. Understanding its mechanisms is crucial, as defects can compromise essential biological functions. The development of pH-sensitive optical reporters alongside fluorescence microscopy enables the assessment of individual vesicle exocytosis events at the cellular level. Manual annotation represents, however, a time-consuming task that is prone to selection biases and human operational errors. Here, we introduce ExoJ, an automated plugin based on Fiji/ImageJ2 software. ExoJ identifies user-defined genuine populations of exocytosis events, recording quantitative features including intensity, apparent size and duration. We designed ExoJ to be fully user-configurable, making it suitable for studying distinct forms of vesicle exocytosis regardless of the imaging quality. Our plugin demonstrates its capabilities by showcasing distinct exocytic dynamics among tetraspanins and vesicular SNARE protein reporters. Assessment of performance on synthetic data shows that ExoJ is a robust tool that is capable of correctly identifying exocytosis events independently of signal-to-noise ratio conditions. We propose ExoJ as a standard solution for future comparative and quantitative studies of exocytosis.

An Open Source Hydroacoustic Benchmarking Framework for Geophonic Signal Detection

Passive hydroacoustic studies have underscored the efficiency and relevance of deploying autonomous hydrophones for the surveillance of underwater geophony. In particular, monitoring networks have been deployed for detecting SOFAR-propagating hydroacoustic waves generated by seismic events and locating their sources. The technique has been extended to study other hydroacoustic signals, such as P-waves from teleseismic events or impulsive waves generated by sea water-lava interactions. A significant challenge in this endeavor lies in the time required for the manual detection and annotation of these signals in long-term records. To address this issue, we tested the feasibility of implementing automated algorithms based on machine learning to detect and identify these various signals, and obtained satisfying classification and time picking accuracies. We incorporated those models in a benchmarking framework, proposing a training dataset, two evaluation datasets, two tasks to solve and the evaluations of the mentionned models on them. The goal of this framework is to foster the development of new models in the community, as it gives a clear way to evaluate them.

Sorghum segmentation and leaf counting using in silico trained deep neural model

AbstractThis paper introduces a novel deep neural model for segmenting and tracking the number of leaves in sorghum plants in phenotyping facilities. Our algorithm inputs a sequence of images of a sorghum plant and outputs the segmented images and the number of leaves. The key novelty of our approach is in training the deep neural model. Manual annotations are tedious, and we have developed a procedural three‐dimensional (3D) sorghum model that provides detailed geometry and texture to generate photorealistic 3D models. The overall shape of the sorghum leaf geometry is determined by its skeleton, and it is detailed by a procedural model that varies its curvature, width, length, and overall shape. The color is determined by using a Monte Carlo path tracer. We mimic the illumination of the phenotyping facility and use reflectance and transmittance on sorghum surfaces to determine the color of the leaves. The 3D procedural model allows us to generate photorealistic and segmented images that we use to train a deep neural model. Our segmentation provides a mean intersection over union score of 0.51, resulting in leaf counting accuracy within the 90% confidence interval for the human counts.

AI-based automated breast cancer segmentation in ultrasound imaging based on Attention Gated Multi ResU-Net

Breast cancer is a leading cause of death among women worldwide, making early detection and diagnosis critical for effective treatment and improved patient outcomes. Ultrasound imaging is a common diagnostic tool for breast cancer, but interpreting ultrasound images can be challenging due to the complexity of breast tissue and the variability of image quality. This study proposed an Attention Gated Multi ResU-Net model for medical image segmentation tasks, that has shown promising results for breast cancer ultrasound image segmentation. The model’s multi-scale feature extraction and attention-gating mechanism enable it to accurately identify and segment areas of abnormality in the breast tissue, such as masses, cysts, and calcifications. The model’s quantitative test showed an adequate degree of agreement with expert manual annotations, demonstrating its potential for improving early identification and diagnosis of breast cancer. The model’s multi-scale feature extraction and attention-gating mechanism enable it to accurately identify and segment areas of abnormality in the breast tissue, such as masses, cysts, and calcifications, achieving a Dice coefficient of 0.93, sensitivity of 93%, and specificity of 99%. These results underscore the model’s high precision and reliability in medical image analysis.

DR-GPT: A large language model for medical report analysis of diabetic retinopathy patients.

Diabetic retinopathy (DR) is a sight-threatening condition caused by diabetes. Screening programmes for DR include eye examinations, where the patient's fundi are photographed, and the findings, including DR severity, are recorded in the medical report. However, statistical analyses based on DR severity require structured labels that calls for laborious manual annotation process if the report format is unstructured. In this work, we propose a large language model DR-GPT for classification of the DR severity from unstructured medical reports. On a clinical set of medical reports, DR-GPT reaches 0.975 quadratic weighted Cohen's kappa using truncated Early Treatment Diabetic Retinopathy Study scale. When DR-GPT annotations for unlabeled data are paired with corresponding fundus images, the additional data improves image classifier performance with statistical significance. Our analysis shows that large language models can be applied for unstructured medical report databases to classify diabetic retinopathy with a variety of applications.

Drug–target prediction through self supervised learning with dual task ensemble approach

Drug–Target interaction (DTI) prediction, a transformative approach in pharmaceutical research, seeks novel therapeutic applications for computational method based virtual screening, existing drugs to address untreated diseases and discovery of existing drugs side effects. The proposed model predict DTI through Heterogeneous biological network by combining drug, genes and disease related knowledge. For the purpose of embedding extraction Self-supervised learning (SSL) has been used which, trains models through pretext tasks, eliminating the need for manual annotations. The pretext tasks are related to either structural based information or similarity based information. To mitigate GNN vulnerability to non-robustness, ensemble learning can be incorporated into GNNs, harnessing multiple models to enhance robustness. This paper introduces a Graph neural network based architecture consisting of task based module and ensemble module for link prediction of DTI. The ensemble module of dual task combinations, both in cold start and warm start scenarios achieve very good performance as it provide 0.960 in cold start and 0.970 in warm start mean AUCROC score with less deviation.

An anthropomorphic diagnosis system of pulmonary nodules using weak annotation-based deep learning

The accurate categorization of lung nodules in CT scans is an essential aspect in the prompt detection and diagnosis of lung cancer. The categorization of grade and texture for nodules is particularly significant since it can aid radiologists and clinicians to make better-informed decisions concerning the management of nodules. However, currently existing nodule classification techniques have a singular function of nodule classification and rely on an extensive amount of high-quality annotation data, which does not meet the requirements of clinical practice. To address this issue, we develop an anthropomorphic diagnosis system of pulmonary nodules (PN) based on deep learning (DL) that is trained by weak annotation data and has comparable performance to full-annotation based diagnosis systems. The proposed system uses DL models to classify PNs (benign vs. malignant) with weak annotations, which eliminates the need for time-consuming and labor-intensive manual annotations of PNs. Moreover, the PN classification networks, augmented with handcrafted shape features acquired through the ball-scale transform technique, demonstrate capability to differentiate PNs with diverse labels, including pure ground-glass opacities, part-solid nodules, and solid nodules. Through 5-fold cross-validation on two datasets, the system achieved the following results: (1) an Area Under Curve (AUC) of 0.938 for PN localization and an AUC of 0.912 for PN differential diagnosis on the LIDC-IDRI dataset of 814 testing cases, (2) an AUC of 0.943 for PN localization and an AUC of 0.815 for PN differential diagnosis on the in-house dataset of 822 testing cases. In summary, our system demonstrates efficient localization and differential diagnosis of PNs in a resource limited environment, and thus could be translated into clinical use in the future.

Phenotyping of Panicle Number and Shape in Rice Breeding Materials Based on Unmanned Aerial Vehicle Imagery.

The number of panicles per unit area (PNpA) is one of the key factors contributing to the grain yield of rice crops. Accurate PNpA quantification is vital for breeding high-yield rice cultivars. Previous studies were based on proximal sensing with fixed observation platforms or unmanned aerial vehicles (UAVs). The near-canopy images produced in these studies suffer from inefficiency and complex image processing pipelines that require manual image cropping and annotation. This study aims to develop an automated, high-throughput UAV imagery-based approach for field plot segmentation and panicle number quantification, along with a novel classification method for different panicle types, enhancing PNpA quantification at the plot level. RGB images of the rice canopy were efficiently captured at an altitude of 15 m, followed by image stitching and plot boundary recognition via a mask region-based convolutional neural network (Mask R-CNN). The images were then segmented into plot-scale subgraphs, which were categorized into 3 growth stages. The panicle vision transformer (Panicle-ViT), which integrates a multipath vision transformer and replaces the Mask R-CNN backbone, accurately detects panicles. Additionally, the Res2Net50 architecture classified panicle types with 4 angles of 0°, 15°, 45°, and 90°. The results confirm that the performance of Plot-Seg is comparable to that of manual segmentation. Panicle-ViT outperforms the traditional Mask R-CNN across all the datasets, with the average precision at 50% intersection over union (AP50) improved by 3.5% to 20.5%. The PNpA quantification for the full dataset achieved superior performance, with a coefficient of determination (R 2) of 0.73 and a root mean square error (RMSE) of 28.3, and the overall panicle classification accuracy reached 94.8%. The proposed approach enhances operational efficiency and automates the process from plot cropping to PNpA prediction, which is promising for accelerating the selection of desired traits in rice breeding.