Annotation Categories Research Articles

Joint testing of rare variant burden scores using non-negative least squares

Gene-based burden tests are a popular and powerful approach for analysis of exome-wide association studies. These approaches combine sets of variants within a gene into a single burden score that is then tested for association. Typically, a range of burden scores are calculated and tested across a range of annotation classes and frequency bins. Correlation between these tests can complicate the multiple testing correction and hamper interpretation of the results. We introduce a method called the sparse burden association test (SBAT) that tests the joint set of burden scores under the assumption that causal burden scores act in the same effect direction. The method simultaneously assesses the significance of the model fit and selects the set of burden scores that best explain the association at the same time. Using simulated data, we show that the method is well calibrated and highlight scenarios where the test outperforms existing gene-based tests. We apply the method to 73 quantitative traits from the UK Biobank, showing that SBAT is a valuable additional gene-based test when combined with other existing approaches. This test is implemented in the REGENIE software.

A New Joint Training Method for Facial Expression Recognition with Inconsistently Annotated and Imbalanced Data

Facial expression recognition (FER) plays a crucial role in various applications, including human–computer interaction and affective computing. However, the joint training of an FER network with multiple datasets is a promising strategy to enhance its performance. Nevertheless, widespread annotation inconsistencies and class imbalances among FER datasets pose significant challenges to this approach. This paper proposes a new multi-dataset joint training method, Sample Selection and Paired Augmentation Joint Training (SSPA-JT), to address these challenges. SSPA-JT models annotation inconsistency as a label noise problem and selects clean samples from auxiliary datasets to expand the overall dataset size while maintaining consistent annotation standards. Additionally, a dynamic matching algorithm is developed to pair clean samples of the tail class with noisy samples, which enriches the tail classes with diverse background information. Experimental results demonstrate that SSPA-JT achieved superior or comparable performance compared with the existing methods by addressing both annotation inconsistencies and class imbalance during multi-dataset joint training. It achieved state-of-the-art performance on RAF-DB and CAER-S datasets with accuracies of 92.44% and 98.22%, respectively, reflecting improvements of 0.2% and 3.65% over existing methods.

One point is all you need for weakly supervised object detection

Object detection with weak annotations has attracted much attention recently. Weakly supervised object detection(WSOD) methods which only use image-level labels to train a detector encounter some severe problems that it cannot cover the whole object and the region proposal methods waste a large amount of time. Meanwhile, point supervised object detection(PSOD) leverages point annotations that remarkably improves the performance. However, point annotation is still complex and increase the costs of annotation. To overcome these issues, we propose a novel method which only requires one point per category for a training image. Compared to point annotation, our method significantly reduces the annotation cost as the number of point annotations is largely reduced. We design a framework to train a detector with one point per category annotation. Firstly, a pseudo box generation module is introduced to generate the corresponding pseudo boxes of the annotated points. Then, inspired by the observation that the features of objects with the same class in an image are very similar, a dense instances mining module is proposed to make use of the similarity between the features of objects with the same class to discover unlabeled instances and generate pseudo category heatmaps. Finally, the pseudo boxes and pseudo category heatmaps are leveraged to train a detector. Experiments conducted on popular open-source datasets verify the effectiveness of our annotation method and framework. Our proposed method outperforms previous WSOD methods and achieves comparable performance with some PSOD methods in a more efficient way.

TCR3d 2.0: expanding the T cell receptor structure database with new structures, toolsand interactions.

Recognition of antigens by T cell receptors (TCRs) is a key component of adaptive immunity. Understanding the structures of these TCR interactions provides major insights into immune protection and diseases, and enables design of therapeutics, vaccinesand predictive modeling algorithms. Previously, we released TCR3d, a database and resource for structures of TCRs and their recognition. Due to the growth of available structures and categories of complexes, the content of TCR3d has expanded substantially in the past 5 years. This expansion includes new tables dedicated to TCR mimic antibody complex structures, TCR-CD3 complexesand annotated Class I and II peptide-MHC complexes. Additionally, tools are available for users to calculate docking geometries for input TCR and TCR mimic complex structures. The core tables of TCR-peptide-MHC complexes have grown by 50%, and include binding affinity data for experimentally determined structures. These major content and feature updates enhance TCR3d as a resource for immunology, therapeuticsand structural biology research, and enable advanced approaches for predictive TCR modeling and design. TCR3d is available at: https://tcr3d.ibbr.umd.edu.

Fast Unsupervised Cross-modal Hashing With Robust Factorization and Dual Projection

Unsupervised hashing has attracted extensive attention in effectively and efficiently tackling large-scale cross-modal retrieval task. Existing methods typically try to mine the latent common subspace across multimodal data without any category annotation. Despite the exciting progress, there are still three challenges that need to be further addressed: 1) efficiently improving the robustness during latent common subspace learning; 2) harmoniously embedding the intra-modal inherence and inter-modal relevance of multimodal data into Hamming space; and 3) effectively reducing the training time complexity and making the model scalable for large-scale datasets. To well address the above challenges, this study proposes a method named Fast Unsupervised Cross-modal Hashing (FUCH). Specifically, FUCH proposes a semantic-aware collective matrix factorization to learn robust representation via exploiting latent category-specific attributes, and introduces Cauchy loss to measure the factorization process. Accordingly, the above process can effectively embed potential discriminative information into common space, while making the model insensitive for outliers. Moreover, FUCH designs a dual projection learning scheme, which not only learns modality-unique hash functions to excavate individual properties, but also learns modality-mutual hash functions to multimodal correlational properties. Experimental results on three benchmark datasets verify the effectiveness of FUCH under various scenarios.

Data-driven Bayesian Gaussian mixture optimized anchor box model for accurate and efficient detection of green citrus

The demand for harvesting green citrus has increased due to the rapid growth of the green citrus industry. However, real-time detection of green citrus is challenging because of its high visual similarity to the background. To address this issue, a new model called EDGC-YOLO (Efficient Detection of Green Citrus based on the YOLO) has been developed, focusing on detecting green citrus effectively and accurately. Specifically, the proposed model uses a data-driven approach that employs the highest confidence level from the aspect ratio of unobstructed citrus fruits to infer anchor boxes of obscured citrus. This method ensures a high degree of coincidence with bounding boxes based on empirical annotation. Furthermore, analyzing the distribution range of various citrus annotation categories within the dataset and the incorporating the highest confidence level of the normal distribution regarding aspect ratios, the initial cluster centers of the BGMM (Bayesian Gaussian Mixture Model) are dynamically adjusted. This strategy enables the anchor boxes to be inferred from the best suited for detecting green citrus. These improved anchor boxes provide a more accurate and effective solution for centroid positioning of obscured green citrus, significantly enhancing the entire network model’s detection accuracy and lightweight efficiency. To further improve detection accuracy and reduce model size, this study integrated the Refined-EfficientNetV2 network as the backbone, enhanced with a Convolutional Block Attention Module (CBAM) for better feature extraction. This integration allows the model to effectively capture relevant channel information and spatial features, increasing the feature extraction capability for green citrus images. Experimental results demonstrate the model’s superior performance: parameters reduced to 4.52 million, computational demand to 7.8 GFLOPs (64.4 % and 49.4 % of the original model, respectively), and model size decreased to 9.4 MB (65.3 % of the original). Additionally, the optimized model improved Precision (P), Recall (R), and mean Average Precision (mAP) by 0.5 %, 1.6 %, and 3.0 %, respectively, compared to the original model. The proposed model achieves higher detection accuracy with a smaller model size, providing theoretical support for the harvesting decision-making of green citrus harvesting robots.

A new dataset for video-based cow behavior recognition

A new video based multi behavior dataset for cows, CBVD-5, is introduced in this paper. The dataset includes five cow behaviors: standing, lying down, foraging,rumination and drinking. The dataset comprises 107 cows from the entire barn, maintaining an 80% stocking density. Monitoring occurred over 96 h for these 20-month-old cows, considering varying light conditions and nighttime data to ensure standardization and inclusivity.The dataset consists of ranch monitoring footage collected by seven cameras, including 687 video segment samples and 206,100 image samples, covering five daily behaviors of cows. The data collection process entailed the deployment of cameras, hard drives, software, and servers for storage. Data annotation was conducted using the VIA web tool, leveraging the video expertise of pertinent professionals. The annotation coordinates and category labels of each individual cow in the image, as well as the generated configuration file, are also saved in the dataset. With this dataset,we propose a slowfast cow multi behavior recognition model based on video sequences as the baseline evaluation model. The experimental results show that the model can effectively learn corresponding category labels from the behavior type data of the dataset, with an error rate of 21.28% on the test set. In addition to cow behavior recognition, the dataset can also be used for cow target detection, and so on.The CBVD-5 dataset significantly influences dairy cow behavior recognition, advancing research, enriching data resources, standardizing datasets, enhancing dairy cow health and welfare monitoring, and fostering agricultural intelligence development. Additionally, it serves educational and training needs, supporting research and practical applications in related fields. The dataset will be made freely available to researchers world-wide.

Category-agnostic semantic edge detection by measuring neural representation randomness

Edge detection plays a fundamental role in computer vision tasks and gains wide applications. In particular, semantic edge detection recently draws more attention due to the high demand for a fine-grained understanding of visual scenes. However, detecting high-level semantic edges hidden in visual scenes is quite challenging. Existing semantic edge detection methods focus on category-aware semantic edges and require elaborate category annotations. Instead, we first propose the category-agnostic semantic edge detection task without additional semantic category annotations. To achieve this goal, we propose to utilize only edge position annotations and leverage the information randomness of semantic edges. Specifically, we align semantic edge positions to the ground truth by maximizing randomness on edge regions and minimizing randomness on non-edge regions in the training process. In the inference process, we first obtain neural representations by the trained network, and then generate semantic edges by measuring neural randomness. We evaluate our method by comparisons with alternative methods on two well-known datasets: Cityscapes (Cordts et al., 2016) and SBD (Hariharan et al., 2014). The results demonstrate our superiority over the alternatives, which is more significant under weak annotations. We also provide comprehensive mechanism studies to verify the generalizability, rationality, and validity of our working mechanism.

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

Weakly supervised object localization (WSOL), which trains object localization models using solely image category annotations, remains a challenging problem. Existing approaches based on convolutional neural networks (CNNs) tend to miss full object extent while activating discriminative object parts. Based on our analysis, this is caused by CNN's intrinsic characteristics, which experiences difficulty to capture object semantics at long distances. In this article, we introduce the vision transformer to WSOL, with the aim to capture long-range semantic dependency of features by leveraging transformer's cascaded self-attention mechanism. We propose the token semantic coupled attention map (TS-CAM) method, which first decomposes class-aware semantics and then couples the semantics with attention maps for semantic-aware activation. To capture object semantics at long distances and avoid partial activation, TS-CAM performs spatial embedding by partitioning an image to a set of patch tokens. To incorporate object category information to patch tokens, TS-CAM reallocates category-related semantics to each patch token. The patch tokens are finally coupled with attention maps which are semantic-agnostic to perform semantic-aware object localization. By introducing semantic tokens to produce semantic-aware attention maps, we further explore the capability of TS-CAM for multicategory object localization. Experiments show that TS-CAM outperforms its CNN-CAM counterpart by 11.6% and 28.9% on ILSVRC and CUB-200-2011 datasets, respectively, improving the state-of-the-art with large margins. TS-CAM also demonstrates superiority for multicategory object localization on the Pascal VOC dataset. The code is available at github.com/yuanyao366/ts-cam-extension.

An Extensible Evaluation Framework Applied to Clinical Text Deidentification Natural Language Processing Tools: Multisystem and Multicorpus Study.

Clinical natural language processing (NLP) researchers need access to directly comparable evaluation results for applications such as text deidentification across a range of corpus types and the means to easily test new systems or corpora within the same framework. Current systems, reported metrics, and the personally identifiable information (PII) categories evaluated are not easily comparable. This study presents an open-source and extensible end-to-end framework for comparing clinical NLP system performance across corpora even when the annotation categories do not align. As a use case for this framework, we use 6 off-the-shelf text deidentification systems (ie, CliniDeID, deid from PhysioNet, MITRE Identity Scrubber Toolkit [MIST], NeuroNER, National Library of Medicine [NLM] Scrubber, and Philter) across 3 standard clinical text corpora for the task (2 of which are publicly available) and 1 private corpus (all in English), with annotation categories that are not directly analogous. The framework is built on shell scripts that can be extended to include new systems, corpora, and performance metrics. We present this open tool, multiple means for aligning PII categories during evaluation, and our initial timing and performance metric findings. Code for running this framework with all settings needed to run all pairs are available via Codeberg and GitHub. From this case study, we found large differences in processing speed between systems. The fastest system (ie, MIST) processed an average of 24.57 (SD 26.23) notes per second, while the slowest (ie, CliniDeID) processed an average of 1.00 notes per second. No system uniformly outperformed the others at identifying PII across corpora and categories. Instead, a rich tapestry of performance trade-offs emerged for PII categories. CliniDeID and Philter prioritize recall over precision (with an average recall 6.9 and 11.2 points higher, respectively, for partially matching spans of text matching any PII category), while the other 4 systems consistently have higher precision (with MIST's precision scoring 20.2 points higher, NLM Scrubber scoring 4.4 points higher, NeuroNER scoring 7.2 points higher, and deid scoring 17.1 points higher). The macroaverage recall across corpora for identifying names, one of the more sensitive PII categories, included deid (48.8%) and MIST (66.9%) at the low end and NeuroNER (84.1%), NLM Scrubber (88.1%), and CliniDeID (95.9%) at the high end. A variety of metrics across categories and corpora are reported with a wider variety (eg, F2-score) available via the tool. NLP systems in general and deidentification systems and corpora in our use case tend to be evaluated in stand-alone research articles that only include a limited set of comparators. We hold that a single evaluation pipeline across multiple systems and corpora allows for more nuanced comparisons. Our open pipeline should reduce barriers to evaluation and system advancement.

Two-Dimensional Thin-Layer Chromatography as an Accessible, Low-Cost Tool for Lipid-Class Profile Screening

The interest in lipid composition profiling is significantly increasing as research reveals the immense importance of lipids in medicine, plant science, food and agriculture. However, lipidomic analysis requires high-end specialty equipment. We used two-dimensional thin-layer chromatography (2D-TLC) as a readily available, low-cost tool for basic lipidomic profiling of lipid classes in algal samples in the models Chlamydomonas reinhardtii, Auxenochlorella protothecoides, and Euglena gracilis, validating lipid class identification using an LC-MS/MS analysis. Algal lipid extracts were separated on a 2D-TLC plate, and TLC analysis was followed by scraping individual TLC spots off the plate, and a subsequent liquid chromatography separation and tandem mass spectrometry (LC-MS/MS) analysis. For comparison, crude lipid extracts were also injected directly to the LC-MS/MS system. Lipid class annotation was achieved by a combination of accurate mass, retention time information, neutral loss and fragment ion analysis by MS2Analyzer, and by matching spectra to LipidBlast MS/MS library. Overall, we were able to identify 15 lipid classes, and to adequately profile the lipid classes in all three organisms. This TLC method is thus suggested as an accessible tool for lipid class profiling of algal, plant, and food lipids, alike, when a rapid and simple analysis is required, e.g., for screening purposes.

Scene-aware data augmentation for ship detection in SAR images

ABSTRACT Ship detection in synthetic aperture radar (SAR) images is widely applied in marine monitoring. In recent years, convolutional neural networks (CNNs) have made significant advances in SAR ship detection. However, the scene imbalance problem, i.e. much more offshore scenes than inshore scenes in SAR images, severely hampers the improvement of detection accuracy. In this article, we propose scene-aware data augmentation to address the problem. First, a novel concept, inshore rate, is introduced to estimate the land area percentage of various scenes in SAR images. We can divide image samples into inshore and offshore ones based on the inshore rate without manual category annotations. Second, to augment inshore samples, we synthesize dummy inshore images by cropping the ground truth bounding boxes (GT bboxes) from offshore images and pasting them onto inshore images. This offline data augmentation strategy can address the scene imbalance problem and improve detection accuracy. The experiments conducted on SAR Ship Detection Dataset (SSDD) and Large-Scale SAR Ship Detection Dataset (LS-SSDD) demonstrate the effectiveness of the proposed method in improving detection performance.

Molecular circadian rhythms are robust in marine annelids lacking rhythmic behavior.

The circadian clock controls behavior and metabolism in various organisms. However, the exact timing and strength of rhythmic phenotypes can vary significantly between individuals of the same species. This is highly relevant for rhythmically complex marine environments where organismal rhythmic diversity likely permits the occupation of different microenvironments. When investigating circadian locomotor behavior of Platynereis dumerilii, a model system for marine molecular chronobiology, we found strain-specific, high variability between individual worms. The individual patterns were maintained for several weeks. A diel head transcriptome comparison of behaviorally rhythmic versus arrhythmic wild-type worms showed that 24-h cycling of core circadian clock transcripts is identical between both behavioral phenotypes. While behaviorally arrhythmic worms showed a similar total number of cycling transcripts compared to their behaviorally rhythmic counterparts, the annotation categories of their transcripts, however, differed substantially. Consistent with their locomotor phenotype, behaviorally rhythmic worms exhibit an enrichment of cycling transcripts related to neuronal/behavioral processes. In contrast, behaviorally arrhythmic worms showed significantly increased diel cycling for metabolism- and physiology-related transcripts. The prominent role of the neuropeptide pigment-dispersing factor (PDF) in Drosophila circadian behavior prompted us to test for a possible functional involvement of Platynereis pdf. Differing from its role in Drosophila, loss of pdf impacts overall activity levels but shows only indirect effects on rhythmicity. Our results show that individuals arrhythmic in a given process can show increased rhythmicity in others. Across the Platynereis population, rhythmic phenotypes exist as a continuum, with no distinct "boundaries" between rhythmicity and arrhythmicity. We suggest that such diel rhythm breadth is an important biodiversity resource enabling the species to quickly adapt to heterogeneous or changing marine environments. In times of massive sequencing, our work also emphasizes the importance of time series and functional tests.

Performance evaluation of three versions of a convolutional neural network for object detection and segmentation using a multiclass and reduced panoramic radiograph dataset

ObjectivesTo evaluate the diagnostic performance of three versions of a deep-learning convolutional neural network in terms of object detection and segmentation using a multiclass panoramic radiograph dataset. MethodsA total of 600 orthopantomographies were randomly selected for this study and manually annotated by a single operator using an image annotation tool (COCO Annotator v.11.0.1) to establish ground truth. The annotation classes included teeth, maxilla, mandible, inferior alveolar nerve, dento- and implant-supported crowns/pontics, endodontic treatment, resin-based restorations, metallic restorations, and implants. The dataset was then divided into training, validation, and testing subsets, which were used to train versions 5, 7, and 8 of You Only Look Once (YOLO) Neural Network. Results were stored, and a posterior performance analysis was carried out by calculating the precision (P), recall (R), F1 Score, Intersection over Union (IoU), and mean average precision (mAP) at 0.5 and 0.5–0.95 thresholds. The confusion matrix and recall precision graphs were also sketched. ResultsYOLOv5s showed an improvement in object detection results with an average R = 0.634, P = 0.781, mAP0.5 = 0.631, and mAP0.5–0.95 = 0.392. YOLOv7m achieved the best object detection results with average R = 0.793, P = 0.779, mAP0.5 = 0.740, and mAP0.5–0.95 = 0,481. For object segmentation, YOLOv8m obtained the best average results (R = 0.589, P = 0.755, mAP0.5 = 0.591, and mAP0.5–0.95 = 0.272). ConclusionsYOLOv7m was better suited for object detection, while YOLOv8m demonstrated superior performance in object segmentation. The most frequent error in object detection was related to background classification. Conversely, in object segmentation, there is a tendency to misclassify True Positives across different dental treatment categories. Clinical SignificanceGeneral diagnostic and treatment decisions based on panoramic radiographs can be enhanced using new artificial intelligence-based tools. Nevertheless, the reliability of these neural networks should be subjected to training and validation to ensure their generalizability.

Leveraging Unidentified Metabolic Features for Key Pathway Discovery: Chemical Classification-driven Network Analysis in Untargeted Metabolomics.

Untargeted metabolomics using liquid chromatography-electrospray ionization-high-resolution tandem mass spectrometry (UPLC-ESI-MS/MS) provides comprehensive insights into the dynamic changes of metabolites in biological systems. However, numerous unidentified metabolic features limit its utilization. In this study, a novel approach, the Chemical Classification-driven Molecular Network (CCMN), was proposed to unveil key metabolic pathways by leveraging hidden information within unidentified metabolic features. The method was demonstrated by using the herbivore-induced metabolic response in corn silk as a case study. Untargeted metabolomics analysis using UPLC-MS/MS was performed on wild corn silk and two genetically modified lines (pre- and postinsect treatment). Global annotation initially identified 256 (ESI-) and 327 (ESI+) metabolites. MS/MS-based classifications predicted 1939 (ESI-) and 1985 (ESI+) metabolic features into the chemical classes. CCMNs were then constructed using metabolic features shared classes, which facilitated the structure- or class annotation for completely unknown metabolic features. Next, 844/713 significantly decreased and 1593/1378 increased metabolites in ESI-/ESI+ modes were defined in response to insect herbivory, respectively. Method validation on a spiked maize sample demonstrated an overall class prediction accuracy rate of 95.7%. Potential key pathways were prescreened by a hypergeometric test using both structure- and class-annotated differential metabolites. Subsequently, CCMN was used to deeply amend and uncover the pathway metabolites deeply. Finally, 8 key pathways were defined, including phenylpropanoid (C6-C3), flavonoid, octadecanoid, diterpenoid, lignan, steroid, amino acid/small peptide, and monoterpenoid. This study highlights the effectiveness of leveraging unidentified metabolic features. CCMN-based key pathway analysis reduced the bias in conventional pathway enrichment analysis. It provides valuable insights into complex biological processes.

PseAAC2Vec protein encoding for TCR protein sequence classification

The classification and prediction of T-cell receptors (TCRs) protein sequences are of significant interest in understanding the immune system and developing personalized immunotherapies. In this study, we propose a novel approach using Pseudo Amino Acid Composition (PseAAC) protein encoding for accurate TCR protein sequence classification. The PseAAC2Vec encoding method captures the physicochemical properties of amino acids and their local sequence information, enabling the representation of protein sequences as fixed-length feature vectors. By incorporating physicochemical properties such as hydrophobicity, polarity, charge, molecular weight, and solvent accessibility, PseAAC2Vec provides a comprehensive and informative characterization of TCR protein sequences. To evaluate the effectiveness of the proposed PseAAC2Vec encoding approach, we assembled a large dataset of TCR protein sequences with annotated classes. We applied the PseAAC2Vec encoding scheme to each sequence and generated feature vectors based on a specified window size. Subsequently, we employed state-of-the-art machine learning algorithms, such as support vector machines (SVM) and random forests (RF), to classify the TCR protein sequences. Experimental results on the benchmark dataset demonstrated the superior performance of the PseAAC2Vec-based approach compared to existing methods. The PseAAC2Vec encoding effectively captures the discriminative patterns in TCR protein sequences, leading to improved classification accuracy and robustness. Furthermore, the encoding scheme showed promising results across different window sizes, indicating its adaptability to varying sequence contexts.

Handling the Challenges of Small-Scale Labeled Data and Class Imbalances in Classifying the N and K Statuses of Rubber Leaves Using Hyperspectroscopy Techniques.

The nutritional status of rubber trees (Hevea brasiliensis) is inseparable from the production of natural rubber. Nitrogen (N) and potassium (K) levels in rubber leaves are 2 crucial criteria that reflect the nutritional status of the rubber tree. Advanced hyperspectral technology can evaluate N and K statuses in leaves rapidly. However, high bias and uncertain results will be generated when using a small size and imbalance dataset to train a spectral estimaion model. A typical solution of laborious long-term nutrient stress and high-intensive data collection deviates from rapid and flexible advantages of hyperspectral tech. Therefore, a less intensive and streamlined method, remining information from hyperspectral image data, was assessed. From this new perspective, a semisupervised learning (SSL) method and resampling techniques were employed for generating pseudo-labeling data and class rebalancing. Subsequently, a 5-classification spectral model of the N and K statuses of rubber leaves was established. The SSL model based on random forest classifiers and mean sampling techniques yielded optimal classification results both on imbalance/balance dataset (weighted average precision 67.8/78.6%, macro averaged precision 61.2/74.4%, and weighted recall 65.7/78.5% for the N status). All data and code could be viewed on the:Github https://github.com/WeehowTang/SSL-rebalancingtest. Ultimately, we proposed an efficient way to rapidly and accurately monitor the N and K levels in rubber leaves, especially in the scenario of small annotation and imbalance categories ratios.

An Attention-Guided Multistream Feature Fusion Network for Early Localization of Risky Traffic Agents in Driving Videos

Detecting dangerous traffic agents in videos captured by vehicle-mounted dashboard cameras (dashcams) is essential to ensure safe navigation in complex environments. Accident-related videos are just a minor portion of the driving-related big data, and the transient pre-accident process is highly dynamic and complex. Besides, risky and non-risky traffic agents can be similar in their appearance. These make risky traffic agent localization in the driving video particularly challenging. To this end, this paper proposes an attention-guided multistream feature fusion network (AM-Net) to localize dangerous traffic agents from dashcam videos ahead of potential accidents. Two Gated Recurrent Unit (GRU) networks use object bounding box and optical flow features extracted from consecutive video frames to capture spatio-temporal cues for distinguishing risky traffic agents. An attention module, coupled with the GRUs, learns to identify traffic agents that are relevant to an accident. Fusing the two streams of global and object-level features, AM-Net predicts the riskiness scores of traffic agents in the video. In supporting this study, the paper also introduces a new benchmark dataset called Risky Object Localization (ROL). The dataset contains spatial, temporal, and categorical annotations of the accident, object, and scene-level attributes. The proposed AM-Net achieves a promising performance of 85.59% AUC on the ROL dataset. Additionally, the AM-Net outperforms the current state-of-the-art for video anomaly detection by 3.5% AUC on the public DoTA dataset. A thorough ablation study further reveals AM-Net's merits by assessing the impact of its constituents.

Robust Fine-Grained Visual Recognition With Neighbor-Attention Label Correction.

Existing deep learning methods for fine-grained visual recognition often rely on large-scale, well-annotated training data. Obtaining fine-grained annotations in the wild typically requires concentration and expertise, such as fine category annotation for species recognition, instance annotation for person re-identification (re-id) and dense annotation for segmentation, which inevitably leads to label noise. This paper aims to tackle label noise in deep model training for fine-grained visual recognition. We propose a Neighbor-Attention Label Correction (NALC) model to correct labels during the training stage. NALC samples a training batch and a validation batch from the training set. It hence leverages a meta-learning framework to correct labels in the training batch based on the validation batch. To enhance the optimization efficiency, we introduce a novel nested optimization algorithm for the meta-learning framework. The proposed training procedure consistently improves label accuracy in the training batch, consequently enhancing the learned image representation. Experimental results demonstrate that our method significantly increases label accuracy from 70% to over 98% and outperforms recent approaches by up to 13.4% in mean Average Precision (mAP) on various fine-grained image retrieval (FGIR) tasks, including instance retrieval on CUB200 and person re-id on Market1501. We also demonstrate the efficacy of NALC on noisy semantic segmentation datasets generated from Cityscapes, where it achieves a significant 7.8% improvement in mIOU score. NALC also exhibits robustness to different types of noise, including simulated noise such as Asymmetric, Pair-Flip, and Pattern noise, as well as practical noisy labels generated by tracklets and clustering.

DeTAL: Open-Vocabulary Temporal Action Localization with Decoupled Networks.

Pre-trained visual-language (ViL) models have demonstrated good zero-shot capability in video understanding tasks, where they were usually adapted through fine-tuning or temporal modeling. However, in the task of open-vocabulary temporal action localization (OV-TAL), such adaption reduces the robustness of ViL models against different data distributions, leading to a misalignment between visual representations and text descriptions of unseen action categories. As a result, existing methods often strike a trade-off between action detection and classification. Aiming at this issue, this paper proposes DeTAL, a simple but effective two-stage approach for OV-TAL. DeTAL decouples action detection from action classification to avoid the compromise between them, and the state-of-the-art methods for close-set action localization can be handily adapted to OV-TAL, which significantly improves the performance. Meanwhile, DeTAL can easily tackle the scenario where action category annotations are unavailable in the training dataset. In the experiments, we propose a new cross-dataset setting to evaluate the zero-shot capability of different methods. And the results demonstrate that DeTAL outperforms the state-of-the-art methods for OV-TAL on both THUMOS14 and ActivityNet1.3. Code and data are publicly available at https://github.com/vsislab/DeTAL.