Visualization Of Datasets Research Articles

Online Active Continual Learning for Robotic Lifelong Object Recognition.

In real-world applications, robotic systems collect vast amounts of new data from ever-changing environments over time. They need to continually interact and learn new knowledge from the external world to adapt to the environment. Particularly, lifelong object recognition in an online and interactive manner is a crucial and fundamental capability for robotic systems. To meet this realistic demand, in this article, we propose an online active continual learning (OACL) framework for robotic lifelong object recognition, in the scenario of both classes and domains changing with dynamic environments. First, to reduce the labeling cost as much as possible while maximizing the performance, a new online active learning (OAL) strategy is designed by taking both the uncertainty and diversity of samples into account to protect the information volume and distribution of data. In addition, to prevent catastrophic forgetting and reduce memory costs, a novel online continual learning (OCL) algorithm is proposed based on the deep feature semantic augmentation and a new loss-based deep model and replay buffer update, which can mitigate the class imbalance between the old and new classes and alleviate confusion between two similar classes. Moreover, the mistake bound of the proposed method is analyzed in theory. OACL allows robots to select the most representative new samples to query labels and continually learn new objects and new variants of previously learned objects from a nonindependent and identically distributed (i.i.d.) data stream without catastrophic forgetting. Extensive experiments conducted on real lifelong robotic vision datasets demonstrate that our algorithm, even trained with fewer labeled samples and replay exemplars, can achieve state-of-the-art performance on OCL tasks.

FLAC: Fairness-Aware Representation Learning by Suppressing Attribute-Class Associations.

Bias in computer vision systems can perpetuate or even amplify discrimination against certain populations. Considering that bias is often introduced by biased visual datasets, many recent research efforts focus on training fair models using such data. However, most of them heavily rely on the availability of protected attribute labels in the dataset, which limits their applicability, while label-unaware approaches, i.e., approaches operating without such labels, exhibit considerably lower performance. To overcome these limitations, this work introduces FLAC, a methodology that minimizes mutual information between the features extracted by the model and a protected attribute, without the use of attribute labels. To do that, FLAC proposes a sampling strategy that highlights underrepresented samples in the dataset, and casts the problem of learning fair representations as a probability matching problem that leverages representations extracted by a bias-capturing classifier. It is theoretically shown that FLAC can indeed lead to fair representations, that are independent of the protected attributes. FLAC surpasses the current state-of-the-art on Biased-MNIST, CelebA, and UTKFace, by 29.1%, 18.1%, and 21.9%, respectively. Additionally, FLAC exhibits 2.2% increased accuracy on ImageNet-A and up to 4.2% increased accuracy on Corrupted-Cifar10. Finally, in most experiments, FLAC even outperforms the bias label-aware state-of-the-art methods.

Unveiling facial kinship: The BioKinVis dataset for facial kinship verification and genetic association studies.

Facial image-based kinship verification represents a burgeoning frontier within the realms of computer vision and biomedicine. Recent genome-wide association studies have underscored the heritability of human facial morphology, revealing its predictability based on genetic information. These revelations form a robust foundation for advancing facial image-based kinship verification. Despite strides in computer vision, there remains a discernible gap between the biomedical and computer vision domains. Notably, the absence of family photo datasets established through biological paternity testing methods poses a significant challenge. This study addresses this gap by introducing the biological kinship visualization dataset, encompassing 5773 individuals from 2412 families with biologically confirmed kinship. Our analysis delves into the distribution and influencing factors of facial similarity among parent-child pairs, probing the potential association between forensic short tandem repeat polymorphisms and facial similarity. Additionally, we have developed a machine learning model for facial image-based kinship verification, achieving an accuracy of 0.80 in the dataset. To facilitate further exploration, we have established an online tool and database, accessible at http://120.55.161.230:88/.

The ripple effect of dataset reuse: Contextualising the data lifecycle for machine learning data sets and social impact

Although there exists a rich literature on data lifecycle, a common framework for data lifecycle depicts reuse as the last stage. However, this framework fails to explain the complex lifecycle of machine learning (ML) data sets, which can have many different afterlives. Data sets for ML can be expanded to supplement previous research, and researchers can concatenate multiple data sets to develop new models. This study discusses ML dataset reuse through the lens of the data–information–knowledge–wisdom pyramid. In social science research, researchers might reuse data to analyse a new research question that is still in the context of the data domain. By contrast, research practices in ML, where researchers layer multiple data sets for training purposes, require us to ask whether the existing data lifecycle model, ending with ‘reuse’, is appropriate for explaining such an iterative and layered lifecycle. This study introduces one case of merging computer vision data set and natural language processing data set and two cases of applying ML models from outside of the ML community (hate speech detection and politeness detection) to justify a framework for a ML dataset lifecycle. Last but not least, this study proposes a ML dataset lifecycle and provides case examples to describe each stage.

Exploring fMRI RDMs: enhancing model robustness through neurobiological data

Artificial neural networks (ANNs) are sensitive to perturbations and adversarial attacks. One hypothesized solution to adversarial robustness is to align manifolds in the embedded space of neural networks with biologically grounded manifolds. Recent state-of-the-art works that emphasize learning robust neural representations, rather than optimizing for a specific target task like classification, support the idea that researchers should investigate this hypothesis. While works have shown that fine-tuning ANNs to coincide with biological vision does increase robustness to both perturbations and adversarial attacks, these works have relied on proprietary datasets—the lack of publicly available biological benchmarks makes it difficult to evaluate the efficacy of these claims. Here, we deliver a curated dataset consisting of biological representations of images taken from two commonly used computer vision datasets, ImageNet and COCO, that can be easily integrated into model training and evaluation. Specifically, we take a large functional magnetic resonance imaging (fMRI) dataset (BOLD5000), preprocess it into representational dissimilarity matrices (RDMs), and establish an infrastructure that anyone can use to train models with biologically grounded representations. Using this infrastructure, we investigate the representations of several popular neural networks and find that as networks have been optimized for tasks, their correspondence with biological fidelity has decreased. Additionally, we use a previously unexplored graph-based technique, Fiedler partitioning, to showcase the viability of the biological data, and the potential to extend these analyses by extending RDMs into Laplacian matrices. Overall, our findings demonstrate the potential of utilizing our new biological benchmark to effectively enhance the robustness of models.

Improving rare relation inferring for scene graph generation using bipartite graph network

Scene graph generation (SGG) task is plagued by insufficient and long-tailed training samples. Thus external text knowledge has been introduced into SGG task, which augments the training dataset. However, rare relations or low-frequency events, such as <dog, standing on, surfboard>, are still hard to be discovered since they also rarely appear in external texts. Addressed these issues, this paper proposes a model-agnostic Bipartite Graph Network with Dual-Group Message Passing (DG-BGN). It extends the relation space of each object pair in training dataset based on the similarity of different object pairs, which cooperates external text knowledge and internal visual information. Specifically, the main framework of DG-BGN consists of three parts, that is, bipartite graph construction (BGC), dual-group message propagation (DG-MP), and pseudo label generation (PLG). For BGC, alignment between external texts and visual dataset is performed to mine information from multimodal resources. Taking object pair as the index of the grouping, dual bipartite graph containing intra/inter-group graph is established to represent predicate label probability distribution of object pair instances. Among them, inter-group bipartite graph is built based on the similarity between object pair instance. For DG-MP, efficient message passing is running on both kinds of bipartite graph through Graph Convolution Neural Networks to refine the graphs, in order to let each object pair instance learn from its similar instances. Finally in PLG, pseudo labels, which are used to train SGG models in fully supervised way, are obtained by aggregating these refined graphs. Systematic experiments on Visual Genome dataset and Conceptual Captions show that our method performs better on discovering rare relations.

Deep Learning-Based Visual Navigation Algorithms for Mobile Robots: A Comprehensive Study

This research addresses the challenges faced by mobile robots in efficiently navigating complex environments. A novel approach is proposed, leveraging deep learning techniques, and introducing the Neo model. The method combines Split Attention with the Res-NeSt50 network to enhance the recognition accuracy of key features in the observed images. Furthermore, improvements have been made in the loss calculation method to improve navigation accuracy across different scenarios. Evaluations conducted on AI2-THOR and active vision datasets demonstrate that the improved model achieves higher average navigation accuracy (92.3%) in scene 4 compared to other methods. The success rate of navigation reached 36.8%, accompanied by a 50% reduction in ballistic length. Additionally, compared to HAUSR and LSTM-Nav, this technology significantly reduced collision rates to 0.01 and reduced time consumption by over 8 seconds. The research methodology addresses navigation model accuracy, speed, and generalization issues, thus making significant advancements for intelligent autonomous robots.

Simultaneous Object Detection and Distance Estimation for Indoor Autonomous Vehicles

Object detection is an essential and impactful technology in various fields due to its ability to automatically locate and identify objects in images or videos. In addition, object-distance estimation is a fundamental problem in 3D vision and scene perception. In this paper, we propose a simultaneous object-detection and distance-estimation algorithm based on YOLOv5 for obstacle detection in indoor autonomous vehicles. This method estimates the distances to the desired obstacles using a single monocular camera that does not require calibration. On the one hand, we train the algorithm with the KITTI dataset, which is an autonomous driving vision dataset that provides labels for object detection and distance prediction. On the other hand, we collect and label 100 images from a custom environment. Then, we apply data augmentation and transfer learning to generate a fast, accurate, and cost-effective model for the custom environment. The results show a performance of mAP0.5:0.95 of more than 75% for object detection and 0.71 m of mean absolute error in distance prediction, which are easily scalable with the labeling of a larger amount of data. Finally, we compare our method with other similar state-of-the-art approaches.

Camera model identification based on dual-path enhanced ConvNeXt network and patches selected by uniform local binary pattern

With the rapid advancement of multimedia technologies, there is a growing demand for reliable methods to verify image integrity. Camera model identification, a passive approach aiming to determine the specific capturing device model, has garnered considerable attention in the field of source camera forensics. In this paper, we first propose a novel patch selection method that enhances the diversity of training data by utilizing the uniform local binary pattern operator to reveal spatial textual information. Secondly, we introduce a complex dual-path enhanced ConvNeXt network for camera model identification, effectively leveraging the multi-frequency information present in the image. Notably, our network demonstrates the ability to learn camera model-related features without relying on a residual prediction module. Finally, extensive experimental results on both Dresden and Vision datasets shown that the proposed network outperforms several state-of-the-art methods in both teams of identification accuracy and computational efficiency.

Language Teacher Educators’ Identity Construction through Teaching and Supporting Action Research: A Trioethnographic Study

Action research is a form of inquiry that allows language educators to understand and improve their own practice. The literature suggests that language teachers can employ it as a self-led and self-directed source of professional development and enhancement of their professional identity and agency. However, little is known about its impact on the development of identity of language teacher educators who teach and support action research. Against this backdrop, this article aims to explore the experience of three language teacher educators regarding the influence of action research on the construction of their professional identity. Drawing on verbal and visual data sets, findings show that when teacher educators teach and mentor action research in language teacher education programmes, they can become guiding knowers, reflective practitioners and agents of change. The roles that teacher educators enact may lead to a reflective process that enhances their professional development. These identities connected to agentic growth demonstrate the relevance of including action research in language teacher education programmes as it can help teacher educators nurture their professional identity, particularly concerning their professional knowledge. This study suggests that teaching and supporting action research can become a meaningful entry point for language teacher educators to reflect on their own practice and to carry out action research themselves in language teacher education programmes.

Multimodal Arabic emotion recognition using deep learning

Emotion Recognition has been an active area for decades due to the complexity of the problem and its significance in human–computer interaction. Various methods have been employed to tackle this problem, leveraging different inputs such as speech, 2D and 3D images, audio signals, and text, all of which can convey emotional information. Recently, researchers have started combining multiple modalities to enhance the accuracy of emotion classification, recognizing that different emotions may be better expressed through different input types. This paper introduces a novel Arabic audio-visual natural-emotion dataset, investigates two existing multimodal classifiers, and proposes a new classifier trained on our Arabic dataset. Our evaluation encompasses different aspects, including variations in visual dataset sizes, joint and disjoint training, single and multimodal networks, as well as consecutive and overlapping segmentation. Through 5-fold cross-validation, our proposed classifier achieved exceptional results with an average F1-score of 0.912 and an accuracy of 0.913 for natural emotion recognition.

Domain Neural Adaptation.

Domain adaptation is concerned with the problem of generalizing a classification model to a target domain with little or no labeled data, by leveraging the abundant labeled data from a related source domain. The source and target domains possess different joint probability distributions, making it challenging for model generalization. In this article, we introduce domain neural adaptation (DNA): an approach that exploits nonlinear deep neural network to 1) match the source and target joint distributions in the network activation space and 2) learn the classifier in an end-to-end manner. Specifically, we employ the relative chi-square divergence to compare the two joint distributions, and show that the divergence can be estimated via seeking the maximal value of a quadratic functional over the reproducing kernel hilbert space. The analytic solution to this maximization problem enables us to explicitly express the divergence estimate as a function of the neural network mapping. We optimize the network parameters to minimize the estimated joint distribution divergence and the classification loss, yielding a classification model that generalizes well to the target domain. Empirical results on several visual datasets demonstrate that our solution is statistically better than its competitors.

Unleashing the Power of Deep Attention Networks: A Comprehensive Approach for Enhanced Artificial Intelligence

Deep learning has revolutionized the field of artificial intelligence by achieving state-of-the-art performance on a variety of complex tasks. Attention mechanisms have emerged as a powerful tool to enhance the capabilities of deep neural networks by enabling them to selectively focus on relevant information. In this article, we propose a novel artificial intelligence algorithm called Deep Attention Networks (DANs), which associate multiple attention mechanisms to improve performance on interesting tasks. We evaluate DANs on benchmark datasets in natural language processing, computer vision, and speech recognition and demonstrate superior results compared to existing state-of-the-art approaches. Our approach opens up new possibilities for advancing the field of artificial intelligence and holds promise for various real-world applications. Overall, our results demonstrate the effectiveness and potential of DANs for various AI applications, and highlight the power of combining deep neural networks with attention mechanisms.

TRAMON: An automated traffic monitoring system for high density, mixed and lane-free traffic

This paper introduces a new visual dataset and framework to facilitate computer-vision-based traffic monitoring in high density, mixed and lane-free traffic (TRAMON). While there are advanced deep learning algorithms that can detect and track vehicles from traffic videos, none of the existing systems provides accurate traffic monitoring in mixed traffic. The mixed traffic flows in developing countries often includes the types of vehicles that are not widely known by the existing visual datasets. The computer vision algorithms also face difficulties in detecting and tracking a high density of vehicles that are not following lanes. This paper proposes a large-scale visual dataset of >282,000 labelled images of traffic vehicles, as well as a comprehensive framework and strategy to train common deep-learning-based computer vision algorithms to detect and track vehicles in high density, heterogeneous and lane-free traffic. A systematic evaluation of results shows that TRAMON, the proposed visual dataset and framework, performs well and better than the common visual dataset at all traffic densities.

EDKSANet: An Efficient Dual-Kernel Split Attention Neural Network for the Classification of Tibetan Medicinal Materials

Tibetan medicine has received wide acclaim for its unique diagnosis and treatment methods. The identification of Tibetan medicinal materials, which are a vital component of Tibetan medicine, is a key research area in this field. However, traditional deep learning-based visual neural networks face significant challenges in efficiently and accurately identifying Tibetan medicinal materials due to their large number, complex morphology, and the scarcity of public visual datasets. To address this issue, we constructed a computer vision dataset with 300 Tibetan medicinal materials and proposed a lightweight and efficient cross-dimensional attention mechanism, the Dual-Kernel Split Attention (DKSA) module, which can adaptively share parameters of the kernel in both spatial and channel dimensions. Based on the DKSA module, we achieve efficient unification of convolution and self-attention under the CNN architecture and develop a new lightweight backbone architecture, EDKSANet, to provide enhanced performance for various computer vision tasks. As compared to RedNet, the top-1 accuracy is improved by 1.2% on an ImageNet dataset, and a larger margin of +1.5 box AP for object detection and an improvement of +1.3 mask AP for instance segmentation on MS-COCO dataset are obtained. Moreover, EDKSANet achieved excellent classification performance on the Tibetan medicinal materials dataset, with an accuracy of up to 96.85%.

Evasion Generative Adversarial Network for Low Data Regimes

A myriad of recent literary works has leveraged generative adversarial networks (GANs) to generate unseen evasion samples. The purpose is to annex the generated data with the original train set for adversarial training to improve the detection performance of machine learning (ML) classifiers. The quality of generated adversarial samples relies on the adequacy of training data samples. However, in low data regimes like medical diagnostic imaging and cybersecurity, the anomaly samples are scarce in number. This paper proposes a novel GAN design called Evasion Generative Adversarial Network (EVAGAN) that is more suitable for low data regime problems that use oversampling for detection improvement of ML classifiers. EVAGAN not only can generate evasion samples, but its discriminator can act as an evasion-aware classifier. We have considered Auxiliary Classifier GAN (ACGAN) as a benchmark to evaluate the performance of EVAGAN on cybersecurity (ISCX-2014, CIC-2017 and CIC2018) botnet and computer vision (MNIST) datasets. We demonstrate that EVAGAN outperforms ACGAN for unbalanced datasets with respect to detection performance, training stability and time complexity. EVAGAN&#x0027;s generator quickly learns to generate the low sample class and hardens its discriminator simultaneously. In contrast to ML classifiers that require security hardening after being adversarially trained by GAN-generated data, EVAGAN renders it needless. The experimental analysis proves that EVAGAN is an efficient evasion hardened model for low data regimes for the selected cybersecurity and computer vision datasets. Code will be available at HTTPS://www.github.com/rhr407/EVAGAN. <i>ImpStatement</i> Artificial Intelligence (AI) applications can help improve the quality of human life. The use of AI is not only limited to medical anomaly detection and drug discovery but can be leveraged in computer networks to keep people safe from malicious activities on the Internet. However, the AI-based models can be biased towards the majority class of data on which they are trained due to data imbalance. Anomaly data samples are always scarce as compared to normal data samples. So this is an open research problem to solve. Our work is an effort to improve the AI-based methods in detection performance, stability and time complexity. Using the proposed technique, we can train our AI model using fewer anomaly samples, improving the cost-efficiency compared to state-of-the-art in anomaly detection.

A Multi-Modal Hypergraph Neural Network via Parametric Filtering and Feature Sampling

In the real world, relationships between objects are often complex, involving multiple variables and modes. Hypergraph neural networks possess the capability to capture and represent such intricate relationships by deriving and inheriting their graph-based counterparts. Nevertheless, both graph and hypergraph neural networks suffer from the problem of over-smoothing when multiple graph convolution layers are stacked. To address this issue, this paper introduces the Multi-modal Hypergraph Neural Network with Parametric Filtering and Feature Sampling (MHNet) to encode complex hypergraph features and mitigate over-smoothing. The proposed approach uses hypergraph structures to model high-order and multi-modal data correlations, a polynomial hypergraph filter to dynamically extract multi-scale node features through parametric polynomial fitting, and a feature sampling strategy to learn from sparse and labeled samples while avoiding overfitting. Experimental results on four hypergraph datasets and two multi-modal visual datasets demonstrate that the proposed MHNet outperforms state-of-the-art algorithms.

Competitive learning to generate sparse representations for associative memory

One of the most well established brain principles, Hebbian learning, has led to the theoretical concept of neural assemblies. Based on it, many interesting brain theories have spawned. Palm’s work implements this concept through multiple binary Willshaw associative memories, in a model that not only has a wide cognitive explanatory power but also makes neuroscientific predictions. Yet, Willshaw’s associative memory can only achieve top capacity when the stored vectors are extremely sparse (number of active bits can grow logarithmically with the vector’s length). This strict requirement makes it difficult to apply any model that uses this associative memory, like Palm’s, to real data. Hence the fact that most works apply the memory to optimal randomly generated codes that do not represent any information. This issue creates the need for encoders that can take real data, and produce sparse representations - a problem which is also raised following Barlow’s efficient coding principle. In this work, we propose a biologically-constrained network that encodes images into codes that are suitable for Willshaw’s associative memory. The network is organized into groups of neurons that specialize on local receptive fields, and learn through a competitive scheme. After conducting auto- and hetero-association experiments on two visual data sets, we can conclude that our network not only beats sparse coding baselines, but also that it comes close to the performance achieved using optimal random codes.

MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC Deployment

Graph Neural Networks (GNNs) are becoming increasingly popular for vision-based applications due to their intrinsic capacity in modeling structural and contextual relations between various parts of an image frame. On another front, the rising popularity of deep vision-based applications at the edge has been facilitated by the recent advancements in heterogeneous multi-processor Systems on Chips (MPSoCs) that enable inference under real-time, stringent execution requirements. By extension, GNNs employed for vision-based applications must adhere to the same execution requirements. Yet contrary to typical deep neural networks, the irregular flow of graph learning operations poses a challenge to running GNNs on such heterogeneous MPSoC platforms. In this paper, we propose a novel unified design-mapping approach for efficient processing of vision GNN workloads on heterogeneous MPSoC platforms. Particularly, we develop MaGNAS, a mapping-aware Graph Neural Architecture Search framework. MaGNAS proposes a GNN architectural design space coupled with prospective mapping options on a heterogeneous SoC to identify model architectures that maximize on-device resource efficiency. To achieve this, MaGNAS employs a two-tier evolutionary search to identify optimal GNNs and mapping pairings that yield the best performance trade-offs. Through designing a supernet derived from the recent Vision GNN (ViG) architecture, we conducted experiments on four (04) state-of-the-art vision datasets using both ( i ) a real hardware SoC platform (NVIDIA Xavier AGX) and ( ii ) a performance/cost model simulator for DNN accelerators. Our experimental results demonstrate that MaGNAS is able to provide 1.57 × latency speedup and is 3.38 × more energy-efficient for several vision datasets executed on the Xavier MPSoC vs. the GPU-only deployment while sustaining an average 0.11% accuracy reduction from the baseline.

Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective

Transferring knowledge from pre-trained deep models for downstream tasks, particularly with limited labeled samples, is a fundamental problem in computer vision research. Recent advances in large-scale, task-agnostic vision-language pre-trained models, which are learned with billions of samples, have shed new light on this problem. In this study, we investigate how to efficiently transfer aligned visual and textual knowledge for downstream visual recognition tasks. We first revisit the role of the linear classifier in the vanilla transfer learning framework, and then propose a new paradigm where the parameters of the classifier are initialized with semantic targets from the textual encoder and remain fixed during optimization. To provide a comparison, we also initialize the classifier with knowledge from various resources. In the empirical study, we demonstrate that our paradigm improves the performance and training speed of transfer learning tasks. With only minor modifications, our approach proves effective across 17 visual datasets that span three different data domains: image, video, and 3D point cloud.