RGB Images Research Articles

Federated Learning Using Multi-Modal Sensors with Heterogeneous Privacy Sensitivity Levels

Data from multi-modal sensors, such as RGB cameras, thermal cameras, microphones, and mmWave radars, have gradually been adopted in various classification problems for better accuracy. Some sensors, like RGB cameras and microphones, however, capture privacy-invasive data, which are less likely to be used in centralized learning. Although the Federated Learning (FL) paradigm frees clients from sharing their sensor data, doing so results in reduced classification accuracy and increased training time. In this article, we introduce a novel Heterogeneous Privacy Federated Learning (HPFL) paradigm to better capitalize on the less privacy-invasive sensor data, such as thermal images and mmWave point clouds, by uploading them to the server for closing the performance gap between FL and centralized learning. HPFL not only allows clients to keep the more privacy-invasive sensor data private, such as RGB images and human voices, but also gives each client total freedom to define the levels of their privacy concern on individual sensor modalities. For example, more sensitive users may prefer to keep their thermal images private, while others do not mind sharing these images. We carry out extensive experiments to evaluate the HPFL paradigm using two representative classification problems: semantic segmentation and emotion recognition. Several key findings demonstrate the merits of HPFL: (i) compared to FedAvg, it improves foreground accuracy by 18.20% in semantic segmentation and boosts the F1-score by 4.20% in emotion recognition, (ii) with heterogeneous privacy concern levels, it achieves an even larger F1-score improvement of 6.17–16.05% in emotion recognition, and (iii) it also outperforms the state-of-the-art FL approaches by 12.04–17.70% in foreground accuracy and 2.54–4.10% in F1-score.

Image Quality Restoration on Historical Artifacts Using Histogram Equalization and Contrast Stretching Methods

This research was carried out to improve the image quality of historical artifacts that experience quality degradation such as noise, unclear or dark image contrast, and poor sharpness. Contrast Stretching and Histogram Equalization are image quality improvement methods to expand and even out image grayscale. This research aims to build a system for improving image quality on historical artifacts using contrast stretching and histogram equalization methods. The sample used in this research is RGB image data with 10 test images in .jpg format, while the process uses the Matlab programming language. The results of improving image quality using the contrast stretching and histogram equalization methods from sample 1 to image sample 10 showed the best results in test image_citra_1 with contrast stretching results based on PSNR = 32.61 dB and MSE = 35.86 Db and PSNR = 19.13 Db and MSE = 794.76 in histogram equalization calculations.

Research into the Applications of a Multi-Scale Feature Fusion Model in the Recognition of Abnormal Human Behavior.

Due to the increasing severity of aging populations in modern society, the accurate and timely identification of, and responses to, sudden abnormal behaviors of the elderly have become an urgent and important issue. In the current research on computer vision-based abnormal behavior recognition, most algorithms have shown poor generalization and recognition abilities in practical applications, as well as issues with recognizing single actions. To address these problems, an MSCS-DenseNet-LSTM model based on a multi-scale attention mechanism is proposed. This model integrates the MSCS (Multi-Scale Convolutional Structure) module into the initial convolutional layer of the DenseNet model to form a multi-scale convolution structure. It introduces the improved Inception X module into the Dense Block to form an Inception Dense structure, and gradually performs feature fusion through each Dense Block module. The CBAM attention mechanism module is added to the dual-layer LSTM to enhance the model's generalization ability while ensuring the accurate recognition of abnormal actions. Furthermore, to address the issue of single-action abnormal behavior datasets, the RGB image dataset RIDS (RGB image dataset) and the contour image dataset CIDS (contour image dataset) containing various abnormal behaviors were constructed. The experimental results validate that the proposed MSCS-DenseNet-LSTM model achieved an accuracy, sensitivity, and specificity of 98.80%, 98.75%, and 98.82% on the two datasets, and 98.30%, 98.28%, and 98.38%, respectively.

Deep learning-based few-shot person re-identification from top-view RGB and depth images

Deep learning-based few-shot person re-identification from top-view RGB and depth images

MosViT: towards vision transformers for moving object segmentation based on Lidar point cloud

Abstract Moving object segmentation is fundamental for various downstream tasks in robotics and autonomous driving, providing crucial information for them. Effectively extracting spatial-temporal information from consecutive frames and addressing the scarcity of dataset is paramount for learning-based 3D LiDAR Moving Object Segmentation (LIDAR-MOS). In this work, we propose a novel deep neural network based on Vision Transformers (ViTs) to tackle this problem. We first validate the feasibility of Transformer networks for this task, offering an alternative to CNNs. Specifically, we utilize a dual-branch structure based on range-image data to extract spatial-temporal information from consecutive frames and fuse it using a motion-guided attention mechanism. Furthermore, we employ the ViT as the backbone, keeping its architecture unchanged from what is used for RGB images. This enables us to leverage pre-trained models from RGB images to improve results, addressing the issue of limited LIDAR point cloud data, which is cheaper compared to acquiring and annotating point cloud data. We validate the effectiveness of our approach on the LIDAR-MOS benchmark of SemanticKitti and achieve comparable results to methods that use CNNs on range image data. The source code and trained models are available at https://github.com/mafangniu/MOSViT.git.

Enhancing healthcare cloud security with intelligent shuffled frog leaping optimized sequential long short-term memory (ISFLO-SLSTM)

Although medical image processing consumes much memory, computing time and specialized platforms are needed. As a result, it is crucial to implement cost-effective solutions to replace outdated ones. Due to these factors, we choose to adopt cloud computing to satisfy our needs for large-scale data processing. Meanwhile, this strategy offers accessibility to services upon request with excellent reliability and versatility. Therefore, employing cloud services rather than internal applications will assist healthcare organizations in outsourcing calculations to a third party, thereby reducing operating costs. However, to avoid malicious data leaks, comprehensive protection of information across unauthorized users and untrustworthy clouds is necessary. Different frameworks are created to allow consumers to save and analyze their information utilizing cloud-based computing. They are developed utilizing distributed systems, cryptosystems, or an amalgam of the two. The primary issue with applying cryptography methods for the enormous analysis of information on the cloud is the computing cost of image processing operations. Preventing unauthorized usage of healthcare records and private health details is the main concern. To secure data processing in a cloud environment by classifying image pixels, we offer a revolutionary intelligent shuffled frog jumping optimized sequential long short-term memory (ISFLO-SLSTM) approach. We added an additional layer, the cloud security module, to lower the danger of future medical data leakage. The effectiveness of the suggested strategy is assessed using medical RGB images taken from the Kaggle database. The findings of this study show how the ISFLO-SLSTM system has the potential to revolutionize healthcare information analysis by protecting patient privacy and security that utilizes the full extent of the latest machine learning methods.

GamaNNet: A Novel Plant Pathologist-Level CNN Architecture for Intelligent Diagnosis

Plant pathologies significantly jeopardise global food security, necessitating the development of prompt and precise diagnostic methods. This study employs advanced deep learning techniques to evaluate the performance of nine convolutional neural networks (CNNs) in identifying a spectrum of phytosanitary issues affecting the foliage of Solanum lycopersicum (tomato). Ten thousand RGB images of leaf tissue were subsampled in training (64%), validation (16%), and test (20%) sets to rank the most suitable CNNs in expediting the diagnosis of plant disease. The study assessed the performance of eight well-known networks under identical hyperparameter conditions. Additionally, it introduced the GamaNNet architecture, a custom-designed model optimised for superior performance on this specific type of dataset. The investigational results were most promising for the innovative GamaNNet and ResNet-152, which both exhibited a 91% accuracy rate, as evidenced by their confusion matrices, ROC curves, and AUC metrics. In comparison, LeNet-5 and ResNet-50 demonstrated lower assertiveness, attaining accuracies of 74% and 69%, respectively. GoogLeNet and Inception-v3 emerged as the frontrunners, displaying diagnostic preeminence, achieving an average F1-score of 97%. Identifying such pathologies as Early Blight, Late Blight, Corynespora Leaf Spot, and Septoria Leaf Spot posed the most significant challenge for this class of problem.

Assessing the Efficacy of the Spectrum-Aided Vision Enhancer (SAVE) to Detect Acral Lentiginous Melanoma, Melanoma In Situ, Nodular Melanoma, and Superficial Spreading Melanoma.

Skin cancer is the predominant form of cancer worldwide, including 75% of all cancer cases. This study aims to evaluate the effectiveness of the spectrum-aided visual enhancer (SAVE) in detecting skin cancer. This paper presents the development of a novel algorithm for snapshot hyperspectral conversion, capable of converting RGB images into hyperspectral images (HSI). The integration of band selection with HSI has facilitated the identification of a set of narrow band images (NBI) from the RGB images. This study utilizes various iterations of the You Only Look Once (YOLO) machine learning (ML) framework to assess the precision, recall, and mean average precision in the detection of skin cancer. YOLO is commonly preferred in medical diagnostics due to its real-time processing speed and accuracy, which are essential for delivering effective and efficient patient care. The precision, recall, and mean average precision (mAP) of the SAVE images show a notable enhancement in comparison to the RGB images. This work has the potential to greatly enhance the efficiency of skin cancer detection, as well as improve early detection rates and diagnostic accuracy. Consequently, it may lead to a reduction in both morbidity and mortality rates.

HPRN: Holistic Prior-Embedded Relation Network for Spectral Super-Resolution.

Spectral super-resolution (SSR) refers to the hyperspectral image (HSI) recovery from an RGB counterpart. Due to the one-to-many nature of the SSR problem, a single RGB image can be reprojected to many HSIs. The key to tackle this ill-posed problem is to plug into multisource prior information such as the natural spatial context prior of RGB images, deep feature prior, or inherent statistical prior of HSIs so as to effectively alleviate the degree of ill-posedness. However, most current approaches only consider the general and limited priors in their customized convolutional neural networks (CNNs), which leads to the inability to guarantee the confidence and fidelity of reconstructed spectra. In this article, we propose a novel holistic prior-embedded relation network (HPRN) to integrate comprehensive priors to regularize and optimize the solution space of SSR. Basically, the core framework is delicately assembled by several multiresidual relation blocks (MRBs) that fully facilitate the transmission and utilization of the low-frequency content prior of RGBs. Innovatively, the semantic prior of RGB inputs is introduced to mark category attributes, and a semantic-driven spatial relation module (SSRM) is invented to perform the feature aggregation of clustered similar ranges for refining recovered characteristics. In addition, we develop a transformer-based channel relation module (TCRM), which breaks the habit of employing scalars as the descriptors of channelwise relations in the previous deep feature prior and replaces them with certain vectors to make the mapping function more robust and smoother. In order to maintain the mathematical correlation and spectral consistency between hyperspectral bands, the second-order prior constraints (SOPCs) are incorporated into the loss function to guide the HSI reconstruction. Finally, extensive experimental results on four benchmarks demonstrate that our HPRN can reach the state-of-the-art performance for SSR quantitatively and qualitatively. Furthermore, the effectiveness and usefulness of the reconstructed spectra are verified by the classification results on the remote sensing dataset. Codes are available at https://github.com/Deep-imagelab/HPRN.

Attention-based fusion network for RGB-D semantic segmentation

RGB-D semantic segmentation can realize a profound comprehension of scenes, which is crucial in various computer vision tasks. However, due to the inherent modal variances and image noise, achieving superior segmentation using existing methods remains challenging. In this paper, we propose an attention-based fusion network for RGB-D semantic segmentation. Specifically, our network employs a forward multi-step propagation strategy and a backward progressive bootstrap fusion strategy based on the encoder–decoder architecture. By aggregating feature maps at different scales, we effectively diminish the uncertainty in the final prediction. Meanwhile, we introduce a Channel and Spatial Rectification Module (CSRM) to enable multi-dimensional interactions and noise removal. In order to achieve comprehensive integration between RGB and depth images, we put the rectified features into the Cross-Attention Fusion Module(CAFM). Extensive experiments show that our network can adeptly manage a diverse array of complex scenarios, demonstrating its innovative strength with superior performance and robust effectiveness across indoor NYU Depth V2 and SUN-RGBD datasets, and extending its capabilities to the outdoor Cityscapes dataset.

SpectralGPT: Spectral Remote Sensing Foundation Model.

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.

Building Shadow Detection Based on Improved Quick Shift Algorithm in GF‐2 Images

Shadows in remote sensing images contain crucial information about various features on the ground. In this study, a method for detecting building shadows in GF‐2 images based on improved quick shift was proposed. First, six feature variables were constructed: first principal component (PC1), brightness component (I), normalized difference shadow index (NDSI), morphological shadow index (MSI), normalized difference water index (NDWI), and normalized difference vegetation index (NDVI). Then, the image was segmented to obtain homogeneous objects, which were then classified using a random forest model. Two improvements were added to the quick shift algorithm: using PC1, I, and MSI as input data instead of RGB images; and adding Canny edge constraints. Validation in six research areas yields Kappa coefficients of 0.928, 0.896, 0.89, 0.913, 0.879, and 0.909, confirming method feasibility. In addition, comparative experiments demonstrate its effectiveness and robustness across different land cover types while mitigating the segmentation scale effect.

A Novel Robotic Pushing and Grasping Method Based on Vision Transformer and Convolution.

Robotic grasping techniques have been widely studied in recent years. However, it is always a challenging problem for robots to grasp in cluttered scenes. In this issue, objects are placed close to each other, and there is no space around for the robot to place the gripper, making it difficult to find a suitable grasping position. To solve this problem, this article proposes to use the combination of pushing and grasping (PG) actions to help grasp pose detection and robot grasping. We propose a pushing-grasping combined grasping network (GN), PG method based on transformer and convolution (PGTC). For the pushing action, we propose a vision transformer (ViT)-based object position prediction network pushing transformer network (PTNet), which can well capture the global and temporal features and can better predict the position of objects after pushing. To perform the grasping detection, we propose a cross dense fusion network (CDFNet), which can make full use of the RGB image and depth image, and fuse and refine them several times. Compared with previous networks, CDFNet is able to detect the optimal grasping position more accurately. Finally, we use the network for both simulation and actual UR3 robot grasping experiments and achieve SOTA performance. Video and dataset are available at https://youtu.be/Q58YE-Cc250.

Employment of conditional random fields for monocular depth estimation

Abstract Estimating depth from a solitary RGB image, known as monocular depth estimation, presents a significant challenge. Currently, most methods for this task involve designing increasingly complex networks to regress the depth map straightforwardly. However, we have adopted a more interpretable approach by using Conditional Random Fields from optimization methods. Additionally, to facilitate better information transfer between nodes, a multi-head attention mechanism is employed to calculate multiple energy functions, which are then optimized by the network into an accurate depth map. Experiments demonstrate that our method can accurately estimate the depth of landscapes.

Invisible gas detection: An RGB-thermal cross attention network and a new benchmark

The widespread use of various chemical gases in industrial processes necessitates effective measures to prevent their leakage during transportation and storage, given their high toxicity. Thermal infrared-based computer vision detection techniques provide a straightforward approach to identify gas leakage areas. However, the development of high-quality algorithms has been challenging due to the low texture in thermal images and the lack of open-source datasets. In this paper, we present the RGB-Thermal Cross Attention Network (RT-CAN), which employs an RGB-assisted two-stream network architecture to integrate texture information from RGB images and gas area information from thermal images. Additionally, to facilitate the research of invisible gas detection, we introduce Gas-DB, an extensive open-source gas detection database including about 1.3K well-annotated RGB-thermal images with eight variant collection scenes. Experimental results demonstrate that our method successfully leverages the advantages of both modalities, achieving state-of-the-art (SOTA) performance among RGB-thermal methods, surpassing single-stream SOTA models in terms of accuracy, Intersection of Union (IoU), and F2 metrics by 4.86%, 5.65%, and 4.88%, respectively. The code and data can be found at https://github.com/logic112358/RT-CAN.

Similar assembly state discriminator for reinforcement learning-based robotic connector assembly

In practice, the process of robot assembly in an unstructured environment faces difficulties due to the presence of unpredictable environmental errors related to vision and pose. Therefore, to minimize the uncertain environmental errors during the robotic assembly process in an unstructured environment, several studies have considered a reinforcement learning (RL)-based approach. However, if assembly parts are changed, it becomes difficult to apply RL-based methods to assemble various parts because additional learning may be required. Especially in the case of connector assembly, fine-tuning is essential because the shape changes depending on the type of connector. In this study, we propose a similar assembly state discriminator that transforms the state information (force, velocity, and RGB image) of reinforcement learning into generalized features to respond various types of connector assembly tasks. This method processes the data to include essential features for assembly regardless of connector type. By learning the RL model with the processed data using this method, the RL model trained for a specific connector can be efficiently applied to other types of connectors without fine-tuning. The assembly success rate for the 7 types of connectors (Harting, HDMI, USB, power, air jack, banana plug and PCIE) using the proposed method was demonstrated to be over 96 %.

Weed Species Identification: Acquisition, Feature Analysis, and Evaluation of a Hyperspectral and RGB Dataset with Labeled Data

Site-specific weed management employs image data to generate maps through various methodologies that classify pixels corresponding to crop, soil, and weed. Further, many studies have focused on identifying specific weed species using spectral data. Nonetheless, the availability of open-access weed datasets remains limited. Remarkably, despite the extensive research employing hyperspectral imaging data to classify species under varying conditions, to the best of our knowledge, there are no open-access hyperspectral weed datasets. Consequently, accessible spectral weed datasets are primarily RGB or multispectral and mostly lack the temporal aspect, i.e., they contain a single measurement day. This paper introduces an open dataset for training and evaluating machine-learning methods and spectral features to classify weeds based on various biological traits. The dataset comprises 30 hyperspectral images, each containing thousands of pixels with 204 unique visible and near-infrared bands captured in a controlled environment. In addition, each scene includes a corresponding RGB image with a higher spatial resolution. We included three weed species in this dataset, representing different botanical groups and photosynthetic mechanisms. In addition, the dataset contains meticulously sampled labeled data for training and testing. The images represent a time series of the weed’s growth along its early stages, critical for precise herbicide application. We conducted an experimental evaluation to test the performance of a machine-learning approach, a deep-learning approach, and Spectral Mixture Analysis (SMA) to identify the different weed traits. In addition, we analyzed the importance of features using the random forest algorithm and evaluated the performance of the selected algorithms while using different sets of features.

Co-scale cross-attentional transformer for rearrangement target detection

Rearranging objects (e.g. vase, door) back in their original positions is one of the most fundamental skills for domestic service robots (DSRs). In rearrangement tasks, it is crucial to detect the objects that need to be rearranged according to the goal and current states. In this study, we focus on Rearrangement Target Detection (RTD), where the model generates a change mask for objects that should be rearranged. Although many studies have been conducted in the field of Scene Change Detection (SCD), most SCD methods often fail to segment objects with complex shapes and fail to detect the change in the angle of objects that can be opened or closed. In this study, we propose a Co-Scale Cross-Attentional Transformer for RTD. We introduce the Serial Encoder which consists of a sequence of serial blocks and the Cross-Attentional Encoder which models the relationship between the goal and current states. We built a new dataset consisting of RGB images and change masks regarding the goal and current states. We validated our method on the dataset and the results demonstrated that our method outperformed baseline methods on F 1 -score and mean IoU.

Depth Image Assisted Aiming for Scoring Goal in Wheeled Soccer Robot

Wheeled soccer robot, as an automatic robot, required to have an advanced decision making system based on information it grasp from its surrounding. One of the most crucial decision making ability is to determine aiming angle when it is scoring goal.This research will enhance the aiming ability for scoring goal by predicting unguarded area of goal. Combination of depth image and RGB image information will be used to predict the position of unguarded space in goal. This position will be converted into aiming angle for the robot. Intel Realsense D435i depth camera will be used to get RGB and depth image simultaneouslyBy using this method, the system is capable to predict unguarded area in all of 60 test points, with 1.3% average error for the predicted coordinate.

3D Industrial anomaly detection via dual reconstruction network

Currently, 2D anomaly detection has demonstrated outstanding performance. However, 2D images limit the improvement of anomaly detection accuracy without utilizing depth information. Therefore, this paper proposes a Dual Reconstruction viAInpainting Network for 3D industrial anomaly detection (DRAIN). Firstly, we design a 3D reconstruction network using an encoder-decoder-based U-shaped network for processing RGB images and depth images. Subsequently, accurate anomaly segmentation is implemented through a 3D segmentation network. We introduce a lightweight MLP module to enhance segmentation performance to capture long-range dependencies in the reconstructed images. Furthermore, we propose a dual attention-based information entropy fusion module to expedite feature fusion in the inference process, aiming for enhanced deployment in the industry. Extensive experiments demonstrate that DRAIN achieves a 94.3% AUROC on the 3D anomaly detection dataset MVTec 3D-AD, surpassing other research methods.Graphical abstractOverall architecture for 3D industrial anomaly detection via dual reconstruction network