RGB Images Research Articles

A duplex transform heterogeneous feature fusion network for road segmentation

Detecting roads in automatic driving environments poses a challenge due to issues such as boundary fuzziness, occlusion, and glare from light. We believe that two factors are instrumental in addressing these challenges and enhancing detection performance: global context dependency and effective feature representation that prioritizes important feature channels. To tackle these issues, we introduce DTRoadseg, a novel duplex Transformer-based heterogeneous feature fusion network designed for road segmentation. DTRoadseg leverages a duplex encoder architecture to extract heterogeneous features from both RGB images and point-cloud depth images. Subsequently, we introduce a multi-source Heterogeneous Feature Reinforcement Block (HFRB) for fusion of the encoded features, comprising a Heterogeneous Feature Fusion Module (HFFM) and a Reinforcement Fusion Module (RFM). The HFFM leverages the self-attention mechanisms of Transformers to achieve effective fusion through token interactions, while the RFM focuses on emphasizing informative features while downplaying less important ones, thereby reinforcing feature fusion. Finally, a Transformer decoder is utilized to produce the final semantic prediction. Furthermore, we employ a boundary loss function to optimize the segmentation structure area, reduce false detection areas, and improve model accuracy. Extensive experiments are carried out on the KITTI road dataset. The results demonstrate that, compared with state-of-the-art methods, DTRoadseg exhibits superior performance, achieving an average accuracy of 97.01%, a Recall of 96.35%, and runs at a speed of 0.09 s per picture.

Using the Multiple-Sensor-Based Frost Observation System (MFOS) for Image Object Analysis and Model Prediction Evaluation in an Orchard

Accurate frost observations are crucial for developing and validating frost prediction models. In 2022, the multi-sensor-based automatic frost observation system (MFOS), including an RGB camera, a thermal infrared camera, a leaf wetness sensor (LWS), LED lighting, and three glass plates, was developed to replace the naked-eye observation of frost. The MFOS, herein installed and operated in an apple orchard, provides temporally high-resolution frost observations that show the onset, end, duration, persistence, and discontinuity of frost more clearly than conventional naked-eye observations. This study introduces recent additions to the MFOS and presents the results of its application to frost weather analysis and forecast evaluation in an orchard in South Korea. The NCAM’s Weather Research and Forecasting (WRF) model was employed as a weather forecast model. The main findings of this study are as follows: (1) The newly added image-based object detection capabilities of the MFOS helped with the extraction and quantitative comparison of surface temperature data for apples, leaves, and the LWS. (2) The resolution matching of the RGB and thermal infrared images was made successful by resizing the images, matching them according to horizontal movement, and conducting apple-centered averaging. (3) When applied to evaluate the frost-point predictions of the numerical weather model at one-hour intervals, the results showed that the MFOS could be used as a much more objective tool to verify the accuracy and characteristics of frost predictions compared to the naked-eye view. (4) Higher-resolution and realistic land-cover and vegetation representation are necessary to improve frost forecasts using numerical grid models based on land–atmosphere physics.

LiDARCapV2: 3D human pose estimation with human–object interaction from LiDAR point clouds

Human–object interactions in open environments are common in the real world. Estimating 3D human pose from data where objects occlude the human is a challenging task in biometrics. However, existing LiDAR-based human motion capture datasets lack occlusion scenarios between humans and objects. To overcome this limitation, we propose LiDARHuman51M, a new human–object interaction dataset captured by LiDAR in long-range outdoor scene. It includes human motion labels acquired by an IMU system and synchronous RGB images. Additionally, we present an occlusion-aware method, LiDARCapV2, for capturing human motion from LiDAR point clouds under human–object interaction settings. Our key insight is to overcome object interference in human feature extraction by introducing a module called AgNoise-Segment. A noise augmentation strategy introduced in the AgNoise-Segment module alleviates the dependency of the segmentation accuracy on the effectiveness of 3D human pose estimations. Furthermore, we propose a skeleton extraction module that integrates features learned from the AgNoise-Segment module and predicts the skeleton locations. Quantitative and qualitative experiments demonstrate that LiDARCapV2 can capture high-quality 3D human motion under human–object interaction settings. Experiments on the KITTI and Waymo datasets demonstrate that our method can be generalized to real-world open scenarios.

MixImages: An Urban Perception AI Method Based on Polarization Multimodalities.

Intelligent urban perception is one of the hot topics. Most previous urban perception models based on semantic segmentation mainly used RGB images as unimodal inputs. However, in natural urban scenes, the interplay of light and shadow often leads to confused RGB features, which diminish the model's perception ability. Multimodal polarization data encompass information dimensions beyond RGB, which can enhance the representation of shadow regions, serving as additional data for assistance. Additionally, in recent years, transformers have achieved outstanding performance in visual tasks, and their large, effective receptive field can provide more discriminative cues for shadow regions. For these reasons, this study proposes a novel semantic segmentation model called MixImages, which can combine polarization data for pixel-level perception. We conducted comprehensive experiments on a polarization dataset of urban scenes. The results showed that the proposed MixImages can achieve an accuracy advantage of 3.43% over the control group model using only RGB images in the unimodal benchmark while gaining a performance improvement of 4.29% in the multimodal benchmark. Additionally, to provide a reference for specific downstream tasks, we also tested the impact of different combinations of polarization types on the overall segmentation accuracy. The proposed MixImages can be a new option for conducting urban scene perception tasks.

An Analytical Study on the Utility of RGB and Multispectral Imagery with Band Selection for Automated Tumor Grading.

The implementation of tumor grading tasks with image processing and machine learning techniques has progressed immensely over the past several years. Multispectral imaging enabled us to capture the sample as a set of image bands corresponding to different wavelengths in the visible and infrared spectrums. The higher dimensional image data can be well exploited to deliver a range of discriminative features to support the tumor grading application. This paper compares the classification accuracy of RGB and multispectral images, using a case study on colorectal tumor grading with the QU-Al Ahli Dataset (dataset I). Rotation-invariant local phase quantization (LPQ) features with an SVM classifier resulted in 80% accuracy for the RGB images compared to 86% accuracy with the multispectral images in dataset I. However, the higher dimensionality elevates the processing time. We propose a band-selection strategy using mutual information between image bands. This process eliminates redundant bands and increases classification accuracy. The results show that our band-selection method provides better results than normal RGB and multispectral methods. The band-selection algorithm was also tested on another colorectal tumor dataset, the Texas University Dataset (dataset II), to further validate the results. The proposed method demonstrates an accuracy of more than 94% with 10 bands, compared to using the whole set of 16 multispectral bands. Our research emphasizes the advantages of multispectral imaging over the RGB imaging approach and proposes a band-selection method to address the higher computational demands of multispectral imaging.

Acne Detection Based on Reconstructed Hyperspectral Images.

Acne Vulgaris is a common type of skin disease that affects more than 85% of teenagers and frequently continues even in adulthood. While it is not a dangerous skin disease, it can significantly impact the quality of life. Hyperspectral imaging (HSI), which captures a wide spectrum of light, has emerged as a tool for the detection and diagnosis of various skin conditions. However, due to the high cost of specialised HS cameras, it is limited in its use in clinical settings. In this research, a novel acne detection system that will utilise reconstructed hyperspectral (HS) images from RGB images is proposed. A dataset of reconstructed HS images is created using the best-performing HS reconstruction model from our previous research. A new acne detection algorithm that is based on reconstructed HS images and RetinaNet algorithm is introduced. The results indicate that the proposed algorithm surpasses other techniques based on RGB images. Additionally, reconstructed HS images offer a promising and cost-effective alternative to using expensive HSI equipment for detecting conditions like acne or other medical issues.

DDSR: Degradation-Aware Diffusion Model for Spectral Reconstruction from RGB Images

The reconstruction of hyperspectral images (HSIs) from RGB images is an attractive low-cost approach to recover hyperspectral information. However, existing approaches focus on learning an end-to-end mapping of RGB images and their corresponding HSIs with neural networks, which makes it difficult to ensure generalization due to the fact that they are trained on data with a specific degradation process. As a new paradigm of generative models, the diffusion model has shown great potential in image restoration, especially in noisy contexts. To address the unstable generalization ability of end-to-end models while exploiting the powerful ability of the diffusion model, we propose a degradation-aware diffusion model. The degradation process from HSI to RGB is modeled as a combination of multiple degradation operators, which are used to guide the inverse process of the diffusion model by utilizing a degradation-aware correction. By integrating the degradation-aware correction to the diffusion model, we obtain an efficient solver for spectral reconstruction, which is robust to different degradation patterns. Experiment results on various public datasets demonstrate that our method achieves competitive performance and shows a promising generalization ability.

Enhanced Pansharpening Using Curvelet Transform Optimized by Multi Population Based Differential Evolution

In this study, a pansharpening process was conducted to merge the color information of low-resolution RGB images with the details of high-resolution panchromatic images to obtain higher quality images. During this process, weight optimization was performed using the Curvelet Transform method and the Multi Population Based Differential Evolution (MDE) algorithm. The proposed method was tested on Landsat ETM satellite image. For Landsat ETM data, the RGB images have a resolution of 30m, while the panchromatic images have a resolution of 15m. To evaluate the performance of the study, the proposed MDE-optimized Curvelet Transform-based pansharpening method was compared with classical IHS, Brovey, PCA, Gram-Schmidt and Simple Mean methods. The comparison process employed metrics such as RMSE, SAM, COC, RASE, QAVE, SID, and ERGAS. The results indicate that the proposed method outperforms classical methods in terms of both visual quality and numerical accuracy.

Real-time testing of AI enabled automatic emergency braking system for ADAS vehicle using 3D point cloud and precise depth information

At the forefront of automobile safety technology, Automatic Emergency Braking (AEB) represents a major advancement in collision avoidance systems. The cutting-edge technology provides an additional layer of security at pivotal times, making it a vital part of the changing landscape of vehicle safety. This research introduces an effective system for automatic emergency braking in ADAS-equipped or Autonomous vehicles using a combination of 3d lidar and stereo vision camera for a swift and robust system in the vehicle that is faster than human drivers in cases of unexpected emergencies. Utilizing the power of clustering algorithms on 3d point clouds and state-of-the-art computer vision algorithms on an RGB image mapped to a depth frame from the stereo vision camera, the system as a whole provides a comprehensive system adding to the safety of the vehicle and the passengers. Further, the efficiency of the system is studied based on various parameters. The data from an external inertial measurement unit is also utilized to derive results that support the claims of the study. The system has been developed and implemented on a passenger car that has been modified into an electric vehicle and further tested in real-world traffic conditions in autonomous driving mode. The findings of the study were having exceptionally good precision in split-second decision-making in emergency maneuvers.

Block Cipher Nonlinear Component Generation via Hybrid Pseudo-Random Binary Sequence for Image Encryption

To analyze the security of encryption, an effectual encryption scheme based on colored images utilizing the hybrid pseudo-random binary sequence (HPRBS) and substitution boxes, known as S-boxes, is proposed. The presented work aims to design S-boxes using pseudo-random binary numbers acquired by Linear Feedback Shift Registers (LFSRs) in combination with a modified quadratic chaotic map. Firstly, cryptographically robust S-boxes are constructed by using binary pseudo-random number sequences, and then the cryptographic properties of the presented S-boxes are tested. The suggested S-boxes showed good results. Secondly, an RGB image encryption algorithm utilizing sequences generated by modified quadratic chaotic maps and S-boxes is offered. The new color image encryption techniques comprise two steps, including a permutation and a substitution step. The key association with the content of the image is also addressed. This strategy can result in a “one-time pad” effect and make the algorithm resistant to chosen-plaintext attack (CPA). The proposed scheme has been confirmed to be more valuable than most of the existing schemes. S-boxes are analyzed by the nonlinearity test, bit independence criterion (BIC), linear and differential approximation probabilities (LPs; DPs), and Strict-Avalanche Criterion (SAC) tests. A comparison with different S-boxes presented in the literature is also carried out. The comparison shows encouraging results about the quality of the proposed box. From security and experimental outcomes, the effectiveness of the presented color image encryption technique is verified. The proposed scheme has evident efficiency benefits, which implies that the proposed colored encryption of the image scheme has better potential for application in encryption schemes in real-time.

Sensor-Fused Nighttime System for Enhanced Pedestrian Detection in ADAS and Autonomous Vehicles.

Ensuring a safe nighttime environmental perception system relies on the early detection of vulnerable road users with minimal delay and high precision. This paper presents a sensor-fused nighttime environmental perception system by integrating data from thermal and RGB cameras. A new alignment algorithm is proposed to fuse the data from the two camera sensors. The proposed alignment procedure is crucial for effective sensor fusion. To develop a robust Deep Neural Network (DNN) system, nighttime thermal and RGB images were collected under various scenarios, creating a labeled dataset of 32,000 image pairs. Three fusion techniques were explored using transfer learning, alongside two single-sensor models using only RGB or thermal data. Five DNN models were developed and evaluated, with experimental results showing superior performance of fused models over non-fusion counterparts. The late-fusion system was selected for its optimal balance of accuracy and response time. For real-time inferencing, the best model was further optimized, achieving 33 fps on the embedded edge computing device, an 83.33% improvement in inference speed over the system without optimization. These findings are valuable for advancing Advanced Driver Assistance Systems (ADASs) and autonomous vehicle technologies, enhancing pedestrian detection during nighttime to improve road safety and reduce accidents.

Dual-stage feedback network for lightweight color image compression artifact reduction

Lossy image coding techniques usually result in various undesirable compression artifacts. Recently, deep convolutional neural networks have seen encouraging advances in compression artifact reduction. However, most of them focus on the restoration of the luma channel without considering the chroma components. Besides, most deep convolutional neural networks are hard to deploy in practical applications because of their high model complexity. In this article, we propose a dual-stage feedback network (DSFN) for lightweight color image compression artifact reduction. Specifically, we propose a novel curriculum learning strategy to drive a DSFN to reduce color image compression artifacts in a luma-to-RGB manner. In the first stage, the DSFN is dedicated to reconstructing the luma channel, whose high-level features containing rich structural information are then rerouted to the second stage by a feedback connection to guide the RGB image restoration. Furthermore, we present a novel enhanced feedback block for efficient high-level feature extraction, in which an adaptive iterative self-refinement module is carefully designed to refine the low-level features progressively, and an enhanced separable convolution is advanced to exploit multiscale image information fully. Extensive experiments show the notable advantage of our DSFN over several state-of-the-art methods in both quantitative indices and visual effects with lower model complexity.

Pseudo-Multispectral Pedestrian Detection with Deep Thermal Feature Guidance

With complementary multi-modal information (i.e. visible and thermal), multispectral pedestrian detection is essential for around-the-clock applications, such as autonomous driving, video surveillance, and vicinagearth security. Despite its broad applications, the requirements for expensive thermal device and multi-sensor alignment limit the utilization in real-world applications. In this paper, we propose a pseudo-multispectral pedestrian detection (called PseudoMPD) method, which employs the gray image converted from the RGB image to replace the real thermal image, and learns the pseudo-thermal feature through deep thermal feature guidance (TFG). To achieve this goal, we first introduce an image base-detail decomposition (IBD) module to decompose image information into base and detail parts. Afterwards, we design a base-detail hierarchical feature fusion (BHFF) module to deeply exploit the information between these two parts, and employ a TFG module to guide pseudo-thermal base and detail feature learning. As a result, our proposed method does not require the real thermal image during inference. The comprehensive experiments are performed on two public multispectral pedestrian datasets. The experimental results demonstrate the effectiveness of our proposed method.

Deformation Estimation of Textureless Objects from a Single Image.

Deformations introduced during the production of plastic components degrade the accuracy of their 3D geometric information, a critical aspect of object inspection processes. This phenomenon is prevalent among primary plastic products from manufacturers. This work proposes a solution for the deformation estimation of textureless plastic objects using only a single RGB image. This solution encompasses a unique image dataset of five deformed parts, a novel method for generating mesh labels, sequential deformation, and a training model based on graph convolution. The proposed sequential deformation method outperforms the prevalent chamfer distance algorithm in generating precise mesh labels. The training model projects object vertices into features extracted from the input image, and then, predicts vertex location offsets based on the projected features. The predicted meshes using these offsets achieve a sub-millimeter accuracy on synthetic images and approximately 2.0 mm on real images.

DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image

Perceiving 3D structures from RGB images based on CAD model primitives can enable an effective, efficient 3D object-based representation of scenes. However, current approaches rely on supervision from expensive yet imperfect annotations of CAD models associated with real images, and encounter challenges due to the inherent ambiguities in the task - both in depth-scale ambiguity in monocular perception, as well as inexact matches of CAD database models to real observations. We thus propose DiffCAD, the first weakly-supervised probabilistic approach to CAD retrieval and alignment from an RGB image. We learn a probabilistic model through diffusion, modeling likely distributions of shape, pose, and scale of CAD objects in an image. This enables multi-hypothesis generation of different plausible CAD reconstructions, requiring only a few hypotheses to characterize ambiguities in depth/scale and inexact shape matches. Our approach is trained only on synthetic data, leveraging monocular depth and mask estimates to enable robust zero-shot adaptation to various real target domains. Despite being trained solely on synthetic data, our multi-hypothesis approach can even surpass the supervised state-of-the-art on the Scan2CAD dataset by 5.9% with 8 hypotheses.

CRLNet: A Multimodal Peach Detection Network Based on Cooperative Asymptotic Enhancement and the Fusion of Granularity Refinement.

Accurate peach detection is essential for automated agronomic management, such as mechanical peach harvesting. However, ubiquitous occlusion makes identifying peaches from complex backgrounds extremely challenging. In addition, it is difficult to capture fine-grained peach features from a single RGB image, which can suffer from light and noise in scenarios with dense small target clusters and extreme light. To solve these problems, this study proposes a multimodal detector, called CRLNet, based on RGB and depth images. First, YOLOv9 was extended to design a backbone network that can extract RGB and depth features in parallel from an image. Second, to address the problem of information fusion bias, the Rough-Fine Hybrid Attention Fusion Module (RFAM) was designed to combine the advantageous information of different modes while suppressing the hollow noise at the edge of the peach. Finally, a Transformer-based Local-Global Joint Enhancement Module (LGEM) was developed to jointly enhance the local and global features of peaches using information from different modalities in order to enhance the percentage of information about the target peaches and remove the interference of redundant background information. CRLNet was trained on the Peach dataset and evaluated against other state-of-the-art methods; the model achieved an mAP50 of 97.1%. In addition, CRLNet also achieved an mAP50 of 92.4% in generalized experiments, validating its strong generalization capability. These results provide valuable insights for peach and other outdoor fruit multimodal detection.

Estimating Bermudagrass Aboveground Biomass Using Stereovision and Vegetation Coverage

Accurate information about the amount of standing biomass is important in pasture management for monitoring forage growth patterns, minimizing the risk of overgrazing, and ensuring the necessary feed requirements of livestock. The morphological features of plants, like crop height and density, have been proven to be prominent predictors of crop yield. The objective of this study was to evaluate the effectiveness of stereovision-based crop height and vegetation coverage measurements in predicting the aboveground biomass yield of bermudagrass (Cynodon dactylon) in a pasture. Data were collected from 136 experimental plots within a 0.81 ha bermudagrass pasture using an RGB-depth camera mounted on a ground rover. The crop height was determined based on the disparity between images captured by two stereo cameras of the depth camera. The vegetation coverage was extracted from the RGB images using a machine learning algorithm by segmenting vegetative and non-vegetative pixels. After camera measurements, the plots were harvested and sub-sampled to measure the wet and dry biomass yields for each plot. The wet biomass yield prediction function based on crop height and vegetation coverage was generated using a linear regression analysis. The results indicated that the combination of crop height and vegetation coverage showed a promising correlation with aboveground wet biomass yield. However, the prediction function based only on the crop height showed less residuals at the extremes compared to the combined prediction function (crop height and vegetation coverage) and was thus declared the recommended approach (R2 = 0.91; SeY= 1824 kg-wet/ha). The crop height-based prediction function was used to estimate the dry biomass yield using the mean dry matter fraction.

A novel swiss-roll counterflow micro-combustor: Experimental investigation of methane-oxygen flame behavior over time

This study introduces a novel Swiss-roll micro-combustor design that employs a non-premixed counterflow flame for efficient and stable micro-combustion. This design addresses practical system challenges by (1) stabilizing the flame, (2) enhancing heat recovery and gas preheating, (3) aligning gas momentum with the exhaust, and (4) minimizing disruptive fluid motion within the channels. This research investigates the dynamic behavior and combustion characteristics of a methane-oxygen flame within the combustor across a range of equivalence ratios (1.50–3.38). A synergistic approach combining flame dynamics analysis, spectroscopy, and RGB image processing explores the interplay between flame temperature, strain rate, and equivalence ratio. The results demonstrate the effectiveness of the RGB method as a cost-efficient tool for predicting variations in C2∗ and CH∗ radicals and light-intensity radiations. While flame thickness and length exhibit complex dependencies on the fuel-to-oxygen ratio and wall temperature, specific radical emissions vary dynamically with the equivalence ratio. Notably, equivalence ratios of 1.94 and 2.26 exhibit minimal temporal variations in flame properties, suggesting optimal stability and predictability for practical applications. The investigated flame is classified as a “cool flame,” and efficiency declines with richer mixtures due to increased exhaust heat loss and incomplete combustion. This work paves the way for further micro-combustor design optimization and unlocks the potential of non-premixed counterflow flames in diverse micro-scale applications.

EASI-Tex: Edge-Aware Mesh Texturing from Single Image

We present a novel approach for single-image mesh texturing , which employs a diffusion model with judicious conditioning to seamlessly transfer an object's texture from a single RGB image to a given 3D mesh object. We do not assume that the two objects belong to the same category, and even if they do, there can be significant discrepancies in their geometry and part proportions. Our method aims to rectify the discrepancies by conditioning a pre-trained Stable Diffusion generator with edges describing the mesh through ControlNet, and features extracted from the input image using IP-Adapter to generate textures that respect the underlying geometry of the mesh and the input texture without any optimization or training. We also introduce Image Inversion , a novel technique to quickly personalize the diffusion model for a single concept using a single image , for cases where the pre-trained IP-Adapter falls short in capturing all the details from the input image faithfully. Experimental results demonstrate the efficiency and effectiveness of our edge-aware single-image mesh texturing approach, coined EASI-Tex, in preserving the details of the input texture on diverse 3D objects, while respecting their geometry. Code : https://github.com/sairajk/easi-tex

Utilizing Silhouette and Head Information for Improved Cloth-changing Person Re-Identification

In recent years, significant progress has been made in person re-identification (ReID). However, as the time span increases, the phenomenon of changing clothes often occur, and the performance of methods based on person re-identification may decrease in this case. In this article, we focus on solving the problem of cloth-changing person re-identification from a single RGB image, which is more challenging, and improving the accuracy and robustness of traditional methods for recognition. We propose a cloth-changing Feature Regularization learning framework by fusing Silhouette and Head information (SHFR), a two-stream framework which transfers the silhouette information and head information learned by auxiliary stream to the main stream and supplements features unrelated to clothing. Specifically, the main stream approach we adopt is the traditional cloth-changing person re-identification network. In the auxiliary stream, we use the DeepLabV3 semantic segmentation model to extract person silhouette features, and the FCHD fully convolutional head detection model to extract head information with advanced semantic information. We concatenate the head information and background information into the silhouette as input for BackBone to continue training the network. In order to utilize silhouette information and head information, we regularize the cloth-changing features on both main stream and auxiliary stream, allowing main stream model to pay more attention to features unrelated to clothing and improve the accuracy of model recognition. We conduct extensive experiments on our model on the PRCC dataset, and the results show that our method has highly competitive performance.