RGB Images Research Articles

Tri-channel visualised malicious code classification based on improved ResNet

As malicious code attacks continue to evolve, attackers leverage techniques like packing and code obfuscation to generate numerous variants, challenging traditional detection methods. Addressing the limitations of current deep learning-based malicious code classification approaches in feature extraction and accuracy, this paper introduces an innovative RGB visualization detection method based on a hybrid multi-head attention mechanism. Initially, a feature representation method utilizing RGB images is introduced. This approach focuses on semantic relationships between a malware’s binary information, assembly details, and API data, generating images with richer textural information. This technique effectively uncovers the deep dependencies between the original and variant versions of malicious code, providing stronger support for subsequent classification tasks. Furthermore, to tackle the issues of malware encryption and obfuscation, a deep neural network framework is adopted, incorporating a modular design philosophy and integrating a multi-head attention mechanism. This design not only enhances the expressiveness of critical features but also helps the model better focus on key aspects of the malicious code, thereby improving classification accuracy. Through comparative experiments and in-depth analysis, the effectiveness and superiority of the proposed RGB visualization method and MSA-ResNet model in the field of malicious code variant classification are validated. The accuracy rates achieved on the Kaggle and DataCon datasets are 99.49% and 97.70%, respectively, representing significant improvements over other methods. This approach demonstrates strong generalization capabilities and resistance to obfuscation, offering a new and effective tool for malicious code detection.Graphical

Revolutionizing tomato disease detection in complex environments.

In the current agricultural landscape, a significant portion of tomato plants suffer from leaf diseases, posing a major challenge to manual detection due to the task's extensive scope. Existing detection algorithms struggle to balance speed with accuracy, especially when identifying small-scale leaf diseases across diverse settings. Addressing this need, this study presents FCHF-DETR (Faster-Cascaded-attention-High-feature-fusion-Focaler Detection-Transformer), an innovative, high-precision, and lightweight detection algorithm based on RT-DETR-R18 (Real-Time-Detection-Transformer-ResNet18). The algorithm was developed using a carefully curated dataset of 3147 RGB images, showcasing tomato leaf diseases across a range of scenes and resolutions. FasterNet replaces ResNet18 in the algorithm's backbone network, aimed at reducing the model's size and improving memory efficiency. Additionally, replacing the conventional AIFI (Attention-based Intra-scale Feature Interaction) module with Cascaded Group Attention and the original CCFM (CNN-based Cross-scale Feature-fusion Module) module with HSFPN (High-Level Screening-feature Fusion Pyramid Networks) in the Efficient Hybrid Encoder significantly enhanced detection accuracy without greatly affecting efficiency. To tackle the challenge of identifying challenging samples, the Focaler-CIoU loss function was incorporated, refining the model's performance throughout the dataset. Empirical results show that FCHF-DETR achieved 96.4% Precision, 96.7% Recall, 89.1% mAP (Mean Average Precision) 50-95 and 97.2% mAP50 on the test set, with a reduction of 9.2G in FLOPs (floating point of operations) and 3.6M in parameters. These findings clearly demonstrate that the proposed method improves detection accuracy and reduces computational complexity, addressing the dual challenges of precision and efficiency in tomato leaf disease detection.

Enhancing Detection of Pedestrians in Low-Light Conditions by Accentuating Gaussian–Sobel Edge Features from Depth Maps

Owing to the low detection accuracy of camera-based object detection models, various fusion techniques with Light Detection and Ranging (LiDAR) have been attempted. This has resulted in improved detection of objects that are difficult to detect due to partial occlusion by obstacles or unclear silhouettes. However, the detection performance remains limited in low-light environments where small pedestrians are located far from the sensor or pedestrians have difficult-to-estimate shapes. This study proposes an object detection model that employs a Gaussian–Sobel filter. This filter combines Gaussian blurring, which suppresses the effects of noise, and a Sobel mask, which accentuates object features, to effectively utilize depth maps generated by LiDAR for object detection. The model performs independent pedestrian detection using the real-time object detection model You Only Look Once v4, based on RGB images obtained using a camera and depth maps preprocessed by the Gaussian–Sobel filter, and estimates the optimal pedestrian location using non-maximum suppression. This enables accurate pedestrian detection while maintaining a high detection accuracy even in low-light or external-noise environments, where object features and contours are not well defined. The test evaluation results demonstrated that the proposed method achieved at least 1–7% higher average precision than the state-of-the-art models under various environments.

FCS-Net: Feather condition scoring of broilers based on dense feature fusion of RGB and thermal infrared images

Assessing the feather condition of broilers is crucial for monitoring the animal welfare status and detecting the occurrence of feather pecking activities. Currently, the feather condition of individual broilers is manually scored by trained experts. To provide a more objective and efficient tool for feather condition scoring, a novel deep learning-based model, named Feather Condition Scoring Network (FCS-Net), was proposed based on RGB and thermal infrared images. The FCS-Net model combined the ResNet18 architecture with the proposed Dense Feature Fusion (DFF) module, which can effectively learn the feature mapping relationship between RGB and thermal infrared images. Before inputting the images into the network, an image registration process was conducted to align the RGB and thermal infrared images. The results showed that the FCS-Net model had a good performance for feather condition scoring, with the Accuracy of 97.02%, the Precision of 96.99%, the Recall of 97.04%, the F1 of 97.01%, and the Inference speed of 15.34 fps. Compared to the ResNet18_RGB model, which only utilise RGB images, the FCS-Net model showed notable improvements in Accuracy by 4.02%, Precision by 3.90%, Recall by 4.08%, and F1 by 4.01%. Moreover, it was observed that the FCS-Net model focused more on the back region of the broilers through heatmap visualization. Furthermore, the algorithm was compared with six typical image recognition algorithms including VGG16, ResNet18, SE-ResNet18, DenseNet121, Mobilenet_V2, and Shufflenet_V2_x1_0, as well as the state-of-the-art (SOTA) feather condition assessment methods. The results showed that the FCS-Net model achieved better performance than the six algorithms and the SOTA feather condition assessment methods. This study provided a valuable reference for automated monitoring of feather condition scoring of broilers in smart farming.

Automatic extraction of geological discontinuities of a tunnel surface by integrating multiple features

In water conservancy, transportation, and mining projects, the timely acquisition of geological structural information from tunnels is critical in the analysis of engineering geological problems during the investigation and construction stages. The acquisition of comprehensive and accurate geological information from a tunnel surface remains challenging. This study provides an automatic extraction method for geological discontinuities on a tunnel surface by integrating 2D textural semantic features and 3D geological semantic features. A dense point cloud is generated using multiline parallel sequence images, after which the 3D geological semantic features, including the local geological attitude, are calculated. Through a virtual projection from 3D to 2D, the red, green, and blue (RGB) images and geological semantic images based on views of the interior umbrella arch and the sidewalls of the tunnel surface are obtained. The feature mapping between the 2D textural semantic features and the 3D geological semantic features is determined accordingly. The virtual RGB images and geological semantic images serve as dual inputs for ensemble learning for pixel block segmentation, and the output is a similarity probability tensor that describes the probability that each pixel will belong to its surrounding pixel blocks. The pixel blocks are clustered on the basis of pole and contour plots of their geological attitudes to extract geological discontinuities. Experiments were conducted to confirm and evaluate the feasibility and veracity of the proposed method. The developed method automatically extracts geological discontinuities of a tunnel surface and extends the scope of surveying and mapping through geological remote sensing.

Deep learning-based sow posture classifier using colour and depth images

Assessing sow posture is essential for understanding their physiological condition and helping farmers improve herd productivity. Deep learning-based techniques have proven effective for image interpretation, offering a better alternative to traditional image processing methods. However, distinguishing transitional postures such as sitting and kneeling is challenging with only conventional top-view RGB images. This study aimed to develop and compare different deep learning-based sow posture classifiers using different architectures and image types. Using Kinect v.2 cameras, RGB and depth images were collected from 9 sows housed individually in farrowing crates. A total of 26,362 images were manually labelled by posture: “standing”, “kneeling”, “sitting”, “ventral recumbency” and “lateral recumbency”. Different deep learning algorithms were developed to detect sow postures from three types of images: colour (RGB), depth (depth image transformed into greyscale), and fused (colour-depth composite images). Results indicated that the ResNet-18 model presented the best results and that including depth information improved the performance of all models tested. Depth and fused models achieved higher accuracies than the models using only RGB images. The best model used only depth images as input and presented an accuracy of 98.3 %. The mean precision and recall values were 97.04 % and 97.32 %, respectively (F1-score = 97.2 %). The study shows improved posture classification using depth images. Future research can improve model accuracy and speed by expanding the database, exploring fused methods and computational models, considering different breeds of sows, and incorporating more postures. These models can be integrated into computer vision systems to automatically characterise sow behavior.

Reconstructing hyperspectral images of textiles from a single RGB image utilizing the multihead self-attention mechanism

Hyperspectral images possess abundant information and play a pivotal role in enhancing the accuracy of color difference detection in textiles. However, traditional hyperspectral imaging methods necessitate costly equipment and intricate operational procedures. A novel deep learning model based on a multihead attention mechanism was proposed in this article to facilitate the extensive application of hyperspectral imaging technology in textile quality inspection. This model enabled the reconstruction of the hyperspectral information of plain weave textiles from a single RGB image. In this model, encoder-decoder architecture and pyramid pooling convolutional operations were employed to integrate multiscale features of plain weave cotton-linen textiles. This could capture details and contextual information in textile images more precisely, enhancing the accuracy of hyperspectral image reconstruction. Simultaneously, an attention mechanism was introduced to increase the model’s receptive field and improve its focus on key regions in the input image and feature maps. This resulted in a reduced weighting of redundant information during network learning, leading to an improved feature extraction capability of the network. Through these methods, successful reconstructions of plain weave textiles hyperspectral information from a single RGB image was achieved. Quantitative and qualitative tests were conducted on two datasets, namely, the NTIRE 2020 dataset and a self-made textile dataset, to evaluate the performance of the proposed method. The approach proposed in this article exhibited promising results on both datasets. Specifically, the reconstructed textile hyperspectral images achieved a root mean square error of 0.0344, a peak signal-to-noise ratio of 29.945, a spectral angle mapper of 3.753, and a structural similarity index measure of 0.955 on the textile dataset. In the reconstructed hyperspectral colorimetric experiment, the maximum value of average color difference was 2.641. These results demonstrate that the method can meet the requirements for textile color measurement applications.

Self-adaptive 2D[sbnd]3D image fusion for automated pixel-level pavement crack detection

Current 2D and 3D image-based crack detection methods in transportation infrastructure often struggle with noise robustness and feature diversity. To overcome these challenges, the paper use CSF-CrackNet, a self-adaptive 2D3D image fusion model utilizes channel and spatial modules for automated pavement crack segmentation. CSF-CrackNet consists of four parts: feature enhanced and field sensing (FEFS) module, channel module, spatial module, and semantic segmentation module. A multi-feature image dataset was established using a vehicle-mounted 3D imaging system, including color images, depth images, and color-depth overlapped images. Results show that the mean intersection over union (mIOU) of most models under the CSF-CrackNet framework can be increased to above 80 %. Compared with original RGB and depth images, the average mIOU increases with image fusion by 10 % and 5 %, respectively. The ablation experiment and weight significance analysis further demonstrate that CSF-CrackNet can significantly improve semantic segmentation performance by balancing information between 2D and 3D images.

MoistNet: Machine vision-based deep learning models for wood chip moisture content measurement

Quick and reliable measurement of wood chip moisture content is an everlasting problem for numerous forest-reliant industries such as biofuel, pulp and paper, and bio-refineries. Moisture content is a critical attribute of wood chips due to its direct relationship with the final product quality. Conventional techniques for determining moisture content, such as oven-drying, possess some drawbacks in terms of their time-consuming nature, potential sample damage, and lack of real-time feasibility. Furthermore, alternative techniques, including NIR spectroscopy, electrical capacitance, X-rays, and microwaves, have demonstrated potential; nevertheless, they are still constrained by issues related to portability, precision, and the expense of the required equipment. Hence, there is a need for a moisture content determination method that is instant, portable, non-destructive, inexpensive, and precise. This study explores the use of deep learning and machine vision to predict moisture content classes from RGB images of wood chips. A large-scale image dataset comprising 1,600 RGB images of wood chips has been collected and annotated with ground truth labels, utilizing the results of the oven-drying technique. Two high-performing neural networks, MoistNetLite and MoistNetMax, have been developed leveraging Neural Architecture Search (NAS) and hyperparameter optimization. The developed models are evaluated and compared with state-of-the-art deep learning models. Results demonstrate that MoistNetLite achieves 87% accuracy with minimal computational overhead, while MoistNetMax exhibits exceptional precision with a 91% accuracy in wood chip moisture content class prediction. With improved accuracy (9.6% improvement in accuracy by MoistNetMax compared to the best baseline model ResNet152V2) and faster prediction speed (MoistNetLite being twice as fast as MobileNet), our proposed MoistNet models hold great promise for the wood chip processing industry to be efficiently deployed on portable devices, such as smartphones.

Attention-Driven Transfer Learning Model for Improved IoT Intrusion Detection

The proliferation of Internet of Things (IoT) devices has become inevitable in contemporary life, significantly affecting myriad applications. Nevertheless, the pervasive use of heterogeneous IoT gadgets introduces vulnerabilities to malicious cyber-attacks, resulting in data breaches that jeopardize the network’s integrity and resilience. This study proposes an Intrusion Detection System (IDS) for IoT environments that leverages Transfer Learning (TL) and the Convolutional Block Attention Module (CBAM). We extensively evaluate four prominent pre-trained models, each integrated with an independent CBAM at the uppermost layer. Our methodology is validated using the BoT-IoT dataset, which undergoes preprocessing to rectify the imbalanced data distribution, eliminate redundancy, and reduce dimensionality. Subsequently, the tabular dataset is transformed into RGB images to enhance the interpretation of complex patterns. Our evaluation results demonstrate that integrating TL models with the CBAM significantly improves classification accuracy and reduces false-positive rates. Additionally, to further enhance the system performance, we employ an Ensemble Learning (EL) technique to aggregate predictions from the two best-performing models. The final findings prove that our TL-CBAM-EL model achieves superior performance, attaining an accuracy of 99.93% as well as high recall, precision, and F1-score. Henceforth, the proposed IDS is a robust and efficient solution for securing IoT networks.

Evaluation of Sugarcane Crop Growth Monitoring Using Vegetation Indices Derived from RGB-Based UAV Images and Machine Learning Models

Crop monitoring with unmanned aerial vehicles (UAVs) has the potential to reduce field monitoring costs while increasing monitoring frequency and improving efficiency. However, the utilization of RGB-based UAV imagery for crop-specific monitoring, especially for sugarcane, remains limited. This work proposes a UAV platform with an RGB camera as a low-cost solution to monitor sugarcane fields, complementing the commonly used multi-spectral methods. This new approach optimizes the RGB vegetation indices for accurate prediction of sugarcane growth, providing many improvements in scalable crop-management methods. The images were captured by a DJI Mavic Pro drone. Four RGB vegetation indices (VIs) (GLI, VARI, GRVI, and MGRVI) and the crop surface model plant height (CSM_PH) were derived from the images. The fractional vegetation cover (FVC) values were compared by image classification. Sugarcane plant height predictions were generated using two machine learning (ML) algorithms—multiple linear regression (MLR) and random forest (RF)—which were compared across five predictor combinations (CSM_PH and four VIs). At the early stage, all VIs showed significantly lower values than later stages (p < 0.05), indicating an initial slow progression of crop growth. MGRVI achieved a classification accuracy of over 94% across all growth phases, outperforming traditional indices. Based on the feature rankings, VARI was the least sensitive parameter, showing the lowest correlation (r < 0.5) and mutual information (MI < 0.4). The results showed that the RF and MLR models provided better predictions for plant height. The best estimation results were observed withthe combination of CSM_PH and GLI utilizing RF model (R2 = 0.90, RMSE = 0.37 m, MAE = 0.27 m, and AIC = 21.93). This study revealed that VIs and the CSM_PH derived from RGB images captured by UAVs could be useful in monitoring sugarcane growth to boost crop productivity.

Integration of a novel 3D chaotic map with ELSS and novel cross-border pixel exchange strategy for secure image communication

This paper proposes a robust image encryption algorithm that utilizes a Novel three-dimensional (3D) Chaotic map and an Enhanced Logistic Sine System (ELSS). We leverage the unpredictability of 3D chaotic dynamics alongside the complexity of ELSS and DNA Sequence to forge a formidable image encryption scheme. Firstly, the image pixels are converted from decimal to hexadecimal notation and sorted in a 1D pixel array carrying a unique sequence of three channels of the RGB image. Secondly, the secret key is appended to XOR, the values with that 1-D pixels array. Thirdly, values are sorted by performing the binary right shift operation and encoded into DNA. Fourthly, a novel chaotic map is used to perform scrambling operations. Lastly, a novel enormous keyspace ELSS is used to perform efficient Border and Cross-Border (B &CB) pixel exchange, further enhancing the encryption quality of the proposed algorithm. Comprehensive security analysis proved that the proposed algorithm exhibits remarkable resilience against powerful known and chosen plaintext attacks and other prevalent cryptanalysis attacks, including differential attacks and exhaustive key search attacks. Henceforth, the proposed algorithm’s superior security and low computational cost make it an ideal choice for real-time secure image communication across various platforms, including satellite, multimedia, and military communications.

Deep learning-based detection of qanat underground water distribution systems using HEXAGON spy satellite imagery

Qanats are a remarkable type of ancient hydraulic structure for sustainable water distribution in arid environments that use subterranean channels to transport water from highland or mountainous areas. The presence of the qanat system is marked by a line of regularly spaced shafts visible from the surface, which can be used to detect qanats using satellite imagery. Typically, qanats have been documented by field mapping or manual digitisation within a Geographic Information System (GIS) environment. This process is time-consuming due to the numerous shafts within each qanat line. However, several automated methods for detecting qanat structures have been explored, using techniques such as morphological filters, custom convolutional neural networks (CNN) and, more recently, YOLOv5 and Mask R-CNN. These approaches used high-resolution RGB images and CORONA images. However, the use of black and white CORONA in CNNs has been limited in its applicability due to a high rate of false positives.This paper explores the potential of YOLOv9 in processing the black and white HEXAGON (KH-9) high-resolution spy satellite system launched in 1971. Two areas in Afghanistan (Maiwand) and Iran (Gorgan Plain) were selected to train the system images extracted from HEXAGON imagery and artificial synthetic data. The training dataset was augmented using the Albumentation library, which increased the number of tiles used. The model was tested using two types of HEXAGON imagery for selected areas in Afghanistan (Maiwand), Iran (Gorgan Plain) and Morocco (Rissani), and CORONA imagery in Iran (Gorgan Plain).Our study provided a model capable of predicting the location of qanat shafts with a precision of over 0.881 and a recall of 0.627 for most of the case studies tested. This is the first case study aimed at detecting qanats in different landscapes using different types of satellite imagery. Using real, augmented, and artificial data allowed us to generalise the representation of qanats into lineal groups of circular features. Thanks to applying labelling for individual qanats and their pairs as separate classes, our approach eliminated most of the isolated and clustered false positives.

Visualization-based comprehensive feature representation with improved EfficientNet for malicious file and variant recognition

Malicious file attacks seriously affect network and data security, and recognizing malicious files and variants is crucial for preventing network attacks. Faced with the challenge of traditional methods in quickly, effectively, and efficiently recognizing malicious files or variants, visualization-based feature representation methods have shown promising results. However, practical applications encounter issues such as loss of crucial information, high spatiotemporal overhead, and the need for model performance improvement. Therefore, this paper introduces a novel recognition framework focusing on feature representation and model performance. The framework uses the proposed visualization-based comprehensive feature representation method (VCFR) to extract file information into the Gray-Level Co-occurrence Matrix (GLCM), 2-gram frequency matrix, and interval 2-gram frequency matrix, followed by feature fusion to generate the three-channel RGB images. Subsequently, the proposed lightweight model is applied for recognizing those files, which utilizes ideas such as group convolution, channel shuffle, and attention mechanisms to improve model performance while significantly reducing model parameters, size, and FLOPs. In summary, through a series of experiments conducted on manually collected malicious file dataset (MFD) and public dataset MMCC, the proposed framework significantly outperformed other state-of-the-art technologies and has F1-Score as high as 94.10% and 98.58%, respectively, further verifying its outstanding effectiveness and efficiency.

IoTSLE: Securing IoT systems in low-light environments through finite automata, deep learning and DNA computing based image steganographic model

The Internet of Things (IoT) is a vast network of interconnected devices and systems, including wearables, smart home appliances, industrial machinery, and vehicles, equipped with sensors and connectivity. The data collected by the IoT devices are transmitted over a network, for processing and analyzing that data, so that appropriate actions can be initiated. Security of IoT systems is a major concern, as IoT devices collect and transmit crucial information. But images captured in low-light environments pose a challenge for IoT security by limiting the ability to accurately identify objects and people, increasing the risk of spoofing, and hindering forensic analysis. This paper unfolds a novel framework for IoT security using Steganography in Low-light Environment (IoTSLE) by image enhancement and data concealment. In proposed IoTSLE, initially, the low-light images, captured by the IoT devices in a low-light environment, are enhanced by band learning with recursion and band recomposition. After that, the secret information is concealed within the enhanced image. This concealment is supervised by using a specially designed finite automata for genome sequence encoding and 2-2-2 embedding. The proposed steganography technique is capable of hiding secret information within a 512 × 512 RGB image with the payload of 2097152 bits. The experiments like, PSNR, SSIM, Q-Index, BER, NCC, and NAE etc. are conducted to analyze the imperceptibility and security of IoTSLE. The proposed IoTSLE is useful for various IoT systems in different private and government fields like, defense agencies, digital forensics, agriculture, healthcare industry, cybersecurity firms, smart home, smart city etc.

PesRec: A parametric estimation method for indoor semantic scene reconstruction from a single image

Reconstructing semantic indoor scenes is a challenging task in augmented and virtual reality. The quality of scene reconstruction is limited by the complexity and occlusion of indoor scenes. This is due to the difficulty in estimating the spatial structure of the scene and insufficient learning for object location inference. To address these challenges, we have developed PesRec, an end-to-end multi-task scene reconstruction network for parameterizing indoor semantic information. PesRec incorporates a newly designed spatial layout estimator and a 3D object detector to effectively learn scene parameter features from RGB images. We modify an object mesh generator to enhance the robustness of reconstructing indoor occluded objects through point cloud optimization in PesRec. Using the analyzed scene parameters and spatial structure, the proposed PesRec reconstructs an indoor scene by placing object meshes scaled to 3D detection boxes in an estimated layout cuboid. Extensive experiments on two benchmark datasets demonstrate that PesRec performs exceptionally well for object reconstruction with an average chamfer distance of 5.24 × 10-3 on the Pix3D dataset including 53.61 % mAP for 3D object detection and 79.7 % 3D IoU for the estimation of layout on the commonly-used SUN RGB-D datasets. The proposed computing network breaks through the limitations caused by complex indoor scenes and occlusions, showing optimization results that improve the quality of reconstruction in the fields of augmented reality and virtual reality.

A lightweight color and geometry feature extraction and fusion module for end-to-end 6D pose estimation

Although advancements in red–green–blue-depth (RGB-D)-based six degree-of-freedom (6D) pose estimation methods, severe occlusion remains challenging. Addressing this issue, we propose a novel feature fusion module that can efficiently leverage the color and geometry information in RGB-D images. Unlike prior fusion methods, our method employs a two-stage fusion process. Initially, we extract color features from RGB images and integrate them into a point cloud. Subsequently, an anisotropic separable set abstraction network-like network is utilized to process the fused point cloud, extracting both local and global features, which are then combined to generate the final fusion features. Furthermore, we introduce a lightweight color feature extraction network to reduce model complexity. Extensive experiments conducted on the LineMOD, Occlusion LineMOD, and YCB-Video datasets conclusively demonstrate that our method significantly enhances prediction accuracy, reduces training time, and exhibits robustness to occlusion. Further experiments show that our model is significantly smaller than the latest popular 6D pose estimation models, which indicates that our model is easier to deploy on mobile platforms.

VPA-Net: A visual perception assistance network for 3d lidar semantic segmentation

The semantic segmentation of 3D point clouds holds paramount importance in visual perception tasks of automatic driving, including obstacle avoidance, decision control, path planning, and map construction, etc. Multi-sensor fusion is a rational and pivotal technique for implementing LiDAR semantic segmentation. However, effectively fusing and utilizing multi-source data remains a challenging task. In this work, we introduce a novel visual perception-assisted point cloud segmentation network, termed VPA-Net. This network architecture employs a dual-branch design to effectively combine spatial information from point clouds and visual cues from images, thereby bolstering the performance of 3D LiDAR semantic segmentation. More specifically, the dual-branch network structure processes and fuses multi-modal data from both point clouds and RGB images. Subsequently, the intermediate features from the two branches are merged via the proposed attention-based feature fusion module. Furthermore, to address the challenge of precise boundary prediction in large-scale point cloud scene segmentation, we introduce a refinement module based on 3D sparse convolution to enhance the spatial information of the LiDAR point clouds. The effectiveness of our method is validated on SemanticKITTI and a more challenging 3D semantic segmentation dataset. Experimental results underscore significant improvement on SemanticKITTI, with our approach surpassing the state-of-the-art method, achieving a 7.1% higher mIoU than SalsaNext.

Fruit classification by assessing slice hardness based on RGB imaging. Case study: apple slices

Fruit classification by assessing slice hardness based on RGB imaging. Case study: apple slices

CAFNet: Context aligned fusion for depth completion

Depth completion aims at reconstructing a dense depth from sparse depth input, frequently using color images as guidance. The sparse depth map lacks sufficient contexts for reconstructing focal contexts such as the shape of objects. The RGB images contain redundant contexts including details useless for reconstruction, which reduces the efficiency of focal context extraction. The unaligned contextual information from these two modalities poses a challenge to focal context extraction and further fusion, as well as the accuracy of depth completion. To optimize the utilization of multimodal contextual information, we explore a novel framework: Context Aligned Fusion Network (CAFNet). CAFNet comprises two stages: the context-aligned stage and the full-scale stage. In the context-aligned stage, CAFNet downsamples input RGB-D pairs to the scale, at which multimodal contextual information is adequately aligned for feature extraction in two encoders and fusion in CF modules. In the full-scale stage, feature maps with fused multimodal context from the previous stage are upsampled to the original scale and subsequentially fused with full-scale depth features by the GF module utilizing a dynamic masked fusion strategy. Ultimately, accurate dense depth maps are reconstructed, leveraging the GF module’s resultant features. Experiments conducted on indoor and outdoor benchmark datasets show that the CAFNet produces results comparable to state-of-the-art methods while effectively reducing computational costs.