Multi-scale Feature Extraction Module Research Articles

Mapping of Rubber Forest Growth Models Based on Point Cloud Data

The point cloud-based 3D model of forest helps to understand the growth and distribution pattern of trees, to improve the fine management of forestry resources. This paper describes the process of constructing a fine rubber forest growth model map based on 3D point clouds. Firstly, a multi-scale feature extraction module within the point cloud column is used to enhance the PointPillars learning capability. The Swin Transformer module is employed in the backbone to enrich the contextual semantics and acquire global features with the self-attention mechanism. All of the rubber trees are accurately identified and segmented to facilitate single-trunk localisation and feature extraction. Then, the structural parameters of the trunks calculated by RANSAC and IRTLS cylindrical fitting methods are compared separately. A growth model map of rubber trees is constructed. The experimental results show that the precision and recall of the target detection reach 0.9613 and 0.8754, respectively, better than the original network. The constructed rubber forest information map contains detailed and accurate trunk locations and key structural parameters, which are useful to optimise forestry resource management and guide the enhancement of mechanisation of rubber tapping.

Diabetic Plantar Foot Segmentation in Active Thermography Using a Two-Stage Adaptive Gamma Transform and a Deep Neural Network.

Pathological conditions in diabetic feet cause surface temperature variations, which can be captured quantitatively using infrared thermography. Thermal images captured during recovery of diabetic feet after active cooling may reveal richer information than those from passive thermography, but diseased foot regions may exhibit very small temperature differences compared with the surrounding area, complicating plantar foot segmentation in such cold-stressed active thermography. In this study, we investigate new plantar foot segmentation methods for thermal images obtained via cold-stressed active thermography without the complementary information from color or depth channels. To better deal with the temporal variations in thermal image contrast when planar feet are recovering from cold immersion, we propose an image pre-processing method using a two-stage adaptive gamma transform to alleviate the impact of such contrast variations. To improve upon existing deep neural networks for segmenting planar feet from cold-stressed infrared thermograms, a new deep neural network, the Plantar Foot Segmentation Network (PFSNet), is proposed to better extract foot contours. It combines the fundamental U-shaped network structure, a multi-scale feature extraction module, and a convolutional block attention module with a feature fusion network. The PFSNet, in combination with the two-stage adaptive gamma transform, outperforms multiple existing deep neural networks in plantar foot segmentation for single-channel infrared images from cold-stressed infrared thermography, achieving an accuracy of 97.3% and 95.4% as measured by Intersection over Union (IOU) and Dice Similarity Coefficient (DSC) respectively.

DRGAN: Dense Residual Generative Adversarial Network for Image Enhancement in an Underwater Autonomous Driving Device.

Underwater autonomous driving devices, such as autonomous underwater vehicles (AUVs), rely on visual sensors, but visual images tend to produce color aberrations and a high turbidity due to the scattering and absorption of underwater light. To address these issues, we propose the Dense Residual Generative Adversarial Network (DRGAN) for underwater image enhancement. Firstly, we adopt a multi-scale feature extraction module to obtain a range of information and increase the receptive field. Secondly, a dense residual block is proposed, to realize the interaction of image features and ensure stable connections in the feature information. Multiple dense residual modules are connected from beginning to end to form a cyclic dense residual network, producing a clear image. Finally, the stability of the network is improved via adjustment to the training with multiple loss functions. Experiments were conducted using the RUIE and Underwater ImageNet datasets. The experimental results show that our proposed DRGAN can remove high turbidity from underwater images and achieve color equalization better than other methods.

U-SMR: U-SwinT & multi-residual network for fabric defect detection

Fabric defect detection methods based on deep networks are widely used in the textile industry, but they often suffer from poor model generalization and blurry edge detection. To resolve these challenges, we propose a novel network called “U-SMR Net”, which integrates global contextual features, defect detail features, and high-level semantic features through the combination of ResNet-50 and Swin Transformer modules. Our U-SMR network includes a lightweight multiscale feature extraction module, the dual-branch pyramid Module (DBPM), which is nested to preserve high-resolution, shallow semantic information. We propose a recursive multi-level residual decoding block for multiscale fusion to refine, filter, and enhance input characteristics, generating prediction maps at multiple stages, and by employing an improved binary cross entropy loss function to supervise saliency mapping. The experimental results based on four groups from ZJU-Leaper dataset demonstrate the superior performance of our approach compared to other competitive methods by achieving an average fmeasure score of 75.33%, and finally testing results from both ZJU-Leaper-Total dataset and the HKU-Fabric dataset further support our U-SMR Net’s validity and generalization ability.

Contrastive learning with feature fusion for unpaired thermal infrared image colorization

Colorizing unpaired thermal infrared images is a challenging task that existing methods struggle to perform effectively, often resulting in blurry details and unclear textures. To address these issues, we propose a novel unpaired infrared image colorization method based on contrastive learning with feature fusion. Our proposed method designs the multi-scale feature extraction module, the expansion enhancement module, and the region-aware attention module in the generator. The multi-scale feature extraction module enlarges the model receptive field, achieving more comprehensive feature extraction and fusing feature information of varying scales. The expansion enhancement module enhances the network recognition ability, capturing more semantic contextual information and image details. The region-aware attention module enables the network to obtain more global and local information, adaptively focusing on the feature of image semantic regions. Furthermore, we introduce a new composite loss function that combines adversarial loss, contrastive loss, and perceptual loss, utilizing texture information extracted from the color image to further enhance the details in the generated colorized image and improve image perception. Extensive experimental results on the KAIST dataset and the FLIR dataset show that our proposed method maximally retains tiny optical details in infrared images, such as surface textures, edge contours, and small targets, making the generated color images physically highly consistent with the Ground Truth, with rich and realistic details.

A histopathological image classification method for cholangiocarcinoma based on spatial-channel feature fusion convolution neural network.

Histopathological image analysis plays an important role in the diagnosis and treatment of cholangiocarcinoma. This time-consuming and complex process is currently performed manually by pathologists. To reduce the burden on pathologists, this paper proposes a histopathological image classification method for cholangiocarcinoma based on spatial-channel feature fusion convolutional neural networks. Specifically, the proposed model consists of a spatial branch and a channel branch. In the spatial branch, residual structural blocks are used to extract deep spatial features. In the channel branch, a multi-scale feature extraction module and some multi-level feature extraction modules are designed to extract channel features in order to increase the representational ability of the model. The experimental results of the Multidimensional Choledoch Database show that the proposed method performs better than other classical CNN classification methods.

An Improved GAN-Based Image Restoration Method for Imaging Logging Images

An improved GAN-based imaging logging image restoration method is presented in this paper for solving the problem of partially missing micro-resistivity imaging logging images. The method uses FCN as the generative network infrastructure and adds a depth-separable convolutional residual block to learn and retain more effective pixel and semantic information; an Inception module is added to increase the multi-scale perceptual field of the network and reduce the number of parameters in the network; and a multi-scale feature extraction module and a spatial attention residual block are added to combine the channel attention. The multi-scale module adds a multi-scale feature extraction module and a spatial attention residual block, which combine the channel attention mechanism and the residual block to achieve multi-scale feature extraction. The global discriminative network and the local discriminative network are designed to gradually improve the content and semantic structure coherence between the restored parts and the whole image by playing off each other and the generative network. According to the experimental results, the average structural similarity measure of the five sets of imaged logging images with different sizes of missing regions in the test set is 0.903, which is an improvement of about 0.3 compared with other similar methods. It is shown that the method in this thesis can be used for the restoration of micro-resistivity imaging log images with good improvement in semantic structural coherence and texture details, thus providing a new deep learning method to ensure the smooth advancement of the subsequent interpretation of micro-resistivity imaging log images.

A deep learning model for steel surface defect detection

Industrial defect detection is a hot topic in the field of computer vision. It is a challenging task due to complex features and many categories of industrial defects. In this paper, a deep learning model based on the multiscale feature extraction module is introduced for steel surface defect detection. The main focus on the feature extraction capability of the model and feature fusion capability to improve the accuracy of the model for steel surface defect detection. First, to improve the feature extraction ability of the model, a multiscale feature extraction (MSFE) module is introduced. The MSFE module can effectively extract multiscale features through three branches that have different convolution kernel sizes. Second, an efficient feature fusion (EFF) module is proposed to optimize feature fusion by adding features from the backbone network to the neck network. Third, this paper puts forward a new Bottleneck module by reducing the normalization layer and activation function in the original Bottleneck module. Finally, the backbone network is deepened to further enhance the feature extraction ability of the model. Extensive experiments are conducted on the public NEU-DET dataset. The experimental results validate the effectiveness of the designed modules and the proposed model. Compared with other state-of-the-art methods, the proposed model achieves optimal accuracy(73.08% mAP@0.5) while maintaining a small number of parameters.

Swin–MRDB: Pan-Sharpening Model Based on the Swin Transformer and Multi-Scale CNN

Pan-sharpening aims to create high-resolution spectrum images by fusing low-resolution hyperspectral (HS) images with high-resolution panchromatic (PAN) images. Inspired by the Swin transformer used in image classification tasks, this research constructs a three-stream pan-sharpening network based on the Swin transformer and a multi-scale feature extraction module. Unlike the traditional convolutional neural network (CNN) pan-sharpening model, we use the Swin transformer to establish global connections with the image and combine it with a multi-scale feature extraction module to extract local features of different sizes. The model combines the advantages of the Swin transformer and CNN, enabling fused images to maintain good local detail and global linkage by mitigating distortion in hyperspectral images. In order to verify the effectiveness of the method, this paper evaluates fused images with subjective visual and quantitative indicators. Experimental results show that the method proposed in this paper can better preserve the spatial and spectral information of images compared to the classical and latest models.

Multi-modality multi-scale cardiovascular disease subtypes classification using Raman image and medical history

Raman spectroscopy (RS) has been widely used for disease diagnosis, e.g., cardiovascular disease (CVD), owing to its efficiency and component-specific testing capabilities. A series of popular deep learning methods have recently been introduced to learn nuance features from RS for binary classifications and achieved outstanding performance than conventional machine learning methods. However, these existing deep learning methods still confront some challenges in classifying subtypes of CVD. For example, the nuance between subtypes is quite hard to capture and represent by intelligent models due to the chillingly similar shape of RS sequences. Moreover, medical history information is an essential resource for distinguishing subtypes, but they are underutilized. In light of this, we propose a multi-modality multi-scale model called M3S, which is a novel deep learning method with two core modules to address these issues. First, we convert RS data to various resolution images by the Gramian angular field (GAF) to enlarge nuance, and a two-branch structure is leveraged to get embeddings for distinction in the multi-scale feature extraction module. Second, a probability matrix and a weight matrix are used to enhance the classification capacity by combining the RS and medical history data in the multi-modality data fusion module. We perform extensive evaluations of M3S and found its outstanding performance on our in-house dataset, with accuracy, precision, recall, specificity, and F1 score of 0.9330, 0.9379, 0.9291, 0.9752, and 0.9334, respectively. These results demonstrate that the M3S has high performance and robustness compared with popular methods in diagnosing CVD subtypes.

Dynamic Neural Network Architecture Design for Predicting Remaining Useful Life of Dynamic Processes

The prediction of remaining useful life is critical in predictive health management. This is done to reduce the expenses associated with operation and maintenance by avoiding errors and failures in dynamic processes. Recently, the abilities of feature classification and automated extraction of neural networks in its convolutional forms have shown fascinating performance when used for estimating the remaining useful life of dynamic processes using deep learning structures. This was accomplished by putting these talents to the task of predicting how long the procedures would be beneficial. Existing network topologies, on the other hand, virtually entirely extract features at a single scale while neglecting important information at other sizes. Meanwhile, because of the architecture of a single network path, the comprehensiveness of the features discovered by these tools is limited. To address these concerns, the authors propose a network structure based on a feature fusion strategy on a parallel multiscale architecture. This structure is then utilized to compute the remaining useful life. This prototype is divided into two sections: the first is a multiscale feature extraction module designed to extract local information features, and the second is a causal convolution module designed to extract global information features by combining multi-layer causal convolution with average pooling. The multiscale feature extraction module is intended for the extraction of local information features, while the causal convolution module is intended for the extraction of global information features. Finally, the two distinct paths are joined to create a fully integrated layer. The simulations and results show that this method has the potential to improve the efficiency and accuracy of estimating the remaining useful life index. Furthermore, the advantages of the established strategy are shown by comparing the results obtained with those produced by applying cutting-edge techniques on a well-known data-set depicting a simulated turbofan engine.

Multi-scale, multi-dimensional binocular endoscopic image depth estimation network

During invasive surgery, the use of deep learning techniques to acquire depth information from lesion sites in real-time is hindered by the lack of endoscopic environmental datasets. This work aims to develop a high-accuracy three-dimensional (3D) simulation model for generating image datasets and acquiring depth information in real-time. Here, we proposed an end-to-end multi-scale supervisory depth estimation network (MMDENet) model for the depth estimation of pairs of binocular images. The proposed MMDENet highlights a multi-scale feature extraction module incorporating contextual information to enhance the correspondence precision of poorly exposed regions. A multi-dimensional information-guidance refinement module is also proposed to refine the initial coarse disparity map. Statistical experimentation demonstrated a 3.14% reduction in endpoint error compared to state-of-the-art methods. With a processing time of approximately 30fps, satisfying the requirements of real-time operation applications. In order to validate the performance of the trained MMDENet in actual endoscopic images, we conduct both qualitative and quantitative analysis with 93.38% high precision, which holds great promise for applications in surgical navigation.

Multi-Scale Cyclic Image Deblurring Based on PVC-Resnet

Aiming at the non-uniform blurring of image caused by optical system defects or external interference factors, such as camera shake, out-of-focus, and fast movement of object, a multi-scale cyclic image deblurring model based on a parallel void convolution-Resnet (PVC-Resnet) is proposed in this paper, in which a multi-scale recurrent network architecture and a coarse-to-fine strategy are used to restore blurred images. The backbone network is built based on Unet codec architecture, where a PVC-Resnet module designed by combinations of parallel dilated convolution and residual network is constructed in the encoder of the backbone network. The convolution receptive field is expanded with parallel dilated convolution to extract richer global features. Besides, a multi-scale feature extraction module is designed to extract the shallow features of different scale targets in blurred images, and then the extracted features are sent to the backbone network for feature refinement. The SSIM loss function and the L1 loss function are combined to construct the SSIM-L1 joint loss function for the optimization of the overall network to ensure that the image restoration at different stages can be optimized. The experimental results show that the average peak signal-to-noise ratio (PSNR) of the proposed model on different data sets is as high as 32.84 dB, and the structural similarity (SSIM) reaches 0.9235. and statistical structural similarity (Stat-SSIM) of 0.9249 on different datasets. Compared with other methods, the deblurred images generated by this method are superior to the methods proposed by Nah et al., Kupyn et al. and Cho S J et al., especially on the calibration board data set. The model proposed in this paper applies parallel dilated convolution and SSIM-L1 joint loss function to improve the performance of the network so that the edge and texture details of the restored image are clearer.

Research on the Impact of Green Technology Innovation on Enterprise Financial Information Management Based on Compound Neural Network

To enhance the early warning level of financial risks in enterprises, mitigate the financial risks arising from diverse adversities, and drive green technological innovation and sustainable development, this study proposes a financial risk prediction model (MS-BGRU) that amalgamates multi-scale convolution and two-way GRU. Firstly, a multi-scale feature extraction module is devised that assimilates financial information from various scales by leveraging hole convolution with distinct expansion rates. This assimilated information is then fused to obtain richer context information. Secondly, the BGRU network is employed to discern the sequence characteristics and time information of financial indicators. The empirical results showcase that the model proposed in this paper exhibits a high identification accuracy, surging up to 98.03%, which surpasses other benchmark models. The model can accurately prophesize the financial risk of enterprises and offer guidance to management decision-makers in averting financial risk.

Anomaly Detection of Underground Transmission-Line through Multiscale Mask DCNN and Image Strengthening

This study proposes an integrated framework to automatically detect anomalies and faults in underground transmission-line connectors (UTLCs) with thermal images because anomaly detection of underground transmission-line connectors (UTLCs) plays a critical role in power line risk management. The proposed framework features three key characteristics. First, the measured thermal images were preprocessed through z-score normalization and image strengthening. Z-score normalization improves the robustness of feature extraction for UTLCs even though noise exists in a thermal image, and image strengthening improves the accuracy of segmentation for UTLCs. Second, a preprocessed thermal image is segmented to detect UTLCs by addressing a multiscale mask deep convolutional neural network (MS mask DCNN). The MS mask DCNN effectively detects UTLCs, enabling anomaly detection only for pixels of UTLCs. Specifically, the multiscale feature extraction module enables the extraction of distinct features of UTLCs and environments, and the skip-layer fusion module concatenates distinct features from the feature extraction module. Furthermore, a half tensor is used to reduce computational resources but maintain the same segmentation accuracy, enhancing the feasibility of the proposed framework in field applications. Third, anomaly detection is performed by addressing the contour method and unsupervised clustering method of DBSCAN. The contour method compensates for the limits of the MS mask DCNN for real-world applications because the neural networks cannot secure perfect accuracy of 100% owing to a lack of sufficient training images and low computational resources. DBSCAN improves the accuracy of diagnosis and ensures robustness to eliminate noise from thermal reflection caused by low-emissivity objects. Field experiments with high-voltage UTLCs demonstrated the effectiveness of the proposed framework. Ablation studies also confirmed that the methods addressed in this study outperform other methods. The proposed framework with a novel automatic non-destructive patrol inspection system would decrease the risks of human casualties during the periodic operation and maintenance of UTLCs, which are currently the most critical concerns.

Industrial x-ray image enhancement network based on a ray scattering model.

X-ray images frequently have low contrast and lost edge features because of the complexity of objects, attenuation of reflected light, and scattering superposition of rays. Image features are frequently lost in traditional enhancement methods. In this paper, we use a ray scattering model to estimate coarsely clear images and an encoder-decoder network and multi-scale feature extraction module to add multi-scale and detail information to the images. To selectively emphasize useful features, a dual attention module and UnsharpMasking with learnable correction factors are used. The results of the experiments demonstrate that the method may significantly enhance the quality of x-ray images.

Gesture image recognition method based on DC-Res2Net and a feature fusion attention module

To extract decisive features from gesture images and solve the problem of information redundancy in the existing gesture recognition methods, we propose a new multi-scale feature extraction module named densely connected Res2Net (DC-Res2Net) and design a feature fusion attention module (FFA). Firstly, based on the new dimension residual network (Res2Net), the DC-Res2Net uses channel grouping to extract fine-grained multi-scale features, and dense connection has been adopted to extract stronger features of different scales. Then, we apply a selective kernel network (SK-Net) to enhance the representation of effective features. Afterwards, the FFA has been designed to remove redundant information in features by fusing low-level location features with high-level semantic features. Finally, experiments have been conducted to validate our method on the OUHANDS, ASL, and NUS-II datasets. The results demonstrate the superiority of DC-Res2Net and FFA, which can extract more decisive features and remove redundant information while ensuring high recognition accuracy and low computational complexity.

Recognition method of maize crop rows at the seedling stage based on MS-ERFNet model

It is crucial to carry out weeding-cultivating and pesticide application for maize during the seedling stage since weeds, diseases, and pests in the field might negatively impact the growth of maize at the seedling stage. Aiming at the problems of traditional machine vision methods to identify seedling maize crop rows which are easily affected by light and weeds, low recognition accuracy and poor real-time performance, an improved Multi-Scale Efficient Residual Factorized ConvNet (MS-ERFNet) model was proposed for the recognition of seedling maize crop rows. The camera was used to collect maize crop rows images at the seedling stage, which were expanded using image enhancement to create the corresponding data set. A Multi-Scale feature extraction module (MS module) was proposed, with 3 × 3 convolution as the main branch, and parallel connection with 1 × 1 convolution and 5 × 5 depth separable convolution, and then connecting it with two 3 × 3 asymmetric group convolutions in series to realize the extraction of multi-scale features of crop rows. Meanwhile, Recurrent Feature-Shift Aggregator (RESA) module was added to the ERFNet model to enhance the model's ability to extract objects with extensibility. Finally, the crop rows centerlines were fitted utilizing the Least Squares Method (LSM) after the crop row center points had been located and classified. On the test set of seedling maize crop rows, the recognition comparison experiments were carried out on the model of DeepLabv3+, Efficient Nerual Network (ENet), ERFNet, Fully Convolutional Network (FCN-8 s), U-Net, and MS-ERFNet. The experimental results showed that the mean intersection over union (mIoU) and the pixel accuracy (PA) of the MS-ERFNet model were 93.40% and 97.54%, respectively, which were higher than other models. The MS-ERFNet model has 4.68 M network parameters, and the segmentation speed is 27.94 frames/s, and the speed of processing for images recognition of seedling maize crop rows using the method in this paper is 15.26 frames/s, which satisfies the real-time requirements of maize crop rows recognition. The MS-ERFNet model proposed in this paper provides technical support for the realization of automatic field navigation of agricultural machines.

MS3Net: Multiscale stratified-split symmetric network with quadra-view attention for hyperspectral image classification

Recently, hyperspectral image (HSI) classification has become a promising research direction in remote sensing image processing. Many HSI classification methods have been proposed based on convolutional neural networks (CNNs) and attention mechanisms (AMs). However, most current CNN-based methods only consider extracting features at a single scale in HSI, which may ignore the delicate features of some objects. Moreover, present AMs primarily focus on one feature dimension, such as spatial or channel attention, while disregarding dimension interaction. To conquer the above issues, a novel multiscale stratified-split symmetric network with quadra-view attention, namely MS3Net, is proposed for HSI classification. Generally, the proposed MS3Net has a dual-stream symmetric pipeline, which can better extract HSI's spectral signatures and spatial features. Specifically, the proposed MS3Net consists of three modules: a multiscale feature extraction module, a feature enhancement module, and a feature fusion module. Firstly, a stratified-split module is designed to extract multiscale spectral and spatial features. In addition, to reduce the complexity of the model, we designed pseudo-3-D spectral and spatial convolution to replace the traditional 3-D convolution operation. Secondly, a novel quadra-view attention module is proposed, guiding the model to focus on important features from multiple dimensions. Finally, the selective kernel feature fusion module is introduced, which can dynamically integrate spectral and spatial features. Experimental results on four benchmark HSI datasets with different scenes and resolutions confirm the visual and quantitative superiority of the proposed MS3Net over the state-of-the-art related methods in this research direction.

A fault diagnosis for rolling bearing based on multilevel denoising method and improved deep residual network

Aiming at the problem that weak faults in rolling bearings make effective fault diagnosis difficult under strong noise, this paper proposes a multilevel denoising technology based on improved singular value decomposition (ISVD) and intrinsic timescale decomposition (ITD), combined with an improved deep residual network (ResNet), for fault diagnosis in rolling bearings. Firstly, the difference ratio (DR) index is introduced to optimize singular value decomposition, combined with ITD for multilevel denoising of strong noise signals. Effective fault information in bearing vibration signals is extracted and converted into grayscale images. Secondly, the multi-scale feature extraction module (MFE-Module) is introduced to enhance the feature extraction capability of ResNet, and the support vector machine (SVM) is used instead of the Softmax function to identify and classify the fault features. The experimental results indicate that, compared with other methods, the proposed method can more accurately realize the fault diagnosis of rolling bearings in strong noise environments.