Convolution Process Research Articles

Multi-loop graph convolutional network for multimodal conversational emotion recognition

Emotion recognition in conversations (ERC) has gained increasing research attention in recent years due to its wide applications in a surge of emerging tasks, such as social media analysis, dialog generation, and recommender systems. Since constituent utterances in a conversation are closely semantic-related, the constituent utterances’ emotional states are also closely related. In our consideration, this correlation could serve as a guide for the emotion recognition of constituent utterances. Accordingly, we propose a novel approach named Semantic-correlation Graph Convolutional Network (SC-GCN) to take advantage of this correlation for the ERC task in multimodal scenario. Specifically, we first introduce a hierarchical fusion module to model the dynamics among the textual, acoustic and visual features and fuse the multimodal information. Afterward, we construct a graph structure based on the speaker and temporal dependency of the dialog. We put forward a novel multi-loop architecture to explore the semantic correlations by the self-attention mechanism and enhance the correlation information via multiple loops. Through the graph convolution process, the proposed SC-GCN finally obtains a refined representation of each utterance, which is used for the final prediction. Extensive experiments are conducted on two benchmark datasets and the experimental results demonstrate the superiority of our SC-GCN.

A new approach to COVID-19 data mining: A deep spatial–temporal prediction model based on tree structure for traffic revitalization index

The outbreak of the COVID-19 epidemic has had a huge impact on a global scale and its impact has covered almost all human industries. The Chinese government enacted a series of policies to restrict the transportation industry in order to slow the spread of the COVID-19 virus in early 2020. With the gradual control of the COVID-19 epidemic and the reduction of confirmed cases, the Chinese transportation industry has gradually recovered. The traffic revitalization index is the main indicator for evaluating the degree of recovery of the urban transportation industry after being affected by the COVID-19 epidemic. The prediction research of traffic revitalization index can help the relevant government departments to know the state of urban traffic from the macro level and formulate relevant policies. Therefore, this study proposes a deep spatial–temporal prediction model based on tree structure for the traffic revitalization index. The model mainly includes spatial convolution module, temporal convolution module and matrix data fusion module. The spatial convolution module builds a tree convolution process based on the tree structure that can contain directional features and hierarchical features of urban nodes. The temporal convolution module constructs a deep network for capturing temporal dependent features of the data in the multi-layer residual structure. The matrix data fusion module can perform multi-scale fusion of COVID-19 epidemic data and traffic revitalization index data to further improve the prediction effect of the model. In this study, experimental comparisons between our model and multiple baseline models are conducted on real datasets. The experimental results show that our model has an average improvement of 21%, 18%, and 23% in MAE, RMSE and MAPE indicators, respectively.

Identification of Grape Diseases Based on Improved YOLOXS

Here we proposed a grape disease identification model based on improved YOLOXS (GFCD-YOLOXS) to achieve real-time detection of grape diseases in field conditions. We build a dataset of 11,056 grape disease images in 15 categories, based on 2566 original grape disease images provided by the State Key Laboratory of Plant Pest Biology data center after pre-processing. To improve the YOLOXS algorithm, first, the FOCUS module was added to the backbone network to reduce the lack of information related to grape diseases in the convolution process so that the different depth features in the backbone network are fused. Then, the CBAM (Convolutional Block Attention Module) was introduced at the prediction end to make the model focus on the key features of grape diseases and mitigate the influence of the natural environment. Finally, the double residual edge was introduced at the prediction end to prevent degradation in the deep network and to make full use of the non-key features. Compared with the experimental results of relevant authoritative literature, GFCD-YOLOXS had the highest identification accuracy of 99.10%, indicating the superiority of the algorithm in this paper.

Multilayer Convolutional Processing Network Based Cryptography Mechanism for Digital Images Infosecurity

Digital images can be easily shared or stored using different imaging devices, storage tools, and computer networks or wireless communication systems. However, these digital images, such as headshots or medical images, may contain private information. Hence, to protect the confidentiality, reliability, and availability of digital images on online processing applications, it is crucial to increase the infosecurity of these images. Therefore, an authorization encryption scheme should ensure a high security level of digital images. The present study aimed to establish a multilayer convolutional processing network (MCPN)-based cryptography mechanism for performing two-round image encryption and decryption processes. In the MCPN layer, two-dimensional (2D) spatial convolutional operations were used to extract the image features and perform scramble operations from grayscale to gray gradient values for the first-image encryption and second-image decryption processes, respectively. In the MCPN weighted network, a sine-power chaotic map (SPCM)-based key generator was used to dynamically produce the non-ordered pseudorandom numbers to set the network-weighted values as secret keys in a sufficiently large key space. It performs the second and first encryption processes using the diffusion method, modifying the image pixel values. Children’s headshots and medical images were used to evaluate the security level by comparing the plain and cipher images using the information entropy, number of pixel change rate, and unified averaged changed intensity indices. Moreover, the plain and decrypted images were compared to verify the decrypted image quality using the structural similarity index measurement and peak signal-to-noise ratio.

Regularized Multi-Output Gaussian Convolution Process With Domain Adaptation.

Multi-output Gaussian process (MGP) has been attracting increasing attention as a transfer learning method to model multiple outputs. Despite its high flexibility and generality, MGP still faces two critical challenges when applied to transfer learning. The first one is negative transfer, which occurs when there exists no shared information among the outputs. The second challenge is the input domain inconsistency, which is commonly studied in transfer learning yet not explored in MGP. In this paper, we propose a regularized MGP modeling framework with domain adaptation to overcome these challenges. More specifically, a sparse covariance matrix of MGP is constructed using convolution process, where penalization terms are added to adaptively select the most informative outputs for knowledge transfer. To deal with the domain inconsistency, a domain adaptation method is proposed by marginalizing inconsistent features and expanding missing features to align the input domains among different outputs. Statistical properties of the proposed method are provided to guarantee the performance practically and asymptotically. The proposed framework outperforms state-of-the-art benchmarks in comprehensive simulation studies and one real case study of a ceramic manufacturing process. The results demonstrate the effectiveness of our method in dealing with both the negative transfer and the domain inconsistency.

A Hybrid Algorithm for Feature Selection and Classification

<p>With a recent spread of intelligent information systems, massive data collections with a lot of repeated and unintentional, unwanted interference oriented data are gathered and a huge feature set are being operated. Higher dimensional inputs, on the other hand, contain more correlated variables, which might have a negative impact on model performance. In our model a Hybrid method of selecting feature was developed by combining Binary Gravitational Search Particle Swarm Optimization (HBGSPSO) method with an Enhanced Convolution Neural Network Bidirectional Long Short Term Memory (ECNN-BiLSTM). In our proposed system, the Bidirectional Long Short Term Memory (BiLSTM) is introduced which extracts the hidden dynamic data and utilizes the memory cells to think of long-term historical data after the convolution process. In this paper, thirteen well-defined datasets are used from the machine learning database of UC Irvine to evaluate the efficiency of the proposed system. The experiments are conducted using K Nearest Neighbor (KNN) and Decision Tree (DT) which are used as classifiers to evaluate the outcome of selected features. The outcomes are contrasted and compared with the bio-enlivened calculations like Genetic Algorithm (GA), Grey Wolf Optimizer (GWO), and Optimization protocol using Particle Swarm Optimization (PSO).</p> <p> </p>

Interest-Aware Message-Passing Layer-Refined Graph Convolutional Network for Recommendation

Graph convolutional networks (GCNs) show great potential in recommendation applications, as they have excellent performance in propagation node information propagation and capturing high-order connectivity in user-item interaction graphs. However, in the current recommendation model based on GCN, incoming information from neighbors is aggregated during information propagation, and some of this information may be noisy due to negative information. Additionally, the over-smoothing problem occurs when the model layers are stacked too high. During the embedding learning of users in the graph convolution operation, an important factor is that high-order neighbor users with no interest are involved, leading to similar embedding for users with no interest. These issues can degrade the recommendation performance. To address these problems, this paper proposes a method called IMPLayerGCN. In this method, high-order graph convolution is performed within subgraphs, which are composed of users with similar interests and their interaction items. The higher-order graph convolution is carried out in the subgraph and the layer representation in the process of GCN re-information propagation and node update is refined. The convolution process uses a symmetrical matrix. This approach avoids the spread of negative information from higher-order neighbors to embedded learning.

Fault Diagnosis of Rotating Machinery Based on 1D–2D Joint Convolution Neural Network

Convolutional neural network has been widely used in fault diagnosis of mechanical devices. In particular, two-dimensional convolutional neural network requires manual selection of multi-scale transformation to transform vibration signal into two-dimensional structure. Although one-dimensional convolution neural network can directly use the vibration signal for convolution processing, it can't make full use of the nonlinear information in one-dimensional space. In order to make full use of the advantages of one-dimensional and two-dimensional convolutional neural networks, this paper develops a one-dimension in tandem with two-dimension joint convolutional neural network (1D-2D JCNN) for rotating machinery fault diagnosis. More specifically, one-dimensional convolution is employed to adaptively obtain the multi-scale feature vectors of the vibration signal, and these feature vectors are constructed into two-dimensional maps, and then these two-dimensional vectors are used as the input of the two-dimensional convolutions neural network. Take the cross-entropy loss function as the loss function, and use the error back propagation algorithm to optimize the filter parameters of the 1D-2D JCNN model to obtain the final fault diagnosis model. Using the motor bearing data set and the worm gearbox data set, the experimental results show the excellent classification performance of bearings and gears under different working conditions. The average diagnostic accuracy of 10 runs on CWRU bearing data set is 99.92%, and the variance is 3.96e-6. The average diagnostic accuracy of 10 runs on worm gear data set is 99.82%, and the variance is 7.82e-6. Compared with the traditional fault diagnosis model and the latest convolution neural network method, the 1D-2D JCNN shows better diagnosis performance.

Formula omitted]-Net: Medical image segmentation using efficient and effective deep supervision

Although the existing deep supervised solutions have achieved some great successes in medical image segmentation, they have the following shortcomings; (i) semantic difference problem: since they are obtained by very different convolution or deconvolution processes, the intermediate masks and predictions in deep supervised baselines usually contain semantics with different depth, which thus hinders the models’ learning capabilities; (ii) low learning efficiency problem: additional supervision signals will inevitably make the training of the models more time-consuming. Therefore, in this work, we first propose two deep supervised learning strategies, U-Net-Deep and U-Net-Auto, to overcome the semantic difference problem. Then, to resolve the low learning efficiency problem, upon the above two strategies, we further propose a new deep supervised segmentation model, called μ-Net, to achieve not only effective but also efficient deep supervised medical image segmentation by introducing a tied-weight decoder to generate pseudo-labels with more diverse information and also speed up the convergence in training. Finally, three different types of μ-Net-based deep supervision strategies are explored and a Similarity Principle of Deep Supervision is further derived to guide future research in deep supervised learning. Experimental studies on four public benchmark datasets show that μ-Net greatly outperforms all the state-of-the-art baselines, including the state-of-the-art deeply supervised segmentation models, in terms of both effectiveness and efficiency. Ablation studies sufficiently prove the soundness of the proposed Similarity Principle of Deep Supervision, the necessity and effectiveness of the tied-weight decoder, and using both the segmentation and reconstruction pseudo-labels for deep supervised learning.

합성곱 신경망과 소음 신호의 연속 웨이블릿 변환을 이용한 주행 노면 및 타이어의 분류

This paper presents the method of classifying the road surface type using tire-pavement interaction noise (TPIN) signal. TPIN refers to a noise caused by interactions between rolling tires and road surfaces and is measured using the rugged microphone installed in the wheel cover of tire. The road surface information is identified using continuous wavelets transform (CWT). CWT images for the measured TPIN are used as the input of Convolutional neural network (CNN). The CNN extracts the feature for road surface throughout convolution and pooling process and classifies the road surface type in fully connected neural network. Two road surfaces, snow road and asphalt road, are classified using a method that combines the CWT and CNN methods The results indicate an accuracy of over 97 %. Two different tires are used for the experiment. Results from the road classification can be used to control the braking systems of autonomous vehicle in future.

Review of deep learning: Convolutional Neural Network Algorithm

Delves into the advanced field of image processing based on the use of neural networks to automatically and efficiently improve the quality and detail of images. The thesis explains that convolutional neural networks are one of the types of deep neural networks, and they are specially designed to gain knowledge from big data and extract complex features and patterns found in images. The different layers of the network are discussed in detail, as they handle images incrementally and extract various attributes in each layer. The thesis also highlights the ability of CNN to detect, learn, and improve important details found in images through convolutional, filtering, and data aggregation processes. The proposed CNN model for image enhancement was developed and tested on both medical and normal images. The images were enhanced using the proposed model and compared with other models. Different quality metrics were used to evaluate the results. The results showed that the proposed model can significantly improve the quality of images. The thesis also explores the potential applications of CNN in various fields such as medicine, photography, and space imaging. The use of CNN in these fields can lead to improved diagnosis and treatment in medicine, better image quality in photography, and more accurate and detailed images in space imaging.

A two-stage downscaling hydrological modeling approach via convolutional conditional neural process and geostatistical bias correction

Hydrological modeling plays an indispensable role in many practical applications, such as hydrology, water resources assessment, and environmental survey. Downscaling approach is able to obtain high-resolution hydrological models according to a prior reference model. In this work, we propose a downscaling hydrological modeling approach based on convolutional conditional neural processes (ConvCNP) and a novel geostatistical bias correction, named ConvCNP-GBC. A two-stage downscaling modeling workflow is presented. In the model building stage, to reduce the training consumption of the ConvCNP, low-resolution conditioning data are used for training. In the model generating stage, the ConvCNP is used to reconstruct coarse-scale realizations based on upscaling sampled conditioning data and to generate high-resolution realizations by downscaling. Besides, a geostatistical bias correction strategy is presented to refine large-scale realizations with high-resolution conditioning data constraints. The experimental results confirm that ConvCNP-GBC can reproduce heterogeneous patterns by using coarse-scale conditioning data and characterize hydrological structures. The downscaled realizations also have a high spatial correspondence with the coarse-scale realizations, which provide fine subsurface heterogeneous structures. The proposed method can be easily extended to reservoir modeling, geophysical inversion, etc.

Improved U-Net3+ With Spatial–Spectral Transformer for Multispectral Image Reconstruction

Multispectral image reconstruction, which aims to recover a three-dimensional (3D) spatial–spectral signal from a two-dimensional measurement in a spectral camera based on ghost imaging via sparsity constraint (GISC), has been attracting much attention recently. However, faced with abundant 3D spectral data, the reconstruction quality cannot meet the visual requirements. Based on the robust data processing capability of deep learning, a novel network called SSTU-Net3+ is constructed by improving U-Net3+ with a spatial–spectral transformer (SST). To enhance the feature representation of images during reconstruction, mixed pooling modules and new convolution processes are proposed to improve the performance of the encoder and decoder, with U-Net3+ as the backbone. To boost the quality of reconstructed images, with split and concatenate (Concat) operations, we construct SST modules by exploiting both spatial and spectral correlations of multispectral images to refine the spatial and spectral features. Furthermore, we employ the SST in the decoder to reconstruct the desired 3D cube. Given similar network parameters, experiments on GISC spectral imaging data show that, compared to convolutional neural network–based methods, the average peak signal-to-noise ratio of images reconstructed using SSTU-Net3+ is improved by 3%, the structural similarity is enhanced by 3%, and the spectral angle mapping is cut by 12%. Particularly, compared to differential ghost imaging and compressed sensing, the reconstruction quality of SSTU-Net3+ has been significantly improved. SSTU-Net3+ can process a large amount of 3D multispectral image data more efficiently and construct the target image more accurately than the abovementioned methods.

Pixel-Coordinate-Induced Human Pose High-Precision Estimation Method

Accurately estimating human pose is crucial for providing feedback during exercises or musical performances, but the complex and flexible nature of human joints makes it challenging. Additionally, traditional methods often neglect pixel coordinates, which are naturally present in high-resolution images of the human body. To address this issue, we propose a novel human pose estimation method that directly incorporates pixel coordinates. Our method adds a coordinate channel to the convolution process and embeds pixel coordinates into the feature map, while also using coordinate attention to capture position- and structure-sensitive features. We further reduce the network parameters and computational cost by using small-scale convolution kernels and a smooth activation function in residual blocks. We evaluate our model on the MPII Human Pose and COCO Keypoint Detection datasets and demonstrate improved accuracy, highlighting the importance of directly incorporating coordinate location information in position-sensitive tasks.

Multiscale cascade recurrent dilation convolution network for fault diagnosis of rolling bearing under cross-load conditions

Recently, deep learning (DL) models based on convolutional neural networks have achieved satisfactory results in rolling bearing fault diagnosis. However, the bearings usually work in variable loading conditions, and their feature distribution could vary with load. The important features cannot be effectively captured in the convolution process using the existing diagnosis models, resulting in poor generalization performance. In this paper, a novel DL model, named multiscale cascade recurrent dilation convolution network, is proposed by introducing the dilated convolution and global average pooling (GAP) layer. Firstly, a new multiscale cascade structure with different convolution kernel sizes is introduced to extract multiscale features contained in the vibration signal. Secondly, a recurrent dilation convolution strategy is designed in each branch of the multiscale cascade structure to extract abundant feature information. Finally, the GAP is employed to reduce redundant feature vectors and output them, while a classifier of multilayer perceptron is used to automatically identify the fault types. The effectiveness of the proposed algorithm is evaluated by two experimental cases. The results show that the proposed method can successfully identify the labels of fault samples under unknown load conditions using the fault samples with labels under existing load conditions. Compared with other methods, this method exhibits excellent robustness and generalization performance for bearing fault diagnosis under cross-load conditions.

Scatter correction for cone-beam CT via scatter kernel superposition-inspired convolutional neural network

Objective. X-ray scatter leads to signal bias and degrades the image quality in Computed Tomography imaging. Conventional real-time scatter estimation and correction methods include the scatter kernel superposition (SKS) methods, which approximate x-ray scatter field as a convolution of the scatter sources and scatter propagation kernels to reflect the spatial spreading of scatter x-ray photons. SKS methods are fast to implement but generally suffer from low accuracy due to the difficulties in determining the scatter kernels. Approach. To address such a problem, this work describes a new scatter estimation and correction method by combining the concept of SKS methods and convolutional neural network. Unlike conventional SKS methods which estimate the scatter amplitude and the scatter kernel based on the value of an individual pixel, the proposed method generates the scatter amplitude maps and the scatter width maps from projection images through a neural network, from which the final estimated scatter field is calculated based on a convolution process. Main Results. By incorporating physics in the network design, the proposed method requires fewer trainable parameters compared with another deep learning-based method (Deep Scatter Estimation). Both numerical simulations and physical experiments demonstrate that the proposed SKS-inspired convolutional neural network outperforms the conventional SKS method and other deep learning-based methods in both qualitative and quantitative aspects. Significance. The proposed method can effectively correct the scatter-related artifacts with a SKS-inspired convolutional neural network design.

A time domain 2D OaA-based convolutional neural networks accelerator

Convolutional neural networks (CNNs) are widely implemented in modern facial recognition systems for image recognition applications. Runtime speed is a critical parameter for real-time systems. Traditional FPGA-based accelerations require either large on-chip memory or high bandwidth and high memory access time that slow down the network. The proposed work uses an algorithm and its subsequent hardware design for a quick CNN computation using an overlap-and-add-based technique in the time domain. In the algorithm, the input images are broken into tiles that can be processed independently without computing overhead in the frequency domain. This also allows for efficient concurrency of the convolution process, resulting in higher throughput and lower power consumption. At the same time, we maintain low on-chip memory requirements necessary for faster and cheaper processor designs. We implemented CNN VGG-16 and AlexNet models with our design on Xilinx Virtex-7 and Zynq boards. The performance analysis of our design provides 48% better throughput than the state-of-the-art AlexNet and uses 68.85% lesser multipliers and other resources than the state-of-the-art VGG-16.

On-Demand Reconfigurable Incoherent Optical Matrix Operator for Real-Time Video Image Display

For emerging application fields such as autonomous driving and computing vision, which demands the real-time response of massive information, advanced image processing techniques on artificial neural networks (ANNs) are crucial. However, with increasing task complexity, the electrical hardware limited by the well-known von Neumann bottleneck is inadequate to meet these extreme requirements in computing speed and computing power. We propose an incoherent optical matrix operator by dynamically manipulating the parameters of the lightwave in terms of the wavelength, time and space to implement the on-demand reconfiguration both in the optical perceptron and optical convolutional processor. For the optical perceptron, it is conducted to perform the case-sensitive classification of 52 English letters. Furthermore, by adjusting the spectral characteristics and output ports, an optical convolution processor is constructed to realize multitask convolution processing for both the image and real-time video. Based on multikernels parallel computing in an optical convolutional layer coupled with an electrically fully connected layer, a convolutional neural network is built to recognize ‘0–9’ handwritten digital images from the Modified National Institute of Standards and Technology (MNIST) database. Owing to the excellent performance of the incoherent beam in terms of the spectral purity, wavelength stability and side-mode suppression ratio, the experimental classification accuracy up to 96.01% greatly matches the theoretical value of 96.67%. Moreover, incoherent information processing is synchronous with data transmission, which releases the requirement of the global clock in digital counterparts. The high-fidelity incoherent convolution processor offers a novel computing framework that serves the next-generation ANN platform.

Capturing Dynamic Interests of Similar Users for POI Recommendation Using Self-Attention Mechanism

The integration of location-based social networks and POI recommendation systems has the potential to enhance the urban experience by facilitating the exploration of new and relevant locales. The deployment of graph neural networks (GNNs) drives the development of POI recommendations, but this approach also brings with it the challenge of over-smoothing, where information propagation between nodes in the graph can lead to an excessive homogenization of the data. In prior works that utilized GNNs for POI recommendation, the bipartite graphs constructed from users and POIs as nodes failed to incorporate temporal dynamics, limiting the scope of the analysis to only spatial structure information. To circumvent this issue, the incorporation of a temporal component can be introduced during the aggregation process of graph convolution. In light of these considerations, the present study proposes a novel regionalized temporal GCN (RST-GCN) recommendation model that leverages self-attention mechanism to capture various levels of temporal information to better reflect the dynamic changes of time. By combining the graph’s spatial structure with geospatial features, similar users are distributed into distinct regional subgraphs, effectively avoiding the influence of non-similar users. The efficacy of the proposed model has been demonstrated through empirical evaluations conducted on two real-world datasets.

Research on tree image retrieval method based on twin network multi feature fusion

In this paper, an image retrieval method based on twin network multi feature fusion is proposed to improve the retrieval accuracy of tree images to meet the detection requirements of power grid line tree obstacles. In the twin network, the color features, texture features and shape features of the reference image and test image are included in the SiamRPN module for convolution processing, and then the similarity is judged through UM module fusion. The experimental results show that this method can achieve accurate retrieval of coconut, banyan and other tree images, and the retrieval accuracy and robustness are significantly higher than other image retrieval methods.