Multi-channel Convolutional Neural Network Research Articles

Prediction of the tensile strength of friction stir welded joints based on one-dimensional convolutional neural network

Friction stir welding (FSW) is a complex thermo-mechanical coupling process. Tensile strength is an important evaluation index of the mechanical properties of welded joints. How to realize the real-time prediction of tensile strength of the friction stir welded joints to reflect the dynamic change of welding state is a problem in the field. To solve this problem, this paper presents a multi-scale one-dimensional convolutional neural network (Multi-scale 1D CNN) prediction model using time series data of temperature and axial force as inputs to realize the online prediction of tensile strength of welded joints. Firstly, FSW experiments are carried out to obtain time series data of temperature and axial force. Tensile strength values of the welded joints is obtained by tensile tests. The time series data and tensile strength values are fused as a dataset. Then Multi-scale 1D CNN, traditional 1D CNN and Multi-channel 1D CNN prediction models are established and trained with the dataset, respectively. Finally, by comparing the prediction performance of the three models, Multi-scale 1D CNN is proved to be more suitable for analyzing time series data to feedback the dynamic change of tensile strength of the joints during welding.

An attention‐based deep learning model for credibility assessment of online health information

AbstractWith the surge of searching and reading online health‐based articles, maintaining the quality and credibility of online health‐based articles has become crucial. The circulation of deceptive health information on numerous social media sites can mislead people and can potentially cause adverse effects on people's health. To address these problems, this work uses deep learning approaches to automate the assessment and scoring of online health‐related articles' credibility. The paper proposed an Attention‐based Recurrent Multichannel Convolutional Neural Network (ARMCNN) model. The proposed model incorporates a BiLSTM layer, a multichannel CNN layer, and an attention layer and predicts the credibility of online health information. To perform a reliable evaluation of the presented model, we utilize the health articles reviewed by the experts, compiled in a labeled dataset termed “Pubhealth,” which consists of thousands of health articles. The results are evaluated using five performance measures, accuracy, precision, recall, f1‐score, and area under the ROC curve (AUC). Furthermore, we extensively compared the proposed model with different deep learning and machine learning models such as Long short‐term memory (LSTM), Bidirectional LSTM, CNN (Convolutional neural network), and RNN‐CNN. The experimental results showed that the proposed model produced state‐of‐the‐art performance on the used dataset by achieving an accuracy of 0.88, precision of 0.92, recall of 0.87, f1‐score of 0.90, and AUC of 0.94. Further, the proposed model yielded better performance than other benchmarked techniques for the credibility assessment of online health articles.

BBC-FND: An ensemble of deep learning framework for textual fake news detection

A wide spread of false news over Online Social Network platforms (OSNs) causes numerous negative consequences. Several researchers proposed different models using machine learning (ML) and deep learning (DL) techniques to detect fake news. The performance analysis of these models exposed that they achieve low accuracy due to the major drawbacks like improper architectural design, unsuitability of the model on different datasets. To address these issues, a hybrid BERT-BiLSTM-CNN (BBC-FND) model is proposed. It comprises of three main layers: an embedding layer, a feature representation layer, and a classification layer. In the first layer, BERT is used to extract the contextual-dependent features among words from the news. A multichannel CNN and a stacked BiLSTM are employed to produce various key features in the feature extraction layer. The former extracts multiple features from the text and captures complex local patterns in word spatial relations, while the latter extracts global semantic features from the contextual feature vector. The resulting features are concatenated and fed as input to the FFN to predict the news as fake or real. The results exhibit that the BBC-FND model outperforms the SoTA techniques, achieving higher accuracy of 97.31%, 98.64%, 99.06% and 98.26% on four datasets respectively.

Deep learning-based local SAR prediction using B1 maps and structural MRI of the head for parallel transmission at 7 T.

To predict subject-specific local specific absorption rate (SAR) distributions of the human head for parallel transmission (pTx) systems at 7 T. Electromagnetic energy deposition in tissues is nonuniform at 7 T, and interference patterns due to individual channels of pTx systems may result in increased local SAR values, which can only be estimated with very high safety margins. We proposed, designed, and demonstrated a multichannel 3D convolutional neural network (CNN) architecture to predict local SAR maps as well as peak-spatial SAR (ps-SAR) levels. We hypothesized that utilizing a three-channel 3D CNN, in which each channel is fed by a map, a phase-reversed map, and an MR image, would improve prediction accuracies and decrease uncertainties in the predictions. We generated 10 new head-neck body models, along with 389 3D pTx MRI data having different RF shim settings, with their B1 and local SAR maps to support efforts in this field. The proposed three-channel 3D CNN predicted ps-SAR10g levels with an average overestimation error of 20%, which was better than the virtual observation points-based estimation error (i.e., 152% average overestimation). The proposed method decreased prediction uncertainties over 20% (i.e., 22.5%-17.7%) compared to other methods. A safety factor of 1.20 would be enough to avoid underestimations for the dataset generated in this work. Multichannel 3D CNN networks can be promising in predicting local SAR values and perform predictions within a second, making them clinically useful as an alternative to virtual observation points-based methods.

Ideal ratio mask estimation based on cochleagram for audio-visual monaural speech enhancement

In this paper, the estimation of the ideal ratio mask (IRM) has been carried out based on speech cochleagram and visual cues using Audio-Visual Multichannel Convolutional Neural Network (AVMCNN) to enhance the speech signal. Recently several researchers have shown that speech enhancement using visual data as an additional input along with the audio data is more effective in minimizing the acoustic noise present in the speech signal. This work proposes a novel CNN-based audio-visual IRM estimation model. In the proposed audio-visual IRM estimation model, the dynamics of both audio and visual signal features are extracted using multichannel CNN and contextually combined for speech enhancement. The enhanced speech obtained using the proposed model is evaluated based on speech quality and intelligibility. The evaluation results signify that the proposed audio-visual mask estimation model shows improved performance over the audio-only, visual-only, and existing audio-visual mask estimation models. In turn, the proposed AVMCNN model proves its effectiveness in combining the dynamics of the audio features with the visual speech features for speech enhancement.

PhysVENeT: a physiologically-informed deep learning-based framework for the synthesis of 3D hyperpolarized gas MRI ventilation

Functional lung imaging modalities such as hyperpolarized gas MRI ventilation enable visualization and quantification of regional lung ventilation; however, these techniques require specialized equipment and exogenous contrast, limiting clinical adoption. Physiologically-informed techniques to map proton (1H)-MRI ventilation have been proposed. These approaches have demonstrated moderate correlation with hyperpolarized gas MRI. Recently, deep learning (DL) has been used for image synthesis applications, including functional lung image synthesis. Here, we propose a 3D multi-channel convolutional neural network that employs physiologically-informed ventilation mapping and multi-inflation structural 1H-MRI to synthesize 3D ventilation surrogates (PhysVENeT). The dataset comprised paired inspiratory and expiratory 1H-MRI scans and corresponding hyperpolarized gas MRI scans from 170 participants with various pulmonary pathologies. We performed fivefold cross-validation on 150 of these participants and used 20 participants with a previously unseen pathology (post COVID-19) for external validation. Synthetic ventilation surrogates were evaluated using voxel-wise correlation and structural similarity metrics; the proposed PhysVENeT framework significantly outperformed conventional 1H-MRI ventilation mapping and other DL approaches which did not utilize structural imaging and ventilation mapping. PhysVENeT can accurately reflect ventilation defects and exhibits minimal overfitting on external validation data compared to DL approaches that do not integrate physiologically-informed mapping.

A Holistic Approach to Power Systems Using Innovative Machine Learning and System Dynamics

The digital revolution requires greater reliability from electric power systems. However, predicting the growth of electricity demand is challenging as there is still much uncertainty in terms of demographics, industry changes, and irregular consumption patterns. Machine learning has emerged as a powerful tool, particularly with the latest developments in deep learning. Such tools can predict electricity demand and, thus, contribute to better decision-making by energy managers. However, it is important to recognize that there are no efficient methods for forecasting peak demand growth. In addition, features that add complexity, such as climate change and economic growth, take time to model. Therefore, these new tools can be integrated with other proven tools that can be used to model specific system structures, such as system dynamics. This research proposes a unique framework to support decision-makers in dealing with daily activities while attentively tracking monthly peak demand. This approach integrates advances in machine learning and system dynamics. This integration has the potential to contribute to more precise forecasts, which can help to develop strategies that can deal with supply and demand variations. A real-world case study was used to comprehend the needs of the environment and the effects of COVID-19 on power systems; it also helps to demonstrate the use of leading-edge tools, such as convolutional neural networks (CNNs), to predict electricity demand. Three well-known CNN variants were studied: a multichannel CNN, CNN-LSTM, and a multi-head CNN. This study found that the multichannel CNN outperformed all the models, with an R2 of 0.92 and a MAPE value of 1.62% for predicting the month-ahead peak demand. The multichannel CNN consists of one main model that processes four input features as a separate channel, resulting in one feature map. Furthermore, a system dynamics model was introduced to model the energy sector’s dynamic behavior (i.e., residential, commercial, and government demands, etc.). The calibrated model reproduced the historical data curve fairly well between 2005 and 2017, with an R2 value of 0.94 and a MAPE value of 4.8%.

Multi-domain-fusion deep learning for automatic modulation recognition in spatial cognitive radio

Automatic modulation recognition (AMR) is a critical technology in spatial cognitive radio (SCR), and building high-performance AMR model can achieve high classification accuracy of signals. AMR is a classification problem essentially, and deep learning has achieved excellent performance in various classification tasks. In recent years, joint recognition of multiple networks has become increasingly popular. In complex wireless environments, there are multiple signal types and diversity of characteristics between different signals. Also, the existence of multiple interference in wireless environment makes the signal characteristics more complex. It is difficult for a single network to accurately extract the unique features of all signals and achieve accurate classification. So, this article proposes a time–frequency domain joint recognition model that combines two deep learning networks (DLNs), to achieve higher accuracy AMR. A DLN named MCLDNN (multi-channel convolutional long short-term deep neural network) is trained on samples composed of in-phase and quadrature component (IQ) signals, to distinguish modulation modes that are relatively easy to identify. This paper proposes a BiGRU3 (three-layer bidirectional gated recurrent unit) network based on FFT as the second DLN. For signals with significant similarity in the time domain and significant differences in the frequency domain that are difficult to distinguish by the former DLN, such as AM-DSB and WBFM, FFT (Fast Fourier Transform) is used to obtain frequency domain amplitude and phase (FDAP) information. Experiments have shown that the BiGUR3 network has superior extraction performance for amplitude spectrum and phase spectrum features. Experiments are conducted on two publicly available datasets, the RML2016.10a and RML2016.10b, and the results show that the overall recognition accuracy of the proposed joint model reaches 94.94% and 96.69%, respectively. Compared to a single network, the recognition accuracy is significantly improved. At the same time, the recognition accuracy of AM-DSB and WBFM signals has been improved by 17% and 18.2%, respectively.

SBI-DHGR: Skeleton-based intelligent dynamic hand gestures recognition

Hand gesture recognition (HGR) plays a significant role in interpreting the meaning of sign language, human–computer interaction, and robot control. This paper proposes a real-time skeleton-based intelligent dynamic hand gesture recognition (SBI-DHGR) approach, which comprises three modules: Palm Centroid (PC), Data augmentation with the Tenet Effect, and Deep learning architecture. The palm Centroid (PC) module is introduced to identify the proposed palm center joint. Data augmentation with a tenet effect module is designed to improve the CNN model’s generalizability. Further, a novel deep learning model is proposed for temporal 3D-HGR by exploiting the capabilities of a multi-channel convolutional neural network (CNN) and long short-term memory (LSTM) recurrent network. The multi-channel CNN is introduced to learn the cardinal positions of hand joints by capturing the low, average, and high-level features. LSTM is embedded to learn the temporal characteristics of hand joints. The effectiveness of the SBI-DHGR framework is evaluated over five challenging datasets: SHREC-14, SHREC-28, DHG-14, DHG-28, and FPHA, by adopting person-dependent and person-independent validation setups.

Quantitative susceptibility mapping using multi-channel convolutional neural networks with dipole-adaptive multi-frequency inputs.

Quantitative susceptibility mapping (QSM) quantifies the distribution of magnetic susceptibility and shows great potential in assessing tissue contents such as iron, myelin, and calcium in numerous brain diseases. The accuracy of QSM reconstruction was challenged by an ill-posed field-to-susceptibility inversion problem, which is related to the impaired information near the zero-frequency response of the dipole kernel. Recently, deep learning methods demonstrated great capability in improving the accuracy and efficiency of QSM reconstruction. However, the construction of neural networks in most deep learning-based QSM methods did not take the intrinsic nature of the dipole kernel into account. In this study, we propose a dipole kernel-adaptive multi-channel convolutional neural network (DIAM-CNN) method for the dipole inversion problem in QSM. DIAM-CNN first divided the original tissue field into high-fidelity and low-fidelity components by thresholding the dipole kernel in the frequency domain, and it then inputs the two components as additional channels into a multichannel 3D Unet. QSM maps from the calculation of susceptibility through multiple orientation sampling (COSMOS) were used as training labels and evaluation reference. DIAM-CNN was compared with two conventional model-based methods [morphology enabled dipole inversion (MEDI) and improved sparse linear equation and least squares (iLSQR) and one deep learning method (QSMnet)]. High-frequency error norm (HFEN), peak signal-to-noise-ratio (PSNR), normalized root mean squared error (NRMSE), and the structural similarity index (SSIM) were reported for quantitative comparisons. Experiments on healthy volunteers demonstrated that the DIAM-CNN results had superior image quality to those of the MEDI, iLSQR, or QSMnet results. Experiments on data with simulated hemorrhagic lesions demonstrated that DIAM-CNN produced fewer shadow artifacts around the bleeding lesion than the compared methods. This study demonstrates that the incorporation of dipole-related knowledge into the network construction has a potential to improve deep learning-based QSM reconstruction.

Semantic feature learning for software defect prediction from source code and external knowledge

Software defects not only reduce operational reliability but also significantly increase overall maintenance costs. Consequently, it is necessary to predict software defects at an early stage. Existing software defect prediction studies work with artificially designed metrics or features extracted from source code by machine learning-based approaches to perform classification. However, these methods fail to make full use of the defect-related information other than code, such as comments in codes and commit messages. Therefore, in this paper, additional information extracted from natural language text is combined with the programming language codes to enrich the semantic features. A novel model based on Transformer architecture and multi-channel CNN, PM2-CNN, is proposed for software defect prediction. Pretrained language model and CNN-based classifier are utilized in the model to obtain context-sensitive representations and capture the local correlation of sequences. A large and widely used dataset is utilized to verify the effectiveness of the proposed method. The results show that the proposed method has improvements in generic evaluation metrics compared with the optimal baseline method. Accordingly, external information can have a positive impact on software defect prediction, and our model effectively incorporates such information to improve detection performance.

A Multichannel Convolutional Neural Network for Multilabel Sentiment Classification Using Abilify Oral User Reviews

Nowadays, patients and caregivers have become very active in social media. They are sharing a lot of information about their medication and drugs in terms of posts or comments. Therefore, sentiment analysis plays an active role to compute those posts or comments. However, each post is associated with multilabel such as ease of use, effectiveness, and satisfaction. To solve this kind of problem, we propose a multichannel convolution neural network for multilabel sentiment classification using Abilify oral user comments. The multichannel represents the multiple versions of the standard model with different strides. Specifically, we use the pre-trained model to generate word vectors. The proposed model is evaluated with multilabel metrics. The results indicate that the proposed multichannel convolutional network model outperforms the traditional machine learning algorithms.

Spectral decomposition and multifrequency joint amplitude-variation-with-offset inversion based on the neural network

The conventional amplitude-variation-with-offset (AVO) inversion method based on the amplitude attribute of prestack gathers calculates the elastic parameters of underground media through the variation of amplitude with offset. However, when the underground medium is a thin interlayer, the tuning effect will occur, which is the aliasing phenomenon of amplitude at the reflection interface. The tuning effect makes the conventional AVO inversion method based on amplitude attributes difficult to solve the problem of thin interlayer recognition. In addition, the same reflection interface will have different AVO characteristics at different frequencies, whereas the frequency factor is not included in conventional AVO inversion methods. Two-stage neural network approaches based on deep learning are combined to improve the resolution of thin interlayers and to accurately invert the elastic parameters. For the first-stage neural network, a fully connected network is used to solve the inversion spectral decomposition problem. It can eliminate the thin interlayer tuning effect, effectively improve the resolution, and obtain reflection coefficients at different frequencies. For the second-stage neural network, a multichannel convolutional neural network is used to establish the mapping relationship between multifrequency reflection coefficients and elastic parameters, so that the multifrequency joint inversion of the elastic parameters could be realized. This procedure is applied to synthetic data (with and without noise) to indicate the resistance to noise interference of the two-stage deep-learning method. Compared with the method of directly predicting elastic parameters using seismic data and the conventional AVO inversion method, the two-stage deep-learning method can describe the elastic parameters of thin interbeds more accurately. The same procedure is applied to the field data, and the inversion results indicate that they can well match with the well-logging data. Hence, it is promising for practical applications.

A multi-channel convolutional neural network based on attention mechanism fusion for facial expression recognition

Abstract Facial expressions can reflect people’s inner emotions to a certain extent, and studying facial expressions can help psychologists capture expression information in time and understand patients’ psychological changes quickly. In this paper, we establish a multi-channel convolutional neural network face expression recognition model based on the fusion of the attention mechanism. With the help of the attention mechanism and multi-channel convolutional neural network, we input expression images and perform average pooling and maximum pooling, output the features with high recognition after pooling, and identify the features with high recognition in expression images throughout the process. And with the help of multi-scale feature fusion, we improve the detection of subtle changes, such as the corners of the mouth and the eyes of the expression image target. The loss function is used to calculate the loss rate of facial expression images, which leads to the correct rate of facial expression recognition by a multi-channel convolutional neural network based on the fusion of attention mechanisms. It is demonstrated that the highest recognition correct rate of the multi-channel convolutional neural network faces expression recognition model with attention mechanism fusion is 93.56% on the FER2013 dataset, which is higher than that of the MHBP model by 23.2%. The highest correct recognition rate on the RAF-DB dataset is 91.34%, which is higher than the SR-VGG19 model by 19.39%. This shows that the multi-channel convolutional neural network face expression recognition based on the fusion of attention mechanisms improves the correct rate of facial expression recognition, which is beneficial to the research and development of psychology.

Hybrid Multichannel-Based Deep Models Using Deep Features for Feature-Oriented Sentiment Analysis

With the rapid growth of user-generated content on social media, several new research domains have emerged, and sentiment analysis (SA) is one of the active research areas due to its significance. In the field of feature-oriented sentiment analysis, both convolutional neural network (CNN) and gated recurrent unit (GRU) performed well. The former is widely used for local feature extraction, whereas the latter is suitable for extracting global contextual information or long-term dependencies. In existing studies, the focus has been to combine them as a single framework; however, these approaches fail to fairly distribute the features as inputs, such as word embedding, part-of-speech (PoS) tags, dependency relations, and contextual position information. To solve this issue, in this manuscript, we propose a technique that combines variant algorithms in a parallel manner and treats them equally to extract advantageous informative features, usually known as aspects, and then performs sentiment classification. Thus, the proposed methodology combines a multichannel convolutional neural network (MC-CNN) with a multichannel bidirectional gated recurrent unit (MC-Bi-GRU) and provides them with equal input parameters. In addition, sharing the information of hidden layers between parallelly combined algorithms becomes another cause of achieving the benefits of their combined abilities. These abilities make this approach distinctive and novel compared to the existing methodologies. An extensive empirical analysis carried out on several standard datasets confirms that the proposed technique outperforms the latest existing models.

PW-360IQA: Perceptually-Weighted Multichannel CNN for Blind 360-Degree Image Quality Assessment.

Image quality assessment of 360-degree images is still in its early stages, especially when it comes to solutions that rely on machine learning. There are many challenges to be addressed related to training strategies and model architecture. In this paper, we propose a perceptually weighted multichannel convolutional neural network (CNN) using a weight-sharing strategy for 360-degree IQA (PW-360IQA). Our approach involves extracting visually important viewports based on several visual scan-path predictions, which are then fed to a multichannel CNN using DenseNet-121 as the backbone. In addition, we account for users' exploration behavior and human visual system (HVS) properties by using information regarding visual trajectory and distortion probability maps. The inter-observer variability is integrated by leveraging different visual scan-paths to enrich the training data. PW-360IQA is designed to learn the local quality of each viewport and its contribution to the overall quality. We validate our model on two publicly available datasets, CVIQ and OIQA, and demonstrate that it performs robustly. Furthermore, the adopted strategy considerably decreases the complexity when compared to the state-of-the-art, enabling the model to attain comparable, if not better, results while requiring less computational complexity.

Spatial infrared objects discrimination based on multi-channel CNN with attention mechanism

Spatial infrared (IR) object discrimination is a critical component of IR imaging systems. However, the existence of detector noise, insufficient utilization of multi-band information, and the lack of robust features at long observation distances make it difficult to cope with the discrimination of spatial objects. For such cases, a multi-channel convolutional neural network with attention mechanism (MCA-CNN) is proposed in this study. The MCA-CNN learns features independently from single-band IR radiation intensity sequences in each channel and temporally captures robust features by introducing temporal attention blocks. Subsequently, the learned multiple single-band features are fused, and the significance of different channel features was calculated using the channel attention block. Training data were generated from simulation models of the IR radiation intensity sequences of six types of objects in a micro-motion situation. The experimental results reveal that the method improves the recognition performance while being robust to noise interference, data missing and band mismatch factors.

An Image Recommendation Algorithm Based on Target Alternating Attention and User Affiliation Network

Currently, how to exploit the deep features of images in image recommender systems to achieve image enhancement still needs further research. In addition, little research has explored the implicit and increasing preferences of users by using the affiliation generated by indirect users and virtual users of the main users, which leads to the phenomenon of information cocoon. An Image Recommendation Algorithm Based on Target Alternating Attention and User Affiliation Network (TAUA) is proposed in this paper that addresses the problems of inadequate extraction of semantic features in an image and information cocoon in image recommender systems. First, to complete the multi-dimensional description of the image, we extract the category, color, and style features of the image through a multi-channel convolutional neural network (MCNN), and we then perform migration and integration on these features. Then, to enhance the pixel-level representation ability of the image and achieve image feature enhancement, we propose target alternating attention to capture the information of surrounding pixels alternately from inside to outside. Finally, a user affiliation network, including indirect users and virtual users, is established according to the user behavior and transaction record, and the users’ increasing preferences and affiliated users are mined through the implicit interaction relationship of users. Experimental results show that compared with baselines on the Amazon dataset, the results of F@10, NDCG@10, and AUC of the proposed algorithm are 4.02%, 5.00%, and 2.14% higher than those of ACF, and 5.76%, 0.86% and 1.16% higher than those of VPOI. On the Flickr dataset, our algorithm outperforms ACF by 5.74%, 5.12%, and 3.68% in F@10, NDCG@10, and AUC, respectively, and outperforms VPOI by 0.45%, 0.47%, and 0.49%. TAUA has better recommendation performance and can significantly improve the recommendation effect.

Energy efficiency and savings analysis with multirate sampling for petrochemical process using convolutional neural network-based transfer learning

Energy efficiency in the petrochemical industry is crucial in reducing energy consumption and environmental impact. An accurate energy efficiency model will provide valuable insight for supporting operational adjustment decisions. In practice, due to inconsistent sampling intervals in the petrochemical industry, the traditional approach for obtaining energy efficiency may be unreliable and difficult to handle these multirate data characteristics. Therefore, in this paper, a multi-channel convolutional neural network model integrating a model parameter-based transfer learning approach is proposed to improve the prediction of energy efficiency under inconsistent sampling intervals. The multi-channel structure aims to recognize a different pattern from the dataset by convolving the information along the time dimension. Concurrently, transfer learning allows the model to learn a new pattern of input after the model is fully trained. Finally, the performance for energy efficiency prediction and saving analysis is validated by applying it to the vinyl chloride monomer production case study. The result shows that the proposed model outperformed traditional models and typical convolutional neural network structures in terms of accuracy and reproducibility, with an r-square of 0.97. The utilization of transfer learning prevents a significant drop in performance and enhances adaptability in model learning on real-time energy tracking. Moreover, the energy gap analysis of the prediction result identified a significant energy-saving potential, which would decrease annual energy consumption by 7.25% on average and a 5,709-ton reduction in carbon dioxide emissions.

Research on Multi-Scale Feature Extraction and Working Condition Classification Algorithm of Lead-Zinc Ore Flotation Foam

To address the problems of difficult online monitoring, low recognition efficiency and the subjectivity of work condition identification in mineral flotation processes, a foam flotation performance state recognition method is developed to improve the issues mentioned above. This method combines multi-dimensional CNN (convolutional neural networks) characteristics and improved LBP (local binary patterns) characteristics. We have divided the foam flotation conditions into six categories. First, the multi-directional and multi-scale selectivity and anisotropy of nonsubsampled shearlet transform (NSST) are used to decompose the flotation foam images at multiple frequency scales, and a multi-channel CNN network is designed to extract static features from the images at different frequencies. Then, the flotation video image sequences are rotated and dynamic features are extracted by the LBP-TOP (local binary patterns from three orthogonal planes), and the CNN-extracted static picture features are fused with the LBP dynamic video features. Finally, classification decisions are made by a PSO-RVFLNs (particle swarm optimization-random vector functional link networks) algorithm to accurately identify the foam flotation performance states. Experimental results show that the detection accuracy of the new method is significantly improved by 4.97% and 6.55%, respectively, compared to the single CNN algorithm and the traditional LBP algorithm, respectively. The accuracy of flotation performance state classification was as high as 95.17%, and the method reduced manual intervention, thus improving production efficiency.