Bilinear Convolutional Neural Networks Research Articles

Remaining useful life prediction and state of health diagnosis of lithium-ion batteries with multiscale health features based on optimized CatBoost algorithm

Due to the large-scale application of electric vehicles, the remaining service life prediction and health status diagnosis of lithium-ion batteries as their power core is particularly important, and the essence of RUL prediction and SOH diagnosis is the prediction of remaining capacity. Through the aging experiment of cycle charging and discharging of lithium-ion batteries, the health features of experimental data are extracted for the prediction of remaining capacity. In this paper, a deep feature extraction method based on Bilinear CNN combined with CatBoost algorithm based on fractional order method optimization particle swarm optimization, and ant colony optimization algorithm is proposed for battery remaining capacity prediction. Seven groups of health features extracted from ten groups of battery data were used to input them into the optimized CatBoost algorithm for regression prediction. The results show that the proposed model achieves accurate SOH and RUL prediction, the three evaluation indicators MAE, RMSE, and MAPE of SOH are all within 1.7 % and the error rate of RUL is not higher than 1.5 %, and the test of multiple batteries also proves its strong robustness.

A Positioning and Recognition Method based on Bilinear Time-Frequency Transform and CNN for Long-Range Asymmetric Interferometer Vibration Sensor

A Positioning and Recognition Method based on Bilinear Time-Frequency Transform and CNN for Long-Range Asymmetric Interferometer Vibration Sensor

Cervical cancer classification based on a bilinear convolutional neural network approach and random projection

Cervical cancer (CC) is one of the most prevalent malignancies affecting women globally, with a particularly notable impact and high mortality in regions with limited economic resources. This underscores the imperative for the expeditious development of techniques that facilitate timely and precise detection, thereby augmenting treatment efficacy, enhancing survival rates, and mitigating the burden of healthcare expenditure. The intricacies of cervical cancer detection are inherently aligned with the challenges of fine-grained visual classification. This study focuses on the integration of bilinear pooling within convolutional neural networks (CNNs) and addresses the problem of the computational complexity of bilinear features with the fortification of the bilinear CNN with a random projection paradigm (RP-BCNN). The aim is the simultaneous achievement of improved classification precision and streamlined processing temporalities. The proposed methodology entails the introduction of a dyadic feature extraction protocol in which the input cellular image is subjected to twin feature extraction modalities. The feature maps obtained from this process undergo element-wise multiplication via an outer-product operation, thereby engendering composite feature representations. Subsequently, a judiciously designed random projection procedure is invoked to reduce dimensionality, yielding a more succinct yet informative image descriptor. Empirical evaluations of the introduced model predicted on the RP-BCNN framework yielded commendable outcomes. Notably, an accuracy of 0.9983 was achieved for the dual-label classification scenarios, and an accuracy of 0.9530 was realized in the context of multiclass classification encompassing seven distinct labels. The proposed model achieves an optimal equilibrium between classification accuracy and processing efficiency, thus constituting a potent instrument for the classification of cervical cancer. Further, it holds promise for the refinement of diagnostic accuracy, thereby providing a vantage point for embracing sophisticated techniques in the realm of medical image analysis.

A quantitative method for the assessment of facial attractiveness based on transfer learning with fine-grained image classification

In this paper, we investigate a new approach based on a combination of three-dimensional (3D) facial images and deep transfer learning (TL) with fine-grained image classification (FGIC) for quantitative evaluation of facial attractiveness. The 3D facial surface images of patients with and without filtering and the publicly available SCUT-FBP5500 dataset was used for transfer training and model pre-training, respectively. Experimental results show that a bilinear CNN model with a Gaussian filter freezing 80 % of the weights exhibit the strongest performance and lowest average error as a deep learning prediction model; the model was subsequently adopted for automatic assessment of facial attractiveness in clinical application. This is the first TL model with FGIC using 3D facial images for automatic quantitative evaluation of facial attractiveness in patients undergoing Orthognathic surgery (OGS). The developed web browser–based user interface enables effective and rapid assessment, thus contributing to effective patient–clinician communication and decision-making.

A Soft-Reference Breast Ultrasound Image Quality Assessment Method That Considers the Local Lesion Area.

The quality of breast ultrasound images has a significant impact on the accuracy of disease diagnosis. Existing image quality assessment (IQA) methods usually use pixel-level feature statistical methods or end-to-end deep learning methods, which focus on the global image quality but ignore the image quality of the lesion region. However, in clinical practice, doctors' evaluation of ultrasound image quality relies more on the local area of the lesion, which determines the diagnostic value of ultrasound images. In this study, a global-local integrated IQA framework for breast ultrasound images was proposed to learn doctors' clinical evaluation standards. In this study, 1285 breast ultrasound images were collected and scored by experienced doctors. After being classified as either images with lesions or images without lesions, they were evaluated using soft-reference IQA or bilinear CNN IQA, respectively. Experiments showed that for ultrasound images with lesions, our proposed soft-reference IQA achieved PLCC 0.8418 with doctors' annotation, while the existing end-to-end deep learning method that did not consider the local lesion features only achieved PLCC 0.6606. Due to the accuracy improvement for the images with lesions, our proposed global-local integrated IQA framework had better performance in the IQA task than the existing end-to-end deep learning method, with PLCC improving from 0.8306 to 0.8851.

INTS-Net: Improved Navigator-Teacher-Scrutinizer Network for Fine-Grained Visual Categorization

Fine-grained image recognition, as a significant branch of computer vision, has become prevalent in various applications in the real world. However, this image recognition is more challenging than general image recognition due to the highly localized and subtle differences in special parts. Up to now, many classic models, including Bilinear Convolutional Neural Networks (Bilinear CNNs), Destruction and Construction Learning (DCL), etc., have emerged to make corresponding improvements. This paper focuses on optimizing the Navigator-Teacher-Scrutinizer Network (NTS-Net). The structure of NTS-Net determines its strong ability to capture subtle information areas. However, research finds that this advantage will lead to a bottleneck of the model’s learning ability. During the training process, the loss value of the training set approaches zero prematurely, which is not conducive to later model learning. Therefore, this paper proposes the INTS-Net model, in which the Stochastic Partial Swap (SPS) method is flexibly added to the feature extractor of NTS-Net. By injecting noise into the model during training, neurons are activated in a more balanced and efficient manner. In addition, we obtain a speedup of about 4.5% in test time by fusing batch normalization and convolution. Experiments conducted on CUB-200-2011 and Stanford cars demonstrate the superiority of INTS-Net.

Crowd Density Level Estimation and Anomaly Detection Using Multicolumn Multistage Bilinear Convolution Attention Network (MCMS-BCNN-Attention)

The detection of crowd density levels and anomalies is a hot topic in video surveillance. Especially in human-centric action and activity-based movements. In some respects, the density level variation is considered an anomaly in the event. Crowd behaviour identification relies on a computer-vision-based approach and basically deals with spatial information of foreground video information. In this work, we focused on a deep-learning-based attention-oriented classification system for identifying several basic movements in public places, especially, human flock movement, sudden motion changes and panic events in several indoor and outdoor places. The important spatial features were extracted from a bilinear CNN and a multicolumn multistage CNN with preprocessed morphological video frames from videos. Finally, the abnormal and crowd density estimation was distinguished by using an attention feature combined with a multilayer CNN feature by modifying the fully connected layer for several categories (binary and multiclass). We validate the proposed method on several video surveillance datasets including PETS2009, UMN and UCSD. The proposed method achieved an accuracy of 98.62, 98.95, 96.97, 99.10 and 98.38 on the UCSD Ped1, UCSD Ped2, PETS2009, UMN Plaza1 and UMN Plaza2 datasets, respectively, with the different pretrained models. We compared the performance between recent modern approaches and the proposed method (MCMS-BCNN-Attention) and achieved the highest accuracy. The anomaly detection performance on the UMN and PETS2009 datasets was compared with that of a state-of-the-art method and achieved the best AUC results as 0.9953 and 1.00 for both scenarios, respectively, with a binary classification.

Automatic Gender and Age Classification from Offline Handwriting with Bilinear ResNet.

This work focuses on automatic gender and age prediction tasks from handwritten documents. This problem is of interest in a variety of fields, such as historical document analysis and forensic investigations. The challenge for automatic gender and age classification can be demonstrated by the relatively low performances of the existing methods. In addition, despite the success of CNN for gender classification, deep neural networks were never applied for age classification. The published works in this area mostly concentrate on English and Arabic languages. In addition to Arabic and English, this work also considers Hebrew, which was much less studied. Following the success of bilinear Convolutional Neural Network (B-CNN) for fine-grained classification, we propose a novel implementation of a B-CNN with ResNet blocks. To our knowledge, this is the first time the bilinear CNN is applied for writer demographics classification. In particular, this is the first attempt to apply a deep neural network for the age classification. We perform experiments on documents from three benchmark datasets written in three different languages and provide a thorough comparison with the results reported in the literature. B-ResNet was top-ranked in all tasks. In particular, B-ResNet outperformed other models on KHATT and QUWI datasets on gender classification.

Histopathological Tissue Segmentation of Lung Cancer with Bilinear CNN and Soft Attention.

Automatic tissue segmentation in whole-slide images (WSIs) is a critical task in hematoxylin and eosin- (H&E-) stained histopathological images for accurate diagnosis and risk stratification of lung cancer. Patch classification and stitching the classification results can fast conduct tissue segmentation of WSIs. However, due to the tumour heterogeneity, large intraclass variability and small interclass variability make the classification task challenging. In this paper, we propose a novel bilinear convolutional neural network- (Bilinear-CNN-) based model with a bilinear convolutional module and a soft attention module to tackle this problem. This method investigates the intraclass semantic correspondence and focuses on the more distinguishable features that make feature output variations relatively large between interclass. The performance of the Bilinear-CNN-based model is compared with other state-of-the-art methods on the histopathological classification dataset, which consists of 107.7 k patches of lung cancer. We further evaluate our proposed algorithm on an additional dataset from colorectal cancer. Extensive experiments show that the performance of our proposed method is superior to that of previous state-of-the-art ones and the interpretability of our proposed method is demonstrated by Grad-CAM.

Medical Ultrasound Image Quality Assessment for Autonomous Robotic Screening

Autonomous ultrasound scanning robots have attracted the attention of researchers, and the real-time quality assessment of ultrasound images is the key technology of them. Existing robot systems usually use pixel-level feature statistical methods such as grayscale, confidence map, etc. However, in clinical practice doctors’ evaluation of ultrasound image quality not only relies on the pixel quality, but also on the image content. In this study, we introduced the deep learning method to the quality assessment of medical breast ultrasound images to learn the doctors’ clinical evaluation standards. We collected 1205 breast ultrasound images of 533 patients and asked experienced doctors to score them. The ResNet18 with a shallow number of layers is adopted to extract features of ultrasound images, and the high-order feature coding model BCNN (Bilinear CNN) in fine-grained categorization is adopted to improve the accuracy of quality assessment. The consistency between the ultrasound image quality assessment results of our method and the doctors’ annotation results reached 0.842 by PLCC (Pearson Linear Correlation Coefficient). Compared with the confidence map method commonly used for quality assessment in the automatic ultrasound scanning robots, our method achieves a higher consistency with the evaluation results of doctors and provides a new image evaluation method and idea for the visual servo control framework of ultrasound autonomous scanning robots.

BCA: Bilinear Convolutional Neural Networks and Attention Networks for legal question answering

The National Judicial Examination of China is an essential examination for selecting legal practitioners. In recent years, people have tried to use machine learning algorithms to answer examination questions. With the proposal of JEC-QA (Zhong et al. 2020), the judicial examination becomes a particular legal task. The data of judicial examination contains two types, i.e., Knowledge-Driven questions and Case-Analysis questions. Both require complex reasoning and text comprehension, thus challenging computers to answer judicial examination questions. We propose Bilinear Convolutional Neural Networks and Attention Networks (BCA) in this paper, which is an improved version based on the model proposed by our team on the Challenge of AI in Law 2021 judicial examination task. It has two essential modules, Knowledge-Driven Module (KDM) for local features extraction and Case-Analysis Module (CAM) for the semantic difference clarification between the question stem and the options. We also add a post-processing module to correct the results in the final stage. The experimental results show that our system achieves state-of-the-art in the offline test of the judicial examination task.

Improving Deep Learning Based Second-Order Side-Channel Analysis With Bilinear CNN

In recent years, deep learning techniques have received significant attention in the side-channel community due to their state-of-the-art performance in profiled attacks against embedded devices. Compared with template attacks, deep learning-based attacks can deal with the high dimensionality of trace and misalignment without pre-processing. However, the performance of attacks is very sensitive to the network architecture, especially when considering masking countermeasures. Although previous works have shown the potential of neural networks to break the first-order masking, the inner-working of how the network combines the leakage of mask and masked value is still unclear and could be suboptimal. To reduce this gap, we propose to embed product combination, which has been proved to be the best combination function in noisy situations, into the design of neural networks. To this end, we introduce a bilinear convolutional neural network (in short, B-CNN) for efficient profiled attacks against the widely used masking countermeasure. In order to interpret the inner-working and decision-making of B-CNN, we propose a new visualization tool called layer-wise correlation that can reveal the points of interest and help to understand the combination of leakages. We evaluate our networks on several public datasets, e.g., ASCAD and CHES CTF 2018. The results indicate that B-CNN converges significantly faster than classic CNN models, even using a very limited number of profiling traces (e.g., 8000 profiling traces for the ASCAD dataset). Moreover, our networks perform even systematically better (w.r.t. the number of attack traces) than the current state-of-the-art results on all the investigated datasets.

Retraction Note: Discussion on ecological design and construction of river landscape based on fine-grained image classification

The fine-grained image classification method does not have a very good training effect when performing random weight training for neural networks. For the fine-grained classification under the condition of convolutional neural network, the main reason is that its upgrading method can better improve the stability of convolutional neural network. Fine-grained classification through bilinear convolutional neural networks is to effectively merge several convolutional neural networks, so as to obtain a high-resolution advantage to improve the method of fine-grained classification. With the development of society and the improvement of urban construction, the artificial reconstruction of rivers in the city will strengthen the connectivity and smoothness of the reconstructed river to a certain extent, but it will also affect the self-healing of the river. At present, the treatment of river pollution is a major aspect of the country’s treatment of environmental pollution. Obtain the transformation and restoration of the ecological environment around the river in the city through data exchange and other methods. Establishing a scientific and complete plant community with a naturalized revetment system can reduce the impact of river water on the riverbank. At the same time, these naturalized revetment plants can also play a role in flood prevention and discharge, and to a certain extent, it can affect the polluted water quality and give purification. While carrying out the ecological environment protection of the river, it also pays attention to the design and planning of the river landscape, so that the river can withstand the impact of flowing water while having a beautiful landscape design, so as to realize the ecological civilization construction of the river landscape.

Classification of Amanita Species Based on Bilinear Networks with Attention Mechanism

The accurate classification of Amanita is helpful to its research on biological control and medical value, and it can also prevent mushroom poisoning incidents. In this paper, we constructed the Bilinear convolutional neural networks (B-CNN) with attention mechanism model based on transfer learning to realize the classification of Amanita. When the model is trained, the weight on ImageNet is used for pre-training, and the Adam optimizer is used to update network parameters. In the test process, images of Amanita at different growth stages were used to further test the generalization ability of the model. After comparing our model with other models, the results show that our model greatly reduces the number of parameters while achieving high accuracy (95.2%) and has good generalization ability. It is an efficient classification model, which provides a new option for mushroom classification in areas with limited computing resources.

Transfer Learning-Based Bilinear Convolutional Networks for Unsupervised Change Detection

With the increasing popularity of deep learning, most recent developments of change detection (CD) approaches have taken advantage of deep learning techniques to improve the detection performance. However, it is usually necessary to elaborately design the network architecture and train the model with a large amount of labeled data, which are difficult to obtain in practice. To overcome these limitations, this letter proposed an unsupervised CD framework for high-resolution remote sensing images integrating transfer learning-based bilinear convolutional neural networks (BCNNs) and object-based change analysis. A difference image is first generated, which is used for the subsequent preclassification and superpixel segmentation. Then, two sets of superpixel samples with reliable labels derived from the bitemporal remote sensing images are input into two pretrained CNNs to extract representative features, respectively. On this basis, the matrix outer product is utilized to generate the combined bilinear features, which are input into the softmax classifier to discriminate the change and no-change information and thus obtaining the final change map by feeding all sample data into the well-trained model. The experimental results on three real data sets demonstrate the effectiveness and superiority of the proposed method over several existing CD approaches.

Classification of neurofibromatosis-related dystrophic or nondystrophic scoliosis based on image features using Bilateral CNN.

We developed a system that can automatically classify cases of scoliosis secondary to neurofibromatosis type 1 (NF1-S) using deep learning algorithms (DLAs) and improve the accuracy and effectiveness of classification, thereby assisting surgeons with the auxiliary diagnosis. Comprehensive experiments in NF1 classification were performed based on a dataset consisting 211 NF1-S (131 dystrophic and 80 nondystrophic NF1-S) patients. Additionally, 100 congenital scoliosis (CS), 100 adolescentidiopathicscoliosis (AIS) patients, and 114 normal controls were used for experiments in primary classification. For identification of NF1-S with nondystrophic or dystrophic curves, we devised a novel network (i.e., Bilateral convolutional neural network [CNN]) utilizing a bilinear-like operation to discover the similar interest features between whole spine AP and lateral x-ray images. The performance of Bilateral CNN was compared with spine surgeons, conventional DLAs (i.e., VGG-16, ResNet-50, and Bilinear CNN [BCNN]), recently proposed DLAs (i.e., ShuffleNet, MobileNet, and EfficientNet), and Two-path BCNN which was the extension of BCNN using AP and lateral x-ray images as inputs. In NF1 classification, our proposed Bilateral CNN with 80.36% accuracy outperformed the other seven DLAs ranging from 61.90% to 76.19% with fivefold cross-validation. It also outperformed the spine surgeons (with an average accuracy of 77.5% for the senior surgeons and 65.0% for the junior surgeons). Our method is highly generalizable due to the proposed methodology and data augmentation. Furthermore, the heatmaps extracted by Bilateral CNN showed curve pattern and morphology of ribs and vertebrae contributing most to the classification results. In primary classification, our proposed method with an accuracy of 87.92% also outperformed all the other methods with varied accuracies between 52.58% and 83.35% with fivefold cross-validation. The proposed Bilateral CNN can automatically capture representative features for classifying NF1-S utilizing AP and lateral x-ray images, leading to a relatively good performance. Moreover, the proposed method can identify other spine deformities for auxiliary diagnosis.

BILSK: A bilinear convolutional neural network approach for skin lesion classification

Background and objective: Skin lesions are areas of the dermis that have an abnormal growth or appearance compared to the surrounding area. They can be harmless such as a small scrape or severe as skin cancer. These dermis abnormalities increase in size over time and cause morbidity problems. Correct diagnosis of these skin lesions is crucial for successful treatment, which is generally too expensive. Convolutional neural networks (CNN) have been investigated for the classification of skin lesions with different training methods and techniques. However, the best results obtained by previous works show that there is a wide range to be achieved regarding detection, precision, and computational costs.Methods: Therefore, in this paper, we present a bilinear CNN approach capable of classifying seven skin lesions classes with the highest accuracy of the state-of-the-art and low computational cost. The framework proposed includes a data augmentation step to correct the data imbalance problem, transfer learning, and fine-tuning to improve the classification performance while reducing the computational cost. Several simulations were executed over the HAM10000 dataset.Results: The results show that a bilinear approach composed of the ResNet50 and the VGG16 architectures, increases accuracy by 2.7% compared to the state-of-the-art. Specifically, the proposed approach achieves an average of 0.9321 accuracy, 0.9292 precision, 0.9300 recall, 0.9321 F1 score, 0.9810 AUC, and requires 238.6 min for training.Conclusions: This performance increase can help to support the clinician diagnosis in order to provide a second opinion which can reduce the morbidity and treatment costs.

Fine-Grained Breast Cancer Classification With Bilinear Convolutional Neural Networks (BCNNs).

Classification of histopathological images of cancer is challenging even for well-trained professionals, due to the fine-grained variability of the disease. Deep Convolutional Neural Networks (CNNs) showed great potential for classification of a number of the highly variable fine-grained objects. In this study, we introduce a Bilinear Convolutional Neural Networks (BCNNs) based deep learning method for fine-grained classification of breast cancer histopathological images. We evaluated our model by comparison with several deep learning algorithms for fine-grained classification. We used bilinear pooling to aggregate a large number of orderless features without taking into consideration the disease location. The experimental results on BreaKHis, a publicly available breast cancer dataset, showed that our method is highly accurate with 99.24% and 95.95% accuracy in binary and in fine-grained classification, respectively.

Bilinear pyramid network for flower species categorization

It is a challenging task to distinguish between numerous species of flowers due to their visually similarities and variations of the pose and structure. Thanks to properly modeling of the local feature interactions, bilinear CNN has succeeded in classifying of many non-rigid fine-grained species including flowers. However, bilinear CNN only computes the feature in a straightforward way without exploring the interactions between features from multiple layers in the network. In this paper, we present a novel Bilinear Pyramid Network (BPN) for flower categorization. Instead of passing through the network and directly feeding the final classifier, features from a convolutional layer are resized and multiplied with that from the former layer, which alternates multiple times to generates prediction vectors using the features from distinct layers. These features encoded from the feature pyramid spontaneously carry multi-level semantic cues, which yields stronger discriminative powers than single-layer features. Experiments show that the proposed network obtains superior classification results on the challenging dataset of flowers.

No-reference quality assessment for image-based assessment of economically important tropical woods.

Image Quality Assessment (IQA) is essential for the accuracy of systems for automatic recognition of tree species for wood samples. In this study, a No-Reference IQA (NR-IQA), wood NR-IQA (WNR-IQA) metric was proposed to assess the quality of wood images. Support Vector Regression (SVR) was trained using Generalized Gaussian Distribution (GGD) and Asymmetric Generalized Gaussian Distribution (AGGD) features, which were measured for wood images. Meanwhile, the Mean Opinion Score (MOS) was obtained from the subjective evaluation. This was followed by a comparison between the proposed IQA metric, WNR-IQA, and three established NR-IQA metrics, namely Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE), deepIQA, Deep Bilinear Convolutional Neural Networks (DB-CNN), and five Full Reference-IQA (FR-IQA) metrics known as MSSIM, SSIM, FSIM, IWSSIM, and GMSD. The proposed WNR-IQA metric, BRISQUE, deepIQA, DB-CNN, and FR-IQAs were then compared with MOS values to evaluate the performance of the automatic IQA metrics. As a result, the WNR-IQA metric exhibited a higher performance compared to BRISQUE, deepIQA, DB-CNN, and FR-IQA metrics. Highest quality images may not be routinely available due to logistic factors, such as dust, poor illumination, and hot environment present in the timber industry. Moreover, motion blur could occur due to the relative motion between the camera and the wood slice. Therefore, the advantage of WNR-IQA could be seen from its independency from a "perfect" reference image for the image quality evaluation.