Compact Bilinear Research Articles

Natural Densitals.

The concept of natural densitals (NDs) and their amplitudes is introduced. These quantities provide the spectral decomposition of the cumulant of the two-electron density that, by definition, quantifies the extent of electron correlation. Consequently, they are ideally suited for a rigorous description of electron correlation effects in Coulombic systems. Spin-summed and spin-resolved versions of the NDs and their amplitudes are defined, and their properties are discussed in detail. Unlike the nonnegative-valued occupation numbers of the natural orbitals (NOs), these amplitudes exhibit diverse sign patterns that emerge within different regimes of electron correlation. The descriptive power of this property is vividly illustrated with the ground state of the H2 molecule, in which the subtle interplay of various types of electron correlation is captured in detail by a straightforward examination of the amplitudes of the NDs alone. Offering the most compact bilinear representations of (a property analogous to that of the NOs with respect to the 1-matrix), the NDs open up entirely new vistas in the analysis of electronic structures of atoms and molecules.

Analysis and identification of nocturnal groaning syndrome based on multimodal data

Nocturnal groaning syndrome is a common sleep disorder characterized by irregular groaning or vocalizations during nighttime sleep, representing a significant area of research in sleep disorders. Nocturnal groaning syndrome is a common sleep disorder characterized by irregular groaning or vocalizations during nighttime sleep, representing a significant area of research in sleep disorders. proposes a multimodal recognition approach based on speech, image, and text modalities. The study analyzes audio features using Mel Frequency Cepstral Coefficients (MFCC), which is the most common method for identifying nocturnal groaning syndrome. Coefficients (MFCC), extracts image features with pretrained MobileNetV2, and identifies key physiological signals from text using TF-IDF algorithm. Subsequently, Multimodal Compact Bilinear Pooling (MCB) is employed to fuse audio and image features, and a Text-Image CNN is used to combine image and text features. Support Vector Machine (SVM) is then used to classify the fused multimodal features, and decision-level fusion is performed using weighting criteria. Experimental results demonstrate an identification accuracy of 89.5% on the test set, significantly enhancing the auxiliary diagnostic effectiveness of nocturnurnal diagnosis. Experimental results demonstrate an identification accuracy of 89.5% on the test set, significantly enhancing the auxiliary diagnostic effectiveness of nocturnal groaning syndrome.

Compact bilinear operators and paraproducts revisited

AbstractWe present a new proof of the compactness of bilinear paraproducts withCMOsymbols. By drawing an analogy to compact linear operators, we first explore further properties of compact bilinear operators on Banach spaces and present examples. We then prove compactness of bilinear paraproducts withCMOsymbols by combining one of the properties of compact bilinear operators thus obtained with vanishing Carleson measure estimates and interpolation of bilinear compactness.

Research on the Wild Mushroom Recognition Method Based on Transformer and the Multi-Scale Feature Fusion Compact Bilinear Neural Network

Wild mushrooms are popular for their taste and nutritional value; however, non-experts often struggle to distinguish between toxic and non-toxic species when foraging in the wild, potentially leading to poisoning incidents. To address this issue, this study proposes a compact bilinear neural network method based on Transformer and multi-scale feature fusion. The method utilizes a dual-stream structure that integrates multiple feature extractors, enhancing the comprehensiveness of image information capture. Additionally, bottleneck attention and efficient multi-scale attention modules are embedded to effectively capture multi-scale features while maintaining low computational costs. By employing a compact bilinear pooling module, the model achieves high-order feature interactions, reducing the number of parameters without compromising performance. Experimental results demonstrate that the proposed method achieves an accuracy of 98.03%, outperforming existing comparative methods. This proves the superior recognition performance of the model, making it more reliable in distinguishing wild mushrooms while capturing key information from multiple dimensions, enabling it to better handle complex scenarios. Furthermore, the development of public-facing identification tools based on this method could help reduce the risk of poisoning incidents. Building on these findings, the study suggests strengthening the research and development of digital agricultural technologies, promoting the application of intelligent recognition technologies in agriculture, and providing technical support for agricultural production and resource management through digital platforms. This would provide a theoretical foundation for the innovation of digital agriculture and promote its sustainable development.

Coreference resolution helps visual dialogs to focus

Visual Dialog is a multi-modal task involving both computer vision and dialog systems. The goal is to answer multiple questions in conversation style, given an image as the context. Neural networks with attention modules are widely used for this task, because of their effectiveness in reasoning the relevance between the texts and images. In this work, we study how to further improve the quality of such reasoning, which is an open challenge. Our baseline is the Recursive Visual Attention (RVA) model, which refines the vision-text attention by iteratively visiting the dialog history. Building on top of that, we propose to improve the attention mechanism with contrastive learning. We train a Matching-Aware Attention Kernel (MAAK) by aligning the deep feature embeddings of an image and its caption, to provide better attention scores. Experiments show consistent improvements from MAAK. In addition, we study the effect of using Multimodal Compact Bilinear (MCB) pooling as a three-way feature fusion for the visual, textual and dialog history embeddings. We analyze the performance of both methods in the discussion section, and propose further ideas to resolve current limitations.

Environmental sound classification based on improved compact bilinear attention network

Aiming at the difficulty in effectively extracting representative features from environmental sounds characterized by the complicated and considerable variation in the time-frequency domain characteristics, an improved compact bilinear attention network (ICBA-Net) is proposed to more precisely focus on the regions that need attention in feature maps in this paper. Firstly, log-mel spectrogram (LMS) extracted from environmental sound is fed into two parallel pre-trained densely connected convolutional network (DenseNet) for deep feature extraction. Afterwards, a coordinate attention module (CAM) is utilized in each branch to integrate key information in both horizontal and vertical directions to achieve a more accurate focus on global feature salient regions. Finally, two-stream enhanced global representations are integrated into a compact bilinear vector, followed by the full connection layer with activation function softmax to output the final prediction result. Experiments on two publicly available datasets (ESC-10 and ESC-50) and some environmental sounds collected from common scenarios in our real life are used to verify the validity of the proposed method. Through comparison and verification, the results show that the proposed network outperformed almost all state-of-the-art systems regarding classification accuracy, and the feasibility and effectiveness of CAM have also been verified.

Weakly compact bilinear operators among real interpolation spaces

We show a necessary and sufficient condition for weak compactness of bilinear operators interpolated by the real method. This characterization does not hold for interpolated operators by the complex method.

On interpolation of weakly compact bilinear operators

AbstractWe study the interpolation properties of weakly compact bilinear operators by the real method and also by the complex method. We also study the factorization property of weakly compact bilinear operators through reflexive Banach spaces.

Efficient Compact Bilinear Pooling via Kronecker Product

Bilinear pooling has achieved excellent performance in fine-grained recognition tasks. Nevertheless, high-dimensional bilinear features suffer from over-fitting and inefficiency. To alleviate these issues, compact bilinear pooling (CBP) methods were developed to generate low-dimensional features. Although the low-dimensional features from existing CBP methods enable high efficiency in subsequent classification, CBP methods themselves are inefficient. Thus, the inefficiency issue of the bilinear pooling is still unsolved. In this work, we propose an efficient compact bilinear pooling method to solve the inefficiency problem inherited in bilinear pooling thoroughly. It decomposes the huge-scale projection matrix into a two-level Kronecker product of several small-scale matrices. By exploiting the ``vec trick'' and the tensor modal product, we can obtain the compact bilinear feature through the decomposed projection matrices in a speedy manner. Systematic experiments on four public benchmarks using two backbones demonstrate the efficiency and effectiveness of the proposed method in fine-grained recognition.

HOS-FingerCode: Bispectral invariants based contactless multi-finger recognition system using ridge orientation and feature fusion

Fingerprints are a commonly used biometric for civil and forensic applications and the majority of them are captured by contact sensors. More recently, there is a trend toward the use of high resolution digital and video cameras to acquire contactless images. However, the processing of such images is challenging because of intra-class variations such as pose, rotation, translation and scaling due to hand orientation and deformation. They can, however, facilitate the use of full finger or multiple fingers or hand information rather than the fingertip alone. Although some recent research has investigated representation of full finger and hand information under relaxed conditions, current state-of-the-art methods are unable to address transformation issues without manual intervention. To overcome the issues of relaxed constrained nature, geometrical variations and invariant feature encoding regardless of finger orientation, we have used higher order spectral features (HOS-FingerCode) from ridge patterns. We employ an automated method to detect key points and constructs a graph of key-lines. Ridge orientation profiles along with the selected key-lines provide a 1D signal from which bispectral invariant features are extracted. In addition, two fusion schemes are proposed where fusion from multiple key lines combinations and feature fusion from multiple key line configurations are investigated to extract ridge pattern profile for a better performance. The features are pooled from multiple fingers by concatenation or compact bilinear pooling to yield the HOS-FingerCode for multi-finger biometrics. For a single-finger system using 3369 high resolution index and middle finger images, the algorithm better performs when different key lines configurations are fused. For geometrical variations of the images, the system is most impacted by pose than rotation and scale changes of the images, and is tolerant to such variations. For the multi-finger system using 3111 images from both fingers, the algorithm has achieved 93.8% TPR and 97.95% classification accuracy at a setting of 2% FPR. For the feature fusion scheme proposed for multi-finger system using concatenation and compact bilinear pooling, concatenation of features performs better than the compact bilinear pooling. Since multi-finger recognition system performed better than the single finger biometrics system, it can help to diminish the high traffic scenario in busy premises and will facilitate the soft identity verification.

Deep Learning-Based Fake Information Detection and Influence Evaluation

With the prevalence of the Internet, a large number of users have participated in OSN (Online Social Networks), which has gradually made it the mainstream way for obtaining news or information from the Internet. However, with the rapid development of the Internet, a large amount of fake information has also been spread on the Internet. Therefore, fake information detection is of great significance at the moment. A multimodal fake information detection method is proposed in this article, which has adopted the textual and visual contents in the piece of information to make the judgments. The textual feature representation vector is firstly obtained through the pretraining of the Bert model, and then the visual feature representation is obtained through the pretraining of the VGG-19 model. From the proposed method, two MCBP (Multimodal Compact Bilinear Pooling) modules are adopted. The first MCBP module is adopted to obtain the visual feature representation vector with attention, and the second MCBP module is adopted to join the visual feature with the attention mechanism and the textual feature vector. Then, the joined vector can be adopted for fake information detection. The proposed method in this article is compared with two baseline methods. The experimental results on the Twitter and Weibo datasets have proved that the proposed method in this article is better than the EANN method and the SpotFake method in terms of accuracy, precision, recall, and F1 score.

Compact bilinear operators on asymmetric normed spaces

The paper is concerned with compact bilinear operators on asymmetric normed spaces. The study of multilinear operators on asymmetric normed spaces was initiated by Latreche and Dahia (2020) [24]. We go further in this direction and prove a Schauder type theorem on the compactness of the adjoint of a compact bilinear operator and study the ideal properties of spaces of compact bilinear operators. These extend some results of Ramanujan and Schock (1985) [34], and Ruch (1989) [36], on compact bilinear operators on Banach spaces. On the space of bilinear forms one introduces the analog of the weak⁎-topology, called the w2-topology, and one proves an Alaoglu-Bourbaki type theorem – the w2-compactness of the closed unit ball.

Knowledge Graph Representation Fusion Framework for Fine-Grained Object Recognition in Smart Cities

Autonomous object detection powered by cutting-edge artificial intelligent techniques has been an essential component for sustaining complex smart city systems. Fine-grained image classification focuses on recognizing subcategories of specific levels of images. As a result of the high similarity between images in the same category and the high dissimilarity in the same subcategories, it has always been a challenging problem in computer vision. Traditional approaches usually rely on exploring only the visual information in images. Therefore, this paper proposes a novel Knowledge Graph Representation Fusion (KGRF) framework to introduce prior knowledge into fine-grained image classification task. Specifically, the Graph Attention Network (GAT) is employed to learn the knowledge representation from the constructed knowledge graph modeling the categories-subcategories and subcategories-attributes associations. By introducing the Multimodal Compact Bilinear (MCB) module, the framework can fully integrate the knowledge representation and visual features for learning the high-level image features. Extensive experiments on the Caltech-UCSD Birds-200-2011 dataset verify the superiority of our proposed framework over several existing state-of-the-art methods.

XMO and Weighted Compact Bilinear Commutators

To study the compactness of bilinear commutators of certain bilinear Calderón–Zygmund operators which include (inhomogeneous) Coifman–Meyer bilinear Fourier multipliers and bilinear pseudodifferential operators as special examples, Torres and Xue (Rev Mat Iberoam 36:939–956, 2020) introduced a new subspace of BMO\(\,(\mathbb {R}^n)\), denoted by XMO\(\,(\mathbb {R}^n)\), and conjectured that it is just the space VMO\(\,(\mathbb {R}^n)\) introduced by D. Sarason. In this article, the authors give a negative answer to this conjecture by establishing an equivalent characterization of XMO\(\,(\mathbb {R}^n)\), which further clarifies that XMO\(\,(\mathbb {R}^n)\) is a proper subspace of VMO\(\,(\mathbb {R}^n)\). This equivalent characterization of XMO\(\,(\mathbb {R}^n)\) is formally similar to the corresponding one of CMO\(\,(\mathbb {R}^n)\) obtained by A. Uchiyama, but its proof needs some essential new techniques on dyadic cubes as well as some exquisite geometrical observations. As an application, the authors also obtain a weighted compactness result on such bilinear commutators, which optimizes the corresponding result in the unweighted setting.

Fast and Compact Bilinear Pooling by Shifted Random Maclaurin

Bilinear pooling has achieved an excellent performance in many computer vision tasks such as fine-grained classification, scene recognition and texture recognition. However, the high-dimension features from bilinear pooling can sometimes be inefficient and prone to over-fitting. Random Maclaurin (RM) is a widely used GPU-friendly approximation method to reduce the dimensionality of bilinear features. However, to achieve good performance, large projection matrices are usually required in practice, making it costly in computation and memory. In this paper, we propose a Shifted Random Maclaurin (SRM) strategy for fast and compact bilinear pooling. With merely negligible extra computational cost, the proposed SRM provides an estimator with a provably smaller variance than RM, which benefits accurate kernel approximation and thus the learning performance. Using a small projection matrix, the proposed SRM achieves a comparable estimation performance as RM based on a large projection matrix, and thus boosts the efficiency. Furthermore, we upgrade the proposed SRM to SRM+ to further improve the efficiency and make the compact bilinear pooling compatible with fast matrix normalization. Fast and Compact Bilinear Network (FCBN) built upon the proposed SRM+ is devised, achieving an end-to-end training. Systematic experiments conducted on four public datasets demonstrate the effectiveness and efficiency of the proposed FCBN.

Diverse Features Fusion Network for video-based action recognition

The two-stream convolutional network has been proved to be one milestone in the study of video-based action recognition. Lots of recent works modify internal structure of two-stream convolutional network directly and put top-level features into a 2D/3D convolution fusion module or a simpler one. However, these fusion methods cannot fully utilize features and the way fusing only top-level features lacks rich vital details. To tackle these issues, a novel network called Diverse Features Fusion Network (DFFN) is proposed. The fusion stream of DFFN contains two types of uniquely designed modules, the diverse compact bilinear fusion (DCBF) module and the channel-spatial attention (CSA) module, to distill and refine diverse compact spatiotemporal features. The DCBF modules use the diverse compact bilinear algorithm to fuse features extracted from multiple layers of the base network that are called diverse features in this paper. Further, the CSA module leverages channel attention and multi-size spatial attention to boost key information as well as restraining the noise of fusion features. We evaluate our three-stream network DFFN on three public challenging video action benchmarks: UCF101, HMDB51 and Something-Something V1. Experiment results indicate that our method achieves state-of-the-art performance.

Representation transfer learning from deep end-to-end speech recognition networks for the classification of health states from speech

Representation transfer learning has been widely used across a range of machine learning tasks. One such notable approach seen in the speech literature is the use of Convolutional Neural Networks, pre-trained for image classification tasks, to extract features from spectrograms of speech signals. Interestingly, despite the strong performance of such approaches, there have been minimal research efforts exploring the suitability of using speech-specific networks to perform feature extraction. In this regard, a novel feature representation learning framework is presented herein. This approach is comprising the use of Automatic Speech Recognition (ASR) deep neural networks as feature extractors, the fusion of several extracted feature representations using Compact Bilinear Pooling (CBP), and finally inference via a specially optimised Recurrent Neural Network (RNN) classifier. To determine the usefulness of these feature representations, they are comprehensively tested on two representative speech-health classification tasks, namely the food-type being eaten and speaker intoxication. Key results indicate the promise of the extracted features, demonstrating comparable results to other state-of-the-art approaches in the literature.

Improved Bilinear CNN Model for Remote Sensing Scene Classification

Remote sensing (RS) scene classification is challenging due to changes in the scale and direction of scenes within a category. Bilinear pooling method can extract higher-order and spatial orderless information and has been shown to achieve impressive performance on various visual tasks. However, bilinear pooled features are high dimensional, which makes them impractical for subsequent processing, especially for the convolutional neural network (CNN) models with more channels in the final convolutional layer. To alleviate this shortcoming, an improved bilinear pooling method is proposed to build the compact bilinear CNN model in this work. Specifically, a joint pooling method is proposed to reduce the high-dimensional bilinear features, and it can be embedded in a bilinear CNN architecture for end-to-end optimization. Through the experimental evaluation of three real RS scene image data sets, it is proved that the improved bilinear pooling method can obtain features with higher discriminative power than the bilinear pooling method but with lower dimensionality. In addition, it also reduces the running time of model training.

On compactness results of Lions–Peetre type for bilinear operators

Let Ā=(A0,A1), B̄=(B0,B1) be Banach couples, let E be a Banach space and let T be a bilinear operator such that ‖T(a,b)‖E≤Mj‖a‖Aj‖b‖Bj for a∈A0∩A1, b∈B0∩B1, j=0,1. If T:Aj∘×Bj∘⟶E compactly for j=0 or 1, we show that T may be uniquely extended to a compact bilinear operator from the complex interpolation spaces generated by Ā and B̄ to E. Furthermore, the corresponding result for the real method is given and we also study the case when E is replaced by a couple (E0,E1) of Banach function spaces on the same measure space.

Interpolation of compact bilinear operators

This paper is devoted to the study of the stability of the compactness property of bilinear operators acting on the products of interpolated Banach spaces. We prove one-sided compactness results for bilinear operators on products of Banach spaces generated by abstract methods of interpolation, in the sense of Aronszajn and Gagliardo. To get these results, we prove a key one-sided bilinear interpolation theorem on compactness for bilinear operators on couples satisfying an extra approximation property. We give applications to general cases, including Peetre’s method and the general real interpolation methods.