Multimodal Dictionary Research Articles

Deep M2CDL: Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning Network.

For multi-modal image processing, network interpretability is essential due to the complicated dependency across modalities. Recently, a promising research direction for interpretable network is to incorporate dictionary learning into deep learning through unfolding strategy. However, the existing multi-modal dictionary learning models are both single-layer and single-scale, which restricts the representation ability. In this paper, we first introduce a multi-scale multi-modal convolutional dictionary learning ( M2CDL) model, which is performed in a multi-layer strategy, to associate different image modalities in a coarse-to-fine manner. Then, we propose a unified framework namely Deep M2CDL derived from the M2CDL model for both multi-modal image restoration (MIR) and multi-modal image fusion (MIF) tasks. The network architecture of Deep M2CDL fully matches the optimization steps of the M2CDL model, which makes each network module with good interpretability. Different from handcrafted priors, both the dictionary and sparse feature priors are learned through the network. The performance of the proposed Deep M2CDL is evaluated on a wide variety of MIR and MIF tasks, which shows the superiority of it over many state-of-the-art methods both quantitatively and qualitatively. In addition, we also visualize the multi-modal sparse features and dictionary filters learned from the network, which demonstrates the good interpretability of the Deep M2CDL network.

The multi-channel signals based tensor sparse representation classification method for fault diagnosis of high-speed train

The transmission system is a key component to ensure the stable operation of high-speed trains. Thus, it is significant to monitor its condition to ensure the operation safety. Nowadays sparse representation is widely used in fault diagnosis. However, as the number of sensors is increasing, the existing method destroys the internal structure of multi-channel signals and cannot effectively deal with the fault diagnosis of multi-channel signals in parallel. Therefore, this article extends the existing sparse representation method to tensor space to extract the coupling information between channels and realize the fault diagnosis of multi-channel. First, a tensor sparse representation model is proposed to achieve data-level multi-channel signal fusion and complete inter-channel fault feature extraction. Then, a multimodal dictionary learning algorithm is proposed to adaptively design the data-driven dictionary to achieve data-driven feature extraction. Finally, a tensor sparse representation classification method is proposed to achieve the purpose of intelligent diagnosis. Fault experiments verify the effectiveness and superiority of the method.

3D reconstruction of unlimited-size real-world porous media by combining a BicycleGAN-based multimodal dictionary and super-dimension reconstruction

Three-dimensional reconstruction from a single two-dimensional image is important in the field of numerical simulation of porous media (including sandstones, composite, and alloys). Owing to limitations of computer storage space, the algorithm itself, and GPU memory, traditional algorithms and increasingly popular deep-learning algorithms cannot reconstruct complete real-size core images. To solve this problem, we propose a generative adversarial network dictionary-based super-dimension (GANDB-SD) algorithm that uses a BicycleGAN model to establish eight multimodal mappings of small-size structures. These mapping relations are used as the mapping dictionary when the super-dimension reconstruction framework is used to reconstruct the real core images without size limitation. Based on this algorithm, we further propose a three-dimensional co-occurrence matrix function as the loss function for the first time and propose a diversity preservation strategy to solve the pattern repetition problem during reconstruction. We performed an experiment to reconstruct a 5123 image, in which the morphological parameters and reconstruction time of different algorithms were compared. Results show that the algorithm has advantages in reconstruction accuracy and time. We further reconstructed 10003 real cores, and the statistical and morphological similarities between the reconstructed and target structures proved that the proposed algorithm is competent in the task of reconstructing real-world cores with unlimited size. We remark that our approach opens the door to a new methodology that incorporating machine learning/deep learning techniques into conventional super-dimension methods to address 3D reconstruction of unlimited-size real-world porous media issues in a wide range of related fields.

MULTIMODAL LEARNER’S DICTIONARY WITH HISTORICAL AND CULTURAL COMPONENTS IN EFL CLASSROOM

This article studies the problem of making a multimodal computer dictionary with historical and cultural components for EFL classrooms. The relevance of the research topic is associated with the development of computer lexicography as a branch of applied linguistics at the intersection with digital humanities and digital pedagogy, as well as with the active development of new methods and tools in the practice of teaching foreign languages. The object of the research is a multimodal learners’ computer dictionary that visualizes semantic changes in lexicon of the English language, in particular, the key lexemes of the thematic field “еducation”. The purpose of this article is to substantiate the problems of creating an educational multimodal dictionary of a historical and cultural type in computer lexicography. One of the objectives of the research is to study the specific semantic features of the units in the thematic field “education” in the multimodal learner’s dictionary. The multimodal dictionary plays an important role in the interaction of the user with the virtual system, where the dictionary is not only an additional reference book but also a multifunctional means of teaching and obtaining information about the culture of the target language countries, including the dynamics of its development. The introduction of such dictionaries into teaching is the key to successful immersion in the culture of the target language country and mastering language competencies. This article presents the elements of typology and principles of building computer dictionaries, notes the role of a multimodal computer dictionary in teaching a foreign language, and proposes a model of a learner’s multimodal dictionary of historical and cultural type for EFL classroom, as well as outlines the stages of dynamic development in English vocabulary.

P.733 Atomoxetine alters stopping networks in healthy controls and participants with cocaine use disorder without affecting stopping performance

We propose a new online approach for multimodal dictionary learning. The method developed in this work addresses the great challenges posed by the computational resource constraints in dynamic environment when dealing with large scale tensor sequences. Given a sequence of tensors, i.e. a set composed of equal-size tensors, the approach proposed in this paper allows to infer a basis of latent factors that generate these tensors by sequentially processing a small number of data samples instead of using the whole sequence at once. Our technique is based on block coordinate descent, gradient descent and recursive computations of the gradient. A theoretical result is provided and numerical experiments on both real and synthetic data sets are performed.

Deep Coupled ISTA Network for Multi-modal Image Super-Resolution.

Given a low-resolution (LR) image, multi-modal image super-resolution (MISR) aims to find the high-resolution (HR) version of this image with the guidance of an HR image from another modality. In this paper, we use a model-based approach to design a new deep network architecture for MISR. We first introduce a novel joint multi-modal dictionary learning (JMDL) algorithm to model cross-modality dependency. In JMDL, we simultaneously learn three dictionaries and two transform matrices to combine the modalities. Then, by unfolding the iterative shrinkage and thresholding algorithm (ISTA), we turn the JMDL model into a deep neural network, called deep coupled ISTA network. Since the network initialization plays an important role in deep network training, we further propose a layer-wise optimization algorithm (LOA) to initialize the parameters of the network before running back-propagation strategy. Specifically, we model the network initialization as a multi-layer dictionary learning problem, and solve it through convex optimization. The proposed LOA is demonstrated to effectively decrease the training loss and increase the reconstruction accuracy. Finally, we compare our method with other state-of-the-art methods in the MISR task. The numerical results show that our method consistently outperforms others both quantitatively and qualitatively at different upscaling factors for various multi-modal scenarios.

Online multimodal dictionary learning

We propose a new online approach for multimodal dictionary learning. The method developed in this work addresses the great challenges posed by the computational resource constraints in dynamic environment when dealing with large scale tensor sequences. Given a sequence of tensors, i.e. a set composed of equal-size tensors, the approach proposed in this paper allows to infer a basis of latent factors that generate these tensors by sequentially processing a small number of data samples instead of using the whole sequence at once. Our technique is based on block coordinate descent, gradient descent and recursive computations of the gradient. A theoretical result is provided and numerical experiments on both real and synthetic data sets are performed.

RADAR: Robust Algorithm for Depth Image Super Resolution Based on FRI Theory and Multimodal Dictionary Learning

Depth image super-resolution is a challenging problem, since normally high upscaling factors are required (e.g., 16×), and depth images are often noisy. In order to achieve large upscaling factors and resilience to noise, we propose a Robust Algorithm for Depth imAge super Resolution (RADAR) that combines the power of finite rate of innovation (FRI) theory with multimodal dictionary learning. Given a low-resolution (LR) depth image, we first model its rows and columns as piece-wise polynomials and propose an FRI-based depth upscaling (FDU) algorithm to super-resolve the image. Then, the upscaled moderate quality (MQ) depth image is further enhanced with the guidance of a registered high-resolution (HR) intensity image. This is achieved by learning multimodal mappings from the joint MQ depth and HR intensity pairs to the HR depth, through a recently proposed triple dictionary learning (TDL) algorithm. Moreover, to speed up the super-resolution process, we introduce a new projection-based rapid upscaling (PRU) technique that pre-calculates the projections from the joint MQ depth and HR intensity pairs to the HR depth. Compared with the state-of-the-art deep learning-based methods, our approach has two distinct advantages: we need a fraction of training data but can achieve the best performance, and we are resilient to mismatches between training and testing datasets. The extensive numerical results show that the proposed method outperforms other state-of-the-art methods on either noise-free or noisy datasets with large upscaling factors up to 16× and can handle unknown blurring kernels well.

Online Data Organizer: Micro-Video Categorization by Structure-Guided Multimodal Dictionary Learning

Micro-videos have rapidly become one of the most dominant trends in the era of social media. Accordingly, how to organize them draws our attention. Distinct from the traditional long videos that would have multi-site scenes and tolerate the hysteresis, a micro-video: 1) usually records contents at one specific venue within a few seconds. The venues are structured hierarchically regarding their category granularity. This motivates us to organize the micro-videos via their venue structure. 2) timely circulates over social networks. Thus, the timeliness of micro-videos desires effective online processing. However, only 1.22% of micro-videos are labeled with venue information when uploaded at the mobile end. To address this problem, we present a framework to organize the micro-videos online. In particular, we first build a structure-guided multi-modal dictionary learning model to learn the concept-level micro-video representation by jointly considering their venue structure and modality relatedness. We then develop an online learning algorithm to incrementally and efficiently strengthen our model, as well as categorize the micro-videos into a tree structure. Extensive experiments on a real-world data set validate our model well. In addition, we have released the codes to facilitate the research in the community.

Multi-modal motion dictionary learning for facial expression recognition

Recently, dictionary learning is actively investigated in image and signal processing. A variety of dictionary learning methods for the classification problems have been proposed. In this paper, we propose a facial expression recognition system using a multi-modal motion dictionary based on motion flow composed of motion flow intensity and motion flow direction. At the dictionary learning stage, two dictionaries having different modalities are learned from motion flow intensity and from motion flow angle data of facial expression image sequences. We made a feature vector by concatenating two weight vectors of individual dictionaries from each image sequence for classification. Experimental result shows higher accuracy than conventional reconstruction-based approaches with extremely reduced classification time. The proposed approach is a promising method for real-time facial expression recognition from motion flow data.

Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

People share their opinions, stories, and reviews through online video sharing websites every day. The automatic analysis of these online opinion videos is bringing new or understudied research challenges to the field of computational linguistics and multimodal analysis. Among these challenges is the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment. This article addresses this question in four ways: introducing the first multimodal dataset with opinion-level sentiment intensity annotations; studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity; proposing a new computational representation, called multimodal dictionary, based on a language-gesture study; and evaluating the authors' proposed approach in a speaker-independent paradigm for sentiment intensity prediction. The authors' study identifies four interaction types between facial gestures and verbal content: neutral, emphasizer, positive, and negative interactions. Experiments show statistically significant improvement when using multimodal dictionary representation over the conventional early fusion representation (that is, feature concatenation).

Could a multimodal dictionary serve as a learning tool? An examination of the impact of technologically enhanced visual glosses on L2 text comprehension

This study examines the efficacy of a multimodal online bilingual dictionary based on cognitive linguistics in order to explore the advantages and limitations of explicit multimodal L2 vocabulary learning. Previous studies have examined the efficacy of the verbal and visual representation of words while reading L2 texts, concluding that it facilitates incidental word retention. This study explores other potentials of multimodal L2 vocabulary learning: explicit learning with a multimodal dictionary could enhance not only word retention, but also text comprehension; the dictionary could serve not only as a reference tool, but also as a learning tool; and technology-enhanced visual glosses could facilitate deeper text comprehension. To verify these claims, this study investigates the multimodal representations’ effects on Japanese students learning L2 locative prepositions by developing two online dictionaries, one with static pictures and one with animations. The findings show the advantage of such dictionaries in explicit learning; however, no significant differences are found between the two types of visual glosses, either in the vocabulary or in the listening tests. This study confirms the effectiveness of multimodal L2 materials, but also emphasizes the need for further research into making the technologically enhanced materials more effective.

Elastic net regularized dictionary learning for image classification

Dictionary learning plays a key role in image representation for classification. A multi-modal dictionary is usually learned from feature samples across different classes and shared in the feature encoding process. Ideally each atom in dictionary corresponds to a single class of images, while each class of images corresponds to a certain group of atoms. Image features are encoded as linear combinations of selected atoms in a given dictionary. We propose to use elastic net as regularizer to select atoms in feature coding and related dictionary learning process, which not only benefits from the sparsity similar as l1 penalty but also encourages a grouping effect that helps improve image representation. Experimental results of image classification on benchmark datasets show that with dictionary learned in the proposed way outperforms state-of-the-art dictionary learning algorithms.

Sparse Multi-Modal Hashing

Learning hash functions across heterogenous high-dimensional features is very desirable for many applications involving multi-modal data objects. In this paper, we propose an approach to obtain the sparse codesets for the data objects across different modalities via joint multi-modal dictionary learning, which we call sparse multi-modal hashing (abbreviated as <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">${\rm SM}^{2}{\rm H}$</tex></formula> ). In <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">${\rm SM}^{2}{\rm H}$</tex></formula> , both intra-modality similarity and inter-modality similarity are first modeled by a hypergraph, then multi-modal dictionaries are jointly learned by Hypergraph Laplacian sparse coding. Based on the learned dictionaries, the sparse codeset of each data object is acquired and conducted for multi-modal approximate nearest neighbor retrieval using a sensitive Jaccard metric. The experimental results show that <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">${\rm SM}^{2}{\rm H}$</tex></formula> outperforms other methods in terms of mAP and Percentage on two real-world data sets.

Multimodal Sparse Representation-Based Classification for Lung Needle Biopsy Images

Lung needle biopsy image classification is a critical task for computer-aided lung cancer diagnosis. In this study, a novel method, multimodal sparse representation-based classification (mSRC), is proposed for classifying lung needle biopsy images. In the data acquisition procedure of our method, the cell nuclei are automatically segmented from the images captured by needle biopsy specimens. Then, features of three modalities (shape, color, and texture) are extracted from the segmented cell nuclei. After this procedure, mSRC goes through a training phase and a testing phase. In the training phase, three discriminative subdictionaries corresponding to the shape, color, and texture information are jointly learned by a genetic algorithm guided multimodal dictionary learning approach. The dictionary learning aims to select the topmost discriminative samples and encourage large disagreement among different subdictionaries. In the testing phase, when a new image comes, a hierarchical fusion strategy is applied, which first predicts the labels of the cell nuclei by fusing three modalities, then predicts the label of the image by majority voting. Our method is evaluated on a real image set of 4372 cell nuclei regions segmented from 271 images. These cell nuclei regions can be divided into five classes: four cancerous classes (corresponding to four types of lung cancer) plus one normal class (no cancer). The results demonstrate that the multimodal information is important for lung needle biopsy image classification. Moreover, compared to several state-of-the-art methods (LapRLS, MCMI-AB, mcSVM, ESRC, KSRC), the proposed mSRC can achieve significant improvement (mean accuracy of 88.1%, precision of 85.2%, recall of 92.8%, etc.), especially for classifying different cancerous types.