Multimodal Version Research Articles

LRMM: Low rank multi-scale multi-modal fusion for person re-identification based on RGB-NI-TI

Person Re-identification is a crucial task in video surveillance, aiming to match person images from non-overlapping camera views. Recent methods introduce the Near-Infrared (NI) modality to alleviate the limitations of traditional single visible light modality under low-light conditions, while they overlook the importance of modality-related information. To incorporate more additional complementary information to assist traditional person re-identification tasks, in this paper, a novel RGB-NI-TI multi-modal person re-identification approach is proposed. First, we design a multi-scale multi-modal interaction module to facilitate cross-modal information fusion across multiple scales. Secondly, we propose a low-rank multi-modal fusion module that leverages the feature and weight parallel decomposition and then employs low-rank modality-specific factors for multimodal fusion. It aims to make the model more efficient in fusing multiple modal features while reducing complexity. Finally, we propose a multiple modalities prototype loss to supervise the network jointly with the cross-entropy loss, enforcing the network to learn modality-specific information by improving the intra-class cross-modality similarity and expanding the inter-class difference. The experimental results on benchmark multi-modal Re-ID datasets (RGBNT201, RGBNT100, MSVR310) and constructed person Re-ID datasets (multimodal version Market1501, PRW) validate the effectiveness of the proposed approach compared with the state-of-the-art methods.

Subpopulations of neurons in the perirhinal cortex enable both modality-specific and modality-invariant recognition of objects.

The perirhinal cortex (PER) supports multimodal object recognition, but how multimodal information of objects is integrated within the PER remains unknown. Here, we recorded single units within the PER while rats performed a PER-dependent multimodal object-recognition task. In this task, audiovisual cues were presented simultaneously (multimodally) or separately (unimodally). We identified 2 types of object-selective neurons in the PER: crossmodal cells, showing constant firing patterns for an object irrespective of its modality, and unimodal cells, showing a preference for a specific modality. Unimodal cells further dissociated unimodal and multimodal versions of the object by modulating their firing rates according to the modality condition. A population-decoding analysis confirmed that the PER could perform both modality-invariant and modality-specific object decoding-the former for recognizing an object as the same in various conditions and the latter for remembering modality-specific experiences of the same object.

Multimodal spatial availability: A singly-constrained measure of accessibility considering multiple modes.

Place-based accessibility measures communicate the potential interaction with opportunities at a zone that populations can access. Recent research has explored the implications of how opportunities are counted by different accessibility methods. In conventional measures, opportunities are multiply counted if more than one zone offers access to the same opportunity. This multi-count of opportunities leads to values of accessibility that are difficult to interpret. A possible solution to enhance the meaning-making of accessibility results is by constraining the calculations to match a known quantity. This ensures all zonal values sum up to a predetermined quantity (i.e., the total number of opportunities). In this way, each value can be meaningfully related to this total. A recent effort that implements this solution is spatial availability, a singly-constrained accessibility measure. In this paper, we extend spatial availability for use in the case of multiple modes or more generally, heterogeneous population segments with distinct travel behaviors. After deriving a multimodal version of spatial availability, we proceed to illustrate its features using a synthetic example. We then apply it to an empirical example of low emission zones in Madrid, Spain. We conclude with suggestions for future research and its use in evaluating policy interventions.

Algorithmic properties of modal and superintuitionistic logics of monadic predicates over finite Kripke frames

Abstract We show that the monadic fragment of the modal predicate logic of a single Kripke frame with finitely many possible worlds, but possibly infinite domains, is decidable. This holds true even for multimodal logics with equality, regardless of whether equality is interpreted as identity or as congruence. By the Gödel–Tarski translation, similar results follow for superintuitionistic predicate logics, with or without equality. Using these observations, we establish upper algorithmic bounds, which match the known lower bounds, for monadic fragments of some modal predicate logics. In particular, we prove that, if $L$ is a propositional modal logic contained in $\textbf{S5}$, $\textbf{GL.3}$ or $\textbf{Grz.3}$ and the class of finite Kripke frames validating $L$ is recursively enumerable, then the monadic fragment with equality of the predicate logic of finite Kripke frames validating $L$ is $\varPi ^{0}_{1}$-complete; this, in particular, holds if $L$ is one of the following propositional logics: $\textbf{K}$, $\textbf{T}$, $\textbf{D}$, $\textbf{KB}$, $\textbf{KTB}$, $\textbf{K4}$, $\textbf{K4.3}$, $\textbf{S4}$, $\textbf{S4.3}$, $\textbf{GL}$, $\textbf{Grz}$, $\textbf{K5}$, $\textbf{K45}$ and $\textbf{S5}$. We also prove that monadic fragments with equality of logics $\textbf{QAlt}^=_{n}$ and $\textbf{QTAlt}^=_{n}$ are decidable. The obtained results are easily extendable to the multimodal versions of the predicate logics we consider and to logics with the Barcan formula.

Modeling the rheological behavior of silica filled rubber compounds

The rheological behavior of styrene–butadiene rubber (SBR) compounds filled with silica is investigated as a function of silica volume fraction. To predict the mechanical response, a continuum model for entangled polymer melts filled with nanoparticles is herein introduced. This model is capable of describing the rheological response in both the linear and nonlinear viscoelastic regimes in the context of non-equilibrium thermodynamics to guarantee its thermodynamic admissibility. The constitutive model describes the polymer nanocomposite melts at a mesoscopic level of description by considering the conformation tensor between successive entanglement points, and the orientation tensor for the, in general, spheroidal nanoparticles that describes their average orientation. Evolution equations are developed for nanoparticles with an arbitrary shape but are eventually specified to the case of spherical ones. The multimode version of the new constitutive model provides a very accurate prediction of the rheological behavior of the processability range of SBR/silica nanocomposites. Thus, the new model is a tool able to provide answers to the several difficulties that rubber-producing manufacturers face when processing rubber compounds.

Multi-modal deep convolutional dictionary learning for image denoising

Leveraging the capabilities of traditional dictionary learning (DicL) and drawing upon the success of deep neural networks (DNNs), the recently proposed framework of deep convolutional dictionary learning (DCDicL) has exhibited remarkable behaviours in image denoising. Note that, the application of the DCDicL method is confined to single modality scenarios, whereas the images in practice often originate from diverse modalities. In this paper, to broaden the application scope of the DCDicL method, we design a multi-modal version of it, dubbed MMDCDicL. Specifically, within the mathematical model of MMDCDicL, we adopt an analytical approach to tackle the sub-problem linked to the guidance modality, harnessing its inherent reliability. Meanwhile, like in DCDicL, we utilize a network-based learning approach for the noisy modality to extract trustworthy information from the data. Based on the solution, we establish an interpretable network structure for MMDCDicL. Additionally, wherein, we design a multi-kernel channel attention block (MKCAB) in the structure to efficiently integrate the information from diverse modalities. Experimental results suggest that MMDCDicL can reconstruct higher-quality outcomes both quantitatively and perceptually. Code is available at http://www.diplab.net/lunwen/mmdcdicl.htm.

Extension of Stochastic Point Location for Multimodal Problems.

Stochastic point location (SPL) involves a learning mechanism (LM) determining an optimal point on the line when the only inputs LM receives are stochastic information about the direction in which LM should move. The complexity of SPL comes from the stochastic responses of the environment, which may lead LM completely astray. SPL is a fundamental problem in optimization and was studied by many researchers during the last two decades, including improvement of its solution and all-pervasive applications. However, all existing SPL studies assume that the whole search space contains only one optimal point. Since a multimodal optimization problem (MMOP) contains multiple optimal solutions, it is significant to develop SPL's multimodal version. This article extends it from a unimodal problem to a multimodal one and proposes a parallel partition search (PPS) solution to address this issue. The heart of the proposed solution involves extracting the feature of the historical sampling information to distinguish the subintervals that contain the optimal points or not. Specifically, it divides the whole search space into multiple subintervals and samples them parallelly, then utilizes the feature of the historical sampling information to adjust the subintervals adaptively and to find the subintervals containing the optimal points. Finally, the optimal points are located within these subintervals according to their respective sampling statistics. The proof of the ε-optimal property for the proposed solution is presented. The numerical testing results demonstrate the power of the scheme.

The 2T-qutrit, a two-mode bosonic qutrit

Quantum computers often manipulate physical qubits encoded on two-level quantum systems. Bosonic qubit codes depart from this idea by encoding information in a well-chosen subspace of an infinite-dimensional Fock space. This larger physical space provides a natural protection against experimental imperfections and allows bosonic codes to circumvent no-go results that apply to states constrained by a 2-dimensional Hilbert space. A bosonic qubit is usually defined in a single bosonic mode but it makes sense to look for multimode versions that could exhibit better performance.In this work, building on the observation that the cat code lives in the span of coherent states indexed by a finite subgroup of the complex numbers, we consider a two-mode generalisation living in the span of 24 coherent states indexed by the binary tetrahedral group 2T of the quaternions. The resulting 2T-qutrit naturally inherits the algebraic properties of the group 2T and appears to be quite robust in the low-loss regime. We initiate its study and identify stabilisers as well as some logical operators for this bosonic code.

Analog holographic wavefront sensor for defocus and spherical aberration measurement recorded in a photopolymer.

An analog holographic wavefront sensor (AHWFS), for measurement of low and high order (defocus and spherical aberration) aberration modes has been developed as volume phase holograms in a photopolymer recording medium. This is the first time that high order aberrations such as spherical aberration can be sensed using a volume hologram in a photosensitive medium. Both defocus and spherical aberration were recorded in a multi-mode version of this AHWFS. Refractive elements were used to generate a maximum and minimum phase delay of each aberration which were multiplexed as a set of volume phase holograms in an acrylamide based-photopolymer layer. The single-mode sensors showed a high degree of accuracy in determining various magnitudes of defocus and spherical aberration generated refractively. The multi-mode sensor also exhibited promising measurement characteristics and similar trends to the single-mode sensors were observed. The method of quantifying defocus was improved upon and a brief study into material shrinkage and sensor linearity is presented.

A Meta-learning Based Generative Model with Graph Attention Network for Multi-Modal Recommender Systems

With increased data in various e-commerce domains or in different online streaming platforms like Amazon Prime, Netflix, etc, it has become more challenging to give a personalized recommendation to a user. In terms of the movie recommendation system, different multimodal information such as audio, video, and text are effectively fused together to tackle this problem. However, in real-world scenarios this information may not always be present, leading to the problem of missing modality. Existing works in Recommender Systems (RSs) may fail to recommend effectively if their model is not susceptible to missing modality. In this work, we propose a meta-learning model to deal with the missing modality problem. Our proposed model tries to reconstruct the missing features with the help of the existing modalities and once reconstructed, we pass them to novel trident architecture for solving the recommendation task. We have performed experiments on the newly created multimodal version of the MovieLens dataset. Additionally, we have incorporated a new modality where we extracted subtitles of the movies to increase the effectiveness of our missing modality model.

Wide-range characteristics of beam perveance and saddle point potential of LIPS-200 ion thruster

Both the long-life and multi-mode versions of LIPS-200 ion thruster are under investigation in LIP (Lanzhou Institute of Physics). To confirm the feasible ranges of the beam current and accel (abbreviation for accelaration) grid potential to apply to the thruster, the wide-range beam perveance (the state of beam focus) and saddle point potential (the lowest potential along beamlet centerline) characteristics of LIPS-200 are studied with a test-verified PIC-MCC (Particle in Cell-Monte Carlo Collisions) model. These characteristics are investigated with both the initial and the eroded states of the accel grid aperture diameter. The results show that the feasible ranges of these parameters with respect to perveance/crossover (overfocused) limit extend as the operating time accumulates, while the feasible range of accel grid potential narrows due to a reduced EBSF (electron backstreaming failure) margin. The feasible ranges determined by the initial condition are: (i) the beam current up to 0.981 A, and (ii) the accel grid potential up to −85 V. A 23% enlargement of the aperture diameter would bring up to 48 V of EBSF margin loss.

Logical Foundations for Hybrid Type-Logical Grammars

This paper explores proof-theoretic aspects of hybrid type-logical grammars, a logic combining Lambek grammars with lambda grammars. We prove some basic properties of the calculus, such as normalisation and the subformula property and also present both a sequent and a proof net calculus for hybrid type-logical grammars. In addition to clarifying the logical foundations of hybrid type-logical grammars, the current study opens the way to variants and extensions of the original system, including but not limited to a non-associative version and a multimodal version incorporating structural rules and unary modes.

Data-Efficient Sensor Upgrade Path Using Knowledge Distillation.

Deep neural networks have achieved state-of-the-art performance in image classification. Due to this success, deep learning is now also being applied to other data modalities such as multispectral images, lidar and radar data. However, successfully training a deep neural network requires a large reddataset. Therefore, transitioning to a new sensor modality (e.g., from regular camera images to multispectral camera images) might result in a drop in performance, due to the limited availability of data in the new modality. This might hinder the adoption rate and time to market for new sensor technologies. In this paper, we present an approach to leverage the knowledge of a teacher network, that was trained using the original data modality, to improve the performance of a student network on a new data modality: a technique known in literature as knowledge distillation. By applying knowledge distillation to the problem of sensor transition, we can greatly speed up this process. We validate this approach using a multimodal version of the MNIST dataset. Especially when little data is available in the new modality (i.e., 10 images), training with additional teacher supervision results in increased performance, with the student network scoring a test set accuracy of 0.77, compared to an accuracy of 0.37 for the baseline. We also explore two extensions to the default method of knowledge distillation, which we evaluate on a multimodal version of the CIFAR-10 dataset: an annealing scheme for the hyperparameter and selective knowledge distillation. Of these two, the first yields the best results. Choosing the optimal annealing scheme results in an increase in test set accuracy of 6%. Finally, we apply our method to the real-world use case of skin lesion classification.

Axiomatizing Hybrid XPath with Data

In this paper we introduce sound and strongly complete axiomatizations for XPath with data constraints extended with hybrid operators. First, we present HXPath=, a multi-modal version of XPath with data, extended with nominals and the hybrid operator @. Then, we introduce an axiomatic system for HXPath=, and we prove it is strongly complete with respect to the class of abstract data models, i.e., data models in which data values are abstracted as equivalence relations. We prove a general completeness result similar to the one presented in, e.g., [BtC06], that ensures that certain extensions of the axiomatic system we introduce are also complete. The axiomatic systems that can be obtained in this way cover a large family of hybrid XPath languages over different classes of frames, for which we present concrete examples. In addition, we investigate axiomatizations over the class of tree models, structures widely used in practice. We show that a strongly complete, finitary, first-order axiomatization of hybrid XPath over trees does not exist, and we propose two alternatives to deal with this issue. We finally introduce filtrations to investigate the status of decidability of the satisfiability problem for these languages.

Visual display for surgical targeting: concepts and usability study

PurposeInteractive image-guided surgery technologies enable accurate target localization while preserving critical nearby structures in many surgical interventions. Current state-of-the-art interfaces largely employ traditional anatomical cross-sectional views or augmented reality environments to present the actual spatial location of the surgical instrument in preoperatively acquired images. This work proposes an alternative, simple, minimalistic visual interface intended to assist during real-time surgical target localization.MethodsThe estimated 3D pose of the interventional instruments and their positional uncertainty are intuitively presented in a visual interface with respect to the target point. A usability study with multidisciplinary participants evaluates the proposed interface projected in surgical microscope oculars against cross-sectional views. The latter was presented on a screen both stand-alone and combined with the proposed interface. The instruments were electromagnetically navigated in phantoms.ResultsThe usability study demonstrated that the participants were able to detect invisible targets marked in phantom imagery with significant enhancements for localization accuracy and duration time. Clinically experienced users reached the targets with shorter trajectories. The stand-alone and multi-modal versions of the proposed interface outperformed cross-sectional views-only navigation in both quantitative and qualitative evaluations.ConclusionThe results and participants’ feedback indicate potential to accurately navigate users toward the target with less distraction and workload. An ongoing study evaluates the proposed system in a preclinical setting for auditory brainstem implantation.

A note on the complexity of S4.2

is the modal logic of directed partial pre-orders and/or the modal logic of reflexive and transitive relational frames with a final cluster. It holds a distinguished position in philosophical logic, where it has been advocated as the ‘correct’ logic of knowledge; it has also found interesting applications in the temporal logic of relativistic spacetime and the metamathematics of forcing in set theory. The satisfiability problem for is -complete: this is a result established in an AiML 2004 paper of Shapirovsky [(2004). On PSPACE-decidability in transitive modal logic. In R. A. Schmidt, I. Pratt-Hartmann, M. Reynolds, & H. Wansing (Eds.), Advances in modal logic 5 (pp. 269–287). King's College Publications] where the complexity classification of emerges as a consequence of a very general method for constructing decision procedures for transitive modal logics. We provide here a ‘classical’ proof in the standard Halpern-Moses style of Halpern and Moses [(1992). A guide to completeness and complexity for modal logics of knowledge and belief. Artificial Intelligence, 54(2), 319–379]. With little extra work, it is shown that the -completeness result extends to , the multimodal version of . We prove that the -completeness characterisation of monomodal persists even if we restrict ourselves to fragments with bounded modal depth, but the problem is -complete when it is restricted to formulae with modal depth at most one. The complexity of satisfiability for the fragment of the language with a finite number of propositional variables (but unbounded modal depth) remains -hard. For a finite language and bounded modal depth, -satisfiability can be checked in linear time.

Noiseless linear amplifiers for multimode states

The entanglement structure between different frequency components within broadband quantum light pulses, forged at entanglement creation, represents a promising route to the practical delivery of many multipartite quantum information applications. However, the scalability of such applications is largely limited by the entanglement decoherence caused by photon loss. One promising method to combat such losses is noiseless linear amplification. However, while there have been various procedures that implement noiseless linear amplification on single-mode states, no realization has thus far been proposed for noiseless linear amplification on quantum states carrying a multimode structure. In this work we close this gap, proposing a novel Noiseless Linear Amplifier (NLA) with Photon Catalysis (PC), namely, the PC-NLA. Constructing a multimode version of an existing NLA that uses Quantum Scissors (QS), the QS-NLA, we then show how the PC-NLA is compatible with the QS-NLA, even though the former uses half the physical resources of the latter. We then apply our newly developed multimode NLA frameworks to the problem of Continuous-Variable (CV) entanglement distillation, determining how the multimode structure of the entanglement impacts the performance of the NLAs. Different from single-mode NLA analyses, we find that a multimode NLA is only effective as a CV entanglement distillation strategy when the channel loss is beyond some threshold - a threshold largely dependent on the multimode structure. The results provided here will be valuable for real-world implementations of multipartite quantum information applications that utilize complex entanglement structure within broadband light pulses.

Start-up and cessation of steady shear and extensional flows: Exact analytical solutions for the affine linear Phan-Thien–Tanner fluid model

Exact analytical solutions for start-up and cessation flows are obtained for the affine linear Phan-Thien–Tanner fluid model. They include the results for start-up and cessation of steady shear flows, of steady uniaxial and biaxial extensional flows, and of steady planar extensional flows. The solutions obtained show that at start-up of steady shear flows, the stresses go through quasi-periodic exponentially damped oscillations while approaching their steady-flow values (so that stress overshoots are present); at start-up of steady extensional flows, the stresses grow monotonically, while at cessation of steady shear and extensional flows, the stresses decay quickly and non-exponentially. The steady-flow rheology of the fluid is also reviewed, the exact analytical solutions obtained in this work for steady shear and extensional flows being simpler than the alternative formulas found in the literature. The properties of steady and transient solutions, including their asymptotic behavior at low and high Weissenberg numbers, are investigated in detail. Generalization to the multimode version of the Phan-Thien–Tanner model is also discussed. Thus, this work provides a complete analytical description of the rheology of the affine linear Phan-Thien–Tanner fluid in start-up, cessation, and steady regimes of shear and extensional flows.

Evaluating predictability of various constitutive equations for MAOS behavior of entangled polymer solutions

To predict the nonlinear viscoelastic behavior of polymer melts and solutions under medium amplitude oscillatory shear (MAOS) flow, we summarized known analytical MAOS material functions and derived new analytical solutions from several continuum- or molecular-based constitutive models [Johnson–Segalman, Phan-Thien–Tanner, Leonov, Larson, Pom-Pom, coupled double-convection-reptation with chain stretch (cDCR-CS), and Rouse linear entangled polymers (Rolie-Poly)]. All constitutive models discussed in this study were classified as time-strain separable or inseparable models. Whereas linear viscoelastic behavior of all constitutive models under small amplitude oscillatory shear flow was identical, the frequency-dependent signatures of four MAOS moduli (first-harmonic elastic and viscous MAOS moduli and third-harmonic elastic and viscous MAOS moduli) were model-specific, though all analytical solutions exhibited negative first-harmonic elastic MAOS moduli. In particular, tube-based constitutive models displayed a sign change in first-harmonic viscous MAOS modulus when chain retraction of the models after deformation was not instantaneous. In addition, two models for entangled linear polymers, that is, the Rolie-Poly and cDCR-CS models, displayed a second sign change in third-harmonic elastic MAOS modulus when finite chain extensibility was introduced. These results suggest that chain relaxation dynamics can be reflected in signatures of MAOS moduli. We extended single-mode predictions into multimode versions to predict the behavior of weakly or moderately entangled polystyrene solutions (effective entanglement number < 5). All models described the two elastic MAOS moduli well, but different predictability was obtained for viscous MAOS moduli. Therefore, viscous MAOS moduli can be a measure of evaluating the predictability of constitutive models for MAOS.

Multimodal optimization using whale optimization algorithm enhanced with local search and niching technique.

For some real-world problems, it is desirable to find multiple global optima as many as possible. The multimodal optimization approach which finds multiple optima in a single run shows significant difference with the single modal optimization approach.The whale optimization algorithm (WOA) is a newly emerging reputable optimization algorithm. Its global search ability has been verified in many benchmark functions and real-world applications. In this paper, we propose a multimodal version of whale optimization algorithm (MMWOA). MMWOA enhances the multimodal search ability of WOA by using the niching technique and improves the local search efficiency of WOA by combining the Gaussian sampling technique. The algorithm has been tested on multimodal optimization benchmark functions recommended by CEC'2013 and on a multimodal optimization problem with non-linear constraints. Experimental results indicate that MMWOA has competitive performance compared with other state-of-the-art multimodal optimization algorithms.