Multimodal Generation Research Articles

A Multi-Modal Story Generation Framework with AI-Driven Storyline Guidance

An automatic story generation system continuously generates stories with a natural plot. The major challenge of automatic story generation is to maintain coherence between consecutive generated stories without the need for human intervention. To address this, we propose a novel multi-modal story generation framework that includes automated storyline decision-making capabilities. Our framework consists of three independent models: a transformer encoder-based storyline guidance model, which predicts a storyline using a multiple-choice question-answering problem; a transformer decoder-based story generation model that creates a story that describes the storyline determined by the guidance model; and a diffusion-based story visualization model that generates a representative image visually describing a scene to help readers better understand the story flow. Our proposed framework was extensively evaluated through both automatic and human evaluations, which demonstrate that our model outperforms the previous approach, suggesting the effectiveness of our storyline guidance model in making proper plans.

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

Deep cross-modal hashing retrieval models inherit the vulnerability of deep neural networks. They are vulnerable to adversarial attacks, especially for the form of subtle perturbations to the inputs. Although many adversarial attack methods have been proposed to handle the robustness of hashing retrieval models, they still suffer from two problems: (1) Most of them are based on the white-box settings, which is usually unrealistic in practical application. (2) Iterative optimization for the generation of adversarial examples in them results in heavy computation. To address these problems, we propose an Efficient Query-based Black-Box Attack (EQB 2 A) against deep cross-modal hashing retrieval, which can efficiently generate adversarial examples for the black-box attack. Specifically, by sending a few query requests to the attacked retrieval system, the cross-modal retrieval model stealing is performed based on the neighbor relationship between the retrieved results and the query, thus obtaining the knockoffs to substitute the attacked system. A multi-modal knockoffs-driven adversarial generation is proposed to achieve efficient adversarial example generation. While the entire network training converges, EQB 2 A can efficiently generate adversarial examples by forward-propagation with only given benign images. Experiments show that EQB 2 A achieves superior attacking performance under the black-box setting.

Multimodal bipedal locomotion generation with passive dynamics via deep reinforcement learning.

Generating multimodal locomotion in underactuated bipedal robots requires control solutions that can facilitate motion patterns for drastically different dynamical modes, which is an extremely challenging problem in locomotion-learning tasks. Also, in such multimodal locomotion, utilizing body morphology is important because it leads to energy-efficient locomotion. This study provides a framework that reproduces multimodal bipedal locomotion using passive dynamics through deep reinforcement learning (DRL). An underactuated bipedal model was developed based on a passive walker, and a controller was designed using DRL. By carefully planning the weight parameter settings of the DRL reward function during the learning process based on a curriculum learning method, the bipedal model successfully learned to walk, run, and perform gait transitions by adjusting only one command input. These results indicate that DRL can be applied to generate various gaits with the effective use of passive dynamics.

Multi-modal data generation with a deep metric variational autoencoder

We present a deep metric variational autoencoder for multi-modal data generation. The variational autoencoder employs triplet loss in the latent space, which allows for conditional data generation by sampling new embeddings in the latent space within each class cluster. The approach is evaluated on a multi-modal dataset consisting of otoscopy images of the tympanic membrane with corresponding wideband tympanometry measurements. The modalities in this dataset are correlated, as they represent different aspects of the state of the middle ear, but they do not present a direct pixel-to-pixel correlation. The approach shows promising results for the conditional generation of pairs of images and tympanograms, and will allow for efficient data augmentation of data from multi-modal sources.

High-Purity Multi-Mode Vortex Beam Generation With Full Complex-Amplitude-Controllable Metasurface

Vortex beam with inherent orbital angular momentum (OAM) is promising in high-capacity communication. On multimode vortex beam generation, metasurface has shown exceptional advantages of integration and miniaturization. For the current widely used phase-only methods on multimode vortex beam generation by metasurface, the purity of the OAM-mode spectrum is severely affected. A new method for generation of multimode vortex beam with high mode purity is proposed in this article by reconstructing the complete complex amplitude information on the meta-device aperture. A 20 dB suppression of the crosstalk modes is experimentally observed for the co-directional multimode vortex beam generation, which is much improved compared to the traditional phase-only scheme. In addition, the proposed scheme also provides the capability of generating high-purity multimode vortex beams with arbitrary preset propagation directions and power allocations. This study provides a platform for high-performance vortex beam communication by increasing the signal-to-noise ratio and enabling the multicasting scenarios with customized capacity requirements.

Gait Based Person Identification Using Deep Learning Model of Generative Adversarial Network

The proliferation of digital age security tools is often attributed to the rise of visual surveillance. Since an individual's gait is highly indicative of their identity, it is becoming an increasingly popular biometric modality for use in autonomous visual surveillance and monitoring. There are various steps used in gait recognition frameworks such as segmentation, feature extraction, feature learning and similarity measurement. These steps are mutually independent with each part fixed, which results in a suboptimal performance in a challenging condition. It can be done independently of the users' involvement. Low-resolution video and straightforward instrumentation can verify an individual's identity, making impersonation a rarity. Using the benefits of the Generative Adversarial Network (GAN), this investigation tackles the problem of unevenly distributed unlabeled data with infrequently performed tasks. To estimate the data circulation in various circumstances using constrained observed gait data, a multimodal generator is applied here. When it comes to sharing knowledge, the variety provided by the data generated by a multimodal generator is hard to beat. The capability to distinguish gait activities with varying patterns due to environmental dynamics is enhanced by this multimodal generator. This system is more stable than other gait-based recognition methods because it can process data that is not equally dispersed throughout a different environment. The system's reliability is enhanced by the multimodal generator's capacity to produce a wide variety of outputs. The testing results show that this algorithm is superior to other gait-based recognition methods because it can adapt to changing environments.

When Hillclimbers Beat Genetic Algorithms in Multimodal Optimization.

This article investigates the performance of multistart next ascent hillclimbing and well-known evolutionary algorithms incorporating diversity preservation techniques on instances of the multimodal problem generator. This generator induces a class of problems in the bitstring domain which is interesting to study from a theoretical perspective in the context of multimodal optimization, as it is a generalization of the classical OneMax and TwoMax functions for an arbitrary number of peaks. An average-case runtime analysis for multistart next ascent hillclimbing is presented for uniformly distributed equal-height instances of this class of problems. It is shown empirically that conventional niching and mating restriction techniques incorporated in an evolutionary algorithm are not sufficient to make them competitive with the hillclimbing strategy. We conjecture the reason for this behavior is the lack of structure in the space of local optima on instances of this problem class, which makes an optimization algorithm unable to exploit information from one optimum to infer where another optimum might be. When no such structure exists, it seems that the best strategy for discovering all optima is a brute-force one. Overall, our study gives insights with respect to the adequacy of hillclimbers and evolutionary algorithms for multimodal optimization, depending on properties of the fitness landscape.

Implementation of Carlson Approximation-Based Fractional Order Universal Filter Using MO-CCCII

In this paper, a new current mode fractional order filter is presented using multi-mode second generation current controlled current conveyors (MO-CCCII).in this method consists of two resistors and grounded capacitors using 3MO-CCCII. This is accomplished with a fractional-order capacitor, also known as a stationary phase element, to form a fractional-order filter. The proposed filter is designed as fractional 0.2, 0.4 orders. The fractional order capacitor was obtained using reduced integer-order models of fractional differentiators using the Carlson method18 using third order Foster I RC network circuit. The filter is simulated using BJT-based MO-CCCII using LTSPICE Simulation for the filter to verify the theory and show its performance of it.

Experimental investigation of multimodal noise generation by ducted low Mach number flows through orifice plates

Noise generation by low Mach number air flows through circular orifices in rectangular ducts is investigated. In particular, the influence of the number and position of the orifices maintaining a constant flow area is addressed. A review of the available theories suggests a certain importance of such parameters in the excitation of higher-order acoustic duct modes. A qualitative coefficient is proposed for a first characterization of the ability to enhance or lessen a given higher-order acoustic mode by the plate geometry. An experimental campaign is performed to measure the total emitted acoustic power by different plate geometries as well as its modal composition. It is found that the orifices' numbers and positions greatly influence the acoustic emissions while the flow pressure drop caused by the obstacles is similar. The proposed qualitative coefficient shows good agreement with the experimental results. A particle image velocimetry measurement campaign is performed to visualize the near-field average flow behavior upstream and downstream of the orifice plates. An increase in the turbulent velocity fluctuations in the vicinity of the orifices is observed on both sides, further validating previous studies on the subject.

Intermodal dispersive waves and soliton collision during multimode supercontinuum generation in chalcogenide glass fiber

Comparing with fused silica fiber, chalcogenide glass fibers are considered as an ideal platform for mid-infrared supercontinuum generation (SCG) because of its higher nonlinear coefficient and wider transparency window. Multimode fiber provides a new way to achieve special spectral characteristics in SCG by introducing intermodal interaction. We investigate the intermodal nonlinear phenomena associated with SCG in multimode chalcogenide fiber by numerically solving multimode generalized nonlinear Schrödinger equation. The results indicate that when the phase matching condition is satisfied, the intermodal interaction leads to the generation of dispersive waves at mid-infrared wavelength. Furthermore, intermodal and intramodal soliton collision are observed, and the bandwidth of the supercontinuum is increased by the intermodal interaction. These results may provide a new way for the generation of mid-infrared supercontinuum light source.

Dynamically tunable multimodal vortex beam generator based on reflective metasurface

How efficiently generating a multi-mode vortex beam is the key to the new system of wireless communication technology based on orbital angular momentum. Due to the advantages of simple structure and flexible tunability of the metasurface, in recent years, people have been devoted to generating dynamic orbital angular momentum more simply and efficiently using programmable metasurfaces. To address the current problem of a single number of modes for generating orbital angular momentum vortex beams from metasurface, the programmable reflective metasurface structure has been designed to realize the function of dynamically tunable multi-mode of vortex beams. By loading two varactor diodes inside each metal patch to construct eight digital coding units, and using the programmable bias circuit to modulate the coding distribution and resonance characteristics on the metasurface in real-time, the metasurface obtains low reflection loss and stable reflection phase, and finally obtains vortex beams with mode numbers of ℓ = ±1, ℓ = ±2, and ℓ = ±3, respectively. The designed metasurface also has the advantages of a simple structure, small electrical size, and low profile, which has a large potential value in the communication of vortex beams and a perfect application in wireless transmission.

Parametric conversion via second harmonic generation and two-hump solitons in phase-matched microresonators

Frequency conversion in microresonators has revolutionised modern-day nonlinear and quantum optics. Here, we present a theory of the multimode second harmonic generation in microresonators under conditions when the parametric conversion back to the pump spectrum dominates through the large domain in the resonator parameter space. We demonstrate that the spectral tunability of the sideband generation in this regime is governed by a discrete sequence of the so-called Eckhaus instabilities. We report the transition to modelocking and generation of solitons, which have a double-pulse structure in the pump field. These solitons exist outside the bistability interval and on a slightly curved background.

A comprehensive review of trip generation models based on land use characteristics

• Existing researches on the impact of land use on trip generation has been explored. • Advanced data collection techniques for travel data collection were investigated. • Numerous approaches to trip generation modelling have been discussed. To assess the impact of any proposed development, it is necessary to estimate the trips that are likely to be generated because of such development. In case of developing countries, land use pattern varies significantly and it becomes very challenging to model the trip rates. In view of this, an effort has been made to synthesize the available literature related to modelling of trip generation rates. This paper provides a comprehensive review of studies related to impact of land use on trip rates, data collection techniques and modelling approaches. Major challenges associated with modelling of trip generation includes the unavailability of standard reference databases for multimodal trip generation, consideration of land use evolvement, less explored machine learning modelling approaches, and the feasibility of using emerging data collection techniques in travel surveys. Finally, a framework is proposed for modelling multimodal trip generation rates considering the above identified issues.

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Video clip retrieval and captioning tasks play an essential role in multimodal research and are the fundamental research problem for multimodal understanding and generation. The CLIP (Contrastive Language-Image Pre-training) model has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the image-text pretrained CLIP model to video-text tasks in an end-to-end manner. Furthermore, we conduct several empirical studies including 1) Whether image feature is enough for video-text retrieval and captioning? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text datasets, including MSR-VTT, MSVD, LSMDC, and DiDeMo for multimodal understanding and generation tasks.

Learning to predict diverse trajectory from human motion patterns

Diverse and reasonable multi-modal trajectory prediction of pedestrian is vital for the safety of intelligent autonomous systems. Currently, most of existing methods primarily focus on modeling the extrinsic interactions of pedestrians, but neglect to take advantage of the intrinsic properties of pedestrian motion, which may generate unrealistic pedestrian movement. We investigate the natural motion of pedestrians and explore future motion patterns of pedestrians with similar observed trajectories, which provides a new perspective for multi-modal prediction of pedestrian trajectories. On this basis, we propose a novel model to learn diverse trajectories from human motion patterns, which is mainly composed of a motion pattern selector and a multi-modal trajectory generator. With a constructed pattern gallery, the motion pattern selector mines the possible future motion patterns based on the similar observed trajectories. Subsequently, the diverse future motion patterns are fed into the multi-modal trajectory generator, in which these motion patterns are selected and refined through a scoring network and a regression network, and then directly generate multiple diverse trajectories. We use two public pedestrian trajectory ETH/UCY datasets to compare our method with several baseline methods. Experimental results show that our model outperforms most of the state-of-the-art approaches both in accuracy and diversity by fully leveraging the intrinsic information of trajectories.

Higher Order Moments Dynamics for Some Multimode Quantum Master Equations

We derive Heisenberg equations for arbitrary high order moments of creation and annihilation operators in the case of the quantum master equation with a multimode generator which is quadratic in creation and annihilation operators and obtain their solutions. Based on them we also derive similar equations for the case of the quantum master equation, which occur after averaging the dynamics with a quadratic generator with respect to the classical Poisson process. This allows us to show that dynamics of arbitrary finite-order moments of creation and annihilation operators is fully defined by finite number of linear differential equations in this case.

SAGA: An Activity-based Multi-modal Mobility ScenarioGenerator for SUMO

In this paper, we define a workflow and a toolchain to support fast mobility scenario prototyping based on open data and open-source software. SAGA is an activity-based multi-modal mobility scenario generator for the Simulation ofUrban MObiltiy (SUMO). Starting from an OpenStreetMap (OSM) file, SAGA extracts the data required to build an amulti-modal scenario, and in a step-by-step fashion, generates the configurations needed to execute it, including the intermediate steps required to refine the scenario with additional data, allowing the iterative improvement of realism and representativeness. The workflow implemented, extended, and automated by SAGA was developed while hand-crafting the Monaco SUMO Traffic (MoST) Scenario. Based on the fast prototyping capabilities added by SAGA, the creation of a multi-modal mobility scenario is readily achievable, and the incremental process to fine-tune it is supported by a workflow instead of being solely based on expert knowledge and experience. Based on previous experience, the generation of the first working prototype of a city-scale multi-modal mobility scenario may take months of work and expert knowledge. SAGAautomatically generates such a prototype, and all the intermediate configuration files are made available for further iterative improvements.

Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric Video

We present Neural Marionette, an unsupervised approach that discovers the skeletal structure from a dynamic sequence and learns to generate diverse motions that are consistent with the observed motion dynamics. Given a video stream of point cloud observation of an articulated body under arbitrary motion, our approach discovers the unknown low-dimensional skeletal relationship that can effectively represent the movement. Then the discovered structure is utilized to encode the motion priors of dynamic sequences in a latent structure, which can be decoded to the relative joint rotations to represent the full skeletal motion. Our approach works without any prior knowledge of the underlying motion or skeletal structure, and we demonstrate that the discovered structure is even comparable to the hand-labeled ground truth skeleton in representing a 4D sequence of motion. The skeletal structure embeds the general semantics of possible motion space that can generate motions for diverse scenarios. We verify that the learned motion prior is generalizable to the multi-modal sequence generation, interpolation of two poses, and motion retargeting to a different skeletal structure.

Hypergraph and Uncertain Hypergraph Representation Learning Theory and Methods

With the advent of big data and the information age, the data magnitude of various complex networks is growing rapidly. Many real-life situations cannot be portrayed by ordinary networks, while hypergraphs have the ability to describe and characterize higher order relationships, which have attracted extensive attention from academia and industry in recent years. Firstly, this paper described the development process, the application areas, and the existing review research of hypergraphs; secondly, introduced the theory of hypergraphs briefly; then, compared the learning methods of ordinary graphs and hypergraphs from three aspects: matrix decomposition, random walk, and deep learning; next, introduced the structural optimization of hypergraphs from three perspectives: dynamic hypergraphs, hyperedge weight optimization, and multimodal hypergraph generation; after that, the applicability of three uncertain hypergraph models were analyzed based on three uncertainty theories: probability theory, fuzzy set, and rough set; finally, the future research directions of hypergraphs and uncertain hypergraphs were prospected.

Wilderness Search for Lost Persons Using a Multimodal Aerial-Terrestrial Robot Team

Mobile robots that are capable of multiple modes of locomotion may have tangible advantages over unimodal robots in unstructured and non-homogeneous environments due to their ability to better adapt to local conditions. This paper specifically considers the use of a team of multimodal robots capable of switching between aerial and terrestrial modes of locomotion for wilderness search and rescue (WiSAR) scenarios. It presents a novel search planning method that coordinates the members of the robotic team to maximize the probability of locating a mobile target in the wilderness, potentially, last seen on an a priori known trail. It is assumed that the search area expands over time and, thus, an exhaustive search is not feasible. Earlier research on search planning methods for heterogeneous though unimodal search teams have exploited synergies between robots with different locomotive abilities through coordination and/or cooperation. Work on multimodal robots, on the other hand, has primarily focused on their mechanical design and low-level control. In contrast, our recent work, presented herein, has two major components: (i) target-motion prediction in the presence of a priori known trails in the wilderness, and (ii) probability-guided multimodal robot search-trajectory generation. For the former sub-problem, the novelty of our work lies in the formulation and use of 3D probability curves to capture target distributions under the influence of a priori known walking/hiking trails. For the latter, the novelty lies in the use of a tree structure to represent the decisions involved in multimodal probability-curve-guided search planning, which enables trajectory generation and mode selection to be optimized simultaneously, for example, via a Monte Carlo tree search technique. Extensive simulations, some of which are included herein, have shown that multimodal robotic search teams, coordinated via the trajectory planning method proposed in this paper, clearly outperform their unimodal counterparts in terms of search success rates.