Multimodal Architecture Research Articles

Channel cloning by multi-mode phase-sensitive parametric mixer.

We investigate high fidelity channel replication approaching the idealized notion of channel cloning with negligible excess noise and distortion. Previously proposed cloning architectures require that the channel carriers to be externally seeded, limiting their ultimate usefulness, whereas the self-seeded approach limits the channel number and signal-to-noise ratio. Specifically, when a single channel is replicated, the noise figure (NF) remains above the well-known 3-dB limit, and multi-channel replication by a dual-pump driven parametric mixer faces a theoretical NF limit of 6-dB. On the other hand, large-channel-count cloning is of particular importance as it allows for rate scaling in generalized signal processing. Recognizing the limits of conventional architectures that rest on homogeneous parametric mixers, we here propose multi-stage, dispersion-managed parametric mixers in multi-mode phase-sensitive architecture to clone the input signal to a substantial number of channels. In particular, when the new mixer is operated in four-mode phase-sensitive architecture, a 17-copy-count channel cloning with maximum NF less than 6-dB and a record-low NF of 2-dB was experimentally implemented and demonstrated in this paper.

Multi-step self-guided pathways for shape-changing metamaterials.

Multi-step pathways-which consist of a sequence of reconfigurations of a structure-are central to the functionality of various natural and artificial systems. Such pathways execute autonomously in self-guided processes such as protein folding1 and self-assembly2-5, but have previously required external control to execute in macroscale mechanical systems, provided by, for example, actuators in robotics6-9 or manual folding in origami8,10-12. Here we demonstrate shape-changing, macroscale mechanical metamaterials that undergo self-guided, multi-step reconfiguration in response to global uniform compression. We avoid the need for external control by using metamaterials that are made purely of passive components. The design of the metamaterials combines nonlinear mechanical elements with a multimodal architecture that enables a sequence of topological reconfigurations caused by the formation of internal self-contacts between the elements of the metamaterial. We realize the metamaterials by using computer-controlled water-jet cutting of flexible materials, and show that the multi-step pathway and final configuration can be controlled by rational design of the nonlinear mechanical elements. We also demonstrate that the self-contacts suppress errors in the pathway. Finally, we create hierarchical architectures to extend the number of distinct reconfiguration steps. Our work establishes general principles for designing mechanical pathways, opening up new avenues for self-folding media11,12, pluripotent materials9,13 and pliable devices14 in areas such as stretchable electronics and soft robotics15.

Is a Picture Worth a Thousand Words? A Deep Multi-Modal Architecture for Product Classification in E-Commerce

Classifying products precisely and efficiently is a major challenge in modern e-commerce. The high traffic of new products uploaded daily and the dynamic nature of the categories raise the need for machine learning models that can reduce the cost and time of human editors. In this paper, we propose a decision level fusion approach for multi-modal product classification based on text and image neural network classifiers. We train input specific state-of-the-art deep neural networks for each input source, show the potential of forging them together into a multi-modal architecture and train a novel policy network that learns to choose between them. Finally, we demonstrate that our multi-modal network improves classification accuracy over both networks on a real-world large-scale product classification dataset that we collected from Walmart.com. While we focus on image-text fusion that characterizes e-commerce businesses, our algorithms can be easily applied to other modalities such as audio, video, physical sensors, etc.

Bayesian optimization on graph-structured search spaces: Optimizing deep multimodal fusion architectures

Abstract A popular testbed for deep learning has been multimodal recognition of human activity or gesture involving diverse inputs like video, audio, skeletal pose and depth images. Deep learning architectures have excelled on such problems due to their ability to combine modality representations at different levels of nonlinear feature extraction. However, designing an optimal architecture in which to fuse such learned representations has largely been a non-trivial human engineering effort. We treat fusion structure optimization as a hyperparameter search and cast it as a discrete optimization problem under the Bayesian optimization framework. We propose two methods to compute structural similarities in the search space of tree-structured multimodal architectures, and demonstrate their effectiveness on two challenging multimodal human activity recognition problems.

Multimodal architecture for video captioning with memory networks and an attention mechanism

Automatically describing videos containing rich and open-domain activities is a very challenging task for computer vision and machine learning research. Obviously, accurate descriptions of video contents need the understanding of both visual concepts and their temporal dynamics. A lot of efforts have been made to understand visual concepts in still image tasks, e.g., image classification and object detection. However, the combination of visual concepts and temporal dynamics has not been given sufficient attention. To delve deeper into the unique characteristic of videos, we propose a novel video captioning architecture to integrate both visual concepts and temporal dynamics. In this paper, an attention mechanism and memory networks are combined together into our multimodal framework with a feature selection algorithm. Specially, we utilize the soft attention mechanism to choose visual concepts relevant frames based on previously generated words, and the memorization of temporal dynamics is implemented by the memory networks, which have great advantages of memorizing long-term information. Then the visual concepts and the temporal dynamics are integrated together into our multimodal architecture. Moreover, the feature selection algorithm is applied to select more relevant features between them according to the part of speech. Finally, we test our proposed framework on both MSVD and MSR-VTT datasets and achieve competitive performance compared with other state-of-the-art methods.

Dynamic configuration management of a multi-standard and multi-mode reconfigurable multi-ASIP architecture for turbo decoding

The multiplication of connected devices goes along with a large variety of applications and traffic types needing diverse requirements. Accompanying this connectivity evolution, the last years have seen considerable evolutions of wireless communication standards in the domain of mobile telephone networks, local/wide wireless area networks, and Digital Video Broadcasting (DVB). In this context, intensive research has been conducted to provide flexible turbo decoder targeting high throughput, multi-mode, multi-standard, and power consumption efficiency. However, flexible turbo decoder implementations have not often considered dynamic reconfiguration issues in this context that requires high speed configuration switching. Starting from this assessment, this paper proposes the first solution that allows frame-by-frame run-time configuration management of a multi-processor turbo decoder without compromising the decoding performances.

A multi-modal architecture for non-intrusive analysis of performance in the workplace

Human performance, in all its different dimensions, is a very complex and interesting topic. In this paper we focus on performance in the workplace which, asides from complex is often controversial. While organizations and generally competitive working conditions push workers into increasing performance demands, this does not necessarily correlates positively to productivity. Moreover, existing performance monitoring approaches (electronic or not) are often dreaded by workers since they either threat their privacy or are based on productivity measures, with specific side effects. We present a new approach for the problem of performance monitoring that is not based on productivity measures but on the workers' movements while sitting and on the performance of their interaction with the machine. We show that these features correlate with mental fatigue and provide a distributed architecture for the non-intrusive and transparent collection of this data. The easiness in deploying this architecture, its non-intrusive nature, the potential advantages for better human resources management and the fact that it is not based on productivity measures will, in our belief, increase the willingness of both organizations and workers to implement this kind of performance management initiatives.

Topology Generation for Hybrid Electric Vehicle Architecture Design

Existing hybrid powertrain architectures, i.e., the connections from engine and motors to the vehicle output shaft, are designed for particular vehicle applications, e.g., passenger cars or city buses, to achieve good fuel economy. For effective electrification of new applications (e.g., heavy-duty trucks or racing cars), new architectures may need to be identified to accommodate the particular vehicle specifications and drive cycles. The exploration of feasible architectures is combinatorial in nature and is conventionally based on human intuition. We propose a mathematically rigorous algorithm to enumerate all feasible powertrain architectures, therefore enabling automated optimal powertrain design. The proposed method is general enough to account for single and multimode architectures as well as different number of planetary gears (PGs) and powertrain components. We demonstrate through case studies that our method can generate the complete sets of feasible designs, including the ones available in the market and in patents.

A novel Li-ion battery charger using multi-mode LDO configuration based on 350 nm HV-CMOS

The design of a novel Li-ion battery charger using multi-mode LDO architecture has been introduced in this paper. The proposed architecture, using an improved multi-mode LDO, not only obtains high accuracy but also reduces power supply noise because of utilizing novel error amplifier and power buffer configuration; while still consuming low power dissipation. To obtain the low power consumption, the Schmitt Trigger technique is applied to the charging controller and an optimized current driven circuit is proposed. Besides, the PSRR parameter is also enhanced by adding pre-regulation circuit in multi-mode LDO circuit. Thus, the proposed Li-ion battery charger achieves 700 mA operation current with 70.9 % efficiency but only dissipates 495 mW in power. During the charging process, the setting time and ripple issues are solved by the use of soft-start circuit which is integrated into the charging controller in order to decrease the chip area. Therefore, the setting time is reduced to 5.5 μs while gaining the load regulation at approximately 0.019 μV/mA and increasing PSRR up to 106 dB at DC level. Moreover, the line regulation is also reduced at 1.3 mV/V. The proposed linear battery charger is designed and implemented, based on High-Voltage CMOS process with using 4.5 V power supply voltage and obtaining 4.2 V battery output voltage.

Kernel Manifold Alignment for Domain Adaptation.

The wealth of sensory data coming from different modalities has opened numerous opportunities for data analysis. The data are of increasing volume, complexity and dimensionality, thus calling for new methodological innovations towards multimodal data processing. However, multimodal architectures must rely on models able to adapt to changes in the data distribution. Differences in the density functions can be due to changes in acquisition conditions (pose, illumination), sensors characteristics (number of channels, resolution) or different views (e.g. street level vs. aerial views of a same building). We call these different acquisition modes domains, and refer to the adaptation problem as domain adaptation. In this paper, instead of adapting the trained models themselves, we alternatively focus on finding mappings of the data sources into a common, semantically meaningful, representation domain. This field of manifold alignment extends traditional techniques in statistics such as canonical correlation analysis (CCA) to deal with nonlinear adaptation and possibly non-corresponding data pairs between the domains. We introduce a kernel method for manifold alignment (KEMA) that can match an arbitrary number of data sources without needing corresponding pairs, just few labeled examples in all domains. KEMA has interesting properties: 1) it generalizes other manifold alignment methods, 2) it can align manifolds of very different complexities, performing a discriminative alignment preserving each manifold inner structure, 3) it can define a domain-specific metric to cope with multimodal specificities, 4) it can align data spaces of different dimensionality, 5) it is robust to strong nonlinear feature deformations, and 6) it is closed-form invertible, which allows transfer across-domains and data synthesis. To authors’ knowledge this is the first method addressing all these important issues at once. We also present a reduced-rank version of KEMA for computational efficiency, and discuss the generalization performance of KEMA under Rademacher principles of stability. Aligning multimodal data with KEMA reports outstanding benefits when used as a data pre-conditioner step in the standard data analysis processing chain. KEMA exhibits very good performance over competing methods in synthetic controlled examples, visual object recognition and recognition of facial expressions tasks. KEMA is especially well-suited to deal with high-dimensional problems, such as images and videos, and under complicated distortions, twists and warpings of the data manifolds. A fully functional toolbox is available at https://github.com/dtuia/KEMA.git.

Electronic health record analysis via deep Poisson factor models

Electronic Health Record (EHR) phenotyping utilizes patient data captured through normal medical practice, to identify features that may represent computational medical phenotypes. These features may be used to identify at-risk patients and improve prediction of patient morbidity and mortality. We present a novel deep multi-modality architecture for EHR analysis (applicable to joint analysis of multiple forms of EHR data), based on Poisson Factor Analysis (PFA) modules. Each modality, composed of observed counts, is represented as a Poisson distribution, parameterized in terms of hidden binary units. Information from different modalities is shared via a deep hierarchy of common hidden units. Activation of these binary units occurs with probability characterized as Bernoulli-Poisson link functions, instead of more traditional logistic link functions. In addition, we demonstrate that PFA modules can be adapted to discriminative modalities. To compute model parameters, we derive efficient Markov Chain Monte Carlo (MCMC) inference that scales efficiently, with significant computational gains when compared to related models based on logistic link functions. To explore the utility of these models, we apply them to a subset of patients from the Duke-Durham patient cohort. We identified a cohort of over 16,000 patients with Type 2 Diabetes Mellitus (T2DM) based on diagnosis codes and laboratory tests out of our patient population of over 240,000. Examining the common hidden units uniting the PFA modules, we identify patient features that represent medical concepts. Experiments indicate that our learned features are better able to predict mortality and morbidity than clinical features identified previously in a large-scale clinical trial.

Slot-Mode Optomechanical Crystals: A Versatile Platform for Multimode Optomechanics.

Cavity optomechanical systems are being studied for their potential in areas such as metrology, communications, and quantum information science. For a number of recently proposed applications in which multiple optical and mechanical modes interact, an outstanding challenge is to develop multimode architectures that allow flexibility in the optical and mechanical sub-system designs while maintaining the strong interactions that have been demonstrated in single-mode systems. To that end, we demonstrate slot-mode optomechanical crystals, devices in which photonic and phononic crystal nanobeams separated by a narrow slot are coupled via optomechanical interactions. These nanobeam pairs are patterned to confine a mechanical breathing mode at the center of one beam and a low-loss optical mode in the slot between the beams. This architecture affords great design flexibility towards multimode optomechanics, as well as substantial optomechanical coupling rates. We show this by producing slot-mode devices in stoichiometric Si3N4, with optical modes in the 980 nm band coupled to mechanical modes at 3.4 GHz, 1.8 GHz, and 400 MHz. We exploit the Si3N4 tensile stress to achieve slot widths down to 24 nm, which leads to enhanced optomechanical coupling, sufficient for the observation of optomechanical self-oscillations at all studied frequencies. We then develop multimode optomechanical systems with triple-beam geometries, in which two optical modes couple to a single mechanical mode, and two mechanical modes couple to a single optical mode. Taken together, these results demonstrate great flexibility in the design of multimode chip-scale optomechanical systems with large optomechanical coupling.

Discovery and registration of components in multimodal systems distributed on the IoT

One of the major gaps in the current HTML5 web platform, is the lack of interoperable means for a multimodal application to discover services and applications available in a given space and network for example, in smart houses or in applications for the Internet of Things. To address this gap, we produced a SOA approach that was submitted to the W3C׳s Multimodal Working Group and validated as a draft recommendation that aims to allow the discovery and registration of components used in multimodal systems for the web of things. In this approach, the components are described and virtualized in a module communicating through two dedicated events, they are registered in a Resources Manager to facilitate the fine management of concurrent multimodal interactions, and the system supports an interoperable discovery, registration and filtering of the features provided by heterogeneous and dynamic components in the web of things. In this paper we will present the W3C׳s Multimodal Architecture and Interfaces standard and explain the Discovery and Registration mechanism adopted to address the need of finding and integrating components into distributed and dynamic multimodal systems.

Configurable Architectures for Multi-Mode Floating Point Adders

This paper presents two architectures for floating point (FP) adders, which operates in multi-mode configuration with multi-precision support. First architecture (named QPdDP) works in dual-mode which can operates either for quadruple precision or two-parallel double precision. The second architecture (named QPdDPqSP) works in tri-mode which is able to compute either of a quadruple precision, two-parallel double precision and four-parallel single precision computations. The architectures are based on the standard state-of-the-art flow for FP adder which supports the computation of normal and sub-normal operands, along with the support for the exceptional case handling. The key components in the architecture, such as comparator, swap, dynamic shifters, leading-one-detector (LOD), mantissa adders/subtractors, and rounding circuit, are re-designed and optimized for multi-mode computation, to enable efficient resource sharing for multi-precision operands. The data-path in each multi-mode architecture is tuned for multi-precision support with minimal multiplexing circuitry overhead. These proposed architectures provide multi-precision SIMD support for lower precision operands, along with high precision computational support, and thus, have a better resource utilization. A fully pipelined version of both adder architectures are presented. The proposed adder architectures are synthesized using UMC 90 nm technology ASIC implementation. The proposed architectures are compared with the best available literature works, and have shown better design metrics in terms of area, delay and $area\times period$ , along with more computational support.

Efficient multi-Gb/s multi-mode LDPC decoder architecture for IEEE 802.11ad applications

This paper presents a novel multi-Gb/s multi-mode LDPC decoder architecture and efficient design techniques for gigabit wireless communications. An efficient dynamic and fixed column-shifting scheme is presented for multi-mode architectures. A novel low-complexity local switch is proposed to implement the dynamic and fixed column-shifting scheme. Furthermore, an efficient quantization method and the usage of a one׳s-complement scheme instead of a two׳s-complement scheme are explored. The proposed decoder achieves very high throughput with minimal area overhead. Post layout results using TSMC 65-nm CMOS technology shows much better throughput, as well as better area- and energy-efficiency, compared to other multi-mode LDPC decoders.

Principles of Construction of Polymodal Info-Communication Systems based on Multimodal Architectures of Subscriber’s Terminals

Analysis of the existing polymodal interfaces, their main characteristics and applications, as well as the results of common investigations in the field of multimodal interaction and interface design led to make a conclusion about the possibility of building a polymodal infocommunication systems based on multimodal architectures of subscriber’s terminals. To solve the tasks of interpersonal communication through technical means of communication the principles of polymodal systems construction and hierarchical system of their models are suggested in the article.

Prototyping using a Pattern Technique and a Context-Based Bayesian Network in Multimodal Systems

Abstract Today, technology allows us to produce extensive multimodal systems which are totally under human control. These systems are equipped with multimodal interfaces, which enable more natural and more efficient interaction between man and machine. End users can take advantage of natural modalities (e.g. audio, eye gaze, speech, gestures, etc.) to communicate or exchange information with applications. In this work, we assume that a number of these modalities are available to the user. In this paper, we present a prototype of a multimodal architecture, and show how modality selection and fission algorithms are implemented in such a system. We use a pattern technique to divide a complex command into elementary subtasks and select suitable modalities for each of them. We integrate a context-based method using a Bayesian network to resolve ambiguous or uncertain situations.

On the development of high-throughput and area-efficient multi-mode cryptographic hash designs in FPGAs

In this paper, area-efficient and high-throughput multi-mode architectures for the SHA-1 and SHA-2 hash families are proposed and implemented in several FPGA technologies. Additionally a systematic flow for designing multi-mode architectures (implementing more than one function) of these families is introduced. Compared to the corresponding architectures that are produced by a commercial synthesis tool, the proposed ones are better in terms of both area (at least 40%) and throughput/area (from 32% up to 175%). Finally, the proposed architectures outperform similar existing ones in terms of throughput and throughput/area, from 4.2× up to 279.4× and from 1.2× up to 5.5×, respectively.

Speech-centric Multimodal Interaction for Easy-to-access Online Services – A Personal Life Assistant for the Elderly

The PaeLife project is a European industry-academia collaboration whose goal is to provide the elderly with easy access to online services that make their life easier and encourage their continued participation in the society. To reach this goal, the project partners are developing a multimodal virtual personal life assistant (PLA) offering a wide range of services from weather information to social networking. This paper presents the multimodal architecture of the PLA, the services provided by the PLA, and the work done in the area of speech input and output modalities, which play a key role in the application.

Area-Efficient Multimode Encoding Architecture for Long BCH Codes

This brief presents a new area-efficient multimode encoder for long Bose-Chaudhuri-Hocquenghen codes. In the proposed multimode encoding architecture, several short linear-feedback shift registers (LFSRs) are cascaded in series to achieve the same functionality that a long LFSR has, and the output of a short LFSR is fed back to the input side to support multimode encoding. Whereas previous multimode architectures necessitate huge overhead due to preprocessing and postprocessing, the proposed architecture completely eliminates the overhead by exploiting an efficient transformation. Without sacrificing the latency, the proposed architecture reduces hardware complexity by up to 97.2% and 49.1% compared with the previous Chinese-remainder-theorem-based and weighted-summation-based multimode architectures, respectively.