Multimodal Constraints Research Articles

MultiSC: a deep learning pipeline for analyzing multiomics single-cell data.

Single-cell technologies enable researchers to investigate cell functions at an individual cell level and study cellular processes with higher resolution. Several multi-omics single-cell sequencing techniques have been developed to explore various aspects of cellular behavior. Using NEAT-seq as an example, this method simultaneously obtains three kinds of omics data for each cell: gene expression, chromatin accessibility, and protein expression of transcription factors (TFs). Consequently, NEAT-seq offers a more comprehensive understanding of cellular activities in multiple modalities. However, there is a lack of tools available for effectively integrating the three types of omics data. To address this gap, we propose a novel pipeline called MultiSC for the analysis of MULTIomic Single-Cell data. Our pipeline leverages a multimodal constraint autoencoder (single-cell hierarchical constraint autoencoder) to integrate the multi-omics data during the clustering process and a matrix factorization-based model (scMF) to predict target genes regulated by a TF. Moreover, we utilize multivariate linear regression models to predict gene regulatory networks from the multi-omics data. Additional functionalities, including differential expression, mediation analysis, and causal inference, are also incorporated into the MultiSC pipeline. Extensive experiments were conducted to evaluate the performance of MultiSC. The results demonstrate that our pipeline enables researchers to gain a comprehensive view of cell activities and gene regulatory networks by fully leveraging the potential of multiomics single-cell data. By employing MultiSC, researchers can effectively integrate and analyze diverse omics data types, enhancing their understanding of cellular processes.

Hierarchical Bayesian spectral regression with shape constraints for multi-group data

We propose a hierarchical Bayesian (HB) model for multi-group analysis with group–specific, flexible regression functions. The lower–level (within group) and upper–level (between groups) regression functions have hierarchical Gaussian process priors. HB smoothing priors are developed for the spectral coefficients. The HB priors smooth the estimated functions within and between groups. The HB model is particularly useful when data within groups are sparse because it shares information across groups, and provides more accurate estimates than fitting separate nonparametric models to each group. The proposed model also allows shape constraints, such as monotone, U and S–shaped, and multi-modal constraints. When appropriate, shape constraints improve estimation by recognizing violations of the shape constraints as noise. The model is illustrated by two examples: monotone growth curves for children, and happiness as a convex, U-shaped function of age in multiple countries. Various basis functions could also be used, and the paper also implements versions with B-splines and orthogonal polynomials.

Multimodal Analysis of Selected Health Awareness Materials: A Visual Grammar Approach

The Visual Grammar approach is effective in examining the communicative value of healthcare posters. As semiotic resources, these posters comprise both images and text for persuasive and informative ends. Previous studies have looked at multilingual health care texts and medical information leaflets using different approaches, but the focus here was monolingual healthcare posters written in English. Hence, this study analyzed five purposefully selected online healthcare posters meant for a Nigerian audience. It looks at the ways image and text cohere as syntagm within narrative and conceptual representations to form persuasive and informative messages, and the multimodal constraints on the healthcare posters as they serve communicative ends. A descriptive research design was adopted to analyse corpus from five online health care materials. Features of the materials were identified and classified according to manifested multimodal features before they were subjected to narrative and conceptual analyses. The way image and text cohere; their functions and communicative values were also discussed to highlight attendant constraints. Findings show that as a multimodal text the healthcare posters employ semiotic resources to project messages as either contextualized or decontextualized. Major and minor semiotic processes, achieved by image and text coherence, form both narrative and conceptual meaning-making units. Added to this informative value are colour variations and font size which are also employed as enhancers of detail, mood and contrast as well as attention drawing devices. Logos of sponsors give the health messages varying credibility values as rhetorical questions and advice-clauses serve persuasive ends. Thus, the study concluded that the communicative potential of a healthcare poster are hinged on semiotic resources that constrain readers’ or viewers’ perceptions on health matters.

Research on the multimodal fusion calibration method of the binocular visual system in a large-scale FOV

Calibration is vital for the binocular visual system. The high-precision measurement volume of the traditional binocular visual system is very small; therefore, it is difficult to measure large objects accurately. Accurate calibration of the binocular visual system is also extremely difficult in a large-scale field-of-view (FOV). Several attempts have been made to calibrate the binocular visual system in a large-scale FOV without recognizing the importance of depth information. A new method based on multimodal constraints has been proposed to calibrate the binocular visual system in a large-scale FOV whose height-width-depth is 2*2*2 . The method fuses multiple constraints, i.e. the distance error constraint, the reprojection error constraint, the epipolar error constraint, the spatial points error constraint, and the regularization constraint. An objective function that integrates the above constraints has been derived. The construction method of the global Jacobi matrix, the adaptive weight adjustment method, and the Levenberg–Marquardt iterative algorithm are used to minimize the objective function. Comparison experiments have been carried out to validate the performance of our proposed method by measuring a 1.4 m long calibration gauge. The results show that the root mean square (RMS) of the residual error of our method is 0.0929 mm, Zhang’s traditional method is 0.523 mm, and Xiao’s photogrammetric method is 0.348 mm. The RMS of our method is 82% less than that of Zhang’s method. Noise experiments show that the RMS of the residual error of our method is 45% less than Zhang’s method with 0.1 level noise inference, which verifies the stability and reliability of our method.

R2DIO: A Robust and Real-Time Depth-Inertial Odometry Leveraging Multimodal Constraints for Challenging Environments

R²DIO: A Robust and Real-Time Depth-Inertial Odometry Leveraging Multimodal Constraints for Challenging Environments

Practices of patient participation: Getting a turn during hospital ward rounds

Patient participation is a fundamental principle in modern Western health care, but not necessarily simple to achieve. During hospital ward rounds, patient participation is further hindered by the multi-party nature of the encounter: at times, members of the medical team talk with each other rather than with the patient. This article examines patients’ opportunities to participate in ward round conversations when the patient is not the addressed recipient. The data consist of 3 hours of video-recorded ward rounds in a Finnish hospital. Using conversation analysis, we study patients’ practices for getting a turn in different sequential environments. The patients monitor the ongoing conversation and exploit its sequential organisation by producing responsive turns and repair initiations, thus becoming active participants. They also produce their own initiatives, although sequential and multimodal constraints affect their possibilities for modifying the participation framework. The results of this study can be exploited to promote patient participation.

A Multi-Objective Scheduling Algorithm for Multi-Mode Resource Constrained Projects in the Presence of Uncertain Resource Availability

Goal: The issue of resource allocation is a major concern for project engineers in the scheduling process of a project. Resources over-allocation are often seen in practice after the scheduling of a project, which makes scheduling unhelpful. Modifying an over-allocated schedule is very complicated and requires a lot of effort and time. Besides, during the scheduling process of resource-constrained projects in the constructing companies, managers should concern more than one objective at the same time. This research aims to propose a new heuristic algorithm for minimizing project completion time, cost or maximizing quality of execution of activities simultaneously while multi-mode activities are taken into consideration.
 Design / Methodology / Approach: In this research, a new heuristic method is proposed for solving multi-objective scheduling problem for multi-mode resource constrained project scheduling problems (MRCPSPs) where the aim is maximizing the net present value (NPV) of project, minimize completion time and maximize the quality of executing activities simultaneously and along with emerging the uncertainty of resources availability and activity durations. The proposed method is then coded by Matlab® 2016.
 Results: The outcomes of solving small, medium and large scale case studies, the following results achieved: (i) the algorithm could solve all problems in different circumstances with no difficulties; (ii) the large scale problems (with 200 activities, 20 resources and 3 execution modes for each activity) could be solved in 4.43 seconds. (iii) in none of the studied cases over-allocation problem. The proposed method can be considered among the fastest scheduling algorithms found in the literature. In addition, it is found that makespan, NPV and quality have co-relation must be taken into consideration during the scheduling process.
 Limitations of the investigation: The main limitations of this research is that it only covers resource constrained project scheduling. Moreover, risk factors associated with the objectives of this research have yet to be addressed in future research studies.
 Practical implications: The performance of the algorithm is validated by using 24 series of dataset that are found in the literature. In order to verify its performance in real practice, it has been applied for a part of a construction project in Malaysia. The outcomes indicated that the algorithm scheduled the problem with 23 activities, 5 constrained resources and 2 execution modes in less than a second and with no over-allocations. The proposed multi-objective algorithm allows the project managers to consider NPV, completion time and quality of activities while scheduling a multi-mode project. In practice, this algorithm can provide a better atmosphere for managers while they aim to consider more than one objective during the scheduling process.
 Originality / Value: The proposed algorithm is original and can be of great value for future studies and managers in preventing resource over-allocation during the scheduling of multi-objective multi-mode resource constraint project scheduling. Moreover, it can help project managers to find near optimum solutions for complex multi-objective resource constraint projects faster and also with more accuracy.

Multi-modal constraint propagation via compatible conditional distribution reconstruction

In many applications, the pairwise constraint is a kind of weaker supervisory information which can be collected more efficiently than the label information. The constraint propagation is a constrained clustering approach, which has been proved to be a success of exploiting such side-information. Recent years have witnessed many methods of multi-modal constraint propagation. However, the problem of reasonably fusing different modalities under the framework of constraint propagation remains unaddressed. In this paper, we first identify a necessary and sufficient condition for compatible conditional distributions under a specific assumption and address the problem of Compatible Conditional Distributions Reconstruction (CCDR). With the help of CCDR, we propose a multi-modal constraint propagation method dubbed Instance Level Multi-Modal Constraint Propagation (ILMCP). ILMCP fuses the affinity of different modalities at the data instance level instead of the modality level and constructs a unified affinity matrix. Extensive experiments on two publicly available multi-modal datasets show the superior performance of the proposed method.

On Multi-modal Fusion Learning in constraint propagation

Constraint propagation methods demonstrate splendid performance in constrained clustering tasks. Although some multi-modal constraint propagation methods have been proposed in recent years, a feasible and robust approach to multi-modal feature fusion in pairwise constraint propagation is still in demand. This paper presents a novel multi-modal fusion approach in order to cope with the constraint propagation on multi-modal datasets, called Multi-modal Fusion Learning (MFL). The proposed method can reach a multi-modal fusion based on the observed constraint information and the propagation process. It is capable of handling any number of modalities without any prior knowledge of each modality. We merge the fusion learning and constraint propagation into one unified problem and solve it by a bound-constrained quadratic optimization. Our proposed method has been tested in clustering tasks on two publicly available multi-modal datasets to show its superior performance.

Developing a new method for modifying over-allocated multi-mode resource constraint schedules in the presence of preemptive resources

Developing a new method for modifying over-allocated multi-mode resource constraint schedules in the presence of preemptive resources

Research on Analysis of Sports Video Multi-Pattern Fusion

In order to effectively integrate multimodal information and multilayer constraints, we present a unified probabilistic framework for sports video analysis. Based the framework, three instances of statistical models are constructed and compared. Experimental results indicate our method with multimodal fusion processes semantic events in sports video more effectively. With the method based on statistics, we'll discuss further sport video content analysis method fusing multimode information. Semantic events in sport videos are in essence multimodal. In television relay, multimode information is integrative used to present video contents like subtitles, narrator's voices, on-site sounds, camera movements, scenarios and images etc. It is incomplete to analyze only one mode. For more effective analysis of events, it's required to study the analytical method which fuses multiple patterns. On the other hand, semantic events in those videos are not isolated. There's some logical or consequential relationship among them. In previous paper, we discussed event detection and recognition with the use of the contextual relationship based on dynamic Bayes network. Now on that basis, we'll explore how to fuse multimode information, which is a key issue we're facing here (1-2). In recent years, the fusion of multimode information has become a hot topic in the field of sport video analysis (3-5). Firstly we introduce the related work. Most of the multimode fusion analysis methods mentioned in previous literatures considered the detection of an isolated event. Unlike them, we propose to detect many events and analyze comprehensively the association among them. Multi-level analysis methods based on statistics are built on the probabilistic graphical model, such as Hidden Markov Model (HMM), dynamic Bayes network (DBN) and their variants. By combining visual graphical model representation and effective reasoning and learning methods, such solutions made fairly good effects. Xie (6-7) et al. applied hierarchical HMM for unsupervised clustering to discover layered structure of video contents. Differently, we introduced the method based on dynamic Bayes network model to do the same work. Through learning of training samples, we fulfilled the detection and recognition of wonderful events in the football match (8). Based on DBN, we developed a common sport video analysis framework. Compared with previous work mentioned above, our approach can not only integrate multimode information for event detection and also deal with hierarchical relationship among events. Although the two functions were stated in previous papers, they as a whole were not paid

Modified shuffled frog leaping algorithm for simulation capability scheduling problem

Based on the analysis of characteristics of simulation capability scheduling problem in cloud simulation platform, this paper gives its mathematical description and introduces a modified shuffled frog leaping (MSFL) algorithm to solve the above optimisation problem with multi-mode constraint. The MFSL introduces GA to code the feasible solution space. During the random execution of coding, decoding and mutation, it increases three layers of coding constraints including simulation capability, task logic and feasible mode, to ensure the randomness of the solving process in the controllable scope. Thus, it can reduce the search range of solution space, get rid of the meaningless illegal solution, and ultimately improve the convergence speed of the algorithm and avoid precocity.

Ensemble mutable smart bee algorithm and a robust neural identifier for optimal design of a large scale power system

The aim of the current study is to probe the potentials of ensemble bio-inspired approaches to handle the deficiencies associated with designing large scale power systems. Ensemble computing has been proven to be a very promising paradigm. The fundamental motivation behind designing such bio-inspired optimization models lies in the fact that interactions among different sole optimizers can afford much better income as compared with an individual optimizer. To do so, the authors propose an optimization technique called ensemble mutable smart bee algorithm (E-MSBA) which is based on the aggregation of several independent low-level optimizers. Here, each low-level unit of the proposed ensemble framework uses mutable smart bee algorithm (MSBA) for optimization procedure. The main provocations behind selecting MSBAs of different properties as components of ensemble are twofold. On the one hand, MSBA proved its capability for handling multimodal constraint problems. On the other hand, based on different experiments, it was demonstrated that MSBA can find the optimum solution with a relatively low computational cost. In this study, the authors intend to indicate that the proposed ensemble paradigm can efficiently optimize the operating parameters of a large scale power system which includes different mechanical components. To this end, E-MSBA and some rival methods are taken into account for the optimization procedure. The obtained results reveal that E-MSBA inherits some positive features of the MSBA algorithm. Additionally, it is observed that the ensembling approach enables the proposed method to effectively tackle the flaws associated with optimization of large scale problems.

Reconstructing shape boundaries with multimodal constraints

Shape primitives are a valuable input for reconstructing 3D models from point clouds. In this paper we present a method for clipping simple shape primitives at reasonable boundaries. The shape primitives, e.g. planes or cylinders, are 2D manifolds which are automatically detected in unstructured point clouds. Shape boundaries are necessary for generating valid 3D models from multiple shape primitives, because shape primitives possibly have dimensions of infinite extent or they are only partially present in the scene. Hints for reasonable boundaries of shape primitives are indicated by different input sources and constraints. Point clouds and range images provide information where shape primitives coincide with measured surface points. Edge detectors offer cues for surface boundaries in color images. The set of shape primitives is analyzed for constraints such as intersections. Due to an iterative approach, intermediate results provide additional constraints such as coplanar boundary points over multiple shape primitives. We present a framework for extracting and optimizing shape boundaries based on the given input data and multiple constraints. Further, we provide a simple user interface for manually adding constraints in order to improve the results. Our approach generates structurally simple 3D models from shape primitives and point clouds. It is useful for reconstructing scenes containing man-made objects, such as buildings, interior scenes, or engineering objects. The application of multiple constraints enables the reconstruction of proper 3D models despite noisy or incomplete point clouds.

Simultaneous localization and mapping with multimodal probability distributions

Simultaneous Localization and Mapping (SLAM) has focused on noisy but unique data associations resulting in linear Gaussian uncertainty models. However, a unique decision is often not possible using only local information, giving rise to ambiguities that have to be resolved globally during optimization. To solve this problem, the pose graph data structure is extended here by multimodal constraints modeled by mixtures of Gaussians (MoG). Furthermore, optimization methods for this novel formulation are introduced, namely (a) robust iteratively reweighted least squares, and (b) Prefilter Stochastic Gradient Descent (SGD) where a preprocessing step determines globally consistent modes before applying SGD. In addition, a variant of the Prefilter method (b) is introduced in form of (c) Prefilter Levenberg–Marquardt. The methods are compared with traditional state-of-the-art optimization methods including (d) Stochastic Gradient Descent and (e) Levenberg–Marquardt as well as (f) Particle filter SLAM and with (g) an optimal exhaustive algorithm. Experiments show that ambiguities significantly impact state-of-the-art methods, and that the novel Prefilter methods (b) and (c) perform best. This is further substantiated with experiments using real-world data. To this end, a method to generate MoG constraints from a plane-based registration algorithm is introduced and used for 3D SLAM under ambiguities.

Optimal design of constraint engineering systems: application of mutable smart bee algorithm

In the current investigation, a new optimisation technique called mutable smart bee algorithm (MSBA) is used for optimal design of real-life engineering systems that are subjected to different types of constraints. MSBA is a memory-based diversified optimisation technique that hires mutable smart bee (MSB) instead of conventional bee. MSB heuristic agents are capable of maintaining their historical memory for the location and quality of food sources and also a little chance of mutation is considered for them. Exerted experiments reveal that these features are really effective for optimising multi-modal constraint problems. To elaborate on the authenticity of MSBA, obtained results are compared to state-of-the-art optimisation techniques.

Self-organizing neural maps for multi-modal associations

To move in a complex and dynamic environment, according to the active perception theory, an agent needs to learn the multi modal correlations between its actions and the changes they induce in the environment [1]. Human beings interact with their environment through several distant organs, whose sensory flow processing are influencing each others as, for example, in the Mc Gurk effect [2]. In the functional view of the cortex, each sensory area processes a specific sensory flow and associative areas integrate these flows in a consistent representation of the world that influences in return uni-modal perceptions as for the ventriloquist effect [3]. At a mesoscopic level, the cortex shows a generic structure composed of multi-layer cortical columns. We propose a bio inspired model of perceptive map which, using a continuous and unsupervised learning, self-organizes to map a sensory data flow. Thus, using a spatial competition mechanism, a perceptive map provides an activity bump representing the current perception. In a multi modal architecture, by connecting multiple perceptive maps to an associative map with reciprocal spatially constrained connectivity, the spatial organization of all perceptions have to relax the multi modal constraints (see figure 1A). Thus, this multimodal architecture provides a way to learn multi modal correlations with generalization. Figure 1 A Example of use of our perceptive map model in a multi modal architecture with two sensors and one actuator. B Generic structure of one cortical column in a perceptive map. More precisely, each perceptive map consists of generic cortical columns with computing and learning which are local and decentralized (see figure1 B). Sensitive layer receives the feed-forward flow coming from a sensor and uses the BCM learning rule [4]. This synaptic learning rule is based on the hebbian one and is able to autonomously raise a selectivity to one stimulus of the upcoming flow. From a computational point of view, the BCM rule uses a sliding threshold between long term potentiation and long term depression, which has been confirmed by biological evidences [5]. The cortical layer receives feedback influence coming from the associative map, whose activity represents the multi modal perception. The perceptive layer is based on the neural field theory [6] and it receives the sensitive activity modulated by the cortical one. Thanks to the competitive mechanism introduced by the lateral connectivity with a difference of Gaussian shape, an activity bump emerges at the map level, where the activity is the most spatially consistent, representing a consensus between the local sensation and the multi modal constraints. To obtain a self-organization of the sensitive layer at the map level, the perceptive layer modulates the BCM activity, so that the spatial consistency of the activity bump is propagated to the selectivity organization. We have also introduced an unlearning term in the BCM learning rule in order to forget the current selectivity if it is not consistent with the received modulation. This unlearning mechanism provides plasticity to the self-organization in order to adapt to the multi modal constraints.