Multi-view Information Research Articles

Dense view synthesis for three-dimensional light-field display based on scene geometric reconstruction

The three-dimensional light-field display is one of the most promising display technologies and has achieved an impressive improvement in past years. To generate real-world dense views efficiently, many virtual view synthesis approaches have been proposed, which take sparse 2D images as input and synthesize dense virtual views. However, in the occlusion areas, the generated results of previous methods are not satisfactory, which usually contain errors and blurs because the occlusion relation cannot​ be recovered correctly. Here, a dense view synthesis method for three-dimensional light-field display based on scene geometric reconstruction is presented, which can synthesize high-quality arbitrary virtual views with correct occlusions. The scene geometric model is firstly reconstructed with several depth maps captured by RGB-D cameras. The geometric model can provide the correct occlusion relation and select the non-occlusion regions as the input data. To relieve the effect of the depth map noise and improve the quality further, the input views semantic feature maps are extracted and reprojected to the virtual view, which encodes the captured multi-view information and represents the virtual view implicitly. The virtual view image can be generated by rendering the reprojected encoded feature map. Compared with other view synthesis approaches, the quality of the synthesized view is significantly improved with our proposed method, especially in the occlusion areas. The experiments are carried out on a 27 inches 3D light-field display device, and the results demonstrate the effectiveness of our proposed method.

Exploiting key points supervision and grouped feature fusion for multiview pedestrian detection

Multiview pedestrian detection detects pedestrians based on the perception of the same environment from multiple perspectives. This task requires feature extraction in a single view with occlusion and aggregation of multiview information. However, existing research is limited by the local occlusion and the multiview feature stitching method, which cannot perform multiview aggregation efficiently. This paper introduces a network that utilizes key points supervision and grouped feature fusion to address these challenges. It uses key points to regress pedestrians in a single view, and augments the pedestrian consistency information in overlapping views by a grouped feature fusion module. Specifically, the proposed key points supervision effectively alleviates false negatives due to occlusion, and the grouped feature fusion module enhances pedestrian location features by computing the similarity and spatial correlation of overlapping views after single view projection to the ground plane, thereby reducing target ambiguity. Quantitative and qualitative results show that the proposed method can reduce false negatives and false positives in multiview pedestrian detection and achieve efficient multiview feature aggregation. Compared to state-of-the-art methods, the proposed model achieves superior performance, achieving the highest MODA of 92.4 and 93.9 on Wildtrack and MultiviewX datasets, respectively. We believe, to the best of our knowledge, that this approach offers a new optimization idea for multiview aggregation.

Multi-Distance Dispersion Entropy for ECG Signal Classification

Automatic detection of heartbeat is critical for early cardiovascular disease prevention and diagnosis. Traditional feature methodologies based on expert knowledge cannot abstract and represent multidimensional and multi-view information. Hence traditional research on heartbeat detection pattern recognition cannot produce adequate results. The proposed method in this research used Dispersion Entropy (DisEn) on Multidistance Signal Level Difference (MSLD) for feature extraction and Support Vector Machine (SVM) method for classifying the ECG signals. DisEn generates 20 DE values as feature vectors for each MSLD signal with a distance D of 1 to 20. The datasets used in this research were obtained from the MIT-BIH Arrhythmia database of ECG signals that consist of forty-five patients was captured using 200 [adu/mV] amplification and a sampling frequency of 360 Hz. The experiments result using 5-fold cross-validation revealed that at distance D= 1-15 had the highest accuracy of 91% to classify the ECG data into Normal Sinus Rhythm (NSR), Left Bundle Branch Block (LBBB), and Atrial Fibrillation (AFIB) from the MIT-BIH Arrhythmias database.
 

Condenser Vacuum Degree Prediction Model with Multi-View Information Fusion

Vacuum degree is a crucial factor for the operation of thermoelectric generating set. Existing approaches typically use machine learning algorithms to link the relationship between unit operating data and condenser vacuum degree by focusing on the temporal information within the data, while ignoring the frequency information implied in the historical condenser vacuum degree. To fully use of frequency information and further improve prediction accuracy, we propose a novel condenser vacuum degree prediction model with multi-view information fusion. In specific, the implicit frequency information in the historical vacuum degree sequence is explored via a combination of Variational Mode Decomposition (VMD) and Convolutional Neural Network (CNN). Furthermore, Transformer encoder is used to extract the temporal information from the unit operating data. Finally, the two views information are fused for condenser vacuum degree prediction. Extensive experiments conducted on the real data collected from a power plant demonstrate the superiority of the proposed method over several state-of-the-art methods.

Fusing Local and Global Information for One-Step Multi-View Subspace Clustering

Multi-view subspace clustering has drawn significant attention in the pattern recognition and machine learning research community. However, most of the existing multi-view subspace clustering methods are still limited in two aspects. (1) The subspace representation yielded by the self-expression reconstruction model ignores the local structure information of the data. (2) The construction of subspace representation and clustering are used as two individual procedures, which ignores their interactions. To address these problems, we propose a novel multi-view subspace clustering method fusing local and global information for one-step multi-view clustering. Our contribution lies in three aspects. First, we merge the graph learning into the self-expression model to explore the local structure information for constructing the specific subspace representations of different views. Second, we consider the multi-view information fusion by integrating these specific subspace representations into one common subspace representation. Third, we combine the subspace representation learning, multi-view information fusion, and clustering into a joint optimization model to realize the one-step clustering. We also develop an effective optimization algorithm to solve the proposed method. Comprehensive experimental results on nine popular multi-view data sets confirm the effectiveness and superiority of the proposed method by comparing it with many state-of-the-art multi-view clustering methods.

Beyond Monocular Deraining: Parallel Stereo Deraining Network Via Semantic Prior

Rain is a common natural phenomenon. Taking images in the rain however often results in degraded quality of images, thus compromises the performance of many computer vision systems. Most existing de-rain algorithms use only one single input image and aim to recover a clean image. Few work has exploited stereo images. Moreover, even for single image based monocular deraining, many current methods fail to complete the task satisfactorily because they mostly rely on per pixel loss functions and ignore semantic information. In this paper, we present a Paired Rain Removal Network (PRRNet), which exploits both stereo images and semantic information. Specifically, we develop a Semantic-Aware Deraining Module (SADM) which solves both tasks of semantic segmentation and deraining of scenes, and a Semantic-Fusion Network (SFNet) and a View-Fusion Network (VFNet) which fuse semantic information and multi-view information respectively. In addition, we also introduce an Enhanced Paired Rain Removal Network (EPRRNet) which exploits semantic prior to remove rain streaks from stereo images. We first use a coarse deraining network to reduce the rain streaks on the input images, and then adopt a pre-trained semantic segmentation network to extract semantic features from the coarse derained image. Finally, a parallel stereo deraining network fuses semantic and multi-view information to restore finer results. We also propose new stereo based rainy datasets for benchmarking. Experiments on both monocular and the newly proposed stereo rainy datasets demonstrate that the proposed method achieves the state-of-the-art performance. https://github.com/HDCVLab/Stereo-Image-Deraining .

Multi-View Visual Relationship Detection with Estimated Depth Map

The abundant visual information contained in multi-view images is widely used in computer vision tasks. Existing visual relationship detection frameworks have extended the feature vector to improve model performance. However, single view information can not fully reveal the visual relationships in complex visual scenes. To solve this problem and explore the multi-view information in a visual relationship detection (VRD) model, a novel multi-view VRD framework based on a monocular RGB image and an estimated depth map is proposed. The contributions of this paper are threefold. First, we construct a novel multi-view framework which fuses information of different views extracted from estimated RGB-D images. Second, a multi-view image generation method is proposed to transfer flat visual space to 3D multi-view space. Third, we redesign the visual relationship balanced classifier which can process multi-view feature vectors simultaneously. Detailed experiments were conducted on two datasets to demonstrate the effectiveness of the multi-view VRD framework. The experimental results showed that the multi-view VRD framework resulted in state-of-the-art zero-shot learning performance in specific depth conditions.

Triple Adversarial Learning and Multi-View Imaginative Reasoning for Unsupervised Domain Adaptation Person Re-Identification

Due to the importance of practical applications, unsupervised domain adaptation (UDA) person re-identification (re-ID) has attracted increasing attention. However, most of existing methods often lack the multi-view information reasoning and ignore the domain discrepancy of the pedestrian images with the same identity, which constrain the further improvement of recognition performance. So, this paper proposes a triple adversarial learning and multi-view imaginative reasoning network (TAL-MIRN) for UDA person re-ID, which consists of a multi-view imaginative reasoning module (IRM) and a triple adversarial learning module (TALM). IRM makes the classified pedestrian identity features from a single-view image extracted by a feature encoder consistent with the classification results of the aggregated multi-view pedestrian identity features, so the strong multi-view imaginative reasoning ability of the feature encoder is obtained. TALM is composed by the adversarial learning between the camera classifier and feature encoder, adversarial learning of joint distribution alignment, and adversarial learning of the difference between two classifiers used in classification. In particular, the domain-invariant features at camera level are guaranteed by the adversarial learning between the feature extractor and camera classifier. The joint alignment of identity and domain is achieved by the competition between the feature extractor and classifier integrated with identity and domain. The discriminability and robustness of the learned features are enhanced by playing a MinMax game between two different identity classifiers. Furthermore, a simple normalization operation named as cross normalization (CN) is proposed to increase both modeling and generalization capability of the proposed TAL-MIRN across multiple domains. The proposed TAL-MIRN is applied to five benchmark datasets, and the comparative experimental results confirm its superiority over the state-of-the-art methods. The related source codes is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lhf12278/TALM-IRM</uri> .

Multiview three-dimensional imaging using a Risley-prism-based spatially adaptive virtual camera field.

We present a three-dimensional (3D) imaging system that incorporates a stationary camera and Risley prisms. By planning prism rotation to generate a spatially adaptive virtual camera field, the system allows multiple virtual cameras in the field to capture any object from different perspectives for 3D reconstruction. An automatic virtual camera calibration method based on perspective projection and geometric optics is developed to enable virtual camera field construction and characterization. Moreover, a 3D computational reconstruction framework is proposed for multiview information fusion using the virtual camera field. This framework combines nonlinear distortion correction with epipolar geometry computation to efficiently perform image rectification and stereo matching, which can further facilitate 3D object reconstruction through multiview triangulation. The experiments on synthetic and real data validate the feasibility and flexibility of our 3D imaging technique.

Research on the Prediction of the Operational Risk Field of Intelligent Vehicles Based on Dual Multiline LiDAR

To effectively evaluate the risk situation between intelligent vehicles and surrounding traffic participants in complex scenes, a complex traffic environment perception technology based on dual multiline light detection and ranging (LiDAR) is proposed in this work. The vehicle motion state is predicted by fusing the multiview characteristics of point cloud timing and multitarget interaction information, and the risk assessment model is constructed via artificial potential field theory. The real-time point cloud information is used to obtain the time-sequence bird’s-eye view and range image. The improved VGG19 network model is used to extract the time-sequence high-level abstract combined features in the multiview scene. The constructed time-sequence feature vector is used as the input data of the attention mechanism, and the attention-bidirectional long short-term memory (Attention-BiLSTM) model is used for training to form the desired input-output mapping relationship. The motion state of the target vehicle can therefore be updated, and the static and dynamic risk fields of traffic participants surrounding the vehicle can be established based on artificial potential field theory, thereby allowing for the evaluation of the operational risk of the intelligent vehicle. The results of experiments demonstrate that the prediction effect of the target vehicle state parameters via the use of the proposed model is better than that of other compared models, and the prediction effect of the risk field of intelligent vehicle operation based on the multiview point cloud features and vehicle interaction information is good.

Edge-enhanced dual discriminator generative adversarial network for fast MRI with parallel imaging using multi-view information

In clinical medicine, magnetic resonance imaging (MRI) is one of the most important tools for diagnosis, triage, prognosis, and treatment planning. However, MRI suffers from an inherent slow data acquisition process because data is collected sequentially in k-space. In recent years, most MRI reconstruction methods proposed in the literature focus on holistic image reconstruction rather than enhancing the edge information. This work steps aside this general trend by elaborating on the enhancement of edge information. Specifically, we introduce a novel parallel imaging coupled dual discriminator generative adversarial network (PIDD-GAN) for fast multi-channel MRI reconstruction by incorporating multi-view information. The dual discriminator design aims to improve the edge information in MRI reconstruction. One discriminator is used for holistic image reconstruction, whereas the other one is responsible for enhancing edge information. An improved U-Net with local and global residual learning is proposed for the generator. Frequency channel attention blocks (FCA Blocks) are embedded in the generator for incorporating attention mechanisms. Content loss is introduced to train the generator for better reconstruction quality. We performed comprehensive experiments on Calgary-Campinas public brain MR dataset and compared our method with state-of-the-art MRI reconstruction methods. Ablation studies of residual learning were conducted on the MICCAI13 dataset to validate the proposed modules. Results show that our PIDD-GAN provides high-quality reconstructed MR images, with well-preserved edge information. The time of single-image reconstruction is below 5ms, which meets the demand of faster processing.

Multi-Relational Graph Convolution Network for Service Recommendation in Mashup Development

With the rapid development of service-oriented computing, an overwhelming number of web services have been published online. Developers can create mashups that combine one or multiple services to meet complex business requirements. To speed up the mashup development process, recommending suitable services for developers is a vital problem. In this paper, we address the data sparsity and cold-start problems faced in service recommendation, and propose a novel multi-relational graph convolutional network framework (MRGCN) for service recommendation. Specifically, we first construct a multi-relational mashup-service graph with three types of relations, namely composition relation, functional relation, and tagging relation. These three relations are indispensable and complement each other for capturing multi-view information. Then, the three relations in the graph are seamlessly fused with various strategies. Next, graph convolution is performed on the fused multi-relational graph to capture the high-order relational information between mashups and services. Finally, the relevance between mashup requirements and services is predicted based on the learned features on the graph. We conduct extensive experiments on the ProgrammableWeb dataset and demonstrate that our proposed method can outperform state-of-the-art methods in recommending services when only mashup requirements are available.

Multiple partitions alignment via spectral rotation

Multi-view spectral clustering has drawn much attention due to the effectiveness of exploiting the similarity relationships among data points. These methods typically reveal the intrinsic structure using a predefined graph for each view. The predefined graphs are fused to a consensus one, on which the final clustering results are obtained. However, such common strategies may lead to information loss because of the inconsistency or noise among multiple views. In this paper, we propose to merge multi-view information in partition level instead of the raw feature space where the data points lie. The partition of each view is treated as a perturbation of the consensus clustering, and the multiple partitions are integrated by estimating a distinct rotation for each partition. The proposed model is formulated as a joint learning framework, i.e., with the input data matrix, our model directly outputs the final discrete clustering result. Hence it is an end-to-end single-stage learning model. An iterative updating algorithm is proposed to solve the learning problem, in which the involved variables can be optimized in a mutual reinforcement manner. Experimental results on real-world data sets illustrate the effectiveness of our model.

Multi-view support vector ordinal regression with data uncertainty

Ordinal regression (OR) is a paradigm which learns a prediction model on the data with ordered classes. Despite much progress in OR, the existing OR works learn the classifier from only one view and the multi-view learning in OR has not been considered. What is more, there may exist uncertain information in the multi-view OR data. In this paper, we put forward a novel approach, called multi-view support vector ordinal regression with uncertain data (MORU), which can improve the OR classifier by incorporating the multi-view information and handling the data uncertainty. In our method, a series of parallel hyperplanes are applied to separate the multi-view ordered data, and the uncertain information is considered in the input data. Then, we adopt a heuristic framework to solve the OR learning problem. Experimental results have illustrated that our method obtains superior performance to the existing OR techniques.

Vehicle Motion State Prediction Method Integrating Point Cloud Time Series Multiview Features and Multitarget Interactive Information

A vehicle motion state prediction algorithm integrating point cloud timing multiview features and multitarget interaction information is proposed in this work to effectively predict the motion states of traffic participants around intelligent vehicles in complex scenes. The algorithm analyzes the characteristics of object motion that are affected by the surrounding environment and the interaction of nearby objects and is based on the complex traffic environment perception dual multiline light detection and ranging (LiDAR) technology. The time sequence aerial view map and time sequence front view depth map are obtained using real-time point cloud information perceived by the LiDAR. Time sequence high-level abstract combination features in the multiview scene are then extracted by an improved VGG19 network model and are fused with the potential spatiotemporal interaction of the multitarget operation state data extraction features detected by the laser radar by using a one-dimensional convolution neural network. A temporal feature vector is constructed as the input data of the bidirectional long-term and short-term memory (BiLSTM) network, and the desired input-output mapping relationship is trained to predict the motion state of traffic participants. According to the test results, the proposed BiLSTM model based on point cloud multiview and vehicle interaction information is better than other methods in predicting the state of target vehicles. The results can provide support for the research to evaluate the risk of intelligent vehicle operation environment.

Hyperspectral Image Classification Based on Semisupervised Self-Learning and Multiview Information Fusion

Hyperspectral images (HSIs) have been utilized in various fields due to abundant information, whose application is to a great extent limited by the number of labeled samples. Hence, how to exploit the diverse and complementary characteristics of multiple information from different views inherent in the HSIs is very critical for improving the classification performance under the condition of small-sized training set. This letter presents a novel method to tackle the small labeled sample size problem with semisupervised self-learning (S³L) and multiview information fusion. First, a sample augmentation scheme based on S³L and high-reliable neighborhood structure is designed for realizing training set enlargement. Thus, the pseudo-labeled samples with high-quality would be automatically picked out from the unlabeled data by exploiting complementary information from multiple views, i.e., semantic information, spectral bands, texture and geospatial information. Then, we retrain the classifier by the enlarged training set and generate an intermediate classification map. Finally, we simultaneously utilize the location and gradient information of the samples to adaptively realize the refined landcover classification. Experimental results on three widely-used data sets compared with several representative methods for small-sized HSI classification validate the effectiveness of the proposed method.

Wind Turbine Clustering and Equivalent Parameter Identification in Multitime Scales Based on the Deep Migration of Multiview Features

To improve the precision of wind farm multi-machine equivalence and multi-scene generalization, this paper proposes a method for wind turbine clustering and equivalent parameter identification in multi-time scales based on the deep migration of multi-view features. The proposed technique carries out multi-machine equivalence by leveraging the multi-view information from each turbine in a wind farm. Specifically, a deep spatio-temporal Improved Auto-Encoder is designed, jointly trained with the target clustering layer. IAE is used for mining multi-view latent characteristics of wind turbines orienting to grouping turbines to improve the model&#x2019;s adaptability to multiple scenarios and divide turbines in an unsupervised manner. This method generates a visual heat map to represent the attended area of characteristics based on transfer learning and Class Activation Map to enable interpretability. In the next phase, this technique constructs a multi-objective optimization model by synthesizing the equivalent deviation of voltage, current, active power, and reactive power to further improve accuracy. It can identify the equivalent parameters of collector lines, the mechanical structure, and the control system at different time scales simultaneously via the black-box paralleled optimization method based on Bayes and Multi-arm Bandit. The proposed approach is evaluated on a typical double-fed wind farm with grid-side faults under various conditions of disturbing winds. Also, an ablation study is conducted to make analysis according to the two phases, i.e., turbine division and parameter identification. The results validate the accuracy and robustness of this method.

Weakly-Supervised Salient Object Detection on Light Fields.

Most existing salient object detection (SOD) methods are designed for RGB images and do not take advantage of the abundant information provided by light fields. Hence, they may fail to detect salient objects of complex structures and delineate their boundaries. Although some methods have explored multi-view information of light field images for saliency detection, they require tedious pixel-level manual annotations of ground truths. In this paper, we propose a novel weakly-supervised learning framework for salient object detection on light field images based on bounding box annotations. Our method has two major novelties. First, given an input light field image and a bounding-box annotation indicating the salient object, we propose a ground truth label hallucination method to generate a pixel-level pseudo saliency map, to avoid heavy cost of pixel-level annotations. This method generates high quality pseudo ground truth saliency maps to help supervise the training, by exploiting information obtained from the light field (including depths and RGB images). Second, to exploit the multi-view nature of the light field data in learning, we propose a fusion attention module to calibrate the spatial and channel-wise light field representations. It learns to focus on informative features and suppress redundant information from the multi-view inputs. Based on these two novelties, we are able to train a new salient object detector with two branches in a weakly-supervised manner. While the RGB branch focuses on modeling the color contrast in the all-in-focus image for locating the salient objects, the Focal branch exploits the depth and the background spatial redundancy of focal slices for eliminating background distractions. Extensive experiments show that our method outperforms existing weakly-supervised methods and most fully supervised methods.

View-Wise Versus Cluster-Wise Weight: Which Is Better for Multi-View Clustering?

Weighted multi-view clustering (MVC) aims to combine the complementary information of multi-view data (such as image data with different types of features) in a weighted manner to obtain a consistent clustering result. However, when the cluster-wise weights across views are vastly different, most existing weighted MVC methods may fail to fully utilize the complementary information, because they are based on view-wise weight learning and can not learn the fine-grained cluster-wise weights. Additionally, extra parameters are needed for most of them to control the weight distribution sparsity or smoothness, which are hard to tune without prior knowledge. To address these issues, in this paper we propose a novel and effective Cluster-weighted mUlti-view infoRmation bottlEneck (CURE) clustering algorithm, which can automatically learn the cluster-wise weights to discover the discriminative clusters across multiple views and thus can enhance the clustering performance by properly exploiting the cluster-level complementary information. To learn the cluster-wise weights, we design a new weight learning scheme by exploring the relation between the mutual information of the joint distribution of a specific cluster (containing a group of data samples) and the weight of this cluster. Finally, a novel draw-and-merge method is presented to solve the optimization problem. Experimental results on various multi-view datasets show the superiority and effectiveness of our cluster-wise weighted CURE over several state-of-the-art methods.

MetaCAR: Cross-Domain Meta-Augmentation for Content-Aware Recommendation

Cold-start has become critical for recommendations, especially for sparse user-item interactions. Recent approaches based on meta-learning succeed in alleviating the issue, owing to the fact that these methods have strong generalization, so they can fast adapt to new tasks under cold-start settings. However, these meta-learning-based recommendation models learned with single and spase ratings are easily falling into the meta-overfitting, since the one and only rating <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$r_{ui}$</tex-math></inline-formula> to a specific item <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$i$</tex-math></inline-formula> cannot reflect a user's diverse interests under various circumstances(e.g., time, mood, age, etc), i.e. if <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$r_{ui}$</tex-math></inline-formula> equals to 1 in the historical dataset, but <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$r_{ui}$</tex-math></inline-formula> could be 0 in some circumstance. In meta-learning, tasks with these single ratings are called Non-Mutually-Exclusive(Non-ME) tasks, and tasks with diverse ratings are called Mutually-Exclusive(ME) tasks. Fortunately, a meta-augmentation technique is proposed to relief the meta-overfitting for meta-learning methods by transferring Non-ME tasks into ME tasks by adding noises to labels without changing inputs. Motivated by the meta-augmentation method, in this paper, we propose a cross-domain meta-augmentation technique for content-aware recommendation systems (MetaCAR) to construct ME tasks in the recommendation scenario. Our proposed method consists of two stages: meta-augmentation and meta-learning. In the meta-augmentation stage, we first conduct domain adaptation by a dual conditional variational autoencoder (CVAE) with a multi-view information bottleneck constraint, and then apply the learned CVAE to generate ratings for users in the target domain. In the meta-learning stage, we introduce both the true and generated ratings to construct ME tasks that enables the meta-learning recommendations to avoid meta-overfitting. Experiments evaluated in real-world datasets show the significant superiority of MetaCAR for coping with the cold-start user issue over competing baselines including cross-domain, content-aware, and meta-learning-based recommendations.