Multi-view Optimization Research Articles

An Effective Augmented Lagrangian Method for Fine-Grained Multi-View Optimization

The significance of multi-view learning in effectively mitigating the intricate intricacies entrenched within heterogeneous data has garnered substantial attention in recent years. Notwithstanding the favorable achievements showcased by recent strides in this area, a confluence of noteworthy challenges endures. To be specific, a majority of extant methodologies unceremoniously assign weights to data points view-wisely. This ineluctably disregards the intrinsic reality that disparate views confer diverse contributions to each individual sample, consequently neglecting the rich wellspring of sample-level structural insights harbored within the dataset. In this paper, we proposed an effective Augmented Lagrangian MethOd for fiNe-graineD (ALMOND) multi-view optimization. This innovative approach scrutinizes the interplay among multiple views at the granularity of individual samples, thereby fostering the enhanced preservation of local structural coherence. The Augmented Lagrangian Method (ALM) is elaborately incorporated into our framework, which enables us to achieve an optimal solution without involving an inexplicable intermediate variable as previous methods do. Empirical experiments on multi-view clustering tasks across heterogeneous datasets serve to incontrovertibly showcase the effectiveness of our proposed methodology, corroborating its preeminence over incumbent state-of-the-art alternatives.

Self-avatar: Monocular 3D human reconstruction from RGB image

3D human position and shape estimate are crucial in many computer vision applications. Despite the fact that there are numerous deep learning techniques designed to handle this problem, they frequently only use training networks with RGB images from a single point of view. In this paper, a unique approach to solve this issue is proposed by combining a regression-based multi-view picture learning loop with an optimization-based multi-view model. This is because some public datasets are collected by multi-view camera systems. A parameterized human body model's position and shape parameters are initially deduced by a convolutional neural network (CNN) from multi-view photos. This work then introduces an enhanced multi-view optimization method called MV-SMPLify, which aligns the SMPL model with multi-view images by using the regressed pose and shape as beginning values. Following that, the CNN model's training can be monitored using the optimum parameters. The Self-avatar project as a whole is a self-supervised framework that combines the advantages of both the CNN method and the optimization-based strategy. Additionally, the use of multi-view photos improves thorough supervision during training. This methodology outperforms earlier methods in a variety of ways, according to qualitative and quantitative testing using open datasets.

Classifier Ensemble Based on Multiview Optimization for High-Dimensional Imbalanced Data Classification.

High-dimensional class imbalanced data have plagued the performance of classification algorithms seriously. Because of a large number of redundant/invalid features and the class imbalanced issue, it is difficult to construct an optimal classifier for high-dimensional imbalanced data. Classifier ensemble has attracted intensive attention since it can achieve better performance than an individual classifier. In this work, we propose a multiview optimization (MVO) to learn more effective and robust features from high-dimensional imbalanced data, based on which an accurate and robust ensemble system is designed. Specifically, an optimized subview generation (OSG) in MVO is first proposed to generate multiple optimized subviews from different scenarios, which can strengthen the classification ability of features and increase the diversity of ensemble members simultaneously. Second, a new evaluation criterion that considers the distribution of data in each optimized subview is developed based on which a selective ensemble of optimized subviews (SEOS) is designed to perform the subview selective ensemble. Finally, an oversampling approach is executed on the optimized view to obtain a new class rebalanced subset for the classifier. Experimental results on 25 high-dimensional class imbalanced datasets indicate that the proposed method outperforms other mainstream classifier ensemble methods.

Scratch-based Reflection Art via Differentiable Rendering

The 3D visual optical arts create fascinating special effects by carefully designing interactions between objects and light sources. One of the essential types is 3D reflection art, which aims to create reflectors that can display different images when viewed from different directions. Existing works produce impressive visual effects. Unfortunately, previous works discretize the reflector surface with regular grids/facets, leading to a large parameter space and a high optimization time cost. In this paper, we introduce a new type of 3D reflection art - scratch-based reflection art , which allows for a more compact parameter space, easier fabrication, and computationally efficient optimization. To design a 3D reflection art with scratches, we formulate it as a multi-view optimization problem and introduce differentiable rendering to enable efficient gradient-based optimizers. For that, we propose an analytical scratch rendering approach, together with a high-performance rendering pipeline, allowing efficient differentiable rendering. As a consequence, we could display multiple images on a single metallic board with only several minutes for optimization. We demonstrate our work by showing virtual objects and manufacturing our designed reflectors with a carving machine.

Interactive Multifactorial Evolutionary Optimization Algorithm with Multidimensional Preference Surrogate Models for Personalized Recommendation

Interactive evolutionary algorithms (IEAs) coupled with a data-driven user surrogate model (USM) have recently been proposed for enhancing personalized recommendation performance. Since the USM relies on only one model to describe the full range of user preferences, existing USM-based IEAs have not investigated how knowledge migrates between preference models to improve the diversity and novelty of recommendations. Motivated by this, an interactive multifactorial evolutionary optimization algorithm with multidimensional preference user surrogate models is proposed here to perform a multi-view optimization for personalized recommendation. Firstly, multidimensional preference user surrogate models (MPUSMs), partial-MPUSMs, and probability models of MPUSMs are constructed to approximate the different perceptions of preferences and serve for population evolution. Next, a modified multifactorial evolutionary algorithm is used for the first time in the IEAs domain to recommend diverse and novel items for multiple preferences. It includes initialization and diversification management of a population with skill factors, recommendation lists of preference grading and interactive model management of inheriting previous information. Comprehensive comparison studies in the Amazon dataset show that the proposed models and algorithm facilitate the mining of knowledge between preferences. Eventually, at the cost of losing only about 5% of the Hit Ratio and Average Precision, the Individual Diversity is improved by 54.02%, the Self-system Diversity by 3.7%, the Surprise Degree by 2.69%, and the Preference Mining Degree by 16.05%.

SDFEst: Categorical Pose and Shape Estimation of Objects From RGB-D Using Signed Distance Fields

Rich geometric understanding of the world is an important component of many robotic applications such as planning and manipulation. In this paper, we present a modular pipeline for pose and shape estimation of objects from RGB-D images given their category. The core of our method is a generative shape model, which we integrate with a novel initialization network and a differentiable renderer to enable 6D pose and shape estimation from a single or multiple views. We investigate the use of discretized signed distance fields as an efficient shape representation for fast analysis-by-synthesis optimization. Our modular framework enables multi-view optimization and extensibility. We demonstrate the benefits of our approach over state-of-the-art methods in several experiments on both synthetic and real data. We open-source our approach at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/roym899/sdfest</uri> .

Multi-view optimization of time-frequency common spatial patterns for brain-computer interfaces

BackgroundCommon spatial pattern (CSP) is a prevalent method applied to feature extraction in motor imagery (MI)-based brain-computer interfaces (BCIs) recorded by electroencephalogram (EEG). The selection of time windows and frequency bands prominently affects the performance of CSP algorithms. Concerning the joint optimization of these two parameters, several studies have utilized a unified framework based on different feature selection strategies and achieved considerable improvement. However, during the feature selection process, useful information could be discarded inevitably and the underlying internal structure of features could be neglected. New methodIn this paper, we proposed a novel framework termed time window filter bank common spatial pattern with multi-view optimization (TWFBCSP-MVO) to further boost the decoding of MI tasks. Concretely, after extracting CSP features from different time-frequency decompositions of EEG signals, a preliminary screening strategy based on variance ratio was devised to filter out the unrelated spatial patterns. We then introduced a multi-view learning strategy for the simultaneous optimization of time windows and frequency bands. A support vector machine classifier was trained to determine the output of the brain. ResultsAn experimental study was conducted on two public datasets to verify the effectiveness of TWFBCSP-MVO. Results showed that the proposed TWFBCSP-MVO could help improve the performance of MI classification. Comparison with existing methodsIn comparison to other competing methods, the proposed method performed significantly better (p<0.01). ConclusionsThe proposed method is a promising contestant to improve the performance of practical MI-based BCIs.

CodeMapping: Real-Time Dense Mapping for Sparse SLAM using Compact Scene Representations

We propose a novel dense mapping framework for sparse visual SLAM systems which leverages a compact scene representation. State-of-the-art sparse visual SLAM systems provide accurate and reliable estimates of the camera trajectory and locations of landmarks. While these sparse maps are useful for localization, they cannot be used for other tasks such as obstacle avoidance or scene understanding. In this letter we propose a dense mapping framework to complement sparse visual SLAM systems which takes as input the camera poses, keyframes and sparse points produced by the SLAM system and predicts a dense depth image for every keyframe. We build on CodeSLAM [1] and use a variational autoencoder (VAE) which is conditioned on intensity, sparse depth and reprojection error images from sparse SLAM to predict an uncertainty-aware dense depth map. The use of a VAE then enables us to refine the dense depth images through multi-view optimization which improves the consistency of overlapping frames. Our mapper runs in a separate thread in parallel to the SLAM system in a loosely coupled manner. This flexible design allows for integration with arbitrary metric sparse SLAM systems without delaying the main SLAM process. Our dense mapper can be used not only for local mapping but also globally consistent dense 3D reconstruction through TSDF fusion. We demonstrate our system running with ORB-SLAM3 and show accurate dense depth estimation which could enable applications such as robotics and augmented reality.

3D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single View.

Automated capture of animal pose is transforming how we study neuroscience and social behavior. Movements carry important social cues, but current methods are not able to robustly estimate pose and shape of animals, particularly for social animals such as birds, which are often occluded by each other and objects in the environment. To address this problem, we first introduce a model and multi-view optimization approach, which we use to capture the unique shape and pose space displayed by live birds. We then introduce a pipeline and experiments for keypoint, mask, pose, and shape regression that recovers accurate avian postures from single views. Finally, we provide extensive multi-view keypoint and mask annotations collected from a group of 15 social birds housed together in an outdoor aviary. The project website with videos, results, code, mesh model, and the Penn Aviary Dataset can be found at https://marcbadger.github.io/avian-mesh.

Accurate and occlusion-robust multi-view stereo

This paper proposes an accurate multi-view stereo method for image-based 3D reconstruction that features robustness in the presence of occlusions. The new method offers improvements in dealing with two fundamental image matching problems. The first concerns the selection of the support window model, while the second centers upon accurate visibility estimation for each pixel. The support window model is based on an approximate 3D support plane described by a depth and two per-pixel depth offsets. For the visibility estimation, the multi-view constraint is initially relaxed by generating separate support plane maps for each support image using a modified PatchMatch algorithm. Then the most likely visible support image, which represents the minimum visibility of each pixel, is extracted via a discrete Markov Random Field model and it is further augmented by parameter clustering. Once the visibility is estimated, multi-view optimization taking into account all redundant observations is conducted to achieve optimal accuracy in the 3D surface generation for both depth and surface normal estimates. Finally, multi-view consistency is utilized to eliminate any remaining observational outliers. The proposed method is experimentally evaluated using well-known Middlebury datasets, and results obtained demonstrate that it is amongst the most accurate of the methods thus far reported via the Middlebury MVS website. Moreover, the new method exhibits a high completeness rate.

A Silhouette-Based Algorithm for Texture Registration and Stitching

In this paper a system is presented that automatically registers and stitches textures acquired from multiple photographic images onto the surface of a given corresponding 3D model. Within this process the camera position, direction, and field of view must be determined for each of the images. For this registration, which aligns a 2D image to a 3D model, we present an efficient hardware-accelerated silhouette-based algorithm working on different image resolutions that accurately registers each image without any user interaction. Besides the silhouettes, the given texture information also can be used to improve accuracy by comparing one stitched texture to already registered images resulting in a global multiview optimization. After the 3D?2D registration for each part of the 3D model's surface the view is determined which provides the best available texture. Textures are blended at the borders of regions assigned to different views.