Gaussian Representation Research Articles

Arbitrary Optics for Gaussian Splatting Using Space Warping.

Due to recent advances in 3D reconstruction from RGB images, it is now possible to create photorealistic representations of real-world scenes that only require minutes to be reconstructed and can be rendered in real time. In particular, 3D Gaussian splatting shows promising results, outperforming preceding reconstruction methods while simultaneously reducing the overall computational requirements. The main success of 3D Gaussian splatting relies on the efficient use of a differentiable rasterizer to render the Gaussian scene representation. One major drawback of this method is its underlying pinhole camera model. In this paper, we propose an extension of the existing method that removes this constraint and enables scene reconstructions using arbitrary camera optics such as highly distorting fisheye lenses. Our method achieves this by applying a differentiable warping function to the Gaussian scene representation. Additionally, we reduce overfitting in outdoor scenes by utilizing a learnable skybox, reducing the presence of floating artifacts within the reconstructed scene. Based on synthetic and real-world image datasets, we show that our method is capable of creating an accurate scene reconstruction from highly distorted images and rendering photorealistic images from such reconstructions.

A Topology Optimization of Electromagnetic Devices Based on Kernel Ridge Regression as a Variant of Gaussian Network-Based Shape Representation

A Topology Optimization of Electromagnetic Devices Based on Kernel Ridge Regression as a Variant of Gaussian Network-Based Shape Representation

Bayesian Inference for Long Memory Stochastic Volatility Models

We explore the application of integrated nested Laplace approximations for the Bayesian estimation of stochastic volatility models characterized by long memory. The logarithmic variance persistence in these models is represented by a Fractional Gaussian Noise process, which we approximate as a linear combination of independent first-order autoregressive processes, lending itself to a Gaussian Markov Random Field representation. Our results from Monte Carlo experiments indicate that this approach exhibits small sample properties akin to those of Markov Chain Monte Carlo estimators. Additionally, it offers the advantages of reduced computational complexity and the mitigation of posterior convergence issues. We employ this methodology to estimate volatility dependency patterns for both the SP&500 index and major cryptocurrencies. We thoroughly assess the in-sample fit and extend our analysis to the construction of out-of-sample forecasts. Furthermore, we propose multi-factor extensions and apply this method to estimate volatility measurements from high-frequency data, underscoring its exceptional computational efficiency. Our simulation results demonstrate that the INLA methodology achieves comparable accuracy to traditional MCMC methods for estimating latent parameters and volatilities in LMSV models. The proposed model extensions show strong in-sample fit and out-of-sample forecast performance, highlighting the versatility of the INLA approach. This method is particularly advantageous in high-frequency contexts, where the computational demands of traditional posterior simulations are often prohibitive.

Multi-level semantics probability embedding for image–text matching

The requirement of image–text matching is to retrieve matching images or texts based on textual or visual queries. However, image–text matching is inherently a many-to-many problem, as an image can correspond to multiple levels of visual semantic scenes, which can be described by different texts. Similarly, textual descriptions can be visualized through multiple visual scenes. This leads to ambiguity in the matching between images and texts. To better capture these matching relationships, we employ graph convolutional networks to extract multi-level semantic information for image–text pairs, and construct Gaussian distribution representations for image and text instead of conventional point representations. Furthermore, we introduce a inter-modal mixture of Gaussian distribution to constrain the matching relationships between image–text pairs, which ensures more precise distribution representations in a shared space and strengthens the correlation between cross-modal. We conducted experiments on Flickr30K and MS-COCO, which are two widely used datasets, demonstrates the superior performance of our approach.

3D Reconstruction with Fast Dipole Sums

We introduce a method for high-quality 3D reconstruction from multi-view images. Our method uses a new point-based representation, the regularized dipole sum, which generalizes the winding number to allow for interpolation of per-point attributes in point clouds with noisy or outlier points. Using regularized dipole sums, we represent implicit geometry and radiance fields as per-point attributes of a dense point cloud, which we initialize from structure from motion. We additionally derive Barnes-Hut fast summation schemes for accelerated forward and adjoint dipole sum queries. These queries facilitate the use of ray tracing to efficiently and differentiably render images with our point-based representations, and thus update their point attributes to optimize scene geometry and appearance. We evaluate our method in inverse rendering applications against state-of-the-art alternatives, based on ray tracing of neural representations or rasterization of Gaussian point-based representations. Our method significantly improves 3D reconstruction quality and robustness at equal runtimes, while also supporting more general rendering methods such as shadow rays for direct illumination.

GroomCap: High-Fidelity Prior-Free Hair Capture

Despite recent advances in multi-view hair reconstruction, achieving strand-level precision remains a significant challenge due to inherent limitations in existing capture pipelines. We introduce GroomCap , a novel multi-view hair capture method that reconstructs faithful and high-fidelity hair geometry without relying on external data priors. To address the limitations of conventional reconstruction algorithms, we propose a neural implicit representation for hair volume that encodes high-resolution 3D orientation and occupancy from input views. This implicit hair volume is trained with a new volumetric 3D orientation rendering algorithm, coupled with 2D orientation distribution supervision, to effectively prevent the loss of structural information caused by undesired orientation blending. We further propose a Gaussian-based hair optimization strategy to refine the traced hair strands with a novel chained Gaussian representation, utilizing direct photometric supervision from images. Our results demonstrate that GroomCap is able to capture high-quality hair geometries that are not only more precise and detailed than existing methods but also versatile enough for a range of applications.

3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.

GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination, which explicitly inject structure priors into the initial optimization process to help build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. We further design a COLMAP-free variant, where pre-given accurate camera poses are not required, which achieves competitive quality and facilitates wider applications. GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, OpenIllumination, and our-collected unposed images, achieving superior performance from only four views and significantly outperforming previous SOTA methods.

LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives

Large garages are ubiquitous yet intricate scenes that present unique challenges due to their monotonous colors, repetitive patterns, reflective surfaces, and transparent vehicle glass. Conventional Structure from Motion (SfM) methods for camera pose estimation and 3D reconstruction often fail in these environments due to poor correspondence construction. To address these challenges, we introduce LetsGo, a LiDAR-assisted Gaussian splatting framework for large-scale garage modeling and rendering. We develop a handheld scanner, Polar, equipped with IMU, LiDAR, and a fisheye camera, to facilitate accurate data acquisition. Using this Polar device, we present the GarageWorld dataset, consisting of eight expansive garage scenes with diverse geometric structures, which will be made publicly available for further research. Our approach demonstrates that LiDAR point clouds collected by the Polar device significantly enhance a suite of 3D Gaussian splatting algorithms for garage scene modeling and rendering. We introduce a novel depth regularizer that effectively eliminates floating artifacts in rendered images. Additionally, we propose a multi-resolution 3D Gaussian representation designed for Level-of-Detail (LOD) rendering. This includes adapted scaling factors for individual levels and a random-resolution-level training scheme to optimize the Gaussians across different resolutions. This representation enables efficient rendering of large-scale garage scenes on lightweight devices via a web-based renderer. Experimental results on our GarageWorld dataset, as well as on ScanNet++ and KITTI-360, demonstrate the superiority of our method in terms of rendering quality and resource efficiency.

A stochastic algorithm for quantile regression models with fixed effects

In this paper, we present a stochastic algorithm for parameter estimation based on panel quantile regression model. We propose an easy-to-implement estimator based on the proposed algorithm. We profile the quantile-specific fixed effects as functions of the parameters of interest based on the Gaussian mixture representation of the asymmetric Laplace (AL) likelihood and eliminate the fixed effects through a data transformation. Parameters of interest can be estimated via quantile regression. Under a set of sufficient conditions, the proposed estimator is consistent and asymptotically normal when n and T both go to infinity. The proposed estimator is illustrated via both simulations and real data examples.

New Measurements of the Lyα Forest Continuum and Effective Optical Depth with LyCAN and DESI Y1 Data

We present the Lyα Continuum Analysis Network (LyCAN), a convolutional neural network that predicts the unabsorbed quasar continuum within the rest-frame wavelength range of 1040–1600 Å based on the red side of the Lyα emission line (1216–1600 Å). We developed synthetic spectra based on a Gaussian mixture model representation of nonnegative matrix factorization (NMF) coefficients. These coefficients were derived from high-resolution, low-redshift (z < 0.2) Hubble Space Telescope/Cosmic Origins Spectrograph (COS) quasar spectra. We supplemented this COS-based synthetic sample with an equal number of DESI Year 5 mock spectra. LyCAN performs extremely well on testing sets, achieving a median error in the forest region of 1.5% on the DESI mock sample, 2.0% on the COS-based synthetic sample, and 4.1% on the original COS spectra. LyCAN outperforms principal component analysis (PCA) and NMF-based prediction methods using the same training set by 40% or more. We predict the intrinsic continua of 83,635 DESI Year 1 spectra in the redshift range of 2.1 ≤ z ≤ 4.2 and perform an absolute measurement of the evolution of the effective optical depth. This is the largest sample employed to measure the optical depth evolution to date. We fit a power law of the form τ(z)=τ0(1+z)γ to our measurements and find τ 0 = (2.46 ± 0.14) × 10−3 and γ = 3.62 ± 0.04. Our results show particular agreement with high-resolution, ground-based observations around z = 2, indicating that LyCAN is able to predict the quasar continuum in the forest region with only spectral information outside the forest.

HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in Urban Scenes

HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in Urban Scenes

FRGEM: Feature integration pre-training based Gaussian embedding model for Chinese word representation

Most Chinese word representation approaches mine the lexeme of Chinese words by predicting the target word or the contextual words. However, Chinese words may have multiple meanings, above ways may only capture one aspect implication. In this paper, for better modeling multiple connotations of Chinese words, a Feature integRation pre-training based Gaussian Embedding Model (FRGEM) is proposed with three stages, feature integration pre-training, relevancy Gaussian representation and similarity-based training. In the first stage, the internal sequences with three inner-character stroke, structure and pinyin features of Chinese words are generated to learn their semantic relevancy, then we combine BERT with internal sequences to extract sentence relevance and integrate the internal and sentence features to learn pre-training representations. In the second stage, relevancy Gaussian representation with pre-training embeddings is proposed to fuse inner-character features with Gaussian through estimating its mean to analyze multiple implications of Chinese words. In the third stage, a similarity-based objective is proposed to distinguish true contextual words and learn Chinese word representations. Extensive experiments on word similarity, word analogy, named entity recognition and text classification show that FRGEM is superior to majority state-of-the-art algorithms.

Deep indoor illumination estimation based on spherical gaussian representation with scene prior knowledge

High dynamic range (HDR) illumination estimation from a single low dynamic range image is a critical task in the fields of computer vision, graphics and augmented reality. However, directly learning the full HDR environment map or parametric lighting information from a single image is extremely difficult and inaccurate. As a result, we propose a two-stage network approach for illumination estimation that integrates spherical gaussian (SG) representation with scene prior knowledge. In the first stage, a convolutional neural network is utilized to generate material and geometric information about the scene, which serves as prior knowledge for lighting prediction. In the second stage, we model indoor environment illumination using 128 SG functions with fixed center direction and bandwidth, allowing only the amplitude to vary. Subsequently, a Transformer-based lighting parameter regressor is employed to capture the complex relationship between the input images with scene prior information and its SG illumination. Additionally, we introduce a hybrid loss function, which combines a masked loss for high-frequency illumination with a rendering loss for improving the visual quality. By training and evaluating the lighting model on the created SG illumination dataset, the proposed method achieves competitive results in both quantitative metrics and visual quality, outperforming state-of-the-art methods.

LGSur‐Net: A Local Gaussian Surface Representation Network for Upsampling Highly Sparse Point Cloud

AbstractWe introduce LGSur‐Net, an end‐to‐end deep learning architecture, engineered for the upsampling of sparse point clouds. LGSur‐Net harnesses a trainable Gaussian local representation by positioning a series of Gaussian functions on an oriented plane, complemented by the optimization of individual covariance matrices. The integration of parametric factors allows for the encoding of the plane's rotational dynamics and Gaussian weightings into a linear transformation matrix. Then we extract the feature maps from the point cloud and its adjoining edges and learn the local Gaussian depictions to accurately model the shape's local geometry through an attention‐based network. The Gaussian representation's inherent high‐order continuity endows LGSur‐Net with the natural ability to predict surface normals and support upsampling to any specified resolution. Comprehensive experiments validate that LGSur‐Net efficiently learns from sparse data inputs, surpassing the performance of existing state‐of‐the‐art upsampling methods. Our code is publicly available athttps://github.com/Rangiant5b72/LGSur-Net.

Superposition and interference of Gaussian wave packets for laser transverse modes

We interpret the intrinsic mechanism of the transition from geometric modes to eigenmodes in the Gaussian wave-packet representation based on numerical simulation. We then construct multiaxial super-geometric modes and polygonal vortex modes based on Gaussian wave packets. We highlight that the superposition of Gaussian wave packets can build the geometric modes described by Hermite polynomials and that the eigenmodes are the result of the interference of Gaussian wave packets. We conclude that Gaussian wave packets might serve as a complete basis for constructing structured light generated in solid-state laser cavities. Our study may provide meaningful insight into Gaussian wave packets for laser transverse modes.

Convolutional Encoding and Normalizing Flows: A Deep Learning Approach for Offshore Wind Speed Probabilistic Forecasting in the Mediterranean Sea

Abstract The safe and efficient execution of offshore operations requires short-term (1–6 h ahead) high-quality probabilistic forecasts of metocean variables. The development areas for offshore wind projects, potentially in high depths, make it difficult to gather measurement data. This paper explores the use of deep learning for wind speed forecasting at an unobserved offshore location. The proposed convolutional architecture jointly exploits coastal measurements and numerical weather predictions to emulate multivariate probabilistic short-term forecasts. We explore both Gaussian and non-Gaussian neural representations using normalizing flows. We benchmark these approaches with respect to state-of-the-art data-driven schemes, including analog methods and quantile forecasting. The performance of the models and resulting forecast quality are analyzed in terms of probabilistic calibration, probabilistic and deterministic metrics, and as a function of weather situations. We report numerical experiments for a real case study off the French Mediterranean coast. Our results highlight the role of regional numerical weather prediction and coastal in situ measurement in the performance of postprocessing. For single-valued forecasts, a 40% decrease in RMSE is observed compared to the direct use of numerical weather predictions. Significant skill improvements are also obtained for the probabilistic forecasts, in terms of various scores, as well as an acceptable probabilistic calibration. The proposed architecture can process a large amount of heterogeneous input data and offers a versatile probabilistic framework for multivariate forecasting.

Gaussian process learning of nonlinear dynamics

One of the pivotal tasks in scientific machine learning is to represent underlying dynamical systems from time series data. Many methods for such dynamics learning explicitly require the derivatives of state data, which are not directly available and can be approximated conventionally by finite differences. However, the discrete approximations of time derivatives may result in poor estimations when state data are scarce and/or corrupted by noise, thus compromising the predictiveness of the learned dynamical models. To overcome this technical hurdle, we propose a new method that learns nonlinear dynamics through a Bayesian inference of characterizing model parameters. This method leverages a Gaussian process representation of states, and constructs a likelihood function using the correlation between state data and their derivatives, yet prevents explicit evaluations of time derivatives. Through a Bayesian scheme, a probabilistic estimate of the model parameters is given by the posterior distribution, and thus a quantification is facilitated for uncertainties from noisy state data and the learning process. Specifically, we will discuss the applicability of the proposed method to several typical scenarios for dynamical systems: identification and estimation with an affine parametrization, nonlinear parametric approximation without prior knowledge, and general parameter estimation for a given dynamical system.

Statistical physics of principal minors: Cavity approach.

Determinants are useful to represent the state of an interacting system of (effectively) repulsive and independent elements, like fermions in a quantum system and training samples in a learning problem. A computationally challenging problem is to compute the sum of powers of principal minors of a matrix which is relevant to the study of critical behaviors in quantum fermionic systems and finding a subset of maximally informative training data for a learning algorithm. Specifically, principal minors of positive square matrices can be considered as statistical weights of a random point process on the set of the matrix indices. The probability of each subset of the indices is in general proportional to a positive power of the determinant of the associated submatrix. We use Gaussian representation of the determinants for symmetric and positive matrices to estimate the partition function (or free energy) and the entropy of principal minors within the Bethe approximation. The results are expected to be asymptotically exact for diagonally dominant matrices with locally treelike structures. We consider the Laplacian matrix of random regular graphs of degree K=2,3,4 and exactly characterize the structure of the relevant minors in a mean-field model of such matrices. No (finite-temperature) phase transition is observed in this class of diagonally dominant matrices by increasing the positive power of the principal minors, which here plays the role of an inverse temperature.

Gaussian distribution representation based knowledge distillation method for oriented SAR ship detection

Gaussian distribution representation based knowledge distillation method for oriented SAR ship detection