Implicit Representation Research Articles

PM-INR: Prior-Rich Multi-Modal Implicit Large-Scale Scene Neural Representation

Recent advancements in implicit neural representations have contributed to high-fidelity surface reconstruction and photorealistic novel view synthesis. However, with the expansion of the scene scale, such as block or city level, existing methods will encounter challenges because traditional sampling cannot cope with the cubically growing sampling space. To alleviate the dependence on filling the sampling space, we explore using multi-modal priors to assist individual points to obtain more global semantic information and propose a priorrich multi-modal implicit neural representation network, Pm-INR, for the outdoor unbounded large-scale scene. The core of our method is multi-modal prior extraction and crossmodal prior fusion modules. The former encodes codebooks from different modality inputs and extracts valuable priors, while the latter fuses priors to maintain view consistency and preserve unique features among multi-modal priors. Finally, feature-rich cross-modal priors are injected into the sampling regions to allow each region to perceive global information without filling the sampling space. Extensive experiments have demonstrated the effectiveness and robustness of our method for outdoor unbounded large-scale scene novel view synthesis, which outperforms state-of-the-art methods in terms of PSNR, SSIM, and LPIPS.

Progressive High-Frequency Reconstruction for Pan-Sharpening with Implicit Neural Representation

Pan-sharpening aims to leverage the high-frequency signal of the panchromatic (PAN) image to enhance the resolution of its corresponding multi-spectral (MS) image. However, deep neural networks (DNNs) tend to prioritize learning the low-frequency components during the training process, which limits the restoration of high-frequency edge details in MS images. To overcome this limitation, we treat pan-sharpening as a coarse-to-fine high-frequency restoration problem and propose a novel method for achieving high-quality restoration of edge information in MS images. Specifically, to effectively obtain fine-grained multi-scale contextual features, we design a Band-limited Multi-scale High-frequency Generator (BMHG) that generates high-frequency signals from the PAN image within different bandwidths. During training, higher-frequency signals are progressively injected into the MS image, and corresponding residual blocks are introduced into the network simultaneously. This design enables gradients to flow from later to earlier blocks smoothly, encouraging intermediate blocks to concentrate on missing details. Furthermore, to address the issue of pixel position misalignment arising from multi-scale features fusion, we propose a Spatial-spectral Implicit Image Function (SIIF) that employs implicit neural representation to effectively represent and fuse spatial and spectral features in the continuous domain. Extensive experiments on different datasets demonstrate that our method outperforms existing approaches in terms of quantitative and visual measurements for high-frequency detail recovery.

Learning Dense Correspondence for NeRF-Based Face Reenactment

Face reenactment is challenging due to the need to establish dense correspondence between various face representations for motion transfer. Recent studies have utilized Neural Radiance Field (NeRF) as fundamental representation, which further enhanced the performance of multi-view face reenactment in photo-realism and 3D consistency. However, establishing dense correspondence between different face NeRFs is non-trivial, because implicit representations lack ground-truth correspondence annotations like mesh-based 3D parametric models (e.g., 3DMM) with index-aligned vertexes. Although aligning 3DMM space with NeRF-based face representations can realize motion control, it is sub-optimal for their limited face-only modeling and low identity fidelity. Therefore, we are inspired to ask: Can we learn the dense correspondence between different NeRF-based face representations without a 3D parametric model prior? To address this challenge, we propose a novel framework, which adopts tri-planes as fundamental NeRF representation and decomposes face tri-planes into three components: canonical tri-planes, identity deformations, and motion. In terms of motion control, our key contribution is proposing a Plane Dictionary (PlaneDict) module, which efficiently maps the motion conditions to a linear weighted addition of learnable orthogonal plane bases. To the best of our knowledge, our framework is the first method that achieves one-shot multi-view face reenactment without a 3D parametric model prior. Extensive experiments demonstrate that we produce better results in fine-grained motion control and identity preservation than previous methods.

Swift-Mapping: Online Neural Implicit Dense Mapping in Urban Scenes

Online dense mapping of urban scenes is of paramount importance for scene understanding of autonomous navigation. Traditional online dense mapping methods fuse sensor measurements (vision, lidar, etc.) across time and space via explicit geometric correspondence. Recently, NeRF-based methods have proved the superiority of neural implicit representations by high-fidelity reconstruction of large-scale city scenes. However, it remains an open problem how to integrate powerful neural implicit representations into online dense mapping. Existing methods are restricted to constrained indoor environments and are too computationally expensive to meet online requirements. To this end, we propose Swift-Mapping, an online neural implicit dense mapping framework in urban scenes. We introduce a novel neural implicit octomap (NIO) structure that provides efficient neural representation for large and dynamic urban scenes while retaining online update capability. Based on that, we propose an online neural dense mapping framework that effectively manages and updates neural octree voxel features. Our approach achieves SOTA reconstruction accuracy while being more than 10x faster in reconstruction speed, demonstrating the superior performance of our method in both accuracy and efficiency.

Latent assimilation with implicit neural representations for unknown dynamics

Data assimilation is crucial in a wide range of applications, but it often faces challenges such as high computational costs due to data dimensionality and incomplete understanding of underlying mechanisms. To address these challenges, this study presents a novel assimilation framework, termed Latent Assimilation with Implicit Neural Representations (LAINR). By introducing Spherical Implicit Neural Representations (SINR) along with a data-driven uncertainty estimator of the trained neural networks, LAINR enhances efficiency in the assimilation process. Experimental results indicate that LAINR holds a certain advantage over existing methods based on AutoEncoders, both in terms of accuracy and efficiency.

Boosting likelihood learning with event reweighting

Extracting maximal information from experimental data requires access to the likelihood function, which however is never directly available for complex experiments like those performed at high energy colliders. Theoretical predictions are obtained in this context by Monte Carlo events, which do furnish an accurate but abstract and implicit representation of the likelihood. Strategies based on statistical learning are currently being developed to infer the likelihood function explicitly by training a continuous-output classifier on Monte Carlo events. In this paper, we investigate the usage of Monte Carlo events that incorporate the dependence on the parameters of interest by reweighting. This enables more accurate likelihood learning with less training data and a more robust learning scheme that is more suited for automation and extensive deployment. We illustrate these advantages in the context of LHC precision probes of new Effective Field Theory interactions.

Reliable and Accurate Implicit Neural Representation of Multiple Swept Volumes with Application to Safe Human–Robot Interaction

In automated production using collaborative robots in a manufacturing cell, a crucial aspect is to avoid collisions to ensure the safety of workers and robots in human–robot interaction. One approach for detecting collisions is using the swept volume (SV) to identify safe protective space for operation. We learn an accurate and reliable signed distance function (SDF) network from raw point clouds of a pre-computed SV to represent a class of linear joint motion trajectories. The network requires only a set of parameters and constant execution time, thus reducing the computational time and memory of collision checking due to the complexity of explicit geometry during task execution. The distance of collision danger foresaw by the learned SDF is exploited to reduce the frequency of collision detection calls in the dynamic environment and reduce the computational cost further. We assess the relative merits of the implicit neural representation of multiple SVs in terms of F1-score, error distance from the surface of the truth geometry, and 3D visualization by comparing favorably with a binary voxel network for learning a single SV with similar inference time. All the predicted errors of the geometry lie within a distance of 4 voxels from the surface of the truth geometry, and most reconstruction errors are within 3 voxels. A simulation of pick-and-place task execution in the human–robot interaction scenarios by leveraging the learned SDF as an efficient continuous collision detector is performed. The improvement in execution time and collision detection number is validated in the simulation.

Periodic implicit representation, design and optimization of porous structures using periodic B-splines

Porous structures are intricate solid materials with numerous small pores, extensively used in fields like medicine, chemical engineering, and aerospace. However, the design of such structures using computer-aided tools is a time-consuming and tedious process. In this study, we propose a novel representation method and design approach for porous units that can be infinitely spliced to form a porous structure. We use periodic B-spline functions to represent periodic or symmetric porous units. Starting from a voxel representation of a porous sample, the discrete distance field is computed. To fit the discrete distance field with a periodic B-spline, we introduce the constrained least squares progressive-iterative approximation algorithm, which results in an implicit porous unit. This unit can be subject to optimization to enhance connectivity and utilized for topology optimization, thereby improving the model’s stiffness while maintaining periodicity or symmetry. The experimental results demonstrate the potential of the designed complex porous units in enhancing the mechanical performance of the model. Consequently, this study has the potential to incorporate remarkable structures derived from artificial design or nature into the design of high-performing models, showing the promise for biomimetic applications.

Deep continuous convolutional networks for fault diagnosis

Convolutional neural network (CNN) architectures have been extensively utilized in data-driven fault diagnosis and have demonstrated significant success. However, there remain certain constraints or restrictions of standard CNN for real-world applications, including discrete convolutions with a priori kernel size that failed to capture richer long-term features, performance deterioration when testing on irregular-length signals, and low accuracy under imbalanced datasets with severe levels of noise. In this paper, a new deep continuous convolutional network (DCCN) based on implicit neural representations is proposed to alleviate these limitations. Multilayer perceptrons with Gaussian masks are developed for the parameterization of continuous convolutional kernels, while multiplicative Gabor filters are used to improve the anti-noise ability. Depth-wise separable convolutions with residual connections are used to improve the computational efficiency of DCCN. The feasibility of DCCN has been verified on two laboratory datasets and one public dataset. The proposed approach achieves average macro F1-scores of over 98.69 % for bearing, gearbox, and mixed fault diagnosis under different noise interferences. Moreover, DCCN achieves an average macro F1-score of 86.43 % when training on 2048 data points and testing on 512 data points. Results show that DCCN has more robust recognition and anti-noise capability than state-of-art methods.

Implicit neural representations in light microscopy.

Three-dimensional stacks acquired with confocal or two-photon microscopy are crucial for studying neuroanatomy. However, high-resolution image stacks acquired at multiple depths are time-consuming and susceptible to photobleaching. In vivo microscopy is further prone to motion artifacts. In this work, we suggest that deep neural networks with sine activation functions encoding implicit neural representations (SIRENs) are suitable for predicting intermediate planes and correcting motion artifacts, addressing the aforementioned shortcomings. We show that we can accurately estimate intermediate planes across multiple micrometers and fully automatically and unsupervised estimate a motion-corrected denoised picture. We show that noise statistics can be affected by SIRENs, however, rescued by a downstream denoising neural network, shown exemplarily with the recovery of dendritic spines. We believe that the application of these technologies will facilitate more efficient acquisition and superior post-processing in the future.

Periodic Composite Function-Based Approach for Designing Architected Materials With Programable Poisson’s Ratios

Abstract Advances in additive manufacturing enable fabrication of architected materials composed of microstructures with extreme mechanical properties. In the design of such architected materials, the parameterization of microstructures determines not just the computational cost but also connectivity between adjacent microstructures. In this paper, we propose a periodic composite function (PCF)-based approach for designing microstructures. The shape of the microstructures is characterized by the value of the periodic composite functions. The proposed method can program microstructures with both positive and negative Poisson’s ratios by a small number of parameters. Furthermore, due to its implicit representation, the proposed method allows for continuously tiling of microstructures with different mechanical properties. Explicit geometric features of the PCF-based microstructures are extracted, and the condition to maintain connectivity between adjacent microstructures is derived. Based on the proposed approach, multiple groups of 2D and 3D microstructures with Poisson’s ratios ranging from negative to positive are presented. Combining with a deep neural network (DNN)-based surrogate model to predict macroscopic material properties of the microstructures, the proposed method is applied to the design of architected materials for elastic deformation control. Numerical examples on both microstructure representation and architected materials design are presented to demonstrate the efficacy of the proposed approach.

Neural Knitworks: Patched neural implicit representation networks

Optimizing images as output of a neural network has been shown to introduce a powerful prior for image inverse tasks, capable of producing solutions of reasonable quality in a fully internal learning context, where no external datasets are involved. Two potential technical approaches involve fitting a coordinate-based Multilayer Perceptron (MLP), or a Convolutional Neural Network to produce the result image as output. The aim of this work is to evaluate the two counterparts, as well as a new framework proposed here, named Neural Knitwork, which maps pixel coordinates to local texture patches rather than singular pixel values. The utility of the proposed technique is demonstrated on the tasks of image inpainting, super-resolution, and denoising. It is shown that the Neural Knitwork can outperform the standard coordinate-based MLP baseline for the tasks of inpainting and denoising, and perform comparably for the super-resolution task.

Lightweight 3-D Convolutional Occupancy Networks for Virtual Object Reconstruction.

The increasing demand for edge devices causes the necessity for recent technologies to be adaptable to nonspecialized hardware. In particular, in the context of augmented, virtual reality, and computer graphics, the 3-D object reconstruction task from a sparse point cloud is highly computationally demanding and for this reason, it is difficult to accomplish on embedded devices. In addition, the majority of earlier works have focused on mesh quality at the expense of speeding up the creation process. In order to find the best balance between time for mesh generation and mesh quality, we aim to tackle the object reconstruction process by developing a lightweight implicit representation. To achieve this goal, we leverage the use of convolutional occupancy networks. We show the effectiveness of the proposed approach through extensive experiments on the ShapeNet dataset using systems with different resources such as GPU, CPU, and an embedded device.

Knowledge Verification From Data.

Knowledge verification is an important task in the quality management of knowledge graphs (KGs). Knowledge is a summary of facts and events based on human cognition and experience. Due to the nature of knowledge, most knowledge quality (KQ) management methods are designed by human experts or the characteristics of existing knowledge, which may be limited by human cognition and the quality of existing knowledge. Numerical data contain a wealth of potential information that may be helpful in verifying knowledge, which is rarely explored. However, due to the implicit representation of numerical data to facts as well as the noise in the data, it is challenging to use data to verify the knowledge. Therefore, this article proposes a knowledge verification method, which discovers the correlation and causality from numerical data to validate knowledge and then evaluate the quality of knowledge. Moreover, to address the impact of noise, the method integrates multisource knowledge to jointly evaluate the KQ. Specifically, an iterative update method is designed to update KQ by utilizing the consistency between multisource knowledge while designing knowledge verification factors based on data causality and correlation to manage update process. The method is validated with multiple datasets, and the results demonstrate that the proposed method could evaluate KQ more accurately and has strong robustness to noise in the data.

Investigation of a neural implicit representation tomography method for flow diagnostics

Abstract In this work, a new gridless approach to tomographic reconstruction of 3D flow fields is introduced and investigated. The approach, termed here as FluidNeRF, is based on the concept of volume representation through Neural Radiance Fields (NeRF). NeRF represents a 3D volume as a continuous function using a deep neural network. In FluidNeRF, the neural network is a function of 3D spatial coordinates in the volume and produces an intensity of light per unit volume at that position. The network is trained using the loss between measured and rendered 2D projections similar to other multi-camera tomography techniques. Projections are rendered using an emission-based integrated line-of-sight method where light rays are traced through the volume; the network is used to determine intensity values along the ray. This paper investigates the influence of the NeRF hyperparameters, camera layout and spacing, and image noise on the reconstruction quality as well as the computational cost. A DNS-generated synthetic turbulent jet is used as a ground-truth representative flow field. Results obtained with FluidNeRF are compared to an adaptive simultaneous algebraic reconstruction technique (ASART), which is representative of a conventional reconstruction technique. Results show that FluidNeRF matches or outperforms ASART in reconstruction quality, is more robust to noise, and offers several advantages that make it more flexible and thus suitable for extension to other flow measurement techniques and scaling to larger-scale problems.

Fast Digital Orthophoto Generation: A Comparative Study of Explicit and Implicit Methods

A digital orthophoto is an image with geometric accuracy and no distortion. It is acquired through a top view of the scene and finds widespread applications in map creation, planning, and related fields. This paper classifies the algorithms for digital orthophoto generation into two groups: explicit methods and implicit methods. Explicit methods rely on traditional geometric methods, obtaining geometric structure presented with explicit parameters with Multi-View Stereo (MVS) theories, as seen in our proposed Top view constrained Dense Matching (TDM). Implicit methods rely on neural rendering, obtaining implicit neural representation of scenes through the training of neural networks, as exemplified by Neural Radiance Fields (NeRFs). Both of them obtain digital orthophotos via rendering from a top-view perspective. In addition, this paper conducts an in-depth comparative study between explicit and implicit methods. The experiments demonstrate that both algorithms meet the measurement accuracy requirements and exhibit a similar level of quality in terms of generated results. Importantly, the explicit method shows a significant advantage in terms of efficiency, with a time consumption reduction of two orders of magnitude under our latest Compute Unified Device Architecture (CUDA) version TDM algorithm. Although explicit and implicit methods differ significantly in their representation forms, they share commonalities in the implementation across algorithmic stages. These findings highlight the potential advantages of explicit methods in orthophoto generation while also providing beneficial references and practical guidance for fast digital orthophoto generation using implicit methods.

Structerf-SLAM: Neural implicit representation SLAM for structural environments

In recent years, research on simultaneous localization and mapping (SLAM) using neural implicit representation has shown promising outcomes due to its smooth mapping and low memory consumption, particularly suitable for structured environments with limited boundaries. However, there is currently no implicit SLAM that can effectively utilize prior structural constraints to accurately build 3D maps. In this study, we propose an RGB-D dense tracking and mapping approach, Structerf-SLAM, that combines visual odometry with neural implicit representation. Our scene representation consists of dual-layer feature grids and pre-trained decoders that decode the interpolated features into RGB and depth values. Moreover, structured planar constraints are integrated. In the tracking stage, utilizing the three-dimensional plane features under the Manhattan assumption achieves more stable and rapid data association, consequently resolving the tracking misalignment issue in textureless regions (e.g., floor, wall, etc.). In the mapping stage, by enforcing planar consistency, the depth predicted by the neural radiation field is well-fitted by a plane, resulting in smoother and more realistic map reconstruction. Experiments on synthetic and real scene datasets demonstrate competitive results of Structerf-SLAM in both mapping and tracking quality.

Self-supervised multicontrast super-resolution for diffusion-weighted prostate MRI.

This study addresses the challenge of low resolution and signal-to-noise ratio (SNR) in diffusion-weighted images (DWI), which are pivotal for cancer detection. Traditional methods increase SNR at high b-values through multiple acquisitions, but this results in diminished image resolution due to motion-induced variations. Our research aims to enhance spatial resolution by exploiting the global structure within multicontrast DWI scans and millimetric motion between acquisitions. We introduce a novel approach employing a "Perturbation Network" to learn subvoxel-size motions between scans, trained jointly with an implicit neural representation (INR) network. INR encodes the DWI as a continuous volumetric function, treating voxel intensities of low-resolution acquisitions as discrete samples. By evaluating this function with a finer grid, our model predicts higher-resolution signal intensities for intermediate voxel locations. The Perturbation Network's motion-correction efficacy was validated through experiments on biological phantoms and in vivo prostate scans. Quantitative analyses revealed significantly higher structural similarity measures of super-resolution images to ground truth high-resolution images compared to high-order interpolation (p 0.005). In blind qualitative experiments, of super-resolution images were assessed to have superior diagnostic quality compared to interpolated images. High-resolution details in DWI can be obtained without the need for high-resolution training data. One notable advantage of the proposed method is that it does not require a super-resolution training set. This is important in clinical practice because the proposed method can easily be adapted to images with different scanner settings or body parts, whereas the supervised methods do not offer such an option.

Robust Deformable Image Registration Using Cycle-Consistent Implicit Representations.

Recent works in medical image registration have proposed the use of Implicit Neural Representations, demonstrating performance that rivals state-of-the-art learning-based methods. However, these implicit representations need to be optimized for each new image pair, which is a stochastic process that may fail to converge to a global minimum. To improve robustness, we propose a deformable registration method using pairs of cycle-consistent Implicit Neural Representations: each implicit representation is linked to a second implicit representation that estimates the opposite transformation, causing each network to act as a regularizer for its paired opposite. During inference, we generate multiple deformation estimates by numerically inverting the paired backward transformation and evaluating the consensus of the optimized pair. This consensus improves registration accuracy over using a single representation and results in a robust uncertainty metric that can be used for automatic quality control. We evaluate our method with a 4D lung CT dataset. The proposed cycle-consistent optimization method reduces the optimization failure rate from 2.4% to 0.0% compared to the current state-of-the-art. The proposed inference method improves landmark accuracy by 4.5% and the proposed uncertainty metric detects all instances where the registration method fails to converge to a correct solution. We verify the generalizability of these results to other data using a centerline propagation task in abdominal 4D MRI, where our method achieves a 46% improvement in propagation consistency compared with single-INR registration and demonstrates a strong correlation between the proposed uncertainty metric and registration accuracy.

Kernel functions embed into the autoencoder to identify the sparse models of nonlinear dynamics

Numerous researches have shown that there are three main challenges in data-driven model identification methods: high-dimensional measurements, system complexity and unknown underlying dynamical properties. For most nonlinear dynamics, the feature space defined by the coefficients of their control equations is sparse. Therefore, sparse regression methods are used to learn the sparse coefficients of the control equations of nonlinear dynamics. However, this method strongly depends on the appropriate selection of the sparse basis vectors. In this essay, the autoencoder is combined with the sparse regression method to simultaneously identify the sparse coordinate and a parsimonious, interpretable and generalizable model of the specified system. It also integrates kernel functions to map the intractable measurements in the hidden space of the autoencoder into a linearly distinguishable kernel space, which kernelizes the candidate function library of the sparse identification of nonlinear dynamics (SINDy) model as the sparse dictionaries. Therefore, the flexible representation of neural networks, the simplicity of sparse regression methods and the implicit non-linear representation of kernel functions are consolidated in this article. To inspect the reliability of the proposed model in this paper, a set of nonlinear dynamics formulated by ordinary differential equations (ODEs), second-order trigonometric functions and partial differential equations (PDEs) are utilized as test cases. And the comparisons between the proposed model and other model identification methods illustrate that the performance of the former is the best.