Efficient Forward Model Research Articles

Fast forward modeling of magnetotelluric data in complex continuous media using an extended Fourier DeepONet architecture

The calculations of magnetotelluric (MT) responses play a fundamental role in the inversion and resolution analysis for MT problems. Conventional numerical methods for forward modeling involve solving a large system of linear equations. Although these methods provide accurate and stable solutions, they are computationally expensive as the model size increases. In this study, we develop an efficient forward modeling network utilizing Extended Fourier DeepONet (EFDO) architecture for fast MT forward modeling in complex continuous media. By efficiently learning the mapping of differential operators from conductivity models to MT responses, the EFDO framework can effectively approximate the MT forward operator. Once trained, it can instantaneously predict the MT responses for complex continuous resistivity models under identical model discretization conditions. We first compare the accuracy of the EFDO with conventional finite volume (FV) based forward modeling solver and existing deep learning (DL) based modeling schemes. Numerical experiments demonstrate that the MT responses predicted by the EFDO exhibit high conformity with the solutions computed by the FV method. Compared to other DL-based solvers, such as the extended Fourier Neural Operator (EFNO) and UNet-Fourier Neural Operator (UFNO), the EFDO shows superior prediction accuracy while significantly reducing training resource requirements. In addition, the EFDO exhibits excellent generalization, maintaining outstanding prediction performance for the untrained frequencies of the same class samples. The EFDO can predict the MT responses at approximately 200 times the speed of the FV solver. The huge improvement in the computational speed makes the EFDO a promising candidate in simulation-intensive tasks such as detailed resolution analysis and high-dimensional MT probabilistic inversions, which involve tremendous forward evaluations.

An Efficient Forward Modeling Method of Electromagnetic Response of Multiscale Hydraulic Fracture Based on Deep Learning

An Efficient Forward Modeling Method of Electromagnetic Response of Multiscale Hydraulic Fracture Based on Deep Learning

Forward modelling low-spectral-resolution Cassini/CIRS observations of Titan

The Composite InfraRed Spectrometer (CIRS) instrument onboard the Cassini spacecraft performed 8.4 million spectral observations of Titan at resolutions between 0.5–15.5 cm-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document}. More than 3 million of these were acquired at a low spectral resolution (SR) (13.5–15.5 cm-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document}), which have excellent spatial and temporal coverage in addition to the highest spatial resolution and lowest noise per spectrum of any of the CIRS observations. Despite this, the CIRS low-SR dataset is currently underused for atmospheric composition analysis, as spectral features are often blended and subtle compared to those in higher SR observations. The vast size of the dataset also poses a challenge as an efficient forward model is required to fully exploit these observations. Here, we show that the CIRS FP3/4 nadir low-SR observations of Titan can be accurately forward modelled using a computationally efficient correlated-k\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varvec{k}$$\\end{document} method. We quantify wavenumber-dependent forward modelling errors, with mean 0.723 nW cm-2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-2}}\\,$$\\end{document}sr-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document}/cm-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document} (FP3: 600–890 cm-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document}) and 0.248 nW cm-2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-2}}\\,$$\\end{document}sr-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}\\,$$\\end{document}/ cm-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document} (FP4: 1240–1360 cm-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{\\varvec{-1}}$$\\end{document}), that can be used to improve the rigour of future retrievals. Alternatively, in cases where more accuracy is required, we show observations can be forward modelled using an optimised line-by-line method, significantly reducing computation time.

Higher-order thin layer method as an efficient forward model for calculating dispersion curves of surface and Lamb waves in layered media

The thin layer method (TLM) is an effective semi-discrete numerical tool for analyzing wave motion in stratified media. The TLM can calculate the eigenvalues and eigenvectors for both propagating and decaying waves, which is essential to simulate waves near the source. The eigenvectors are particularly useful in various applications, such as estimating modal contributions, calculating synthetic seismograms, determining Green’s function, analyzing site response, and solving wave amplification problems in earthquake engineering, to name a few. In practice, most engineering applications use linear element-based TLM. However, the overall performance of TLM depends upon the type of element and the number of thin layers. The linear element TLM requires relatively thin sub-layers to obtain an accurate solution, making it computationally expensive and time-consuming. This study explores the potential of using non-linear higher-order TLM (HTLM) for forward modeling. The mathematical framework of TLM is used to formulate the generalized HTLM in determining the multimodal dispersion curve of Rayleigh, Love, and Lamb waves. The convergence and computational efficiency of different orders of HTLM have been investigated on a diverse set of published soil profiles and plate-like structures. Since non-linear interpolation functions better capture the change in displacement and stress between the layers, the findings reveal that the HTLM consistently outperforms the linear element TLM in terms of accuracy. Furthermore, the computational efficiency of different order HTLM is analyzed by comparing the total degrees of freedom and CPU time. Remarkably, higher-order HTLMs exhibit improved accuracy without increasing the computational burden, making them an attractive option for practical applications such as global search-based inversion and full spectrum inversions where fast and economical forward modeling is essential. Therefore, the HTLM will become helpful in seismic wavefield modeling, dynamic soil characterization, non-destructive evaluation methods, wave propagation analysis through waveguides, seismic site response analysis, etc.

An efficient forward semi-Lagrangian model

An efficient forward trajectory model is proposed, in which the property and position of the fluids advected from the Euler coordinates to the Lagrangian coordinates can be accurately evaluated. After sorting and aligning those fluid elements on the irregular Lagrangian curves, we apply the cubic or other high-degree polynomials to interpolate the properties of the elements from the irregular curves to the regular grids. There is no need to solve the cubic equations and the associated coefficients as proposed previously. The model is quite simple, accurate, and much more efficient than the previous models. It also allows higher-order polynomials to be employed in the interpolations. It is suitable for simulating the multi-dimensional fast-moving flows with large Courant Numbers, the transport of pollutants in the atmosphere and ocean, and movement of raindrops in atmospheric models.

Efficient 3D Frequency Semi-Airborne Electromagnetic Modeling Based on Domain Decomposition

Landslides are common geological hazards that often result in significant casualties and economic losses. Due to their occurrence in complex terrain areas, conventional geophysical techniques face challenges in early detection and warning of landslides. Semi-airborne electromagnetic (SAEM) technology, utilizing unmanned aerial platforms for rapid unmanned remote sensing, can overcome the influence of complex terrain and serve as an effective approach for landslide detection and monitoring. In response to the low computational efficiency of conventional semi-airborne EM 3D forward modeling, this study proposes an efficient forward modeling method. To handle arbitrarily complex topography and geologic structures, the unstructured tetrahedron mesh is adopted to discretize the earth. Based on the vector finite element formula, the Dual–Primal Finite Element Tearing and Interconnecting (FETI-DP) method is further applied to enhance modeling efficiency. It obtains a reduced order subsystem and avoids directly solving huge overall linear equations by converting the entirety problem into the interface problem. We check our algorithm by comparing it with 1D semi-analytical solutions and the conventional finite element method. The numerical experiments reveal that the FETI-DP method utilities less memory and exhibits higher computation efficiency than the conventional methods. Additionally, we compare the electromagnetic responses with the topography model and flat earth model. The comparison results indicate that the effect of topography cannot be ignored. Further, we analyze the characteristic of electromagnetic responses when the thickness of the aquifer changes in a landslide area. We demonstrate the effectiveness of the 3D SAEM method for landslide detection and monitoring.

Characterization of Induced Polarization Parameters from Electromagnetic Data using Evolutionary Approach

Electromagnetic methods are one of the most important tools for exploring the physical properties of scatterers. When the scattering object is polarizable, more information can be gained from electromagnetic measurements to determine the scatterer’s electrical properties. In this paper, an inversion scheme is introduced to extract induced polarization information from electromagnetic measurements. The inverse scattering problem is reformulated as an optimization problem that utilizes an efficient forward model solver to compute the scattered field. Furthermore, a simulated annealing approach is considered in the presented scheme, wherein different cooling schedules are evaluated to solve the inverse problem. The results of applying the proposed scheme to various case studies are reported.

Implementation and validation of swept density reflectometry for integrated data analysis at ASDEX Upgrade

In the tokamak ASDEX Upgrade, Integrated Data Analysis (IDA) is used to infer plasma quantities, such as electron density, using heterogeneous data sources. Essential is forward modeling from the parameter space into the data space with physically reasonable models for probabilistic evaluation. This paper presents a new forward model for O-mode profile reflectometry, a necessary prerequisite for Bayesian inference and inclusion in IDA. An efficient forward model based on the analytic solution for a piece-wise linear density description allows IDA to overcome problems associated with the established determination of cut-off locations via Abel inversion and Bottollier-Curtet’s method. Instead of using a hard-coded initialization for densities below the first measured cut-off density, other diagnostics, such as the lithium beam, are used to analyze the shape of the initial part of the profile. Error propagation from the measured data, and other uncertain sources, to the uncertainties in the density profile and also its gradient is an intrinsic property of the probabilistic approach, which benefits from the joint analysis. Missing or ambiguous data do not prevent the profile evaluation, but only increase the uncertainty for densities in the affected range. Density profiles together with their uncertainties are determined by the joint analysis of complementary diagnostics, with the newly added reflectometry closing a gap in the outer core region. A stand-alone inversion based on the new forward model, including uncertainty quantification, is introduced, optionally providing an n(R) profile with uncertainties and a gradient. This method is a candidate for real-time analysis, providing error bars.

Efficient computation of the steady-state and time-domain solutions of the photon diffusion equation in layered turbid media

Accurate and efficient forward models of photon migration in heterogeneous geometries are important for many applications of light in medicine because many biological tissues exhibit a layered structure of independent optical properties and thickness. However, closed form analytical solutions are not readily available for layered tissue-models, and often are modeled using computationally expensive numerical techniques or theoretical approximations that limit accuracy and real-time analysis. Here, we develop an open-source accurate, efficient, and stable numerical routine to solve the diffusion equation in the steady-state and time-domain for a layered cylinder tissue model with an arbitrary number of layers and specified thickness and optical coefficients. We show that the steady-state (< 0.1 ms) and time-domain (< 0.5 ms) fluence (for an 8-layer medium) can be calculated with absolute numerical errors approaching machine precision. The numerical implementation increased computation speed by 3 to 4 orders of magnitude compared to previously reported theoretical solutions in layered media. We verify our solutions asymptotically to homogeneous tissue geometries using closed form analytical solutions to assess convergence and numerical accuracy. Approximate solutions to compute the reflected intensity are presented which can decrease the computation time by an additional 2–3 orders of magnitude. We also compare our solutions for 2, 3, and 5 layered media to gold-standard Monte Carlo simulations in layered tissue models of high interest in biomedical optics (e.g. skin/fat/muscle and brain). The presented routine could enable more robust real-time data analysis tools in heterogeneous tissues that are important in many clinical applications such as functional brain imaging and diffuse optical spectroscopy.

A parallel adaptive finite-element approach for 3-D realistic controlled-source electromagnetic problems using hierarchical tetrahedral grids

SUMMARY To effectively and efficiently interpret or invert controlled-source electromagnetic (CSEM) data which are recorded in areas with the kind of complex geological environments and arbitrary topography that are typical, 3-D CSEM forward modelling software that can quickly solve large-scale problems, provide accurate electromagnetic responses for complex geo-electrical models and can be easily incorporated into inversion algorithms are required. We have developed a parallel goal-oriented adaptive mesh refinement finite-element approach for frequency-domain 3-D CSEM forward modelling with hierarchical tetrahedral grids that can offer accurate electromagnetic responses for large-scale complex models and that can efficiently serve for inversion. The approach uses the goal-oriented adaptive vector finite element method to solve the total electric field vector equation. The geo-electrical model is discretized by unstructured tetrahedral grids which can deal with complex underground geological models with arbitrary surface topography. Different from previous adaptive finite element software working on unstructured tetrahedral grids, we have utilized a novel mesh refinement technique named the longest edge bisection method to generate hierarchically refined grids. As the refined grids are nested into the coarse grids, the refinement technique can precisely map the electrical parameters of inversion grids onto the forward modelling grids so that the extra numerical errors generated by the inconsistency of electrical parameters between inversion grids and forward modelling grids are eliminated. In addition, we use the parallel domain-decomposition technique to further accelerate the computations, and the flexible generalized minimum residual solver (FGMRES) with an auxiliary Maxwell solver pre-conditioner to solve the final large-scale system of linear equations. In the end, we validate the performance of the proposed scheme using two synthetic models and one realistic model. We demonstrate that accurate electromagnetic fields can be obtained by comparison with the analytic solutions and that the code is highly scalable for large-scale problems with millions or even hundreds of millions of unknowns. For the synthetic 3-D model and the realistic model with complex geometry, our solutions match well with the results calculated by an existing 3-D CSEM forward modelling code. Both synthetic and realistic examples demonstrate that our newly developed code is an effective, efficient forward modelling engine for interpreting CSEM field data acquired in areas of complex geology and topography.

Study on laser beam butt-welding of NiTinol sheet and input-output modelling using neural networks trained by metaheuristic algorithms

Though NiTinol, the equiatomic Ni-Ti alloy, is revolutionizing different fields of engineering, manufacturing and medicine since its discovery as a functionally advanced material, poor formability and machinability, and the limitations of available fabrication techniques applied for this alloy are restricting its applications in the form of various desired products. Now-a-days, laser welding is the mostly reported joining technique applied to the alloy. Though the difficulties associated with fabrication method and the cost of the alloy are high enough, a computationally efficient model for quick and precise prediction of operating parameters in order to obtain the specified attributes of the laser welded NiTinol sheets is significantly missing in the literature. In the present work, basic metallographic study related to laser welding of NiTinol sheets in butt-joint configuration was investigated. At the same time, an efficient forward and reverse model had been developed with the help of artificial neural network (ANN). These ANNs were trained with the help of regression equation model using five different metaheuristic techniques separately for the development of both forward and reverse models. Laser power, scan speed, frequency, duty factor and focal positions were treated as the input process variables and three different weld attributes, namely bead-area, micro hardness of the bead and tensile strength of the bead were considered as output variables of the process. The developed model was capable enough of predicting the outputs, which were in good agreement with the experimental data, for both forward and reverse models. The novelty of this study deals with the development and testing of feed-forward neural network and recurrent neural network trained using five different metaheuristic techniques for both the forward and reverse modelling of fibre laser welding of NiTinol alloys.

An efficient multigrid solver based on a four-color cell-block Gauss-Seidel smoother for 3D magnetotelluric forward modeling

Practical application of 3D magnetotelluric inversion requires efficient forward modeling of electromagnetic (EM) fields in the earth. To resolve realistic 3D structures, large computational domains and extremely large linear systems of equations are required. The iterative solvers, which are almost exclusively used to solve these systems, can be inefficient due to the abundant null space of the curl-curl operator. Multigrid (MG) solvers are considered a potentially efficient technique for solving such problems. However, due to the abundant null solution space and existence of the air layer, MG solvers can still converge slowly or even diverge. We have developed an efficient MG solver for finite-difference frequency-domain EM solution. In this algorithm, the excellent smoothing property of an efficient four-color cell-block Gauss-Seidel (GS) is exploited to remove the short-range errors effectively, and the interpolation and prolongation operators are used to handle the long-range errors. They work as a whole to speed the convergence of our algorithm remarkably. Because all of the nodes for the four-color cell-block GS are grouped into four colors and the edge components attached to different nodes in each color are completely decoupled, this can be used to develop a highly vectorized or parallelized algorithm. Another important property is that our algorithm is locally current divergence free, effectively eliminating spurious solutions in the null space of the curl-curl operator. The accuracy and efficiency of the algorithm are verified by comparing the numerical solutions obtained with our MG solver to those from the biconjugate gradient stabilized solver with different preconditioners based on synthetic models and a model from 3D inversion. Comparisons, in terms of iteration number and computational time, indicate that our algorithm is extremely stable and efficient relative to the other solvers. Our MG algorithm will be suitable for massively parallel computing as well.

Reduced order models for uncertainty quantification of gas plumes from leakages during LNG bunkering

The impacts of uncertainty in wind conditions on the spread of hazardous plume resulting from a jet leak during a Liquefied Natural Gas (LNG) bunkering operation were investigated. Computational Fluid Dynamics (CFD) using the Reynolds-Averaged Navier Stokes (RANS) solver with multi-species transport and a transient leak model for keyhole leak was used for the simulation of a simplified bunkering station. Following detailed validation & verification, the sensitivity of the safety zone extents to the wind conditions was demonstrated. CFD results reinforced the strong dependence of the maximum spread distance on wind conditions and enclosure geometry. To quantify the impact of input uncertainty from wind conditions on the plume spread, a reduced-order model (ROM) was developed using the proper orthogonal decomposition (POD) of CFD results on sampled conditions. ROM-POD enables a fast evaluation of the plume under changing wind conditions and acts as an efficient forward model for uncertainty quantification using Polynomial Chaos Expansion (PCE) technique. The spatial distribution of plume residence time under the same input uncertainty was also obtained from the proposed approach showing its potential in risk assessment and design of bunkering facilities.

Comparison of Different Neural Network Architectures for Plasmonic Inverse Design.

The merge between nanophotonics and a deep neural network has shown unprecedented capability of efficient forward modeling and accurate inverse design if an appropriate network architecture and training method are selected. Commonly, an iterative neural network and a tandem neural network can both be used in the inverse design process, where the latter is well known for tackling the nonuniqueness problem at the expense of more complex architecture. However, we are curious to compare these two networks’ performance when they are both applicable. Here, we successfully trained both networks to inverse design the far-field spectrum of plasmonic nanoantenna, and the results provide some guidelines for choosing an appropriate, sufficiently accurate, and efficient neural network architecture.

PINNeik: Eikonal solution using physics-informed neural networks

The eikonal equation is utilized across a wide spectrum of science and engineering disciplines. In seismology, it regulates seismic wave traveltimes needed for applications like source localization, imaging, and inversion. Several numerical algorithms have been developed over the years to solve the eikonal equation. However, these methods require considerable modifications to incorporate additional physics, such as anisotropy, and may even breakdown for certain complex forms of the eikonal equation, requiring approximation methods. Moreover, they suffer from computational bottleneck when repeated computations are needed for perturbations in the velocity model and/or the source location, particularly in large 3D models. Here, we propose an algorithm to solve the eikonal equation based on the emerging paradigm of physics-informed neural networks (PINNs). By minimizing a loss function formed by imposing the eikonal equation, we train a neural network to output traveltimes that are consistent with the underlying partial differential equation. We observe sufficiently high traveltime accuracy for most applications of interest. We also demonstrate how the proposed algorithm harnesses machine learning techniques like transfer learning and surrogate modeling to speed up traveltime computations for updated velocity models and source locations. Furthermore, we use a locally adaptive activation function and adaptive weighting of the terms in the loss function to improve convergence rate and solution accuracy. We also show the flexibility of the method in incorporating medium anisotropy and free-surface topography compared to conventional methods that require significant algorithmic modifications. These properties of the proposed PINN eikonal solver are highly desirable in obtaining a flexible and efficient forward modeling engine for seismological applications.

Multimodal guided wave inversion for arterial stiffness: methodology and validation in phantoms

Arterial stiffness is an important biomarker for many cardiovascular diseases. Shear wave elastography is a recent technique aimed at estimating local arterial stiffness using guided wave inversion (GWI), i.e. matching the computed and measured wave dispersion. This paper develops and validates a new GWI approach by synthesizing various recent observations and algorithms: (a) refinements to signal processing to obtain more accurate experimental dispersion curves; (b) an efficient forward model to compute theoretical dispersion curves for immersed, incompressible cylindrical waveguides; (c) an optimization framework based on the recent observation that the measured dispersion curve is multimodal, i.e. it matches for not one but two different wave modes in two different frequency ranges. The resulting inversion approach is validated using extensive experimental data from rubber tube phantoms, not only for modulus estimation but also to simultaneously estimate modulus and wall thickness. The observations indicate that the modulus estimates are best performed with the information on wall thickness. The approach, which takes less than half a minute to run, is shown to be accurate, with the modulus estimated with less than 4% error for 70% of the experiments.

Dynamic simulation of FWD tests on flexible transversely isotropic pavements with imperfect interfaces

Developing an accurate and efficient forward model of the dynamic response of pavement structure is of great significance for obtaining the design parameters of the layered pavement structure through an inverse calculation based on the measured pavement data. However, the existing algorithms rarely consider the interlayer contact. Based on the traditional spectral element method (SEM), this study proposes a new forward model to simulate the falling weight deflectometer (FWD) dynamic tests for the transversely isotropic layered pavement structure. The interfaces between the pavement layers can be imperfect, and the displacement boundary conditions are discontinuous. Moreover, the developed model considerably enhances the efficiency of the model in solving the dynamic response of transversely isotropic layered media by avoiding the infinite integration in integral transformation. Numerical examples verify the accuracy and efficiency of the model developed in this study. At the same time, an extensive parameter analysis is conducted on the interlayer contact of the pavement structure and inhomogeneity of the layered soil, which provides a reliable, numerical, theoretical basis for the design of pavement structure.

Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy

Miniature fluorescence microscopes are a standard tool in systems biology. However, widefield miniature microscopes capture only 2D information, and modifications that enable 3D capabilities increase the size and weight and have poor resolution outside a narrow depth range. Here, we achieve the 3D capability by replacing the tube lens of a conventional 2D Miniscope with an optimized multifocal phase mask at the objective’s aperture stop. Placing the phase mask at the aperture stop significantly reduces the size of the device, and varying the focal lengths enables a uniform resolution across a wide depth range. The phase mask encodes the 3D fluorescence intensity into a single 2D measurement, and the 3D volume is recovered by solving a sparsity-constrained inverse problem. We provide methods for designing and fabricating the phase mask and an efficient forward model that accounts for the field-varying aberrations in miniature objectives. We demonstrate a prototype that is 17 mm tall and weighs 2.5 grams, achieving 2.76 μm lateral, and 15 μm axial resolution across most of the 900 × 700 × 390 μm3 volume at 40 volumes per second. The performance is validated experimentally on resolution targets, dynamic biological samples, and mouse brain tissue. Compared with existing miniature single-shot volume-capture implementations, our system is smaller and lighter and achieves a more than 2× better lateral and axial resolution throughout a 10× larger usable depth range. Our microscope design provides single-shot 3D imaging for applications where a compact platform matters, such as volumetric neural imaging in freely moving animals and 3D motion studies of dynamic samples in incubators and lab-on-a-chip devices.

Fourier DiffuserScope: single-shot 3D Fourier light field microscopy with a diffuser.

Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane. Placing the MLA near the pupil plane of the microscope, instead of the image plane, can mitigate the artifacts and provide an efficient forward model, at the expense of field-of-view (FOV). Here, we demonstrate improved resolution across a large volume with Fourier DiffuserScope, which uses a diffuser in the pupil plane to encode 3D information, then computationally reconstructs the volume by solving a sparsity-constrained inverse problem. Our diffuser consists of randomly placed microlenses with varying focal lengths; the random positions provide a larger FOV compared to a conventional MLA, and the diverse focal lengths improve the axial depth range. To predict system performance based on diffuser parameters, we, for the first time, establish a theoretical framework and design guidelines, which are verified by numerical simulations, and then build an experimental system that achieves < 3 µm lateral and 4 µm axial resolution over a 1000 × 1000 × 280 µm3 volume. Our diffuser design outperforms the MLA used in LFM, providing more uniform resolution over a larger volume, both laterally and axially.

Learning Structured Representations of Spatial and Interactive Dynamics for Trajectory Prediction in Crowded Scenes

Context plays a significant role in the generation of motion for dynamic agents in interactive environments. This work proposes a modular method that utilises a learned model of the environment for motion prediction. This modularity explicitly allows for unsupervised adaptation of trajectory prediction models to unseen environments and new tasks by relying on unlabelled image data only. We model both the spatial and dynamic aspects of a given environment alongside the per agent motions. This results in more informed motion prediction and allows for performance comparable to the state-of-the-art. We highlight the model's prediction capability using a benchmark pedestrian prediction problem and a robot manipulation task and show that we can transfer the predictor across these tasks in a completely unsupervised way. The proposed approach allows for robust and label efficient forward modelling, and relaxes the need for full model re-training in new environments.