Sparse Input Data Research Articles

A Transformer Approach for Cognitive Impairment Classification and Prediction.

Early classification and prediction of Alzheimer disease (AD) and amnestic mild cognitive impairment (aMCI) with noninvasive approaches is a long-standing challenge. This challenge is further exacerbated by the sparsity of data needed for modeling. Deep learning methods offer a novel method to help address these challenging multiclass classification and prediction problems. We analyzed 3 target feature-sets from the National Alzheimer Coordinating Center (NACC) dataset: (1) neuropsychological (cognitive) data; (2) patient health history data; and (3) the combination of both sets. We used a masked Transformer-encoder without further feature selection to classify the samples on cognitive status (no cognitive impairment, aMCI, AD)-dynamically ignoring unavailable features. We then fine-tuned the model to predict the participants' future diagnosis in 1 to 3 years. We analyzed the sensitivity of the model to input features via Feature Permutation Importance. We demonstrated (1) the masked Transformer-encoder was able to perform prediction with sparse input data; (2) high multiclass current cognitive status classification accuracy (87% control, 79% aMCI, 89% AD); (3) acceptable results for 1- to 3-year multiclass future cognitive status prediction (83% control, 77% aMCI, 91% AD). The flexibility of our methods in handling inconsistent data provides a new venue for the analysis of cognitive status data.

Sparsity-Robust Feature Fusion for Vulnerable Road-User Detection with 4D Radar

Detecting vulnerable road users is a major challenge for autonomous vehicles due to their small size. Various sensor modalities have been investigated, including mono or stereo cameras and 3D LiDAR sensors, which are limited by environmental conditions and hardware costs. Radar sensors are a low-cost and robust option, with high-resolution 4D radar sensors being suitable for advanced detection tasks. However, they involve challenges such as few and irregularly distributed measurement points and disturbing artifacts. Learning-based approaches utilizing pillar-based networks show potential in overcoming these challenges. However, the severe sparsity of radar data makes detecting small objects with only a few points difficult. We extend a pillar network with our novel Sparsity-Robust Feature Fusion (SRFF) neck, which combines high- and low-level multi-resolution features through a lightweight attention mechanism. While low-level features aid in better localization, high-level features allow for better classification. As sparse input data are propagated through a network, the increasing effective receptive field leads to feature maps of different sparsities. The combination of features with different sparsities improves the robustness of the network for classes with few points.

Radiomics on spatial-temporal manifolds via Fokker-Planck dynamics.

Delta radiomics is a high-throughput computational technique used to describe quantitative changes in serial, time-series imaging by considering the relative change in radiomic features of images extracted at two distinct time points. Recent work has demonstrated a lack of prognostic signal of radiomic features extracted using this technique. We hypothesize that this lack of signal is due to the fundamental assumptions made when extracting features via delta radiomics, and that other methods should be investigated. The purpose of this work was to show a proof-of-concept of a new radiomics paradigm for sparse, time-series imaging data, where features are extracted from a spatial-temporal manifold modeling the time evolution between images, and to assess the prognostic value on patients with oropharyngeal cancer (OPC). To accomplish this, we developed an algorithm to mathematically describe the relationship between two images acquired at time and . These images serve as boundary conditions of a partial differential equation describing the transition from one image to the other. To solve this equation, we propagate the position and momentum of each voxel according to Fokker-Planck dynamics (i.e., a technique common in statistical mechanics). This transformation is driven by an underlying potential force uniquely determined by the equilibrium image. The solution generates a spatial-temporal manifold (3 spatial dimensions + time) from which we define dynamic radiomic features. First, our approach was numerically verified by stochastically sampling dynamic Gaussian processes of monotonically decreasing noise. The transformation from high to low noise was compared between our Fokker-Planck estimation and simulated ground-truth. To demonstrate feasibility and clinical impact, we applied our approach to 18F-FDG-PET images to estimate early metabolic response of patients (n=57) undergoing definitive (chemo)radiation for OPC. Images were acquired pre-treatment and 2-weeks intra-treatment (after 20Gy). Dynamic radiomic features capturing changes in texture and morphology were then extracted. Patients were partitioned into two groups based on similar dynamic radiomic feature expression via k-means clustering and compared by Kaplan-Meier analyses with log-rank tests (p<0.05). These results were compared to conventional delta radiomics to test the added value of our approach. Numerical results confirmed our technique can recover image noise characteristics given sparse input data as boundary conditions. Our technique was able to model tumor shrinkage and metabolic response. While no delta radiomics features proved prognostic, Kaplan-Meier analyses identified nine significant dynamic radiomic features. The most significant feature was Gray-Level-Size-Zone-Matrix gray-level variance (p=0.011), which demonstrated prognostic improvement over its corresponding delta radiomic feature (p=0.722). We developed, verified, and demonstrated the prognostic value of a novel, physics-based radiomics approach over conventional delta radiomics via data assimilation of quantitative imaging and differential equations.

Sparse convolutional neural network for high-resolution skull shape completion and shape super-resolution

Traditional convolutional neural network (CNN) methods rely on dense tensors, which makes them suboptimal for spatially sparse data. In this paper, we propose a CNN model based on sparse tensors for efficient processing of high-resolution shapes represented as binary voxel occupancy grids. In contrast to a dense CNN that takes the entire voxel grid as input, a sparse CNN processes only on the non-empty voxels, thus reducing the memory and computation overhead caused by the sparse input data. We evaluate our method on two clinically relevant skull reconstruction tasks: (1) given a defective skull, reconstruct the complete skull (i.e., skull shape completion), and (2) given a coarse skull, reconstruct a high-resolution skull with fine geometric details (shape super-resolution). Our method outperforms its dense CNN-based counterparts in the skull reconstruction task quantitatively and qualitatively, while requiring substantially less memory for training and inference. We observed that, on the 3D skull data, the overall memory consumption of the sparse CNN grows approximately linearly during inference with respect to the image resolutions. During training, the memory usage remains clearly below increases in image resolution—an times 8 increase in voxel number leads to less than times 4 increase in memory requirements. Our study demonstrates the effectiveness of using a sparse CNN for skull reconstruction tasks, and our findings can be applied to other spatially sparse problems. We prove this by additional experimental results on other sparse medical datasets, like the aorta and the heart. Project page at https://github.com/Jianningli/SparseCNN.

Super Sparse 3D Object Detection.

As the perception range of LiDAR expands, LiDAR-based 3D object detection contributes ever-increasingly to the long-range perception in autonomous driving. Mainstream 3D object detectors often build dense feature maps, where the cost is quadratic to the perception range, making them hardly scale up to the long-range settings. To enable efficient long-range detection, we first propose a fully sparse object detector termed FSD. FSD is built upon the general sparse voxel encoder and a novel sparse instance recognition (SIR) module. SIR groups the points into instances and applies highly-efficient instance-wise feature extraction. The instance-wise grouping sidesteps the issue of the center feature missing, which hinders the design of the fully sparse architecture. To further enjoy the benefit of fully sparse characteristic, we leverage temporal information to remove data redundancy and propose a super sparse detector named FSD++. FSD++ first generates residual points, which indicate the point changes between consecutive frames. The residual points, along with a few previous foreground points, form the super sparse input data, greatly reducing data redundancy and computational overhead. We comprehensively analyze our method on the large-scale Waymo Open Dataset, and state-of-the-art performance is reported. To showcase the superiority of our method in long-range detection, we also conduct experiments on Argoverse 2 Dataset, where the perception range ( 200m) is much larger than Waymo Open Dataset ( 75m). Code is open-sourced at https://github.com/tusen-ai/SST.

CLONeR: Camera-Lidar Fusion for Occupancy Grid-Aided Neural Representations

Recent advances in neural radiance fields (NeRFs) achieve state-of-the-art novel view synthesis and facilitate dense estimation of scene properties. However, NeRFs often fail for outdoor, unbounded scenes that are captured under very sparse views with the scene content concentrated far away from the camera, as is typical for field robotics applications. In particular, NeRF-style algorithms perform poorly: (1) when there are insufficient views with little pose diversity, (2) when scenes contain saturation and shadows, and (3) when finely sampling large unbounded scenes with fine structures becomes computationally intensive. This paper proposes <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CLONeR</i> , which significantly improves upon NeRF by allowing it to model large unbounded outdoor driving scenes that are observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for volumetric rendering in metric space. Through extensive quantitative and qualitative experiments on scenes from the KITTI dataset, this paper demonstrates that the proposed method outperforms state-of-the-art NeRF models on both novel view synthesis and dense depth prediction tasks when trained on sparse input data.

Investigation of effect of logjam series for varying channel and barrier physical properties using a sparse input data 1D network model

Targeted design and placement of natural flood management leaky barriers, or engineered logjams, and accurate representation of logjams in hydraulic models across different scales is necessary to generate desired water storage and flood peak attenuation for effective river restoration and natural flood management projects. We systematically assess the effect of varying logjam spacing, extent of logjam-generated change in water surface profile, vertical height of lower gap, overflow to local floodplain, and varying channel slope, representing a series of logjams in a sparse input data 1D network model including the jam-generated backwater, which depends on loss of momentum within the jam. To guide catchment-scale representation, best-fit increased channel resistance was found to increase with number of jams, approaching an analytically determined maximum for close inter-jam spacing, useful for setting an upper envelope for efficacy.

CODEX: Stochastic Encoding Method to Relax Resistive Crossbar Accelerator Design Requirements

A stochastic input encoding scheme (CODEX) is presented that aims to relax the analog-to-digital converter (ADC) design requirements in memristor crossbar systems. CODEX reduces the ADC input range by encoding the input bits using Bernoulli statistics so that the bit-line current distribution becomes a narrow Gaussian. By reducing ADC input range, CODEX can be used to reduce ADC power and area or increase ADC resolution to reduce the number of epochs required for in-situ training. Besides input data encoding, CODEX includes probability thresholding for sparse input data as well as a random re-sampling method for dealing with ADC overflow. CODEX is evaluated on CIFAR-10 dataset image classification and reconstruction, sentiment classification, and audio classification. The results show an averaged 68.5% reduction in ADC power, 35.5% reduction in ADC area, and 25.8% reduction in training epochs required for in-situ training when applied to the state-of-the-art ISAAC and PUMA accelerators.

Aggregation Operators with Undefined Inputs or Outputs

We consider a generalization of aggregation and fusion functions when some of their inputs or outputs are undefined. For simplicity, we represent undefined values by a single NaN (not-a-number) value. We define four main methods of treating undefined values, observe some of their properties, and compare them with several mainstream implementations and norms for handling NaN values. Finally, as a case study demonstrating the apparatus's usefulness, the formalism is applied to the discrete fuzzy transform technique. The formalism allows us to prove a generalized approximation theorem for fuzzy transforms, which removes previously required restrictions on fuzzy partitions for sparse input data.

3D Cascade RCNN: High Quality Object Detection in Point Clouds.

Recent progress on 2D object detection has featured Cascade RCNN, which capitalizes on a sequence of cascade detectors to progressively improve proposal quality, towards high-quality object detection. However, there has not been evidence in support of building such cascade structures for 3D object detection, a challenging detection scenario with highly sparse LiDAR point clouds. In this work, we present a simple yet effective cascade architecture, named 3D Cascade RCNN, that allocates multiple detectors based on the voxelized point clouds in a cascade paradigm, pursuing higher quality 3D object detector progressively. Furthermore, we quantitatively define the sparsity level of the points within 3D bounding box of each object as the point completeness score, which is exploited as the task weight for each proposal to guide the learning of each stage detector. The spirit behind is to assign higher weights for high-quality proposals with relatively complete point distribution, while down-weight the proposals with extremely sparse points that often incur noise during training. This design of completeness-aware re-weighting elegantly upgrades the cascade paradigm to be better applicable for the sparse input data, without increasing any FLOP budgets. Through extensive experiments on both the KITTI dataset and Waymo Open Dataset, we validate the superiority of our proposed 3D Cascade RCNN, when comparing to state-of-the-art 3D object detection techniques. The source code is publicly available at https://github.com/caiqi/Cascasde-3D.

Generating characteristic maps of battery cell parameters from sparse input data

Describing characteristic maps based on sparse input data (data points) is a problem that occurs in a variety of different disciplines. For the conventional interpolation of input data, various standard interpolation methods are available in most applications. Depending on the data’s complexity to be modeled and the available input data, these methods often produce results that do not meet the required accuracy. This paper presents an interpolation method to generate characteristic maps in which data points are added iteratively evident. The data can be arranged arbitrarily in an n-dimensional space. A comparison with standard interpolation methods for map generation is presented. The presented method shows a better fit with selected reference maps compared to applied standard interpolation methods. As an application example, the method is shown in the context of a Hardware-in-the-Loop application. Electrical loads resulting from two different dynamic load scenarios from a plug-in hybrid electric vehicle simulation were applied simultaneously to the simulated and existing battery module to compare the simulated and measured thermal behavior that directly depends on the generated characteristic maps’ accuracy. The results show that the battery module’s simulated temperature deviates by a maximum of approximately 0.3 K from the measured temperature.

Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms.

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Depth Completion can produce a dense depth map from a sparse input and provide a more complete 3D description of the environment. Despite great progress made in depth completion, the sparsity of the input and low density of the ground truth still make this problem challenging. In this work, we propose DenseLiDAR, a novel real-time pseudo-depth guided depth completion neural network. We exploit dense pseudo-depth map obtained from simple morphological operations to guide the network in three aspects: (1) Constructing a residual structure for the output; (2) Rectifying the sparse input data; (3) Providing dense structural loss for training the network. Thanks to these novel designs, higher performance of the output could be achieved. In addition, two new metrics for better evaluating the quality of the predicted depth map are also presented. Extensive experiments on KITTI depth completion benchmark suggest that our model is able to achieve the state-of-the-art performance at the highest frame rate of 50Hz. The predicted dense depth is further evaluated by several downstream robotic perception or positioning tasks. For the task of 3D object detection, 3~5 percent performance gains on small objects categories are achieved on KITTI 3D object detection dataset. For RGB-D SLAM, higher accuracy on vehicle's trajectory is also obtained in KITTI Odometry dataset. These promising results not only verify the high quality of our depth prediction, but also demonstrate the potential of improving the related downstream tasks by using depth completion results.

Editorial for Special Issue “Digital Mapping in Dynamic Environments”

It is widely acknowledged that the global stock of soil and environmental resources are diminishing and under threat. This issue stems from current and historical unsustainable management practices, leading to degraded landscapes, which is further compounded by increased pressures upon them from ever-increasing anthropogenic activities. To curb the trajectory toward a collapse of our ecosystems, systematic ways are needed to assess the condition of our natural resources, how much they might have changed, and to what extent this might impact on the life sustaining functions we derive from our environment and the extent of our food producing systems. Some solutions to these issues come in the form of measurement, mapping and monitoring technology, which facilitates powerful ways in which to be informed about and to understand and assess the condition of our landscapes so that they can be managed strategically or simply improved. This Special Issue showcases from several locations across the globe, detailed examples of what is achievable at the convergence of big data brought about by remote and proximal sensing platforms, advanced statistical modelling and computing infrastructure to understand and monitor our ecosystems better. These utilities not only provide high-resolution abilities to map the extent and changes to our food producing systems, they also have yielded new ways to determine land-use and climate effects on the fate of soil carbon across living generations and to identify hydrological risk strategies in otherwise data-poor urban environments. Leveraging the availability of remote sensing data is telling, but the papers in this Special Issue also highlight the sophistication of modelling capabilities to deliver not only highly detailed maps of temporal dynamic soil phenomena but ways to draw new inferences from sparse and disparate model input data. The challenges of restoring our ecosystems are immense and sobering. However, we are well equipped and capable of confronting these pervasive issues in objective and data-informed ways that have previously never been possible.

Modeling magnetic fields with helical solutions to Laplace’s equation

The series solution to Laplace’s equation in a helical coordinate system is derived and refined using symmetry and chirality arguments. These functions and their more commonplace counterparts are used to model solenoidal magnetic fields via linear, multidimensional curve-fitting. A judicious choice of functional forms motivated by geometry, a small number of free parameters, and sparse input data can lead to highly accurate, fine-grained modeling of solenoidal magnetic fields. These models capture the helical features arising from the winding of the solenoid, with overall field accuracy at better than one part per million.

Asynchronous Spiking Neurons, the Natural Key to Exploit Temporal Sparsity

Inference of Deep Neural Networks for stream signal (Video/Audio) processing in edge devices is still challenging. Unlike the most state of the art inference engines which are efficient for static signals, our brain is optimized for real-time dynamic signal processing. We believe one important feature of the brain (asynchronous state-full processing) is the key to its excellence in this domain. In this work, we show how asynchronous processing with state-full neurons allows exploitation of the existing sparsity in natural signals. This paper explains three different types of sparsity and proposes an inference algorithm which exploits all types of sparsities in the execution of already trained networks. Our experiments in three different applications (Handwritten digit recognition, Autonomous Steering and Hand-Gesture recognition) show that this model of inference reduces the number of required operations for sparse input data by a factor of one to two orders of magnitudes. Additionally, due to fully asynchronous processing this type of inference can be run on fully distributed and scalable neuromorphic hardware platforms.

People Infer Recursive Visual Concepts from Just a Few Examples

Machine learning has made major advances in categorizing objects in images, yet the best algorithms miss important aspects of how people learn and think about categories. People can learn richer concepts from fewer examples, including causal models that explain how members of a category are formed. Here, we explore the limits of this human ability to infer causal “programs”—latent generating processes with nontrivial algorithmic properties—from one, two, or three visual examples. People were asked to extrapolate the programs in several ways, for both classifying and generating new examples. As a theory of these inductive abilities, we present a Bayesian program learning model that searches the space of programs for the best explanation of the observations. Although variable, people’s judgments are broadly consistent with the model and inconsistent with several alternatives, including a pretrained deep neural network for object recognition, indicating that people can learn and reason with rich algorithmic abstractions from sparse input data.

Frequency-splitting dynamic MRI reconstruction using multi-scale 3D convolutional sparse coding and automatic parameter selection.

In this paper, we propose a novel image reconstruction algorithm using multi-scale 3D convolutional sparse coding and a spectral decomposition technique for highly undersampled dynamic Magnetic Resonance Imaging (MRI) data. The proposed method recovers high-frequency information using a shared 3D convolution-based dictionary built progressively during the reconstruction process in an unsupervised manner, while low-frequency information is recovered using a total variation-based energy minimization method that leverages temporal coherence in dynamic MRI. Additionally, the proposed 3D dictionary is built across three different scales to more efficiently adapt to various feature sizes, and elastic net regularization is employed to promote a better approximation to the sparse input data. We also propose an automatic parameter selection technique based on a genetic algorithm to find optimal parameters for our numerical solver which is a variant of the alternating direction method of multipliers (ADMM). We demonstrate the performance of our method by comparing it with state-of-the-art methods on 15 single-coil cardiac, 7 single-coil DCE, and a multi-coil brain MRI datasets at different sampling rates (12.5%, 25% and 50%). The results show that our method significantly outperforms the other state-of-the-art methods in reconstruction quality with a comparable running time and is resilient to noise.

Optimized Hierarchical Cascaded Processing

Recently, there has been an increasing demand for advanced classification capabilities embedded on wearable battery constrained devices, such as smartphones or watches. Achieving such functionality with a tight power and energy budget has proven a real challenge, specifically for large-scale neural network-based applications. Previously, cascaded systems have been proposed to minimize energy consumption for such applications, either through using a single wake-up stage, or by using a linear- or tree based cascade of consecutive classifiers that allow early termination. In this paper, we expand upon these concepts by generalizing cascades to hierarchical cascaded processing, where a hierarchy of increasingly complex classifiers, each designed and trained for a specific subtask is used. This hierarchical approach significantly outperforms the wake-up based approach by up to 2 orders of magnitude in energy consumption at iso-accuracy, specifically in systems with sparse input data such as speech recognition and visual object detection. This paper presents a general design framework for such systems and illustrates how to optimize them toward minimum energy consumption. The text further proposes a roofline model for cascaded systems, derives system level trade-offs and proves the approaches validity through a visual classification case-study.

Window opening model using deep learning methods

Occupant behavior (OB) and in particular window openings need to be considered in building performance simulation (BPS), in order to realistically model the indoor climate and energy consumption for heating ventilation and air conditioning (HVAC). However, the proposed OB window opening models are often biased towards the over-represented class where windows remained closed. In addition, they require tuning for each occupant which can not be efficiently scaled to the increased number of occupants. This paper presents a window opening model for commercial buildings using deep learning methods. The model is trained using data from occupants from an office building in Germany. In total, the model is evaluated using almost 20 mio. data points from 3 independent buildings, located in Aachen, Frankfurt and Philadelphia. Eventually, the results of 3100 core hours of model development are summarized, which makes this study the largest of its kind in window states modeling. Additionally, the practical potential of the proposed model was tested by incorporating it in the Modelica-based thermal building simulation. The resulting evaluation accuracy and F1 scores on the office buildings ranged between 86 and 89% and 0.53–0.65 respectively. The performance dropped around 15% points in case of sparse input data, while the F1 score remained high.