Sparse Data Research Articles

Dissipative imitation learning for discrete dynamic output feedback control with sparse data sets

Dissipative imitation learning for discrete dynamic output feedback control with sparse data sets

Book Recommendation System

In the digital age, the vast availability of books poses a significant challenge for readers to discover titles that match their interests and preferences. This report explores the use of machine learning for a Book Recommendation System (BRS) to help readers discover books suited to their interests in the digital age. It will utilize advanced algorithms and techniques, including collaborative filtering, content-based filtering, and hybrid models, to offer personalized recommendations based on users' reading history and behavior. The project aims to tackle challenges like data sparsity and the cold-start problem while ensuring scalability. Evaluation will be based on standard metrics like precision, recall, accuracy, and novelty, with a focus on ethical considerations such as fairness and user privacy. Ultimately, the goal is to develop a robust and user-centric system that enhances the book discovery process across digital platforms. A Graphical User Interface is also developed to discover information and display the recommended books. Keywords—Machine Learning, Book Recommendation System, K-Nearest Neighbor Algorithm

Filtering After Shading With Stochastic Texture Filtering

2D texture maps and 3D voxel arrays are widely used to add rich detail to the surfaces and volumes of rendered scenes, and filtered texture lookups are integral to producing high-quality imagery. We show that applying the texture filter after evaluating shading generally gives more accurate imagery than filtering textures before BSDF evaluation, as is current practice. These benefits are not merely theoretical, but are apparent in common cases. We demonstrate that practical and efficient filtering after shading is possible through the use of stochastic sampling of texture filters. Stochastic texture filtering offers additional benefits, including efficient implementation of high-quality texture filters and efficient filtering of textures stored in compressed and sparse data structures, including neural representations. We demonstrate applications in both real-time and offline rendering and show that the additional error from stochastic filtering is minimal. We find that this error is handled well by either spatiotemporal denoising or moderate pixel sampling rates.

Spectral Spatial Neighborhood Attention Transformer for Hyperspectral Image Classification

Hyperspectral image (HSI) can provide rich spectral information, which can be helpful for accurate classification in many applications. However, the hyperspectral image classification task has challenges, including limited labeled data, data redundancy, data sparsity, and imbalanced class samples. Over time, various methods have been proposed to solve the above-mentioned problems. To mitigate the issue mentioned above in this paper, we present the neighborhood attention transformer with a channel-wise shift technique for hyperspectral image classification. The neighborhood attention transformer leverages the power of the attention mechanism to capture the spatial relationship between neighboring pixels and extract discriminative features. The channel-wise shift techniques empower the model to adjust the spectral characteristics of each channel adaptively, enhancing its ability to handle the spectral variations present in hyperspectral data. To validate the effectiveness of the proposed model, we conduct comprehensive experiments on the publically available dataset. The results demonstrate that our model consistently outperforms other state-of-the-art methods. The overall accuracy of the proposed model reached 99.41 % , 99.93 % , 98.35 % 99.59 % four datasets.

Extracting Latent Variables From Forecast Ensembles and Advancements in Similarity Metric Utilizing Optimal Transport

AbstractThis study presents a novel methodology for extracting latent variables from high‐dimensional sparse data, particularly emphasizing spatial distributions such as precipitation distribution. This approach utilizes multidimensional scaling with a distance matrix derived from a new similarity metric, the Unbalanced Optimal Transport Score (UOTS). UOTS effectively captures discrepancies in spatial distributions while preserving physical units. This is similar to mean absolute error, however it considers location errors, providing a more robust measure crucial for understanding differences between observations, forecasts, and ensembles. Probability distribution estimation of these latent variables enhances the analytical utility, quantifying ensemble characteristics. The adaptability of the method to spatiotemporal data and its ability to handle errors suggest its potential as a promising tool for diverse research applications.

An evaluation of computational methods for aggregate data meta-analyses of diagnostic test accuracy studies

BackgroundA Generalized Linear Mixed Model (GLMM) is recommended to meta-analyze diagnostic test accuracy studies (DTAs) based on aggregate or individual participant data. Since a GLMM does not have a closed-form likelihood function or parameter solutions, computational methods are conventionally used to approximate the likelihoods and obtain parameter estimates. The most commonly used computational methods are the Iteratively Reweighted Least Squares (IRLS), the Laplace approximation (LA), and the Adaptive Gauss-Hermite quadrature (AGHQ). Despite being widely used, it has not been clear how these computational methods compare and perform in the context of an aggregate data meta-analysis (ADMA) of DTAs.MethodsWe compared and evaluated the performance of three commonly used computational methods for GLMM - the IRLS, the LA, and the AGHQ, via a comprehensive simulation study and real-life data examples, in the context of an ADMA of DTAs. By varying several parameters in our simulations, we assessed the performance of the three methods in terms of bias, root mean squared error, confidence interval (CI) width, coverage of the 95% CI, convergence rate, and computational speed.ResultsFor most of the scenarios, especially when the meta-analytic data were not sparse (i.e., there were no or negligible studies with perfect diagnosis), the three computational methods were comparable for the estimation of sensitivity and specificity. However, the LA had the largest bias and root mean squared error for pooled sensitivity and specificity when the meta-analytic data were sparse. Moreover, the AGHQ took a longer computational time to converge relative to the other two methods, although it had the best convergence rate.ConclusionsWe recommend practitioners and researchers carefully choose an appropriate computational algorithm when fitting a GLMM to an ADMA of DTAs. We do not recommend the LA for sparse meta-analytic data sets. However, either the AGHQ or the IRLS can be used regardless of the characteristics of the meta-analytic data.

Physics‐constrained symbolic model discovery for polyconvex incompressible hyperelastic materials

AbstractWe present a machine learning framework capable of consistently inferring mathematical expressions of hyperelastic energy functionals for incompressible materials from sparse experimental data and physical laws. To achieve this goal, we propose a polyconvex neural additive model (PNAM) that enables us to express the hyperelastic model in a learnable feature space while enforcing polyconvexity. An upshot of this feature space obtained via the PNAM is that (1) it is spanned by a set of univariate basis functions that can be re‐parametrized with a more complex mathematical form, and (2) the resultant elasticity model is guaranteed to fulfill the polyconvexity, which ensures that the acoustic tensor remains elliptic for any deformation. To further improve the interpretability, we use genetic programming to convert each univariate basis into a compact mathematical expression. The resultant multi‐variable mathematical models obtained from this proposed framework are not only more interpretable but are also proven to fulfill physical laws. By controlling the compactness of the learned symbolic form, the machine learning‐generated mathematical model also requires fewer arithmetic operations than its deep neural network counterparts during deployment. This latter attribute is crucial for scaling large‐scale simulations where the constitutive responses of every integration point must be updated within each incremental time step. We compare our proposed model discovery framework against other state‐of‐the‐art alternatives to assess the robustness and efficiency of the training algorithms and examine the trade‐off between interpretability, accuracy, and precision of the learned symbolic hyperelastic models obtained from different approaches. Our numerical results suggest that our approach extrapolates well outside the training data regime due to the precise incorporation of physics‐based knowledge.

Higher-order embedded learning for heterogeneous information networks and adaptive POI recommendation

Traditional POI recommendation results tend to be homogeneous due to data sparsity, and cannot adapt to the dynamic switching of user locations. Regarding the issues, we propose an adaptive Point of Interest (POI) recommendation algorithm that integrates the user social attributes and POI natural attributes. On the one hand, to fully utilize the complex node interactions and semantic information of heterogeneous networks, an effective higher-order embedding learning framework based on multi-head attention mechanism and multi-task shared features, called UI-HEHo, is presented. It divides hierarchical embedding learning into inter-order and intra-order aggregation to learn richer and more comprehensive representations. Based on UI-HEHo framework, a POI-user social attribute model is constructed to obtain the relational distance between interest points. On the other hand, a POI-natural attributes model based on user preference activity range prediction is constructed to further draw the influence between interest points. Furthermore, a tensor-based dynamic update scheme is designed to reduce the high cost of model updates. Finally, the Radviz graph is utilized to obtain the final recommendation results. Experimental results demonstrate our proposed approach significantly improves the POI recommendation performance, especially achieves 20.7% and 23.8% improvement on the Foursquare and Yelp datasets in terms of the newly defined diversity and novelty metrics.

Snow Disaster Risk Assessment Based on Long-Term Remote Sensing Data: A Case Study of the Qinghai–Tibet Plateau Region in Xizang

The risk analysis and assessment of snow disasters are essential foundational tasks in natural disaster management and profoundly impact the scientific and precise formulation of disaster prevention, preparedness, and mitigation strategies. Employing the theory and methodology of snow disaster assessment, this research focuses on historical and potential snow disasters in the Qinghai–Tibetan Plateau (QTP) Region. Utilizing a long-time-series snow depth remote sensing dataset, we extracted six assessment indicators for historical snow disaster risk factors and potential snow disaster risk factors. We determined the weights of these six assessment indicators using the entropy weight method. Subsequently, we established a snow disaster assessment model to evaluate the grade distribution of snow disasters in the study area. This method can effectively solve the problem of the sparse data distribution of meteorological stations and reflect degrees of snow disaster risk on a large spatial scale. The findings reveal that areas with a relatively high snow disaster risk are primarily concentrated in the western part of the Ali Region, the central part of Chamdo, and near the border in Southern Xizang. Additionally, regions with a high frequency of snow disasters are predominantly located at the junction of Nagchu, Chamdo, and Nyingchi in the eastern part of Xizang. These results contribute valuable insights into the risk assessment of snow disasters and facilitate the development of effective strategies for disaster management in the region.

2D signal estimation for sparse distributed target photon counting data

In this study, we explore the utilization of penalized likelihood estimation for the analysis of sparse photon counting data obtained from distributed target lidar systems. Specifically, we adapt the Poisson Total Variation processing technique to cater to this application. By assuming a Poisson noise model for the photon count observations, our approach yields denoised estimates of backscatter photon flux and related parameters. This facilitates the processing of raw photon counting signals with exceptionally high temporal and range resolutions (demonstrated here to 50 Hz and 75 cm resolutions), including data acquired through time-correlated single photon counting, without significant sacrifice of resolution. Through examination involving both simulated and real-world 2D atmospheric data, our method consistently demonstrates superior accuracy in signal recovery compared to the conventional histogram-based approach commonly employed in distributed target lidar applications.

AI Efficacy in Sparse Data Environments: Exploring Approximate Knowledge Interpolation for Practical Applications

AI Efficacy in Sparse Data Environments: Exploring Approximate Knowledge Interpolation for Practical Applications

Fluviomorphic trajectories for dryland ephemeral stream channels following extreme flash floods

AbstractEphemeral alluvial streams pose globally significant flood hazards to human habitation in drylands, but sparse data for these regions limit understanding of the character and impacts of extreme flooding. In this study, we document decadal changes in dryland ephemeral channel patterns at two sites in the lower Colorado River Basin (southwestern United States) that were ravaged by extraordinary flash floods in the 1970s: Bronco Creek, Arizona (1971), and Eldorado Canyon, Nevada (1974). We refer to these two floods as ‘fluviomorphic erasure events’, because they produced blank slates for the channels that were gradually moulded by more frequent but much smaller flood events. We studied georectified aerial photos that span ~60 years at each site to show that both study sites recovered to their pre‐flood condition after ~25 years. We employ channel network metrics: stream‐link area (SLA), geometric braiding index and junction‐node density. Each metric decreased during the short‐duration extreme flood erasure events. Subsequently, a fluviomorphic trajectory at a decadal tempo returned the channels to pre‐flood values. The SLA decreased at rates of 3.6%–4.1% per year in the decade following the floods. The extreme flood events decreased the pre‐flood geometric braiding index at the two sites by 56%–68%, and it took 15–24 years for this index to recover to pre‐flood values. In contrast, it took 30–35 years for the channels to recover to a uniform pre‐flood channel form, as indicated by the spatial distribution of bars and junction nodes. Our results document baseline examples of ephemeral stream channel evolution trajectories, as future climatic change will likely accelerate increases in the magnitudes and frequencies of extreme floods and geomorphic erasure events.

AAGCN: a graph convolutional neural network with adaptive feature and topology learning

In recent years, there has been a growing prevalence of deep learning in various domains, owing to advancements in information technology and computing power. Graph neural network methods within deep learning have shown remarkable capabilities in processing graph-structured data, such as social networks and traffic networks. As a result, they have garnered significant attention from researchers.However, real-world data often face challenges like data sparsity and missing labels, which can hinder the performance and generalization ability of graph convolutional neural networks. To overcome these challenges, our research aims to effectively extract the hidden features and topological information of graph convolutional neural networks. We propose an innovative model called Adaptive Feature and Topology Graph Convolutional Neural Network (AAGCN). By incorporating an adaptive layer, our model preprocesses the data and integrates the hidden features and topological information with the original data’s features and structure. These fused features are then utilized in the convolutional layer for training, significantly enhancing the expressive power of graph convolutional neural networks.To evaluate the effectiveness of the adaptive layer in the AAGCN model, we conducted node classification experiments on real datasets. The results validate its ability to address data sparsity and improve the classification performance of graph convolutional neural networks.In conclusion, our research primarily focuses on addressing data sparsity and missing labels in graph convolutional neural networks. The proposed AAGCN model, which incorporates an adaptive layer, effectively extracts hidden features and topological information, thereby enhancing the expressive power and classification performance of these networks.

Abstract PO1-20-01: A Prospective Single-Arm Pilot Interventional Trial to Assess Individual Breast Cancer Risk Prior to Gender-Affirming Chest Masculinization Surgery

Abstract Background: The number of individuals who identify as transgender and/or gender-diverse (TGD) is steadily increasing and the number of individuals seeking chest masculinization surgery, commonly called “top-surgery,” is also increasing. In persons assigned female at birth, top surgery removes most breast tissue to allow the chest to appear masculine. However, top surgery is not equivalent to oncologic risk-reducing mastectomies (RRM), which aim to remove all breast tissue to reduce future breast cancer (BC) risk. While top surgery has significant psychological and health benefits, its impact on future BC risk is unclear, especially for those individuals who are at high-risk for future BC development (defined as ≥17% by international consensus guidelines) and/or have a pathogenic germline mutation for BC. The aims of this study are to 1) determine the percentage of TGD persons assigned female at birth who are considering undergoing top surgery and are at elevated BC risk using validated risk models, 2) determine the percentage of TGD persons at elevated BC risk who choose to undergo RRM as part of top surgery, and 3) assess and compare self-perceived BC risk compared to calculated risk. Methods: A single-institution prospective single-arm pilot study is currently underway from March 2023 through January 2024. Individuals assigned female at birth, who are age ≥18 years, and considering undergoing top surgery, but have not yet undergone top surgery, are eligible to enroll (goal accrual N=35). Participants are being actively recruited form our institution’s Comprehensive LGBTQIA+ Inclusion Health Clinic and Plastic Surgery Clinic. Consented participants undergo a personalized BC risk assessment the validated IBIS and Gail (for those ≥35 years old) BC risk models. Those with an average lifetime BC risk (< 17%) are recommended to continue top surgery as planned/no intervention, participants with a moderate lifetime BC risk (17-30%) are counseled they may consider RRM as part of their top surgery operation given that sparse data exists in this space, while those with a high lifetime BC risk ( >30% or pathogenic germline variant) are counseled on the potential benefits of RRM. Participants with a family history suggestive of a hereditary cancer syndrome meet with a certified Genetics counselor for consideration of genetic testing, per NCCN guidelines. Results: We anticipate reporting 1) general demographic description of the cohort, 2) the percentage who have an elevated lifetime BC risk and/or a pathogenic genetic mutation, 3) the percentage of those with an elevated risk who choose to undergo RRM as part of top surgery, and 4) report the differences between self-perceived vs calculated lifetime BC risk. Conclusion: Our findings will inform TGD persons and surgeons on the utility of a personalized BC risk assessment prior to top surgery to assist in surgical decision-making, report the accuracy of self-perceived BC risk in TGD persons, and potentially change the standard pre-operative evaluation for TGD persons considering top surgery to include a formal BC risk assessment to ensure high-risk individuals have the option to undergo RRM to appropriately reduce their future BC risk. Citation Format: Chandler Cortina, Anna Purdy, Samantha Stachowiak, Kristen Klement, Sarah Sasor, Kate Krucoff, Andrew Petroll, Kevin Robertson, Annie Lakatos, Erin Doren. A Prospective Single-Arm Pilot Interventional Trial to Assess Individual Breast Cancer Risk Prior to Gender-Affirming Chest Masculinization Surgery [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO1-20-01.

The Experimental Method of Sparse Measurement Based on Target’s Radar Cross Section Data

In experiments of testing targets’ radar cross section (RCS), RCS is a measure of the target’s ability to reflect radar signals in the direction of radar reception. In order to take non blurry spatial-domain imaging of the target, sampling or measuring the target’s RCS signal must meet the Nyquist sampling theorem. However, RCS testing in microwave darkroom would produce a significant amount of sampling data, which requires a very long testing period. Based on the sparsity of spatial-domain RCS data after sparsifying transformation, a sparse projection measurement experimental method is designed for frequency-domain RCS data by sparsely sampling frequency-domain signals, fully utilizing the correlation between row and column information of two-dimensional frequency-domain data and the global sparsity of two-dimensional spatial-domain data. Finally, based on sparsely sampled data in the frequency-domain, the original RCS data is reconstructed by reconstruction algorithms, achieving the goal of effectively reducing RCS sampling data in RCS testing experiments.

Learning continuous shape priors from sparse data with neural implicit functions.

Statistical shape models are an essential tool for various tasks in medical image analysis, including shape generation, reconstruction and classification. Shape models are learned from a population of example shapes, which are typically obtained through segmentation of volumetric medical images. In clinical practice, highly anisotropic volumetric scans with large slice distances are prevalent, e.g., to reduce radiation exposure in CT or image acquisition time in MR imaging. For existing shape modeling approaches, the resolution of the emerging model is limited to the resolution of the training shapes. Therefore, any missing information between slices prohibits existing methods from learning a high-resolution shape prior. We propose a novel shape modeling approach that can be trained on sparse, binary segmentation masks with large slice distances. This is achieved through employing continuous shape representations based on neural implicit functions. After training, our model can reconstruct shapes from various sparse inputs at high target resolutions beyond the resolution of individual training examples. We successfully reconstruct high-resolution shapes from as few as three orthogonal slices. Furthermore, our shape model allows us to embed various sparse segmentation masks into a common, low-dimensional latent space - independent of the acquisition direction, resolution, spacing, and field of view. We show that the emerging latent representation discriminates between healthy and pathological shapes, even when provided with sparse segmentation masks. Lastly, we qualitatively demonstrate that the emerging latent space is smooth and captures characteristic modes of shape variation. We evaluate our shape model on two anatomical structures: the lumbar vertebra and the distal femur, both from publicly available datasets.

Development of a physiologically based pharmacokinetic model of fostemsavir and its pivotal application to support dosing in pregnancy.

It is critical to understand the impact of significant physiological changes during pregnancy on the extent of maternal and fetal drug exposure. Fostemsavir (FTR) is a prodrug of temsavir (TMR) and is approved in combination with other antiretrovirals for multi-drug-resistant human immunodeficiency virus (HIV) infections. This physiologically based pharmacokinetic model (PBPK) study was used to estimate TMR PK in pregnant populations during each trimester of pregnancy to inform FTR dosing. A PBPK model was developed and validated for TMR using PK data collected following intravenous TMR and oral FTR dosing (immediate-release and extended-release tablets) in healthy volunteers. Predicted TMR concentration-time profiles accurately predicted the reported clinical data and variability in healthy (dense data) and pregnant (sparse data) populations. Predicted versus observed TMR geometric mean (CV%) clearance following intravenous administration was 18.01 (29) versus 17 (21) (L/h). Predicted versus observed TMR AUC0-inf (ng.h/mL) in healthy volunteers following FTR administration of the extended-release tablet were 9542 (66) versus 7339 (33). The validated TMR PBPK model was then applied to predict TMR PK in a population of pregnant individuals during each trimester. Simulations showed TMR AUC in pregnant individuals receiving FTR 600 mg twice daily was decreased by 25% and 38% in the second and third trimesters, respectively. However, TMR exposure remained within the range observed in nonpregnant adults with no need for dose adjustment. The current PBPK model can also be applied for the prediction of local tissue concentrations and drug-drug interactions in pregnancy.

State inference for low-observable distribution system based on graph convolutional network

Distribution network state inference refers to the process of calculating the state variables of each node by using measurement data and network models in the operation of the distribution system. However, the uneven measurement layout and insufficient measurement accuracy in the distribution network have brought great challenges to the state inference of the distribution network. This paper proposes a low-observable distribution network state inference method based on a graph convolution network (GCN), which uses sparse measurement data to infer missing measurement information. Firstly, the observability of the distribution network is analyzed by the numerical probability analysis method. Secondly, the GCN is employed to extract feature information from measurement data and integrate these features. The state inference model of the distribution network based on the GCN is established. Subsequently, power flow constraints of the distribution network are incorporated into the GCN training process to enhance the precision of the generated data. Ultimately, the efficacy of the proposed method is validated using the IEEE 33-node distribution system.

A multiple criteria Bayesian hierarchical model for analyzing heterogeneous consumer preferences

We introduce a novel Bayesian hierarchical model for consumer preference analysis, addressing two significant challenges in this domain. First, it accommodates preference heterogeneity at both individual and segment levels. This enables actionable insights for targeting and pricing decisions while quantifying uncertainty. Second, it incorporates probabilistic value-based ranking to handle inconsistent and sparse preference data. This way, it mitigates the impact of cognitive biases and alleviates uncertainty in estimates. The proposed method performs robust inference of consumers’ preferences through hierarchical priors, allowing for flexible parameter learning and borrowing statistical strength from well-informed individuals. We demonstrate its practical usefulness by analyzing the real preferences of almost one hundred consumers considering mobile phone contracts. We also report the results of an extensive experimental study. The proposed method outperforms its counterpart, executing an independent estimation and the state-of-the-art approaches regarding predictive accuracy and preference similarity within identified customer groups. The performance improvements are more pronounced with larger sample sizes, smaller sets of items, and in contexts with reduced heterogeneity and increased consistency among consumers.

Focus-Attention Approach in Optimizing DETR for Object Detection from High-Resolution Images

Transformer-based detectors have recently achieved remarkable success in object detection, revolutionizing the field with their efficiency and accuracy. However, applying these models to high-resolution images presents significant challenges due to the increased computational demands and complexity of processing dense, high-resolution data. In this paper, we introduce a novel model specifically designed for object detection from high-resolution imagery. This model incorporates a multi-layer object-focus network along with a transformer encoder-decoder structure. Specifically, the model employs a dual-head strategy in the object-focus network to balance the detailed analysis of small objects with computational efficiency. This is achieved by leveraging data sparsity to reduce unnecessary computations in massive background regions. Additionally, to improve detection performance for small objects, we propose a method to effectively apply the transformer encoder-decoder structure combined with the object-focus network on the multi-layer feature maps of the feature pyramid. Our extensive evaluations of the VisDrone, MS-COCO, and UAVid datasets demonstrate that our model outperforms other DETR-based detectors in both detection accuracy and computational speed, highlighting its superior performance. These results indicate a significant advancement in the field of high-resolution object detection utilizing transformer architecture.