Distributed Data Sets Research Articles

Neural fields for rapid aircraft aerodynamics simulations

This paper presents a methodology to learn surrogate models of steady state fluid dynamics simulations on meshed domains, based on Implicit Neural Representations (INRs). The proposed models can be applied directly to unstructured domains for different flow conditions, handle non-parametric 3D geometric variations, and generalize to unseen shapes at test time. The coordinate-based formulation naturally leads to robustness with respect to discretization, allowing an excellent trade-off between computational cost (memory footprint and training time) and accuracy. The method is demonstrated on two industrially relevant applications: a RANS dataset of the two-dimensional compressible flow over a transonic airfoil and a dataset of the surface pressure distribution over 3D wings, including shape, inflow condition, and control surface deflection variations. On the considered test cases, our approach achieves a more than three times lower test error and significantly improves generalization error on unseen geometries compared to state-of-the-art Graph Neural Network architectures. Remarkably, the method can drastically accelerate high fidelity numerical solver, achieving a five order of magnitude speedup on the RANS transonic airfoil dataset.

Measuring the community dream: A social capital lens of capacity building initiatives

ABSTRACT In 2010, San Antonio leaders sought to leverage networks and collective action to create a future in which each resident was responsible to the whole community in helping to reach a shared vision for 2020 to become a world-class city. In doing so, city administrators applied social capital theories to utilize the active networks in a social capital-rich community, which creates a wider capacity for dynamic actions and more opportunity for community practitioners to meet development goals. This case study compiles secondary sources from publicly distributed data sets, utilizing data mining techniques and social network analysis to describe community capacity building outcomes. This case study examines community development outcomes through a social-network analysis of organizational leadership, combined with an examination of diversity, civic engagement, and participation within the decade-long initiative, to provide insights for other leaders seeking to utilize a social capital theory approach to capacity building.

Deep learning for pore-scale two-phase flow: Modelling drainage in realistic porous media

In order to predict phase distributions within complex pore structures during two-phase capillary-dominated drainage, we select subsamples from computerized tomography (CT) images of rocks and simulated porous media, and develop a pore morphology-based simulator (PMS) to create a diverse dataset of phase distributions. With pixel size, interfacial tension, contact angle, and pressure as input parameters, convolutional neural network (CNN), recurrent neural network (RNN) and vision transformer (ViT) are transformed, trained and evaluated to select the optimal model for predicting phase distribution. It is found that commonly used CNN and RNN have deficiencies in capturing phase connectivity. Subsequently, we develop a higher-dimensional vision transformer (HD-ViT) that drains pores solely based on their size, regardless of their spatial location, with phase connectivity enforced as a post-processing step. This approach enables inference for images of varying sizes and resolutions with inlet-outlet setup at any coordinate directions. We demonstrate that HD-ViT maintains its effectiveness, accuracy and speed advantage on larger sandstone and carbonate images, compared with the microfluidic-based displacement experiment. In the end, we train and validate a 3D version of the model.

Spatial and Temporal Patterns of Maize Phenology in China From 2001 to 2020

AbstractClimate change has significantly altered crop phenology, which has further impacted crop growth and yield. Accurate monitoring of crop phenology is essential for managing agricultural production in response. However, regional monitoring requires high spatial resolution distribution data, as medium resolution data suffers from mixed pixel issues. This study based on a long‐term high spatiotemporal resolution fusion data set of Normalized Difference Vegetation Index and an annually updated maize distribution data set, used the relative threshold method to identify the maize phenology in 22 provinces of China from 2001 to 2020. We further analyzed the trend of maize phenology and assessed its responses to climate change. The results reveal large inter‐annual fluctuations and spatial variability in maize phenology from 2001 to 2020. The length of the growth season (LOS) of spring maize has prolonged by 4.28 days in the northern maize zone and has shortened by 4.90 days in the southern maize zone. Additionally, the LOS of summer maize in the Huang‐Huai‐Hai region has shortened by 2.24 days. We also found a positive correlation between the length of the vegetative growth stage and the mean temperature and a negative correlation between the length of the reproductive growth stage and accumulated precipitation. This study utilized large‐scale, high‐resolution maize phenology data to analyze the trend of maize phenology and its response to climate change. These findings are expected to provide valuable support for assessing maize growth status and developing agricultural adaptive practices.

A checklist of moths in Bilaspur district, Himachal Pradesh, in the western Himalayan foothills, India

In the first ever enumeration of moth diversity of Bilaspur district, Himachal Pradesh, India, located mostly in the Shivalik range, 82 species/morphospecies were reported, at least 22 of which are new records for Himachal Pradesh, and five are new records for Western Himalayas. In addition to a list of moths for the district supplemented with photographs, identification keys for similar species, larval host plants for species, and a near exhaustive dataset of distribution of the species/genera within and outside India are also provided.

Hardware optimization for effective switching power reduction during data compression in GOLOMB rice coding.

Loss-less data compression becomes the need of the hour for effective data compression and computation in VLSI test vector generation and testing in addition to hardware AI/ML computations. Golomb code is one of the effective technique for lossless data compression and it becomes valid only when the divisor can be expressed as power of two. This work aims to increase compression ratio by further encoding the unary part of the Golomb Rice (GR) code so as to decrease the amount of bits used, it mainly focuses on optimizing the hardware for encoding side. The algorithm was developed and coded in Verilog and simulated using Modelsim. This code was then synthesised in Cadence Encounter RTL Synthesiser. The modifications carried out show around 6% to 19% reduction in bits used for a linearly distributed data set. Worst-case delays have been reduced by 3% to 8%. Area reduction varies from 22% to 36% for different methods. Simulation for Power consumption shows nearly 7% reduction in switching power. This ideally suggest the usage of Golomb Rice coding technique for test vector compression and data computation for multiple data types, which should ideally have a geometrical distribution.

RTG-GNN: A novel rock topology-guided approach for permeability prediction using graph neural networks

Digital core permeability is a crucial factor in rock engineering, reservoir simulation, and underground applications, serving as the foundation for evaluating fluid flow underground. However, predicting digital core permeability using artificial intelligence presents challenges such as high model calculation complexity, strong resource dependence, and low efficiency. To overcome these challenges, this paper presents a novel hybrid enhanced rock topology-guided graph neural network (RTG-GNN) to accurately characterize the pore structure of digital cores and predict permeability. This model takes advantage of the physical attributes of pores and throats, utilizing them as features for nodes and edges. It improves information exchange through a gating mechanism and incorporates an attention mechanism for weighted aggregation of neighboring data, culminating in accurate permeability predictions. Furthermore, a scheme for constructing a comprehensive permeability distribution dataset (CPDD) is proposed for digital cores. This scheme effectively mitigates challenges like inadequate training data, subjective sampling bias, and data drift. Multiple experimental results demonstrate that RTG-GNN surpasses most methods in various performance indicators. Additionally, the RTG-GNN model boasts a lightweight design, occupying only 17% to 44% of the size of mainstream models and requiring just 15% to 38% of GPU resources for training. Remarkably, while maintaining high accuracy and low error rates, the computational load is reduced by a factor ranging from 19.06 to 636.41, the training speed is enhanced by 12.12 to 86.47 times, and the inference speed is increased by 6.76 to 16.14 times. The experiment further confirms that both single core and mixed rock datasets, acquired through the CPDD scheme, exhibit excellent generalization and reliability.

Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theory

This work introduces a novel and efficient Bayesian federated learning algorithm, namely, the Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC), for parameter estimation and uncertainty quantification. We establish rigorous convergence guarantees of FA-HMC on non-iid distributed datasets, under the strong convexity and Hessian smoothness assumptions. Our analysis investigates the effects of parameter space dimension, noise on gradients and momentum, and the frequency of communication (between the central node and local nodes) on the convergence and communication costs of FA-HMC. Beyond that, we establish the tightness of our analysis by showing that the convergence rate cannot be improved even for continuous FA-HMC process. Moreover, extensive empirical studies demonstrate that FA-HMC outperforms the existing Federated Averaging-Langevin Monte Carlo (FA-LD) algorithm. Supplementary materials for this article are available online.

SparkDWM: a scalable design of a Data Washing Machine using Apache Spark.

Data volume has been one of the fast-growing assets of most real-world applications. This increases the rate of human errors such as duplication of records, misspellings, and erroneous transpositions, among other data quality issues. Entity Resolution is an ETL process that aims to resolve data inconsistencies by ensuring entities are referring to the same real-world objects. One of the main challenges of most traditional Entity Resolution systems is ensuring their scalability to meet the rising data needs. This research aims to refactor a working proof-of-concept entity resolution system called the Data Washing Machine to be highly scalable using Apache Spark distributed data processing framework. We solve the single-threaded design problem of the legacy Data Washing Machine by using PySpark's Resilient Distributed Dataset and improve the Data Washing Machine design to use intrinsic metadata information from references. We prove that our systems achieve the same results as the legacy Data Washing Machine using 18 synthetically generated datasets. We also test the scalability of our system using a variety of real-world benchmark ER datasets from a few thousand to millions. Our experimental results show that our proposed system performs better than a MapReduce-based Data Washing Machine. We also compared our system with Famer and concluded that our system can find more clusters when given optimal starting parameters for clustering.

Characterizing Sentinel Lymph Node Status in Breast Cancer Patients Using a Deep-Learning Model Compared With Radiologists' Analysis of Grayscale Ultrasound and Lymphosonography.

The objective of the study was to use a deep learning model to differentiate between benign and malignant sentinel lymph nodes (SLNs) in patients with breast cancer compared to radiologists' assessments.Seventy-nine women with breast cancer were enrolled and underwent lymphosonography and contrast-enhanced ultrasound (CEUS) examination after subcutaneous injection of ultrasound contrast agent around their tumor to identify SLNs. Google AutoML was used to develop image classification model. Grayscale and CEUS images acquired during the ultrasound examination were uploaded with a data distribution of 80% for training/20% for testing. The performance metric used was area under precision/recall curve (AuPRC). In addition, 3 radiologists assessed SLNs as normal or abnormal based on a clinical established classification. Two-hundred seventeen SLNs were divided in 2 for model development; model 1 included all SLNs and model 2 had an equal number of benign and malignant SLNs. Validation results model 1 AuPRC 0.84 (grayscale)/0.91 (CEUS) and model 2 AuPRC 0.91 (grayscale)/0.87 (CEUS). The comparison between artificial intelligence (AI) and readers' showed statistical significant differences between all models and ultrasound modes; model 1 grayscale AI versus readers, P = 0.047, and model 1 CEUS AI versus readers, P < 0.001. Model 2 r grayscale AI versus readers, P = 0.032, and model 2 CEUS AI versus readers, P = 0.041.The interreader agreement overall result showed κ values of 0.20 for grayscale and 0.17 for CEUS.In conclusion, AutoML showed improved diagnostic performance in balance volume datasets. Radiologist performance was not influenced by the dataset's distribution.

A dataset of spatial distribution of tourism resources in the Yellow River Basin in 2022

A dataset of spatial distribution of tourism resources in the Yellow River Basin in 2022

A dataset of annual surface water distribution in the Yellow River Basin (1986&amp;ndash;2023)

A dataset of annual surface water distribution in the Yellow River Basin (1986&amp;ndash;2023)

A Dataset of Distribution and Characterization of Underground Wastewater Treatment Plants in China

Underground wastewater treatment plants (U-WWTPs) have emerged as a novel paradigm for urban wastewater pollutants management, offering benefits such as alleviating the Not-in-my-backyard (NIMBY) effect and utilizing land resources efficiently. China stands at the forefront, witnessing swift advancements in U-WWTP technology and deployment. However, the absence of a thorough understanding of their geographical distribution and operational characteristics could lead to misaligned planning and construction, resulting in inefficient resource allocation and treatment capacities for urban wastewater treatment. This dataset provides an up-to-date overview of the spatial distribution, process selection, and discharge standards for all U-WWTPs in China (with a total number of 201) constructed since 1995. To enhance comparative analysis, the dataset has been supplemented with information on conventional aboveground wastewater treatment plants (A-WWTPs), comprising a total of 2464 records, which enriches a more comprehensive evaluation of different wastewater treatment approaches. Utilizing this dataset can provide essential data support for the strategic management of urban wastewater systems and serve as a valuable reference for the paradigmatic renovation of existing wastewater treatment plants.

A High-Resolution Distribution Dataset of Paddy Rice in India Based on Satellite Data

India, as the world’s second-largest rice producer, accounting for 21.7% of global rice production, plays a crucial role in ensuring global food supply stability. However, creating high-resolution rice maps for India, such as those at 10 to 30 m, poses significant challenges due to frequent cloudy weather conditions and the complexities of its agricultural systems. This study used a sample-independent mapping method for rice in India using the synthetic aperture radar (SAR)-based Rice Index (SPRI). We produced 10 m spatial resolution rice distribution maps for three years (i.e., 2018, 2020, and 2022) for 23 states in India, covering 98% of Indian rice production. The method effectively utilized the unique characteristics of rice in the vertical–horizontal (VH) backscatter coefficient time series of Sentinel-1, from ttransplantation to the maturity stage, combined with cloud-free Sentinel-2 imagery. By calculating the SPRI values for each agricultural field object using adaptive parameters, the planting locations of rice were accurately identified. On average, the user, producer, and overall accuracy over all investigated states and union territories was 84.72%, 82.31%, and 84.40%, respectively. Additionally, the regional-scale validation based on the statistical area at the district level showed that the coefficient of determination (R2) ranged from 0.53 to 0.95 for each state, indicating that the spatial distribution of the statistical planted area at the district level was reproduced well.

Physics-guided federated learning as an enabler for digital twins

Digital twins bring the potential to increase the efficiency of assets, systems, and processes by building virtual replicas through real-time data and modeling. However, data are often confidential and distributed, high-fidelity models based on physical principles tend to be slow and require detailed knowledge about the asset design that may not be available, simplified models lack accuracy, and data-driven models are not interpretable and limited to their training space. We demonstrate the interplay of two enabling technologies, namely federated learning and hybrid analysis and modeling, to combine the strengths of physics-based and data-driven models on distributed data sets that are kept confidential by different stakeholders. Throughout the work, an ensemble of physics-guided neural networks is designed, optimized, and validated to infer the parameters of digital twin components. The physics-guided neural network is injected with output from a simplified physics-based model in intermediate layers for increased performance, and the injection layer is optimized through weight regularization. The model is then integrated into a simulation of the digital twin asset and validated there. It is shown that the model outperforms the simplified physics-based and the data-driven models both on component and on asset level. In the final step, the physics-guided neural network is trained using federated methods, which allows training on confidential data owned by different stakeholders without requiring the stakeholders to share their data. We show that federated hybrid models can – with the right training strategy – by design achieve results that are always identical and potentially superior to models trained on a central data set. The problem chosen for this demonstration is the prediction of airfoil lift and drag coefficients based on the airfoil shape and the angle of attack, which is a task crucial for simulating wind turbines. Using the physics-guided neural network, the estimation of the airfoil coefficients experiences a speedup of several orders of magnitude compared to the high-fidelity simulation, and integration into blade element momentum theory shows that the turbine thrust, torque, and mechanical power can be accurately calculated. The federated approach makes it possible to extend the training of the model to any number of distributed confidential data sets from both simulations and experiments to further increase the model’s accuracy.

A Localized Quantitative Precipitation Estimation for S-band Polarimetric Radar in Taiwan

Abstract A polarimetric radar quantitative precipitation estimation to estimate rain rate (R) from specific attenuation (A) has been applied in Taiwan's operational Quantitative Precipitation Estimation and Segregation Using Multiple Sensors (QPESUMS) system since 2016. A 3-yrs' (2016-2018) drop size distribution dataset from an operational Parsivel network was used to derive a localized coefficient as well as the α(K) function in the R(A) scheme for S-band radar, where α is a key parameter in the estimation of A and K is the linear fitted slope of differential reflectivity (ZDR) versus reflectivity (Z). The local DSD data was also used to derive localized R(Z) and R(KDP) relationships, and the relationships were evaluated using radar observations in heavy rain cases. A synthetic QPE combining the localized R(A), R(Z), and R(KDP) relationships is compared to its operational counterpart and showed about 8 % reduction in normalized mean error for the Mei-Yu cases. Typhoon cases exhibited similar improvements by the localized QPE relationships, but showed higher uncertainties than in the Mei-Yu cases. The higher uncertainties in the typhoon QPE verification was likely due to the stronger winds in typhoons than in the Mei-Yu events that caused greater mismatches between the radar observations at an altitude and the gauges at the ground. Overall, the results demonstrated advantages of localized radar rainfall relationships derived from the disdrometer data to improve the accuracy of the operational rainfall estimation products.

Analysis of lake changes and their influence factors in the three river regions from 2000 to 2020 in the Sanjiangyuan Region, China

An important factor for investigating climate change in the Sanjiangyuan is the evolution of the spatio-temporal pattern of lakes in this region. The present study used the Google Earth Engine (GEE) platform to extract lakes from 2000 to 2020. The present approach created a lake distribution dataset yearly and analyzed spatial and temporal patterns over 20 years. The analysis of lakes focused on the reaction of the Sanjiangyuan Lakes area to changes in climate, glaciers, and permafrost. The findings indicated that the Sanjiangyuan region contains 143 lakes, the majority of which are predominantly small, measuring 1–10 km2. The small lakes account for 60.14 % of the total and are primarily located in the source regions of the Yangtze River and Yellow River. The findings demonstrated that the Sanjiangyuan lakes experienced a significant expansion over the past two decades, particularly from 2011 to 2020. These lakes are divided into expanded, atrophic, and stable categories. Expanded lakes showed significant inter-annual trends in expansion, while atrophic lakes showed smaller fluctuations. The area of stable lakes experienced a consistent decline after 2010, despite a consistent expansion tendency from 2001 to 2010. Moreover, the results indicated that alterations in the size of glaciers and ice reserves in the Sanjiangyuan region have had the greatest influence on the fluctuation in lake area. Among the factors that affect the climate, temperature had the most significant effect on the change in lake area, followed by precipitation.

Annual time-series 1 km maps of crop area and types in the conterminous US (CropAT-US): cropping diversity changes during 1850–2021

Abstract. Agricultural activities have been recognized as an important driver of land cover and land use change (LCLUC) and have significantly impacted the ecosystem feedback to climate by altering land surface properties. A reliable historical cropland distribution dataset is crucial for understanding and quantifying the legacy effects of agriculture-related LCLUC. While several LCLUC datasets have the potential to depict cropland patterns in the conterminous US, there remains a dearth of a relatively high-resolution datasets with crop type details over a long period. To address this gap, we reconstructed historical cropland density and crop type maps from 1850 to 2021 at a resolution of 1 km × 1 km by integrating county-level crop-specific inventory datasets, census data, and gridded LCLUC products. Different from other databases, we tracked the planting area dynamics of all crops in the US, excluding idle and fallow farm land and cropland pasture. The results showed that the crop acreages for nine major crops derived from our map products are highly consistent with the county-level inventory data, with a residual less than 0.2×103 ha (0.2 kha) in most counties (&gt;75 %) during the entire study period. Temporally, the US total crop acreage has increased by 118×106 ha (118 Mha) from 1850 to 2021, primarily driven by corn (30 Mha) and soybean (35 Mha). Spatially, the hot spots of cropland distribution shifted from the Eastern US to the Midwest and the Great Plains, and the dominant crop types (corn and soybean) expanded northwestward. Moreover, we found that the US cropping diversity experienced a significant increase from the 1850s to the 1960s, followed by a dramatic decline in the recent 6 decades under intensified agriculture. Generally, this newly developed dataset could facilitate spatial data development, with respect to delineating crop-specific management practices, and enable the quantification of cropland change impacts on the environment. Annual cropland density and crop type maps are available at https://doi.org/10.6084/m9.figshare.22822838.v2 (Ye et al., 2023).

A random feature mapping method based on the AdaBoost algorithm and results fusion for enhancing classification performance

The feature mapping method can improve data separability, enhance data representation ability, and reduce data processing complexity. However, on the one hand, the existing feature mapping methods have difficulty processing datasets of different dimensions and distributions adaptively, limiting the scope of application; on the other hand, a single feature mapping method has the problem of instability and poor generalization ability, weakening the classification ability of subsequent classifiers. This paper proposes a random feature mapping method based on the AdaBoost algorithm and results fusion to enhance classification performance. The method adopts horizontal expansion, fine-tuning weights through sparse autoencoders, and uses input-mapped features as feature nodes to generate multiple feature subsets for increasing stability. After training weak classifiers on each multiple feature subset, the weights of classifiers are adjusted adaptively by the Adaboost ensemble algorithm. Finally, the method fuses weak classifiers twice to enhance classification performance, which abandons the traditional voting method and uses the weighted probability selection method. Experiments on twenty classic datasets show that the proposed method can effectively mine essential features and enhance classification accuracy compared with original datasets. For instance, the Balance dataset has an average classification accuracy of more than 20% higher than the original dataset on the KNN classifier. The proposed method outperforms alternative feature mapping methods in terms of performance and efficiency on different classifiers in most cases.

How budgets change: punctuations, trends, and super-trends

AbstractPunctuated equilibrium theory (PET) describes policy change as occurring mostly through incremental movements with infrequent periods of dramatic change. An impressive body of empirical literature relating to budgeting supports this view, but virtually all empirical tests have focused on examining distributions of annual changes, thus nullifying chronology. In this article, we focus on the time element. Using the same databases as previously used in canonical PET studies, we explore multi-year trends, not only annual observations. For our analyses, we identify directional series of changes (while allowing for one-year changes in direction if these are immediately offset in the following year) on a U.S. budget distribution dataset covering the period of 1947 through 2014, with 60 categories of spending consistently defined over time and adjusted for inflation. We then assess the robustness of the PET findings when incorporating a longer time units of trending series of annual changes into the analysis. We find that almost 65% of changes occur in series of 4 years or more. Nonetheless, the signature PET literature pattern of high kurtosis is equally present in these series as well as in shorter series. Moreover, within growing and trending series, we find that 21% of these series generate 80% of positive budget change. Within these series, we identify a small group of “super-trends” that account for a large share of the overall change. We conclude that expanding methodologies for the study of budgetary change to incorporate longer-term dynamics helps to better understand policy change, but such findings remain consistent with the PET perspective.