Dispersed Data Research Articles

A multi-layer perceptron neural network for varied conditional attributes in tabular dispersed data.

The paper introduces a novel approach for constructing a global model utilizing multilayer perceptron (MLP) neural networks and dispersed data sources. These dispersed data are independently gathered in various local tables, each potentially containing different objects and attributes, albeit with some shared elements (objects and attributes). Our approach involves the development of local models based on these local tables imputed with some artificial objects. Subsequently, local models are aggregated using weighted techniques. To complete, the global model is retrained using some global objects. In this study, the proposed method is compared with two existing approaches from the literature-homogeneous and heterogeneous multi-model classifiers. The analysis reveals that the proposed approach consistently outperforms these existing methods across multiple evaluation criteria including classification accuracy, balanced accuracy, F1-score, and precision. The results demonstrate that the proposed method significantly outperforms traditional ensemble classifiers and homogeneous ensembles of MLPs. Specifically, the proposed approach achieves an average classification accuracy improvement of 15% and a balanced accuracy enhancement of 12% over the baseline methods mentioned above. Moreover, in practical applications such as healthcare and smart agriculture, the model showcases superior properties by providing a single model that is easier to use and interpret. These improvements underscore the model's robustness and adaptability, making it a valuable tool for diverse real-world applications.

Objects Diversity and its Impact on Classification Quality in Dispersed Data Environments

This paper studies a classification model dedicated to dispersed data, employing the k-nearest neighbors method as a base classifier and a Radial Basis Function (RBF) network as a fusion method. Focusing on classifying objects described by different attributes, the study systematically reduces common objects in local tables to assess the robustness of the model. Surprisingly, the proposed approach shows resilience in reducing common objects without significantly affecting key metrics such as F-measure, balanced accuracy and overall accuracy. Moreover, the studied model performs better on balanced data. This research contributes valuable insights into dispersed data classification, demonstrating the model’s effectiveness in handling diverse objects and attributes. The findings have implications for fields reliant on dispersed data storage, such as healthcare, banking, and surveillance, showcasing the model’s potential for real-world applications.

A cGAN-based fatigue life prediction of 316 austenitic stainless steel in high-temperature and high-pressure water environments

The thermo-mechanical-chemical coupling effect presents significant challenges in accurately predicting the fatigue life of 316 austenitic stainless steel in high-temperature and high-pressure water environments (referred to hereafter as environmental fatigue). The complexity of environmental fatigue experiments results in limited and dispersed data, further making the life prediction difficult. Traditional fatigue life prediction models are often constrained by specific loading conditions and do not adequately account for the complex environmental influences. To address these issues, this paper proposes a novel environmental fatigue life prediction model of 316 stainless steel utilizing conditional Generative Adversarial Networks. The proposed model incorporates critical environmental factors, loading conditions and stacking fault energy, allowing direct prediction of environmental fatigue life. A comparative analysis on the predicted and experimental results reveals that the cGAN-based model significantly improves the prediction accuracy, reducing the fatigue life prediction error from a factor of 5 to within 3. To quantify the uncertainty in fatigue life prediction, the Monte Carlo Dropout method is employed to enable a probabilistic assessment of fatigue life. Furthermore, four environmental and loading conditions are established to evaluate the model’s extrapolation capability. The results demonstrate that the probabilistic fatigue assessment effectively captures data distribution and achieves high prediction accuracy.

Theoretical framework for the difference of two negative binomial distributions and its application in comparative analysis of sequencing data.

High-throughput sequencing (HTS) technologies have been instrumental in investigating biological questions at the bulk and single-cell levels. Comparative analysis of two HTS data sets often relies on testing the statistical significance for the difference of two negative binomial distributions (DOTNB). Although negative binomial distributions are well studied, the theoretical results for DOTNB remain largely unexplored. Here, we derive basic analytical results for DOTNB and examine its asymptotic properties. As a state-of-the-art application of DOTNB, we introduce DEGage, a computational method for detecting differentially expressed genes (DEGs) in scRNA-seq data. DEGage calculates the mean of the sample-wise differences of gene expression levels as the test statistic and determines significant differential expression by computing the P-value with DOTNB. Extensive validation using simulated and real scRNA-seq data sets demonstrates that DEGage outperforms five popular DEG analysis tools: DEGseq2, DEsingle, edgeR, Monocle3, and scDD. DEGage is robust against high dropout levels and exhibits superior sensitivity when applied to balanced and imbalanced data sets, even with small sample sizes. We utilize DEGage to analyze prostate cancer scRNA-seq data sets and identify marker genes for 17 cell types. Furthermore, we apply DEGage to scRNA-seq data sets of mouse neurons with and without fear memory and reveal eight potential memory-related genes overlooked in previous analyses. The theoretical results and supporting software for DOTNB can be widely applied to comparative analyses of dispersed count data in HTS and broad research questions.

A new two-parameter over-dispersed discrete distribution with mathematical properties, estimation, regression model and applications

This paper focuses on the derivation of a new two-parameter discrete probability distribution. The new model is derived by mixing Poisson and Loai distributions and is named “Poisson Loai Distribution”. The paper explores various mathematical properties of the new model, introducing a count-regression model based on this distribution. The parameters of the model are estimated using the maximum likelihood estimation method. A comprehensive simulation study is utilized to assess the behavior of derived estimators. The importance of the proposed distribution is confirmed through the analysis of three real datasets. It is found that the suggested distribution has the greatest match when compared to all rival distributions, and it may be a viable alternative for assessing dispersed count data.

A Comparative Analysis of Federated and Centralized Learning for SpO2 Prediction in Five Critical Care Databases.

This study explores the potential of federated learning (FL) to develop a predictive model of hypoxemia in intensive care unit (ICU) patients. Centralized learning (CL) and local learning (LL) approaches have been limited by the localized nature of data, which restricts CL approaches to the available data due to data privacy regulations. A CL approach that combines data from different institutions, could offer superior performance compared to a single-institution approach. However, the use of this method raises ethical and regulatory concerns. In this context, FL presents a promising middle ground, enabling collaborative model training on geographically dispersed ICU data without compromising patient confidentiality. This study is the first to use all five public ICU databases combined. The findings demonstrate that FL achieved comparable or even slightly improved performance compared to local or centralized learning approaches.

Rethinking Urban Water Management Through Drivers-Pressures-States-Impacts-Responses Framework Application in Chennai, India

Cities suffering water scarcity are projected to increase in the following decades. However, the application of standardized indicator frameworks for assessing urban water resource management problems is on an early stage. India is expected to have the highest urban population facing water scarcity in the world by 2050. In this study, the authors assess how the Drivers-Pressures-States-Impacts-Responses framework, a causal framework adopted by the European Environment Agency, can contribute to evaluate water management challenges in cities and apply it to Chennai, India´s fourth-largest urban agglomeration. The framework proved to be a helpful tool for the evaluation of water management challenges in cities by disentangling relationships between environmental indicators and structuring dispersed data that allows a better understanding for policymakers. The main drivers identified in Chennai were population growth and economic development which generated impacts such as loss of aquatic ecosystems, low water table, low water quality, and reduction of biodiversity and human health. As a response, better urban planning, projects for new water infrastructure, and water bodies restoration have been implemented. Nevertheless, Chennai keeps facing difficulties to achieve proper water management. The severe hit of the COVID-19 pandemic on the Indian economy and its future management will be key for achievements related to water management.

Rapid identification method for on-road high-emission vehicle based on deep semi-supervised anomaly detection

This study aims to identify high-emission vehicles in urban traffic management. Existing high-emission vehicle identification models typically overlook the dispersed anomaly data distribution, and machine learning anomaly detection methods face difficulties in learning decision boundaries, leading to reduced detection performance. To address these issues, this study proposes a semi-supervised learning method based on data fusion and deep anomaly detection. This method integrates the chassis and engine dynamometer test data with on-road remote sensing system data to obtain more comprehensive vehicle emission information. Experimental results demonstrate that the proposed method introduces a penalty mechanism for anomaly samples, encouraging the model to increase the dissimilarity in similarity between normal and abnormal data at the latent data distribution level. In vehicle emission datasets from different regions, this method achieves over 95% AUC, demonstrating strong applicability and accuracy.

Domain Adaptation for Bearing Fault Diagnosis Based on SimAM and Adaptive Weighting Strategy.

Domain adaptation techniques are crucial for addressing the discrepancies between training and testing data distributions caused by varying operational conditions in practical bearing fault diagnosis. However, transfer fault diagnosis faces significant challenges under complex conditions with dispersed data and distinct distribution differences. Hence, this paper proposes CWT-SimAM-DAMS, a domain adaptation method for bearing fault diagnosis based on SimAM and an adaptive weighting strategy. The proposed scheme first uses Continuous Wavelet Transform (CWT) and Unsharp Masking (USM) for data preprocessing, and then feature extraction is performed using the Residual Network (ResNet) integrated with the SimAM module. This is combined with the proposed adaptive weighting strategy based on Joint Maximum Mean Discrepancy (JMMD) and Conditional Adversarial Domain Adaption Network (CDAN) domain adaptation algorithms, which minimizes the distribution differences between the source and target domains more effectively, thus enhancing domain adaptability. The proposed method is validated on two datasets, and experimental results show that it improves the accuracy of bearing fault diagnosis.

An Overview of the Efficiency and Long-Term Viability of Powered Hydrogen Production

This work studies the efficiency and long-term viability of powered hydrogen production. For this purpose, a detailed exploration of hydrogen production techniques has been undertaken, involving data collection, information authentication, data organization, and analysis. The efficiency trends, environmental impact, and hydrogen production costs in a landscape marked by limited data availability were investigated. The main contribution of this work is to reduce the existing data gap in the field of hydrogen production by compiling and summarizing dispersed data. The findings are expected to facilitate the decision-making process by considering regional variations, energy source availability, and the potential for technological advancements that may further enhance the economic viability of electrolysis. The results show that hydrogen production methods can be identified that do not cause significant harm to the environment. Photolysis stands out as the least serious offender, producing 0 kg of CO2 per kg of H2, while thermolysis emerges as the major contributor to emissions, with 20 kg of CO2 per kg of H2 produced.

Citizen labor

AbstractIn the last two decades, India's system of agricultural land management has been transitioning from paper to digital records. In effecting this shift, engineers and bureaucrats in the city of Bangalore, India's “Silicon Valley,” have tacitly and invisibly shifted the responsibility for maintaining data from the state to individuals. Moreover, the new digital databases of land records have fragmented offices and dispersed data across new sites and actors. Under these transformed conditions, people can access services only through what I call citizen labor. That is, when digitization is applied to land and property—which are quintessential sites for the making and unmaking of citizenship—people are interpellated into laboring on their own data. This shows that the digitization of government extracts a form of labor, one whose benefits accrue to groups and organizations beyond the laboring individual. As a result, people have an increasingly degraded experience of substantive citizenship.

Negative Binomial Three Parameter Lindley Distribution and Its Properties

Many researchers have proposed mixed distributions as one of the most important methods for obtaining new probability distributions. Several studies have shown that mixed Negative Binomial distributions fits count data better than Poisson and Negative Binomial distribution itself. In this paper, we introduce a mixed distribution by mixing the distributions of negative binomial and three Parameter Lindley distribution. This new distribution has a thick tail and may be considered as an alternative for fitting count data with over dispersion. The parameters of the new distribution are estimated using MLE method and properties studied. Special cases of the new distribution and also identified. A simulation study carried out shows that the ML estimators give the parameter estimates close to the parameter when the sample is large, that is, the bias and variance of the parameter estimates decrease with increase in sample size showing the consistent nature of the new compound distribution. The study also compares the performance of the new distribution over distributions of Poisson, Negative Binomial, Negative Binomial oneParameter Lindley Distribution, Negative Binomial two Parameter distribution, three parameter Lindley distribution using a real count over dispersed dataset and the results shows that Negative Binomial three parameter Lindley distribution gave the smallest Kolmogorov Smirnov test statistic, AIC and BIC as compared to other distributions, hence the new distribution provided a better fit compared to other distributions under study for fitting over dispersed count data.

UReslham: Radar reflectivity inversion for smart agriculture with spatial federated learning over geostationary satellite observations

AbstractThe frequent occurrence of severe convective weather has certain adverse effects on the smart agriculture industry. To enhance the prediction of severe convective weather, the inversion model effectively fills radar reflectivity data gaps by leveraging geostationary satellite data, offering more comprehensive and accurate support for meteorological information in smart agriculture systems. Nevertheless, collaborative cross‐regional inversion driven by dispersed radar data faces challenges in efficiency, privacy, and model accuracy. To this end, we employ an U‐shaped residual network with an embedded light hybrid attention mechanism and utilize a federated averaging algorithm for efficient distributed training across multiple devices which could preserve the privacy of data from different locations, thereby improving inversion performance. In addition, to address the unbalanced nature of radar data, a weighted loss function is designed to enhance the model's sensitivity to high radar reflectivity. Experimental results demonstrate that the proposed model exhibits a certain level of improvement in evaluating radar reflectivity inversion performance across different thresholds compared to other models, thus substantiating the superiority of the proposed approach.

Regression to the mean effect for bi-variate dispersed count data

Regression to the mean (RTM) occurs when extreme observations upon remeasurement are found closer to the population mean. Interventions are usually applied to subjects based on a cut-off point to check it effectiveness. The average difference of the pre-post observations called the total effect is the sum of RTM and intervention effects and needs to be accounted for RTM to avoid incorrect conclusions and unbiasedly estimate the intervention effect. The existing methods assume equidispersion and strict positive correlation for bivariate count data. This study considers a bivariate Poisson distribution which takes both the dispersion and negative correlation into consideration. Expressions for the total effect are derived and partitioned into RTM and intervention effects. The maximum likelihood estimators of the RTM and intervention effects are found numerically, and their asymptotic properties are verified via simulations. Data on the English premier league are used to estimate the RTM and intervention effects.

Estimating spawning green sturgeon (Acipenser medirostris Ayres, 1854) abundance in the Sacramento River using side-scan sonar and N-mixture models

Current estimates of the threatened southern distinct population segment of the North American green sturgeon ( Acipenser medirostris) combine a plot-sampling density estimator with Dual frequency IDentification SONar (DIDSON) and adaptive resolution imaging sonar (ARIS) sonar data. From 2020 to 2022, we annually collected images of all known green sturgeon aggregations and compared the established method to an N-mixture model using side-scan sonar images. We compared 18 different N-mixture model combinations and chose an overdispersed Poisson model that produced estimated abundances of 742, 1286, and 1208 for 2020–2022, respectively. These numbers are ∼2 times greater than the previous method and, if sustained, would fulfill a key criterion for green sturgeon recovery. N-mixture models are known to be sensitive to violations of assumptions, such as the highly dispersed data from our study that caused serious issues, and we recommend practitioners make judicious use of overdispersion and goodness-of-fit tests and be able to identify parameter confounding between detectability and abundance estimates. For our green sturgeon, we recommend simpler population estimates and to focus future energy on reducing variability in the data collection process.

Federated Causality Learning with Explainable Adaptive Optimization

Discovering the causality from observational data is a crucial task in various scientific domains. With increasing awareness of privacy, data are not allowed to be exposed, and it is very hard to learn causal graphs from dispersed data, since these data may have different distributions. In this paper, we propose a federated causal discovery strategy (FedCausal) to learn the unified global causal graph from decentralized heterogeneous data. We design a global optimization formula to naturally aggregate the causal graphs from client data and constrain the acyclicity of the global graph without exposing local data. Unlike other federated causal learning algorithms, FedCausal unifies the local and global optimizations into a complete directed acyclic graph (DAG) learning process with a flexible optimization objective. We prove that this optimization objective has a high interpretability and can adaptively handle homogeneous and heterogeneous data. Experimental results on synthetic and real datasets show that FedCausal can effectively deal with non-independently and identically distributed (non-iid) data and has a superior performance.

Dual-drive coordinated operation strategy for internet data centers by exploiting spatiotemporal flexibility

With the rapid expansion of computational infrastructure, the electricity consumption of geographically dispersed Internet Data Centers (IDCs) has imposed a substantial influence on the power grid. Geographically distributed IDCs, which present a versatile electricity demand with significant operational flexibility, hold potential for mitigating cost issues and balancing regional supply and load demand. To exploit this flexibility, this paper has developed a dual-drive coordinated operational strategy, which integrates locational electricity prices and unit carbon emission costs based on the dynamic carbon emission intensity (DCEI) of various regions as interaction signals to improve coordination effectiveness. The upper-level determines the consumption behaviour of IDCs, the lower-level provides feedback on dual-drive signals factoring in IDCs' involvement in grid operation, whilst an iterative algorithm is suggested to solve the bi-level model. Case studies based on real-world datasets demonstrate that the proposed operation strategy can achieve arbitrage through multi-regional carbon-electricity markets, significantly reduce operating costs, and support the rapid development of IDCs. Additionally, this strategy facilitates optimal matching of power load in IDCs with renewable energy sources, thereby mitigating imbalances in regional renewable energy supply and demand.

The visualization of dialect data with VDM

Abstract Visualization of the results is the final and essential step in dialect data analysis. For geographically dispersed data such as dialect data, colorful maps have long been the standard of visualization. The focus is on polygon maps, which can be used to represent contiguous areas particularly well. The Dialectometric visualization of script data is quite similar to the visualization of data from dialect atlases. For this reason, the VDM application (Visual DialectoMetry) can be used for the analysis of text documents from the DocLing1 corpus with few modifications. After a short tour of the visualizations in VDM and an explanation of the data structure, the focus of this paper is on these modifications and the new features which are currently being added to VDM. This new version of VDM will be better suited for scripta data and will streamline the generation of maps for VDM.

Dispersed Data Classification Model with Conflict Analysis and Parameterized Allied Relations

In the paper, a classification model for dispersed data is proposed. By dispersed data we mean a set of local tables that are collected independently by different units. This model uses conflict analysis, which considers the similarity of the values of conditional attributes occurring in decision classes of local tables. Tables having compatible values within classes are arranged into coalitions. The paper proposes the use of a parameter that steers the intensity of conflict between tables that are in allied relation. It was experimentally confirmed using 25 dispersed data that the proposed approach gives better results than the baseline approach in which coalitions are not used. The statistical significance of the differences was also proven. In addition, an advanced analysis of the impact of the parameter’s value steering the allied relations on the form of coalitions and the quality of classification was carried out.

Ensembles of random trees with coalitions - a classification model for dispersed data

The paper delves into the challenge of classification using dispersed data gathered from independent sources. The examined approach involves local models as ensembles of random trees constructed based on local data and randomly selected attributes. In the proposed model, a conflict analysis is used to identify the coalitions of local models. Finally, four different strategies for generating final decisions are explored: utilizing coalitions with and without weights and allowing one or two of the strongest coalitions to make decisions. The paper demonstrates that, regardless of the chosen method for making the final decision, the proposed model with conflict analysis and coalitions obtained better results, particularly in terms of the F1 measure and accuracy, compared to approaches from the literature, such as random forest or a single tree generated based on each local table separately with majority voting.