Large-scale Data Research Articles

Diabetes Prediction Based on Machine Learning

Diabetes mellitus is a growing global health issue with an increasing incidence rate. Traditional methods for predicting diabetes, which rely heavily on clinical data and physical examinations, present challenges such as high costs, time-consuming processes, and difficulties in providing timely and personalized risk assessments. However, with the rise of machine learning (ML), new opportunities have emerged in diabetes prediction, utilizing large-scale data and advanced pattern recognition techniques. This study examines the application of ML in diabetes risk assessment by leveraging electronic health records (EHR) and big data, leading to significant improvements in accuracy and efficiency. The results demonstrate that ML methods can more effectively identify high-risk individuals, facilitating early intervention and contributing to the advancement of diabetes prediction.

Exploring the self-perceived causes of eating disorders among Chinese social media users with self-reported eating disorders

BackgroundEven though robust evidence suggests the high prevalence of eating disorders (EDs) in China, EDs in China are characterized by low diagnosis rates, delayed treatment-seeking, and ineffective treatments. Given that listening to patients’ perspectives and lived experiences is crucial to improving our understanding of EDs in the Chinese context, an investigation of the perceived causes of EDs in Chinese individuals with EDs represents a key step in improving the prevention and treatment of EDs in China.AimsTo explore the perceived causes of EDs based on data from a sample of Chinese social media users with self-reported EDs, with a particular focus on the Zhihu platform.MethodsWe extracted and analyzed data through content analysis. Eight specific causes that could be classified into two groups were coded, including individual factors (e.g., “body image and eating”) and sociocultural factors (e.g., “media and cultural ideals”).ResultsA total of 2079 entries regarding self-reported EDs were retained for content analysis (14.7% were anorexia nervosa, 37.6% were bulimia nervosa, and 47.7% were binge-eating disorder). More than 90% of users with self-reported EDs claimed causes belonging to individual factors, while 35–51% of users claimed sociocultural factors. “Body image and eating” (68–87%) and “psychological and emotional problems” (65–67%) were the most commonly claimed specific causes, while “traumatic life events” (13–14%), “genetics and biology” (7–13%), and “sports and health” (9–12%) were the least claimed. Chi-square independent tests showed that users with different self-reported EDs disproportionately claimed certain causes.ConclusionsUsing large-scale social media data, findings provide a deeper understanding of the perceived causes of EDs in the Chinese context from individuals with self-reported EDs and highlight the variations in perceived causes across different self-reported ED types.

Word embeddings on ideology and issues from Swedish parliamentarians’ motions: a comparative approach

ABSTRACT Quantitative analysis of large-scale political text data in the form of word embeddings has great potential for systematising differences between political parties. We examine the differences between embeddings obtained from speakers from the two competitors for the PM position in Sweden (Social Democrats and Moderates) over a 30-year period. The goal is to compare how off-the-shelf general pre-trained models perform relative to pre-training on a smaller dataset from the same domain. In the analysis, we focus on two types of concepts: issues and ideological terms. We find that generally, the off-the-shelf pre-trained models lead to more reliable results and greater emphasis on ideological differences between the studied parties.

Prediction of preterm birth using machine learning: a comprehensive analysis based on large-scale preschool children survey data in Shenzhen of China

BackgroundPreterm birth (PTB) is a significant cause of neonatal mortality and long-term health issues. Accurate prediction and timely prevention of PTB are essential for reducing associated child mortality and morbidity. Traditional predictive methods face challenges due to heterogeneous risk factors and their interaction effects. This study aims to develop and evaluate six machine learning (ML) models to predict PTB using large-scale children survey data from Shenzhen, China, and to identify key predictors through Shapley Additive Explanations (SHAP) analysis.MethodsData from 84,050 mother–child pairs, collected in 2021 and 2022, were processed and divided into training, validation, and test sets. Six ML models were tested: L1-Regularised Logistic Regression, Light Gradient Boosting Machine (LightGBM), Naive Bayes, Random Forests, Support Vector Machine, and Extreme Gradient Boosting (XGBoost). Model performance was evaluated based on discrimination, calibration and clinical utility. SHAP analysis was used to interpret the importance and impact of individual features on PTB prediction.ResultsThe XGBoost model demonstrated the best overall performance, with the area under the receiver operating characteristic curve (AUC) scores of 0.752 and 0.757 in the validation and test sets, respectively, along with favorable calibration and clinical utility. Key predictors identified were multiple pregnancies, threatened abortion, and maternal age of conception. SHAP analysis highlighted the positive impacts of multiple pregnancies and threatened abortion, as well as the negative impact of micronutrient supplementation on PTB.ConclusionOur study found that ML models, particularly XGBoost, show promise in accurately predicting PTB and identifying key risk factors. These findings provide the potential of ML for enhancing clinical interventions, personalizing prenatal care, and informing public health initiatives.

PRODUCTION OF WEBGL BASED DIGITAL DOCUMENTATION OF PONTE MOLINO GATE FROM POINT CLOUD

In recent years, it has become increasingly important to rapidly present and share data from photogrammetric techniques in digital documentation projects. In this study, Potree, a WebGL-based software, is employed to digitally document the Ponte Molino Gate in Padova, Italy. This historically significant gate, dating back to between 40 and 30 BC, was scanned using a terrestrial laser scanner (TLS) to generate a comprehensive point cloud. The study demonstrates the advantages of using WebGL technology for the presentation of large-scale data by processing and visualizing the point cloud with Potree. It also highlights Potree’s effectiveness in enabling virtual exchange and interactive exploration of cultural heritage assets, while discussing its potential applications in GIS, terrain modeling, and urban planning.

Human interventions have enhanced the net ecosystem productivity of farmland in China

Human interventions, such as farmland management, have long been considered crucial for soil carbon sequestration, but little is known about the exact impact of these interventions on the net carbon flux, represented by net ecosystem productivity (NEP). Here, using multiple long-term, large-scale data and statistical data, we reveal that 75.54% of farmland NEP in China experiences an increase, with northern regions showing the greatest potential for future farmland carbon sequestration. This growth is primarily attributed to the role of farmland management, especially the enhancement of no-tillage, land consolidation and multiple cropping level (17.02%, major grain-producing areas in 2020). Notably, the current unreasonable practices of mechanized straw returning and irrigation have a negative impact on farmland NEP. Our results show that it is imperative to acknowledge the crucial role of human interventions on farmland NEP to strike a balance between food security and farmland carbon sequestration.

LearningViz: a dashboard for visualizing, analyzing and closing learning performance gaps—a case study approach

The availability of large-scale learning data presents unprecedented opportunities for investigating student learning processes. However, it is challenging for instructors to fully make sense of this data and effectively support their teaching practices. This study introduces LearningViz, an interactive learning analytics dashboard to help instructors identify, analyze, and close performance gaps among students in their classes. In this dashboard, we incorporated three modules to enhance human and computer interactions for better supporting the teaching practices: the Student Overall Performance Analysis Module, which provides a comprehensive understanding of students’ learning in the course; the Student Group Performance Analysis Module, which examines performance gaps across different groups and identifies factors contributing to these gaps; and the Final Exam Item Analysis Module, which evaluates the quality of exam questions and identifies strategies for closing performance gaps. The overall design of the platform follows a user-centered approach, integrating data analysis with various visualization strategies in a unified platform. A case study is then conducted to highlight the effectiveness of LearningViz in supporting instructors analyzing students’ learning patterns and associated factors impacting learning performance. We further conduct a usability test with several domain experts, to evaluate the usefulness and effectiveness of this platform in supporting the teaching practices. Our findings underscore the platform's ability to support instructors in detecting performance gaps among students, investigating influential factors, evaluating assessment quality and implementing targeted instructional strategies for closing performance gaps.

The Impact of Proton Pump Inhibitor Use on Fusion Rates Following Single-level Anterior Cervical Discectomy and Fusion.

Cohort study. This study explores how proton pump inhibitors (PPIs) affect fusion rates following anterior cervical discectomy and fusion (ACDF), using a large-scale data analysis. ACDF is essential for treating cervical disc herniation leading to myelopathy and radiculopathy, involving disc removal and vertebral fusion, crucial for long-term stability and symptom relief. Notably, PPIs, which are commonly prescribed for acid-related disorders, have been linked to altered bone health and healing processes. Utilizing the TriNetX network database spanning from 2008 to 2023, we identified patients undergoing single-level ACDF, classified into cohorts based on PPI usage following ACDF. A 1:1 propensity score matching was performed to balance demographics and comorbidities between the two groups. The study focused on the incidence of non-fusion, indicated by the International Classification of Disease-10 code M96.0, within 6 months to 2 years postoperatively. The initial cohort comprised 1269 PPI users and 23,932 non-users, adjusted to 1266 per group after matching. Postmatching analysis indicated minimal differences in demographics and comorbidities between the cohorts. Our results showed that postoperative PPI users have a significantly higher risk of non-fusion following single-level ACDF surgery at 1 year (odds ratio: 1.35, 95% CI: 1.05-1.73) and 2 years (odds ratio: 1.42, 95% CI: 1.11-1.81) follow-up. Our study showed a significant link between postoperative PPI use and increased long-term pseudarthrosis risk after ACDF surgery. These findings suggest careful consideration of PPI use in these patients.

A Blockchain Semantic-based Approach for Secure and Traceable Agri-Food Supply Chain

Ensuring food security is crucial for maintaining food quality and enhancing consumer services by guaranteeing both safety and satisfaction. However, traditional methods to ensure food security are often susceptible to various forms of fraud and require significant processing overhead, making them inefficient for the evolving demands of modern food supply chains. To address these shortcomings, blockchain technology has emerged as a robust and efficient solution to enhance food security. This paper presents a novel lightweight blockchain-based signature mechanism designed for the rapid detection of food fraud. It also includes a domain-specific ontology to serve as a structured knowledge model, allowing systematic analysis and detection of different types of fraud within the food supply chain. This approach uses smart contracts built on lightweight blockchain technology to initiate and manage transactions related to food fraud. Then, semantic rules are applied to detect and identify fraudulent activities. Once fraud is detected, associated transactions are encrypted and tracked, ensuring visibility and traceability among consortium members. Experimental results based on large-scale transaction data demonstrated ~7.5× speed improvement over iterative search algorithms while maintaining high transaction traceability and significantly reducing storage costs.

GPThingSim: A IoT Simulator Based GPT Models Over an Edge-Cloud Environments

The Internet of Things encapsulates a vision of a world in which billions of objects, called connected devices, are endowed with intelligence, communication capabilities, and integrated sensing and actuation capabilities. They often outsource their storage and computing needs to more powerful resources in the cloud. In Edge computing, it is the device itself that collects and processes data and must make real-time decisions. A model that combines the edge and the cloud allows for instant processing of large amounts of data, making devices increasingly intelligent. One fundamental aspect of Artificial Intelligence, particularly Machine Learning (ML), is that it requires a substantial amount of data to learn. The GPT model can be adapted to operate within a cloud environment, which enables instant processing of large quantities of data. In this article, our approach relies on a hybrid of Cloud computing and Edge computing, merging the best features of both approaches by using Edge for real-time processing and the cloud for storage and large-scale data analysis. We will explore the possibilities of integrating the GPT model into IoT devices. This extension allows devices to offer processing capabilities at the local level, which can be extremely beneficial for many applications. We propose a platform for simulating connected objects augmented by the LSTM neural network model to predict the energy consumption of IoT devices in a smart city scenario.

Spatial Statistics of Three-Dimensional Growth Dynamics of Spindle Microtubules.

The latest high-resolution 3D live-cell imaging technology, lattice light-sheet microscopy (LLSM), has successfully tracked the dynamics of microtubule growth throughout the entire mitotic spindle with unparalleled precision. By using green fluorescent protein-labeled end-binding protein 1 (EB1-GFP) as a marker for growing microtubule ends, LLSM has generated an extensive collection of multidimensional datasets mapping the positions and trajectories of these growing microtubule ends. Processing this data requires statistical analysis in three-dimensional space. This chapter describes the spatial statistical methods developed for this purpose, illustrated with practical examples. Finally, we discuss future prospects for analyzing complex, large-scale image data.

ICESat-2 data denoising and forest canopy height estimation using Machine Learning

Supervised classification methods can distinguish between noise and signal in ice, cloud, and land elevation satellite-2 (ICESat-2) data across various feature perspectives and autonomously optimize parameters. Nevertheless, model generalization remains a significant limitation for practical applications. This study focuses on developing a universal denoising model for ICESat-2 using machine learning algorithms and analyzing its spatial transferability under various forest and terrain conditions. A photon-denoising feature parameter system is developed based on the analysis of the three-dimensional distribution of photons in forested regions. This system reduces the parameters dependent on absolute physical quantities and increases those that are less influenced by terrain and forest features to enhance the model’s transferability. Subsequently, automated machine learning algorithms (AutoML) are used for model selection and parameter optimization across six non-parametric regression models. We evaluate the accuracies of the local, global, and transfer models in estimating canopy height across four representative forested areas in China. Results show that the algorithm can effectively distinguish between signal and noise photons. The estimated canopy heights from signal photons are highly consistent with heights obtained using airborne laser scanning (ALS), exhibiting a Pearson correlation coefficient (r) of 0.89, root mean square errors (RMSE) of 3.75 m, relative root mean square error (rRMSE) of 0.27, relative bias (rBias) of −0.11 and mean Bias of −1.45 m. Notably, the accuracy of canopy height estimation by the global model has increased by an average of 21 % compared to ICESat-2 land-vegetation along-track products (ATL08). Furthermore, the model exhibits significant spatial transfer capabilities, with the accuracies of the transfer model exceeding those of ATL08 by margins ranging from 4 % to 41 %. This study marks a significant advancement in photon-denoising methodologies, providing a robust and transferable solution for large-scale environmental data analysis.

Research and application of key technologies for large-scale time-series data collection and sharing in power grid WAMS

Research and application of key technologies for large-scale time-series data collection and sharing in power grid WAMS

A least-squares framework for developing interval type-2 fuzzy semantics

The developing of IT2 fuzzy semantics is critical for computing with words (CWW), but exiting approaches lack flexibility and fail to adapt user’s diversified demands. The study proposes a least-squares framework for designing CWW encoders that construct interval type-2 fuzzy sets (IT2 FSs) to represent the semantic meanings of linguistic words. In the least-squares framework, an CWW encoder is characterized by two elements: an intra-uncertain semantic mapping and an inter-uncertain semantic family. The intra-uncertain semantic mapping transforms data intervals into type-1 fuzzy sets (T1 FSs), then an optimal IT2 FS is derived from the inter-uncertain semantic family using the least-squares method. Furthermore, several intra-uncertain semantic mappings are introduced, and a compatibility measure is defined to facilitate model selection. The least-squares framework benefits from the flexible selection of intra-uncertain semantic mappings and least-squares optimization-based construction of IT2 FSs. In experiments, the least-squares framework is applied to handle real-world online survey data and the large-scale online review data set of a Chinese life service review site, Dianping.com. Compared to the enhanced interval approach and the Hao-Mendel approach, the least-squares framework shows its favorable efficiency in experiments and statistical tests, and adapts to user-defined intra- and inter-uncertain semantic families.

Learning at a Glance: Towards Interpretable Data-Limited Continual Semantic Segmentation via Semantic-Invariance Modelling.

Continual semantic segmentation (CSS) based on incremental learning (IL) is a great endeavour in developing human-like segmentation models. However, current CSS approaches encounter challenges in the trade-off between preserving old knowledge and learning new ones, where they still need large-scale annotated data for incremental training and lack interpretability. In this paper, we present Learning at a Glance (LAG), an efficient, robust, human-like and interpretable approach for CSS. Specifically, LAG is a simple and model-agnostic architecture, yet it achieves competitive CSS efficiency with limited incremental data. Inspired by human-like recognition patterns, we propose a semantic-invariance modelling approach via semantic features decoupling that simultaneously reconciles solid knowledge inheritance and new-term learning. Concretely, the proposed decoupling manner includes two ways, i.e., channel-wise decoupling and spatial-level neuron-relevant semantic consistency. Our approach preserves semantic-invariant knowledge as solid prototypes to alleviate catastrophic forgetting, while also constraining sample-specific contents through an asymmetric contrastive learning method to enhance model robustness during IL steps. Experimental results in multiple datasets validate the effectiveness of the proposed method. Furthermore, we introduce a novel CSS protocol that better reflects realistic data-limited CSS settings, and LAG achieves superior performance under multiple data-limited conditions.

Genomic insights for personalised care in lung cancer and smoking cessation: motivating at-risk individuals toward evidence-based health practices

Lung cancer and tobacco use pose significant global health challenges, necessitating a comprehensive translational roadmap for improved prevention strategies such as cancer screening and tobacco treatment, which are currently under-utilised. Polygenic risk scores (PRSs) may further motivate health behaviour change in primary care for lung cancer in diverse populations. In this work, we introduce the GREAT care paradigm, which integrates PRSs within comprehensive patient risk profiles to motivate positive health behaviour changes. We developed PRSs using large-scale multi-ancestry genome-wide association studies and standardised PRS distributions across all ancestries. We validated our PRSs in 561,776 individuals of diverse ancestry from the GISC Trial, UK Biobank (UKBB), and All of Us Research Program (AoU). Significant odds ratios (ORs) for lung cancer and difficulty quitting smoking were observed in both UKBB and AoU. For lung cancer, the ORs for individuals in the highest risk group (top 20% versus bottom 20%) were 1.85 (95% CI: 1.58-2.18) in UKBB and 2.39 (95% CI: 1.93-2.97) in AoU. For difficulty quitting smoking, the ORs (top 33% versus bottom 33%) were 1.36 (95% CI: 1.32-1.41) in UKBB and 1.32 (95% CI: 1.28-1.36) in AoU. Our PRS-based intervention model leverages large-scale genetic data for robust risk assessment across populations, which will be evaluated in two cluster-randomised clinical trials. This approach integrates genomic insights into primary care, promising improved outcomes in cancer prevention and tobacco treatment. National Institutes of Health, NIH Intramural Research Program, National Science Foundation.

Automatic registration of large-scale building point clouds with high outlier rates

Point cloud registration plays a crucial role in processing large-scale building point cloud data. However, existing registration algorithms face challenges in effectively handling outliers in descriptor-based correspondence. This paper presents an automatic registration method for large-scale building point clouds that is capable of achieving swift and accurate registration without the need for initial guessing. The method employs a two-step matching optimization approach: coarse (two-point)-to-fine (three-point), selecting matches based on two-point reliability and three-point consistency. Spatial transformation parameters are broken down into rotations and translations. A progressively optimized kernel function is proposed for estimating rotation, while a clustering confidence algorithm computes translation. Comprehensive experiments were conducted using real-world data. The results indicate that the approach swiftly and accurately estimates optimal outcomes when processing large-scale building point clouds with outlier rates up to 99%. Compared to six existing registration methods, the proposed approach reduces rotation error by 6.15% and translation error by 12.83%, while improving efficiency by 2.57%.

The Coming Era of Large-scale Data Calls for a Fast and Accurate Data Analytics Platform

The Coming Era of Large-scale Data Calls for a Fast and Accurate Data Analytics Platform

Character region extraction of wheel water meter based on object detection

Currently, research on automatic meter reading mainly focuses on meter reading recognition, while neglecting the fundamental role of counter detection in the entire automatic meter reading system. In fact, only by accurately locating the counter area can the influence of dial factors be completely eliminated, thus ensuring the accuracy and reliability of subsequent water meter reading recognition. In view of this phenomenon, the focus of this study is on the counter detection stage. Firstly, a target detection-based image skew correction method is proposed to solve the problem of image skew caused by shooting angle and other reasons. This method ensures the accuracy of subsequent counter area positioning and the neatness of cutting effect. Secondly, a semi-supervised target detection training method is proposed to solve the problem of time and manpower costs required in large-scale data situations. In addition, we have made publicly available a dataset containing 1070 water meter images for non-commercial purposes, which can be obtained from the Github11https://github.com/QuanhuiZhao/water-datasets.. Finally, we evaluated our model on three completely different datasets and compared it with the best positioning results of other models. The experimental results show that compared with other models, the proposed model in this paper has improved the positioning accuracy by 5.82%, 5.96%, and 9.20% on three datasets respectively. Furthermore, in the final visualization comparison, the model accurately identifies the counter region even when faced with complex real-world environments.

Extraction of association rules in a diabetic dataset using parallel FP-growth algorithm under apache spark

This research paper focuses on enhancing the frequent pattern growth (FP-growth) algorithm, an advanced version of the Apriori algorithm, by employing a parallelization approach using the Apache Spark framework. Association rule mining, particularly in healthcare data for predicting and diagnosing diabetes, necessitates the handling of large datasets which traditional methods may not process efficiently. Our method improves the FP-growth algorithm’s scalability and processing efficiency by leveraging the distributed computing capabilities of apache spark. We conducted a comprehensive analysis of diabetes data, focusing on extracting frequent itemsets and association rules to predict diabetes onset. The results demonstrate that our parallelized FP-growth (PFP-growth) algorithm significantly enhances prediction accuracy and processing speed, offering substantial improvements over traditional methods. These findings provide valuable insights into disease progression and management, suggesting a scalable solution for large-scale data environments in healthcare analytics.<p> </p>