Scale Data Research Articles

Data extension of high-resolution wind speed database by fusing meteorological observation and local objective analysis data with POD–LSE

Newly generated high-resolution meteorological reanalysis data have been published over the past few years on short records. Producing high-resolution data in long-term time scales is challenging due to high computational costs and complex weather models. In contrast, urban meteorological observations have been continuously recorded for decades, although at a low spatial resolution. This study proposes a data fusion method using Proper Orthogonal Decomposition (POD) and Linear Stochastic Estimation (LSE). It fuses meteorological observations and high spatial resolution local objective analysis (LA) data to reproduce a high-resolution and long-term fusion database for urban areas without observations. First, the accuracy of LA data was verified by comparison with meteorological observations. Subsequently, the accuracy of the generated fusion data was investigated by comparison with Doppler lidar observations. Finally, the performance of single- and multi-time LSE in data fusion is discussed. These results indicate that the proposed method generates high-resolution and long-term fusion wind speed data that are similar to the Doppler lidar observations. In addition, the fusion model built using hourly data can accurately generate a highly sampled database at 10-min intervals. The single-time LSE method is recommended to be applied over the multi-time delay method for data fusion.

P‑Phenylenediamine antioxidants in PM2.5: New markers for traffic in positive matrix factorization source apportionment

The extensive utilization of rubber-related products can lead to a substantial release of p-phenylenediamine (PPD) antioxidants into the environment. In recent years, studies mainly focus on the pollution characteristics and health risks of PM2.5-bound PPDs. This study presents long-time scale data of PPDs and N-(1,3-dimethylbutyl)-N′-phenyl-p-phenylenediamine quinone (6PPD-Q) in PM2.5 and proposes the innovative use of PPDs as new markers for vehicular emissions in the Positive Matrix Factorization (PMF) source apportionment. The results indicate that PPDs and 6PPD-Q were detectable in 100 % of the winter PM2.5 samples, and the concentration ranges of PPDs and 6PPD-Q are 15.6–2.92 × 103 pg·m−3 and 3.90–27.4 pg·m−3, respectively, in which 6PPD and DNPD are the main compounds. Moreover, a competitive formation mechanism between sulfate, nitrate, ammonium (SNA) and 6PPD-Q was observed. The source apportionment results show that the incorporation of PPDs in PMF reduced the contribution of traffic source to PM2.5 from 13.5 % to 9.5 %. In the traffic source factor profiles, the load of IPPD, CPPD, DPPD, DNPD and 6PPD reaches 91.8 %, 91.6 %, 92.9 %, 80.6 % and 87.2 %, respectively. It`s amazing that traditional markers of traffic source, which often overlap with coal burning and industrial sources, over-estimated the contribution of vehicles by one third or more. The discovery of PPDs as specific markers for vehicular emissions holds significant utility, particularly considering the growing proportion of new energy vehicles in the future. The results may prove more accurate policy implications for pollution control. SynopsisPPDs are excellent indicators of vehicle emissions, and PMF without PPDs over-estimated the contribution of traffic source to PM2.5.

Corporate network anomaly detection methodology utilizing machine learning algorithms

ABSTRACT This study addresses the critical need for securing corporate networks against anomalies, a pressing concern in ensuring the comprehensive security of these networks. It aims to develop and validate a new machine learning-based methodology for anomaly detection that is adaptable across various corporate network environments, highlighting the method’s potential practical applications. Employing a systematic approach, the research integrates system analysis of anomaly detection methodologies with an analytical review of machine learning techniques tailored for high-security measures and attack prevention in corporate networks. This dual approach ensures a robust framework for identifying and addressing network anomalies efficiently. The methodology demonstrated notable efficacy, with the proposed machine learning-based anomaly detection techniques achieving an efficiency rate upwards of 90% in identifying and categorizing network traffic types. This high level of precision allows for the effective tracking of network anomalies across diverse corporate networks and their respective devices and equipment. The findings underscore the substantial practical value of the developed methodology, offering a promising avenue for enhancing corporate network security. The implementation of this machine learning-based approach not only facilitates the timely detection of anomalies but also significantly contributes to the improvement of machine learning applications within the realm of network security. Future research could further refine these techniques, exploring scalability and real-time data analysis enhancements to bolster their effectiveness across various network configurations.

Rule learning by modularity

In this paper, we present a modular methodology that combines state-of-the-art methods in (stochastic) machine learning with well-established methods in inductive logic programming (ILP) and rule induction to provide efficient and scalable algorithms for the classification of vast data sets. By construction, these classifications are based on the synthesis of simple rules, thus providing direct explanations of the obtained classifications. Apart from evaluating our approach on the common large scale data sets MNIST, Fashion-MNIST and IMDB, we present novel results on explainable classifications of dental bills. The latter case study stems from an industrial collaboration with Allianz Private Krankenversicherung which is an insurance company offering diverse services in Germany.

A Novel Scalable Digital Data Acquisition System for Industrial Condition Monitoring

A Novel Scalable Digital Data Acquisition System for Industrial Condition Monitoring

A robust CETSA data analysis automation workflow for routine screening

The Cellular Thermal Shift Assay (CETSA) enables the study of protein-ligand interactions in a cellular context. It provides valuable information on the binding affinity and specificity of both small and large molecule ligands in a relevant physiological context, hence forming a unique tool in drug discovery. Though high-throughput lab protocols exist for scaling up CETSA, subsequent data analysis and quality control remain laborious and limit experimental throughput. Here, we introduce a scalable and robust data analysis workflow which allows integration of CETSA into routine high throughput screening (HT-CETSA). This new workflow automates data analysis and incorporates quality control (QC), including outlier detection, sample and plate QC, and result triage. We describe the workflow and show its robustness against typical experimental artifacts, show scaling effects, and discuss the impact of data analysis automation by eliminating manual data processing steps.

Continuous measurement of university students stress level during COVID-19 lockdown: Longitudinal study

Despite numerous studies documenting high stress levels among university students, few have explored the potential benefits of continuous and systematic monitoring of stress-related or other psychological data. The COVID-19 pandemic and policies such as lockdowns, online education, and other restrictions have made it necessary to explore the impact of these measures on students' stress levels. This study aims to describe our experience with collecting longitudinal stress data in a university setting and to investigate the impact of COVID-19 policies on students' stress levels.We employed a mixed-methods approach, collecting quantitative data using the Arabic-translated version of the Perceived Stress Scale (PSS-14) and qualitative data through interviews. Baseline PSS data collection began on October 20, 2019, before the COVID-19 pandemic, to assess students' stress levels during different academic timeframes, such as enrollment, midterm exams, and final exams. Following the outbreak of the COVID-19 pandemic, a series of follow-up PSS collections was conducted from February 9, 2020, to November 15, 2021. After quantitative data collection ended, we conducted in-person interviews with 96 students, exploring domains such as health, academic life, sleep, diet, financial situation, living environments, work, social relationships, depression, and social thoughts.We received 404 responses with a nearly equal distribution of male and female participants. Our results revealed that female participants reported higher mean stress scores than male participants, while male participants had higher mean coping scores.Moreover, continuous and systematic monitoring of stress-related data can be valuable for coping with unpredictable events such as the COVID-19 pandemic. The mixed-methods approach enabled us to obtain a more comprehensive understanding of the complex impact of COVID-19 and lockdowns on students' stress levels. These results have important implications for developing interventions and support systems to help students manage stress during times of uncertainty and disruption.To improve health outcomes in both workplace and academic settings, we recommend integrating health-related measures. This could include the collection of diverse health-related data, such as psychological, psychometric, social, and behavioral information. Future studies should explore guidelines and standards to help organizations determine the most effective measures based on their unique environments and objectives.

Measuring Restrictive Lung Disease Severity Using FEV1 vs TLC.

No clear parameters currently exist to grade severity in restrictive lung disease as for other ventilatory diseases. This article evaluates whether total lung capacity (TLC) or forced expiratory volume in 1 second (FEV1) better correlates with the symptomatology of patients with restrictive lung disease. A retrospective review of 6461 patient records at Veterans Affairs Caribbean Healthcare System in Puerto Rico was conducted, and 414 patients met the inclusion criteria. Pulmonary function test, Modified Medical Research Council Dyspnea Scale, FEV1, and TLC data were collected for each patient. We identified a stronger correlation between FEV1 (r = 0.25, P < .001) vs TLC (r = 0.15, P < .001) when related to the degree of dyspnea as measured with the Modified Medical Research Council Dyspnea Scale. Results of this study suggest that compared with TLC, FEV1 may provide a more accurate measure of restrictive lung disease severity. Further research should look for more accurate measures of patient dyspnea in restrictive lung disease.

Toward parametric Bayesian adaptive procedures for multi-frequency categorical loudness scaling.

A series of Bayesian adaptive procedures to estimate loudness growth across a wide frequency range from individual listeners was developed, and these procedures were compared. Simulation experiments were conducted based on multinomial psychometric functions for categorical loudness scaling across ten test frequencies estimated from 61 listeners with normal hearing and 87 listeners with sensorineural hearing loss. Adaptive procedures that optimized the stimulus selection based on the interim estimates of two types of category-boundary models were tested. The first type of model was a phenomenological model of category boundaries adopted from previous research studies, while the other type was a data-driven model derived from a previously collected set of categorical loudness scaling data. An adaptive procedure without Bayesian active learning was also implemented. Results showed that all adaptive procedures provided convergent estimates of the loudness category boundaries and equal-loudness contours between 250 and 8000 Hz. Performing post hoc model fitting, using the data-driven model, on the collected data led to satisfactory accuracies, such that all adaptive procedures tested in the current study, independent of modeling approach and stimulus-selection rules, were able to provide estimates of the equal-loudness-level contours between 20 and 100 phons with root-mean-square errors typically under 6 dB after 100 trials.

Feature Constrained Encoding Generative Adversarial Prediction Network for Quality Measurement Modeling Under Multitime Scales Data Imbalance In Process Industry

Feature Constrained Encoding Generative Adversarial Prediction Network for Quality Measurement Modeling Under Multitime Scales Data Imbalance In Process Industry

Advanced Parallel Laser Line-Camera System for Automatic and Precise Crack Width Measurement

Accurate quantification of cracks is crucial in evaluating building conditions. However, current non-contact measurement technologies encounter challenges in measuring crack widths, particularly in hard-to-reach areas. This paper presents a novel portable parallel laser line-camera system that is specifically designed for automatic and precise measurement of crack widths from different angles and over long distances. The system is composed of a digital camera, parallel line laser diodes, and rigid positioning arms, forming a triangulation-based configuration. A newly developed image processing algorithm is at the core of this innovation, allowing for the extraction of accurate pixel scale information from laser strips, thus overcoming the challenge of pixel distortion in non-perpendicular photography. The U-Net algorithm is utilized to efficiently identify crack regions, followed by the Canny algorithm for meticulous extraction of crack features. By utilizing the pixel scale data obtained from the laser strips, the actual width of cracks can be accurately determined. The system's efficacy and precision have been validated through laboratory and field tests. This automatic, precise, and portable solution significantly advances the field of on-site crack measurement, offering substantial potential for practical applications in building condition assessments.

Utilizing Python for Scalable Data Processing in Cloud Environments

In the age of big data and cloud computing, enterprises need to effectively analyze enormous datasets to get meaningful insights and stay ahead. Python, a popular programming language, is a strong tool for cloud-scale data processing. This research study examines Python's integration with cloud platforms and its effects on performance and efficiency in scalable data processing. The study introduces scalable data processing and cloud computing. It then discusses Python's ecosystem, including Dask, Apache Spark with PySpark, TensorFlow, and PyTorch for data processing and machine learning. The study also examines Python's interoperability with cloud services like AWS, Google Cloud Platform, and Microsoft Azure in data input, transformation, and analysis. Many case studies and real-world applications demonstrate how Python has been used in banking, healthcare, and e-commerce. Python is useful for managing massive amounts of data, streamlining processing processes, and scaling cloud applications, as shown in the case studies. The report also analyzes Python-based cloud systems' performance indicators and cost consequences, revealing best practices and possible issues. The article explores Python's involvement in cloud computing trends and technology. Serverless architectures, Docker and Kubernetes, and Python interaction with cloud-native tools and services are examples. These patterns show how data processing is changing and how Python is improving to meet current data needs. This study concludes that Python is a reliable and scalable cloud data processing option. The language's strengths, alignment with cloud technologies, and practical applications in many areas are covered in detail. The results indicate that Python's versatility and cloud scalability provide a robust foundation for handling and analyzing massive datasets, enabling better decision-making and innovation across fields.

(Vision Paper) A Vision for Spatio-Causal Situation Awareness, Forecasting, and Planning

Successfully tackling many urgent challenges in socio-economically critical domains, such as public health and sustainability, requires a deeper understanding of causal relationships and interactions among a diverse spectrum of spatio-temporally distributed entities. In these applications, the ability to leverage spatio-temporal data to obtain causally based situational awareness and to develop informed forecasts to provide resilience at different scales is critical. While the promise of a causally grounded approach to these challenges is apparent, the core data technologies needed to achieve these are in the early stages and lack a framework to help realize their potential. In this article, we argue that there is an urgent need for a novel paradigm of spatio-causal research built on computational advances in spatio-temporal data and model integration, causal learning and discovery, large scale data- and model-driven simulations, emulations, and forecasting, as well as spatio-temporal data-driven and model-centric operational recommendations, and effective causally driven visualization and explanation. We thus provide a vision, and a road map, for spatio-causal situation awareness, forecasting, and planning.

Assertive Ability Based on Gender Differences Students of Tahfidz Quran Middle School Pontianak, West Kalimantan

Islamic-based secondary schools, with a boarding school education concept, aim to train children to deepen their knowledge of religion. In reality, boarding schools are the trigger for widespread cases of bullying. Busy activities at school and continued rote activities in the dormitory create boredom and pressure for students. This becomes a stimulus for emotional explosions for students and female students. This research aims to describe the findings of the assertive level abilities of santriwan and santriwati based on gender differences, because life in the dormitory is led by ustad and ustadzah with different characters. Apart from that, it describes an understanding of the form of an assertive attitude and reflects the feelings of santri and female students in being assertive. The mix method research method includes qualitative and quantitative. Qualitative descriptive data analysis activities include collecting data in in-depth interviews, observation and assertive scales, reflecting on experiences and at the same time training students. The research subjects were 80 people consisting of 2 classes, namely class VII and VIII with a gender division of 37 santri and 43 santriwati. The researcher carried out data triangulation, both time, place and data until the data was saturated to process the interview data. Research findings show that students are more able to be assertive by avoiding situations they don't like, refusing with the word "don't want to", because they are afraid of the rules and being angry with the ustad. female students have a weak assertive ability to refuse, because they are embarrassed and feel loyal to their friendship group, but some of the female students are very firm in refusing and reporting threatening and unpleasant situations to the ustadzah. In conclusion, a form of assertive training is needed for self-strength and self-confidence for students and female students to avoid bullying.

Custom Biomedical FAIR Data Analysis in the Cloud Using CAVATICA.

The historically fragmented biomedical data ecosystem has moved towards harmonization under the findable, accessible, interoperable, and reusable (FAIR) data principles, creating more opportunities for cloud-based research. This shift is especially opportune for scientists across diverse domains interested in implementing creative, nonstandard computational analytic pipelines on large and varied datasets. However, executing custom cloud analyses may present difficulties, particularly for investigators lacking advanced computational expertise. Here, we present an accessible, streamlined approach for the cloud compute platform CAVATICA that offers a solution. We outline how we developed a custom workflow in the cloud, for analyzing whole genome sequences of case-parent trios to detect sex-specific genetic effects on orofacial cleft risk, which required several programming languages and custom software packages. The approach involves just three components: Docker to containerize software environments, tool creation for each analysis step, and a visual workflow editor to weave the tools into a Common Workflow Language (CWL) pipeline. Our approach should be accessible to any investigator with basic computational skills, is readily extended to implement any scalable high-throughput biomedical data analysis in the cloud, and is applicable to other commonly used compute platforms such as BioData Catalyst. We believe our approach empowers versatile data reuse and promotes accelerated biomedical discovery in a time of substantial FAIR data.

Evaluation of GPM IMERG Satellite Precipitation Products in Event-Based Flood Modeling over the Sunshui River Basin in Southwestern China

This study evaluates the applicability of hourly Global Precipitation Measurement Mission (GPM) Integrated Multi-satellitE Retrievals for GPM (IMERG) data for event-based flood modeling in the Sunshui River Basin, southwestern China, using the hydrologic modeling system (HEC-HMS) model. The accuracies of IMERG V6, IMERG V7, and the corrected IMERG V7 satellite precipitation products (SPPs) were assessed against ground rainfall observations. The performance of flood modeling based on the original and the corrected SPPs was then evaluated and compared. In addition, the ability of different numbers (one–eight) of ground stations to correct IMERG V7 data for flood modeling was investigated. The results indicate that IMERG V6 data generally underestimate the actual rainfall of the study area, while IMERG V7 and the corrected IMERG V7 data using the geographical discrepancy analysis (GDA) method overestimate rainfall. The corrected IMERG V7 data performed best in capturing the actual rainfall events, followed by IMERG V7 and IMERG V6 data, respectively. The IMERG V7-generated flood hydrographs exhibited the same trend as those of the measured data, yet the former generally overestimated the flood peak due to its overestimation of rainfall. The corrected IMERG V7 data led to superior event-based flood modeling performance compared to the other datasets. Furthermore, when the number of ground stations used to correct the IMERG V7 data in the study area was greater than or equal to four, the flood modeling performance was satisfactory. The results confirm the applicability of IMERG V7 data for fine time scales in event-based flood modeling and reveal that using the GDA method to correct SPPs can greatly enhance the accuracy of flood modeling. This study can act as a basis for flood research in data-scarce areas.

OPTIMIZING SQL DATABASES FORBIG DATA WORKLOADS: TECHNIQUES AND BEST PRACTICES

In the era of big data, SQL databases face significant challenges in handling vast volumes of data efficiently. This article explores optimization techniques and best practices for enhancing the performance and scalability of SQL databases in handling big data workloads. The study addresses the significant challenges faced by traditional SQL databases, including scalability issues, performance bottlenecks, resource constraints, and data integration complexities. Through a comprehensive methodology involving literature review, case studies, expert interviews, and performance testing, the research identifies effective strategies such as indexing, partitioning, sharding, and caching. Findings from case studies in e-commerce and financial services sectors demonstrate substantial improvements in query performance and resource utilization, validating the practical benefits of these optimization techniques. The study underscores the importance of a multifaceted approach to database optimization, integrating both theoretical and practical insights to address the complexities of big data environments. By staying informed and adopting the latest optimization strategies, database administrators and IT professionals can ensure their SQL databases remain efficient, scalable, and capable of managing the increasing demands of large-scale data processing, ultimately enabling organizations to derive valuable insights from their data.

A Brief Systematics Review and Visualization of Smart Cities based on Fog and Big Data Analysis Technology

The fog computing is based on IoT, and same as clouds are on the edge of a network between a device and the cloud, which can provide very close service to the device in real time. cloud computing frameworks also extend support to emerging application paradigms such as IoT, Fog computing, Edge, and Big Data through service and infrastructure. the ubiquitous deployment of sensors in smart cities requires a high-performance computing paradigm to support big data analysis with smart technologies and communications in IoT, providing location-awareness and latency-sensitive computing near the data sources (i.e., at the edge of the network). Cloud Computing paradigms widely used in enterprises to address the emerging challenges of big data analysis because of its scalable and distributed data management scheme. However, data centers in the Cloud faces great challenges on the burden of exploding amount of big data and the additional requirements of location awareness and low latency at the edge of network necessary for smart cites. the widely distributed sensor networks generate massive volume of data, which leads to a “Big Data “analysis challenge. The machine-to-machine(M2M) communication among massive numbers of sensors will dominate future communication network traffic, namely Internet of Things, instead of traditional Internet of Contents (IoC) in human-to-human and human-to-machine communication. In this review system where we took the research article around 33 from the different database and also taken 1987 datasets form the PubMed database from 1959 to 2024 (02 June 2024). In addition, depending on the different requirement which is partially processed the data. Then the gathered data is stored at the ledger unit through the smart cities based on the fog computing and big data technology. The hashing identity-based data encryption is a public key encryption which uses a digital signature to encrypt the data to prevent the information from the third party.

A scalable big data approach for remotely tracking rangeland conditions

Rangelands, covering half of the global land area, are critically degraded by unsustainable use and climate change. Despite their extensive presence, global assessments of rangeland condition and sustainability are limited. Here we introduce a novel analytical approach that combines satellite big data and statistical modeling to quantify the likelihood of changes in rangeland conditions. These probabilities are then used to assess the effectiveness of management interventions targeting rangeland sustainability. This approach holds global potential, as demonstrated in Mongolia, where the shift to a capitalist economy has led to increased livestock numbers and grazing intensity. From 1986 to 2020, heavy grazing caused a marked decline in Mongolia’s rangeland condition. Our evaluation of diverse management strategies, corroborated by local ground observations, further substantiates our approach. Leveraging globally available yet locally detailed satellite data, our proposed condition tracking approach provides a rapid, cost-effective tool for sustainable rangeland management.

Effects of Early Gait Training on Inpatient Frailty After Transcatheter Aortic Valve Implantation.

This study aimed to clarify the relationship between the number of days of early gait training and frailty in in-hospital patients undergoing transcatheter aortic valve implantation (TAVI) for aortic stenosis, focusing on the Clinical Frailty Scale (CFS) and clinical laboratory data. Sixty-nine patients admitted to the Ichinomiya West Hospital from November 1, 2019 to November 30, 2023 were included in the study. Of the 69 patients, those who started gait training on postoperative day 0 or 1 were defined as the early gait training group and those who started gait training later than postoperative day 1 were defined as the delayed gait training group. There was a significant difference in the number of days to gait training initiation, which was 3.9 days in the delayed gait training group and 0.9 days in the early gait training group. The early gait training group started early mobilization and had a significantly shorter postoperative hospital stay than the delayed gait training group. Clinical laboratory data showed that walking speed was significantly faster and grip strength was significantly higher in the early group. The number of days to gait training initiation was an independent predictor of changes in CFS scores. Early gait training in patients after TAVI may predict early improvements in physical function and movement, shorter hospital stay, and frailty at discharge.