Scale Data Research Articles

Knowledge Extraction from LLMs for Scalable Historical Data Annotation

This paper introduces a novel approach to extract knowledge from large language models and generate structured historical datasets. We investigate the feasibility and limitations of this technique by comparing the generated data against two human-annotated historical datasets spanning from 10,000 BC to 2000 CE. Our findings demonstrate that generative AI can successfully produce historical annotations for a wide range of variables, including political, economic, and social factors. However, the model’s performance varies across different regions, influenced by factors such as data granularity, historical complexity, and model limitations. We highlight the importance of high-quality instructions and effective prompt engineering to mitigate issues like hallucinations and improve the accuracy of generated annotations. The successful application of this technique can significantly accelerate the development of reliable structured historical datasets, with a potentially high impact on comparative and computational history.

Least angle sparse principal component analysis for ultrahigh dimensional data

AbstractPrincipal component analysis (PCA) has been a widely used technique for dimension reduction while retaining essential information. However, the ordinary PCA lacks interpretability, especially when dealing with large scale data. To address this limitation, sparse PCA (SPCA) has emerged as an interpretable variant of ordinary PCA. However, the ordinary SPCA relies on solving a challenging non-convex discrete optimization problem, which maximizes explained variance while constraining the number of non-zero elements in each principal component. In this paper, we propose an innovative least angle SPCA technique to address the computational complexity associated with SPCA, particularly in ultrahigh dimensional data, by sequentially identifying sparse principal components with minimal angles to their corresponding components extracted through ordinary PCA. This sequential identification enables solving the optimization problem in polynomial time, significantly reducing computational challenges. Despite its efficiency gains, our proposed method also preserves the main attributes of SPCA. Through comprehensive experimental results, we demonstrate advantages of our approach as a viable alternative for dealing with the computational difficulties inherent in ordinary SPCA. Notably, our method emerges as an efficient and effective solution for conducting ultrahigh dimensional data analysis, enabling researchers to extract meaningful insights and streamline data interpretation.

Faking in Self‐Report Personality Scales: A Qualitative Analysis and Taxonomy of the Behaviors That Constitute Faking Strategies

ABSTRACTFaking in self‐report personality scales (SRPSs) is not sufficiently understood. This limits its detection and prevention. Here, we introduce a taxonomy of faking behaviors that constitute faking strategies in SRPSs, reflecting the stages (comprehension, retrieval, judgment, and response) of the general response process model (GRPM). We reanalyzed data from two studies investigating the faking of high and low scores on Extraversion (E) and Need for Cognition (NFC) scales (Data Set 1; N = 305) or on an E scale (Data Set 2; N = 251). Participants were asked to explain exactly what they did to fake, and their responses (N = 533) were examined via a qualitative content analysis. The resulting taxonomy included 22 global and 13 specific behaviors that (in combination) constitute faking strategies in SRPSs. We organized the behaviors into four clusters along the stages of the GRPM. The behaviors held irrespective of the construct (E or NFC), and with two exceptions, also irrespective of the data set (Data Sets 1 or 2). Eight exceptions concerning faking direction (high or low) indicate direction‐specific differences in faking behaviors. Respondents reported using not only different faking behaviors (e.g., role‐playing, behaviors to avoid being detected) but also multiple combinations thereof. The suggested taxonomy is necessarily limited to the specified context, and, thus, additional faking behaviors are possible. To fully understand faking, further research in other contexts should be conducted to complement the taxonomy. Still, the complexity shown here explains why adequate detection and prevention of faking in SRPSs is so challenging.

Architecting Enterprise-Scale Data Products: A Framework for Advanced Data Science and AI/ML Operations

This article presents a comprehensive framework for building enterprise-scale data products that power modern Customer & Product Analytics, Data Science, artificial intelligence, and machine learning initiatives. The article examines the foundational architecture patterns, pipeline engineering strategies, and advanced distributed computing approaches in both on-prem and cloud. These are essential for developing robust data infrastructure capable of handling complex Data Analytics, Data Science, and AI/ML workflows. The article explores critical aspects of feature engineering at scale, real-time processing capabilities, and the implementation of feature stores, while addressing the challenges of data quality, governance, legal, and security in regulated environments. The article introduces a systematic approach to integrating data products with MLOps pipelines, emphasizing the importance of automated workflows, monitoring systems, and feedback loops in production environments. The findings demonstrate that successful implementation of scalable data products requires a careful balance of architectural decisions, technology selection, and operational practices. The article contributes to the field by providing actionable insights and architectural patterns that organizations can adopt to build resilient, scalable, and efficient data products for their Data Analytics, Data Science, and AI/ML use cases. This article establishes a foundational framework that bridges the gap between theoretical data architecture principles and practical implementation challenges in enterprise settings.

Web Data Mining and Crawling: A Detailed Overview in the Aspect of Googlebot, Apache Nutch and Bingbot

This paper explores the integration of web data mining with web crawlers, focusing on how various web crawlers—Apache Nutch, G-Boats, and Bing Bot— collect and process web data. Apache Nutch, with its scalable distributed crawling capabilities and integration with Hadoop, excels in handling large- scale data collections. G-Boats offers real-time data processing with a flexible framework, while Bing Bot demonstrates advanced algorithms for web indexing and search engine optimization. The paper compares these crawlers, highlighting their strengths and weaknesses in scalability, fault tolerance, content extraction, and big data integration. The conclusion provides insights into choosing the appropriate web crawler based on specific web data mining needs.

Genetic Variability and Character Association Analysis for Yield and Its Related Traits in Rain Fed Lowland Rice (<i>Oryza sativa </i>L.)<i> </i>Genotypes at Teppi and Fogera, Ethiopia

In Ethiopia, rice variety development started very recently and is mainly dependent on the introduction breeding materials in which characterization and understanding of the genetic variability of genotypes is the critical step. Knowledge of the extent of genetic variability and interrelationships of yield and yield components is pre-request for designing an effective selection-based rice improvement programs for evolving high yielding rice genotypes. Therefore, this study was carried out with all objectives of assessing genetic variation for yield and yield related traits in lowland rice genotypes. Thirty-six selected lowland rice genotypes were evaluated in simple lattice design with two replications during 2017 cropping season and data on 15 quantitative morphological traits were collected and subjected to various statistical analysis. Combined analysis of variance across location revealed significant location, genotype and genotype × location interaction effects for the traits evaluated at p ≤0.01. The phenotypic coefficient of variation was high for un fertile tiller per plant (88.16 %), followed by infertile grain per panicle (50.66%), and grain yield (21.57%), and the genotypic coefficient of variation was high for un fertile tiller per plant (50.38 %), followed by infertile grain per panicle (27.22%), and culm length (17.7%). The estimate of broad sense heritability ranges from 15.2% for biological yield to 91.5% for plant height. The expected genetic advance as present of mean varied from 5.42% for biological yield to 59.3% for unfertile tiller per plant. Phenotypic and genotypic correlation revealed grain yield had a a significant and positive association with primary branches per panicle (rp=0.4, rg=0.7), fertile grain per panicle (rp=0.5, rg=0.72) and harvest index (rp=0.76, rg=0.73) at both genotypic and phenotypic. Path coefficient analysis of grain yield revealed that harvest index, total tiller per plant and thousand grain weights showed positive phenotypic direct effect. A Positive direct effect was found for total tiller per plant, harvest index and biological yield at genotypic level. The present study indicated sufficient genetic variability for the characters studied in rain fed low land rice genotype, which will create opportunity for future improvement. To have full scale data, further study at various locations and years with the help of molecular techniques should be done.

Improving of computed tomography images for effective diagnosis of gunshot wounds

Background. The purpose of the work is to develop gradation correction models, methods, and test software that can be quickly applied to improve X-ray and computed tomography (CT) images in automatic mode under conditions of time constraints, as well as significant distortion of the dynamic range of density due to the presence of high-density metal objects, significant manifestation of the gloss effect. Materials and methods. To achieve this, experiments were planned and conducted to obtain CT scans of the organs of interest with metal fragments of the given types placed in them. All the fragments were photographed and measured by a group of independent experts for the possibility of further comparative analysis by classical and computer methods, evaluation of the quality of improved images by numerical and visual methods. Results. Based on experimental data, a new unified model, method and test software for gradation correction of X-ray and CT images have been developed, the use of which allows you to significantly improve the quality of images (up to 5 times in the sense of increasing contrast) in real time (most often, the processing time of one image does not exceed 1 second). The proposed model, method and software are unified in terms of file formats, automatically adjust to the density scale and process data. Conclusions. The developed model, method and software make it possible to achieve the goal of the work — to provide prompt improvement of X-ray and CT images in automatic mode in conditions of lack of time, as well as significant distortion of the dynamic range of density, significant manifestation of the gloss effect. This makes it possible to quickly and qualitatively diagnose gunshot wounds. High efficiency is confirmed by the results of numerous experiments and implementation. At the same time, thanks to a one-time adjustment by a highly qualified expert, the proposed software can be applied with high efficiency in a mass order using preset parameters.

Data Modeling Essentials: Techniques, Best Practices, and Future Trends

Data modeling, in essence, is based on the process of data structuring and organization to effectively handle, analyze, and make informed decisions across organizations. This paper reviews the major concepts and methods significant for modern data architecture in "Data Modeling Essentials: Techniques, Best Practices, and Future Trends.". We start by considering the base techniques, namely Entity-Relationship and Dimensional modeling, and look into their applications in operational and analytical data systems. The paper also covers more advanced topics that include Data Vault modeling for agile data warehousing and NoSQL modeling to accommodate unstructured and semi-structured data, increasingly important in today's big data era. The paper also gives an overview of best practices in developing scalable, flexible data models: stakeholder communication, documentation, and the use of automation tools in accelerating model development and consistency. Data quality, adherence to regulations, and alignment of models with business objectives are among the practices identified as crucial in today's fast-moving data environments. New trends are taking precedence, and data modeling is continuously evolving. We will review the rise of AI- assisted data modeling, graph database structures for highly interconnected data, and model-driven development to accelerate system deployments. This paper examines actionable insights for data professionals, IT teams, and business leaders on how to design robust, agile models that improve the quality of data, enhance decision-making, and foster innovation in an ever-changing data ecosystem. Keywords Data Modeling, Entity-Relationship (ER) Modeling, Dimensional Modeling, Data Quality, Scalable Data Models, Model Driven Development (MDD), Data Management, Structured & Unstructured Data, Future Trends in Data Modeling.

Large scale paired antibody language models.

Antibodies are proteins produced by the immune system that can identify and neutralise a wide variety of antigens with high specificity and affinity, and constitute the most successful class of biotherapeutics. With the advent of next-generation sequencing, billions of antibody sequences have been collected in recent years, though their application in the design of better therapeutics has been constrained by the sheer volume and complexity of the data. To address this challenge, we present IgBert and IgT5, the best performing antibody-specific language models developed to date which can consistently handle both paired and unpaired variable region sequences as input. These models are trained comprehensively using the more than two billion unpaired sequences and two million paired sequences of light and heavy chains present in the Observed Antibody Space dataset. We show that our models outperform existing antibody and protein language models on a diverse range of design and regression tasks relevant to antibody engineering. This advancement marks a significant leap forward in leveraging machine learning, large scale data sets and high-performance computing for enhancing antibody design for therapeutic development.

Music Preference and Emotional Expressivity Among Emerging Adults

Emerging adults, are in the process of evolution from adolescence to complete adulthood. Music preference refers to what kind of music you like at a specific moment, comparing one song to another. It’s about your overall style of music over a long period, showing what kind of music you tend to enjoy consistently. Emotional expressivity refers to how someone shows or communicates their emotions, whether through facial expression, body language, and voice tone or even actions. The purpose of the present study aimed to explore the relationship between music preference and emotional expressivity among emerging adults. The sample chosen for the study conducted among 107 emerging adults ages between 18-25 (Females =53, Males =54) in Kollam, Pathanamthitta and Kottayam of Kerala through convenience sampling method. The measures used were Short Test of Music Preference (Rentfrow& Gosling, 2003), Emotional Expressivity Scale (Kring et.al, 1994) and personal data schedule. Statistical procedure of Spearman’s Correlation analysis was done using IBM SPSS Statistics 25. The results of the study showed that there is no significant relationship between music preference and emotional expressivity among emerging adults.

Depression and anxiety at 1- and 12-months post ischemic stroke: methods for examining individual change over time.

Background Depression is commonly studied post stroke, while anxiety is less studied. This study presents prevalence of depression and anxiety at 1- and 12-months post ischemic stroke alongside three methods for examining within-subjects change over time. Methods Participants were ischemic stroke patients of the Auckland Regional Community Stroke Study (ARCOS-V) with Hospital Anxiety and Depression Scale data at 1- (n =343) and 12-months (n =307). Change over time was examined using within-subjects repeated measures ANOVA, calculation of the Reliable Change Index, and a Sankey diagram of those meeting cut-off scores (>7) for caseness over time. Results Using repeated measures ANOVA, depression scores didn't change significantly over time, while anxiety symptoms decreased significantly. When reliable change was calculated, 4.2% of individuals had reliable decreases in anxiety symptoms, while 5.7% had reliable decreases in depression symptoms. Those who had a reliable decrease in one tended to have a reliable decrease in the other. In the Sankey, the proportion of those meeting the cut-off score for anxiety did not change over time (12.8 and 12.7% at 1- and 12-months), while those meeting the cut-off for depression increased slightly (3.7-4.5%) and those meeting cut-offs for both decreased from 10.4 to 8.1%. Conclusion The three methods produced very different findings. Use of cut-off scores is common but has limitations. Calculation of clinically reliable change is recommended. Further work is needed to ensure depression and anxiety are monitored over time post-stroke, and both should be the subject of intervention efforts in both acute and late stages post-stroke.

Scalable Photonic Nose Development through Corona Phase Molecular Recognition.

Breath sensors promise early disease diagnosis through noninvasive, rapid analysis, but have struggled to reach clinical use due to challenges in scalability and multivariate data extraction. The current breath sensor design necessitates various channel materials and surface functionalization methods, which delays the process. Additionally, the limited options for channel materials that provide optimum sensitivity and selectivity further restrict the array size to a maximum of only 10 to 20 channels. To address these limitations, we propose a breath sensing array design process based on Corona Phase Molecular Recognition (CoPhMoRe), which enables the creation of an expansive library of nanoparticle interfaces and broad fingerprints for multiple analytes in the breath. Although CoPhMoRe has predominantly been utilized for liquid-phase sensing, its recent application to gas-phase sensing has shown significant potential for breath sensing. We introduce the recent demonstrations in the field and present the concept of a CoPhMoRe-based photonic-nose sensor array, leveraging fluorescent nanomaterials such as near-infrared single-walled carbon nanotubes. Additionally, we identified four critical milestones for translating CoPhMoRe into breath sensors for practical clinical applications.

Examining Engineering Students’ Attitudes Towards Mathematics Using Fuzzy Conjoint Analysis

Attitudes are an important factor influencing individuals’ academic success and learning processes. Research indicates that a positive attitude towards mathematics is associated with higher achievement and motivation, while a negative attitude is linked to difficulties. Attitudes towards mathematics shape the academic performance and professional success of students in engineering, as in many other fields. One of the methods used to analyze Likert-type scale data, which is employed to determine ambiguous perceptions and beliefs such as self-efficacy and attitudes, is the Fuzzy Conjoint Analysis. In this study, the attitudes of engineering students towards mathematics were examined using Fuzzy Conjoint Analysis according to various variables. The study was conducted with 382 undergraduate students from different departments of the engineering faculty at a foundation university in Türkiye. The “Attitude Towards Mathematics Scale” was used as the data collection tool. The analysis of the data revealed that the attitudes of engineering students towards mathematics were positive. In addition, it was determined that attitudes towards mathematics showed a low level of difference in favor of females according to the gender variable, in favor of the Industrial Engineering department according to the department variable, and against third-year students according to the grade variable.

Blockchain empowered dynamic access control for secure data sharing in collaborative emergency management

In the realm of emergency management involving multi-party collaboration, securely sharing data among multiple entities poses a significant challenge. Traditional centralized data platforms struggle with fine-grained access control for data from different sources and are susceptible to single-point failures and risks of forged authorization. This paper delves into the prospective application of blockchain technology for decentralized data governance. We propose a dynamic access control technology for multi-party collaborative emergency response, which combines blockchain and attribute-based access control technologies to enable dynamic access control in multi-party data sharing scenarios. A data sharing system for collaborative emergency management built with Hyperledger Fabric has been developed and subjected to a series of evaluation tests. The experimental results reveal that our blockchain-based system not only significantly enhances data security and privacy by leveraging decentralized control but also improves the efficiency and reliability of data sharing in emergency situations. Specifically, our findings demonstrate the system's ability to dynamically adapt access control policies based on changing emergency scenarios, effectively mitigating risks associated with single-point failures and unauthorized access. These outcomes underscore the viability and reliability of our approach, providing a robust framework for secure, scalable, and flexible data sharing in the demanding field of emergency management.

A review of primary school students attitudes towards visual arts course and their painting performances

This study aims to examine the attitudes of fourth-grade primary school students towards the Visual Arts Course and the success levels of their painting performances. The study is a descriptive scanning model with a quantitative research approach. The study was conducted in two class hours at a public primary school located in Hatay/Reyhanlı center in the fall semester of the 2022-2023 academic year. The study group consisted of 55 students aged 8-11 from classes 4-A and 4-B. While collecting the research data, the "Attitude Scale Towards Visual Arts Course" and the "Student Products (Paintings) Evaluation Form (Rubric)" were used. Descriptive analyses and t-tests were applied to the Attitude Scale Data. Descriptive statistics were applied to the data obtained in the evaluation of the paintings. Based on the findings, it is seen that the attitudes of the fourth-grade students towards the visual arts course are at a medium level. The attitudes of the students towards the course were compared according to their gender and age and no statistically significant difference was found. As a result of the paintings evaluated according to the Student Products (Paintings) Evaluation Form (Rubric), the students generally have a medium level of success in all criteria. While the highest average score was received from the criterion of "The work being appropriate to the subject", the lowest score was received from the sub-dimensions of "Image richness", "Using colours (originality, richness, colour transitions in the use of colours) and "Movement-Rhythm". As a result, both the attitudes of the fourth-grade students towards the visual arts course and the paintings they made in the course are consistent within themselves, but the fact that they are at a medium level in both cases reveals that the issue needs to be emphasized.

A novel energy efficient data gathering algorithm for wireless sensor networks using artificial intelligence

Energy efficiency is challenging task in wireless sensor network (WSN), it is the main barrier in extending network lifespan. In WSN, maximum energy is wasted during data gathering, hence energy efficient algorithms using artificial intelligence can be designed, that preserves energy while data gathering. Thus, our proposed methodology, A novel energy efficient data gathering algorithm using artificial intelligence for wireless sensor networks (NDGAI), uses novel artificial intelligence algorithms and addresses issue of energy consumption while gathering data. In our proposed work, mobile element is utilized to gather information from sensor nodes in the clusters, formed using amended-expectation-maximization. Each cluster should have a cluster leader and a virtual-point. These cluster leaders are formed utilizing fuzzy logic technique. Virtual-points are formed in the range of cluster leader, only when cluster leader has data. The mobile element reaches virtual point by taking the optimal path, that determined by the hybrid artificial intelligence algorithms, such as artificial-bee-colony (ABC) technique and particle swarm optimization (PSO) algorithms. Thus, by properly performing clustering, cluster leader selection, virtual-point selection and optimal path determination, lead to improved network lifetime and energy saving while gathering the data. Results are simulated and compared with scalable gridbased data gathering algorithm for environmental monitoring wireless sensor networks (SGBDN) and proposed algorithm performs better.

Constructing phylogenetic trees for microbiome data analysis: A mini-review

As next-generation sequencing technologies advance rapidly and the cost of metagenomic sequencing continues to decrease, researchers now face an unprecedented volume of microbiome data. This surge has stimulated the development of scalable microbiome data analysis methods and necessitated the incorporation of phylogenetic information into microbiome analysis for improved accuracy. Tools for constructing phylogenetic trees from 16S rRNA sequencing data are well-established, as the highly conserved regions of the 16S gene are limited, simplifying the identification of marker genes. In contrast, metagenomic and whole genome shotgun (WGS) sequencing involve sequencing from random fragments of the entire gene, making identification of consistent marker genes challenging owing to the vast diversity of genomic regions, resulting in a scarcity of robust tools for constructing phylogenetic trees. Although bacterial sequence tree construction tools exist for upstream bioinformatics, many downstream researchers—those integrating these trees into statistical models or machine learning—are either unaware of these tools or find them difficult to use due to the steep learning curve of processing raw sequences. This is compounded by the fact that public datasets often lack phylogenetic trees, providing only abundance tables and taxonomic classifications. To address this, we present a comprehensive review of phylogenetic tree construction techniques for microbiome data (16S rRNA or whole-genome shotgun sequencing). We outline the strengths and limitations of current methods, offering expert insights and step-by-step guidance to make these tools more accessible and widely applicable in quantitative microbiome data analysis.

Channel Mixer Layer: Multimodal Fusion Toward Machine Reasoning for Spatiotemporal Predictive Learning of Ionospheric Total Electron Content

AbstractThe spatiotemporal distribution of Total Electron Content (TEC) in ionosphere determines the refractive index of electromagnetic wave leading to the radio signal scintillation and deterioration. Thanks to the development of machine learning for video prediction, spatiotemporal predictive models are applied on the future TEC map prediction based on the graphic features of past frames. However, output result of graphic prediction is unable to properly respond to the external factor variations such as solar or geomagnetic activity. Meanwhile, there is still neither standard data ‐set nor comprehensive evaluation framework for spatiotemporal predictive learning of TEC map sequences leading to the comparisons unfair and insights inconclusive. In this research, a new feature‐level multimodal fusion method named as channel mixer layer for machine reasoning is proposed that can be embedded into the existing advanced spatiotemporal sequence prediction models. Meanwhile, all performance benchmarks are accomplished on the same running environment and newly proposed largest scale data set. Experiment results suggest that the multimodal fusion prediction of existing model backbones by proposed method improves the prediction accuracy up to 15% with almost the same computational complexity compared to that of graphic prediction without auxiliary factors input, having the real‐time inference speed of 34 frames/second and minimum mean absolute error of 0.94/2.63 TEC unit during low/high solar activity period respectively. The channel mixer layer embedded models can respond to the variations of auxiliary external factors more correctly than previous multimodal fusion methods such as concatenation and arithmetic, which is regarded as the evidence of state‐of‐the‐art machine reasoning ability.

Cloud data centers and networks: Applications and optimization techniques

As the volume and complexity of big data continue to escalate, optimizing the performance, scalability, and energy efficiency of big data applications within cloud data centers has become increasingly crucial. This journal presents a comprehensive survey of current optimization techniques, focusing on data placement, job scheduling, and network configurations tailored for cloud environments. We explore the impact of various data center topologies on the performance of big data frameworks like Hadoop, emphasizing the trade-offs between performance and energy efficiency. Advanced methodologies, including dynamic data placement strategies, locality-aware scheduling, and innovative reduce task placement techniques, are reviewed in depth. Additionally, we highlight the importance of network power effectiveness (NPE) and examine the role of optical and electronic switching technologies in enhancing data center efficiency. By synthesizing findings from recent studies, this paper provides valuable insights into the optimization of cloud data centers, offering recommendations for improving resource utilization and reducing job completion times while maintaining energy efficiency. The findings contribute to the ongoing efforts to scale and adapt cloud data infrastructures for the rapidly growing demands of big data applications.

Enhancing financial portfolio management with predictive analytics and scalable data modeling techniques

The financial industry faces an increasingly complex landscape for portfolio management, where data-driven insights are crucial for optimizing asset allocation and managing risk. This paper explores the integration of predictive analytics and scalable data modeling techniques in enhancing financial portfolio management. By leveraging machine learning algorithms, big data architectures, and real-time data processing, predictive analytics can forecast asset trends, detect market anomalies, and assess portfolio risk with high precision. We evaluate several predictive models, including time-series forecasting, neural networks, and ensemble methods, for their efficacy in financial prediction. The study also discusses the role of scalable data modeling frameworks, such as Apache Spark and cloud-based data lakes, in handling vast volumes of unstructured data across different markets and asset classes. Findings indicate that predictive analytics, when paired with robust data models, can deliver real-time, actionable insights, enhancing decision-making for fund managers and institutional investors. Furthermore, this paper highlights best practices for implementing scalable models in financial institutions, addressing challenges like data latency, model interpretability, and system scalability. By adopting these advanced analytics frameworks, portfolio managers can achieve improved risk-adjusted returns, better asset diversification, and enhanced adaptability to market volatility.