Large-scale Processes Research Articles

Optimizing Cloud Data Storage: Evaluating File Formats for Efficient Data Warehousing

This paper presents a detailed analysis of three widely-used data storage formats—Parquet, Avro, and ORC— evaluating their performance across key metrics such as query execution, compression efficiency, data skipping, schema evolution, and throughput. Each format offers distinct advantages depending on the nature of the workload. Parquet is optimized for read-heavy analytical queries, providing excellent compression and efficient query performance through its columnar structure. Avro excels in write-heavy, real-time data streaming scenarios, where schema flexibility and backward compatibility are crucial. ORC balances the two, offering strong support for analytical and transactional workloads, especially in handling complex queries and nested data structures. This comparative study highlights the contexts in which each format performs best, providing valuable insights into the trade-offs associated with their use in cloud data warehouses and large-scale data processing environments.

Enhancing Bucket Sort Efficiency: A Combinatorial Approach to Algorithm Optimization

Abstract: The efficiency of sorting algorithms is critical to numerous computational processes, including data management, search operations, and real-time applications. Among these algorithms, Bucket Sort is notable for its linear-time performance when data is uniformly distributed over a defined range. However, its efficiency suffers significantly with non-uniform data distributions, resulting in unbalanced buckets and increased sorting time. In our research, we explore how combinatorial algorithms can enhance Bucket Sort’s efficiency by optimizing bucket partitioning, dynamic allocation, and internal sorting strategies. Through rigorous mathematical proofs and analysis, we demonstrate that the application of combinatorial techniques yields substantial improvements in average-case performance, particularly in handling skewed data distributions. Practical case studies in large-scale data processing illustrate the adaptability and effectiveness of these optimizations, supporting their potential in real-world applications.

Research on the influence of cutter overhang length on robotic milling chatter stability

Serial industrial robots are widely in demand in the field of large-scale component processing and manufacturing. However, the structure characteristic of serial robots makes it easy to generate chatter during milling and the overhang length of milling cutter has an important influence on the stability of robotic milling. Milling cutter overhang length refers to the length from the tip of the cutter to the protrusion of the spindle, to study the influence of milling cutter overhang length on the stability behavior of robotic milling, the modal parameters of milling cutter under different overhang length were obtained by hammer experiment, and its dynamic characteristics were analyzed. The stability behavior for robotic milling under different cutter overhang lengths was analyzed by using the stability lobe diagrams, and the stability lobe diagrams were verified by robotic milling experiments. The results show that the rigidity and natural frequency of milling cutter decrease with the increase of overhang length. There is a big gap between the stability lobe diagrams and the actual machining state when the cutter overhang length is short, and the low frequency vibration is easy to occur in the robotic milling process when the spindle speed is low, which is most probably because that the absorbed vibration energy by the cutter has been transferred to the robot body before the milling cutter vibrates. When the cutter overhang is increased and the feed direction is along the y-axis of the robot global coordinate system, the stability lobe diagrams by considering the multi-mode coupling effect is more consistent with the actual milling state.

UNDER THE INFLUENCE OF THE INDUSTRIAL REVOLUTION: A RETROSPECTIVE ANALYSIS

The article is intended to analyze the cyclicality of the development of construction, caused by fundamental qualitative changes in the basic generations of equipment, machines, technologies, vehicles and large structures in the leading sectors of the economy, i.e. industrial revolutions. The first industrial revolution created the organizational-economic, scientific-technical and institutional prerequisites for a conceptual rethinking of the principles and norms of construction and brought to life the pre-monopolist-industrial model of the construction sector (Construction 1.0). Its main feature was the active use of metal elements (cast iron, wrought iron, steel), glass, concrete and reinforced concrete in construction production. The industrial-extensive model of Construction 2.0 covers the period from the beginning of the 20th century. until the end of the 1960s and coincides with the Second Industrial Revolution. This model is distinguished by large-scale processes of territorial and spatial concentration of construction based on the accumulation of means of production, labor and capital investments in large construction organizations and complex processes of industrialization of the construction sector. From the first half of the 1970s to the first half of the 2000s, the energy-efficient construction model (Construction 3.0), brought to life by the Third Industrial Revolution, prevails in the world. This period is characterized by the systematic orientation of construction production to the optimization of thermal energy provision of objects, the rationalization of the processes of using energy resources at all stages of the construction and production process, and the minimization of the level of consumption of energy and material resources. Today, marked by the unfolding of the Fourth Industrial Revolution, is characterized by the global development of an ecosystem-type green construction model (Construction 4.0) – an environmentally responsible and resource-saving process of social reproduction of construction products. This model is based on deep integration of green and digital transition technologies and is ensured by large-scale digitization of construction works.

Novel Genetic Optimization Techniques for Accurate Social Media Data Summarization and Classification Using Deep Learning Models

In the era of big data, effectively processing and understanding the vast quantities of brief texts on social media platforms like Twitter (X) is a significant challenge. This paper introduces a novel approach to automatic text summarization aimed at improving accuracy while minimizing redundancy. The proposed method involves a two-step process: first, feature extraction using term frequency–inverse document frequency (TF–IDF), and second, summary extraction through genetic optimized fully connected convolutional neural networks (GO-FC-CNNs). The approach was evaluated on datasets from the Kaggle collection, focusing on topics like FIFA, farmer demonstrations, and COVID-19, demonstrating its versatility across different domains. Preprocessing steps such as tokenization, stemming, stop word s removal, and keyword identification were employed to handle unprocessed data. The integration of genetic optimization into the neural network significantly improved performance compared to traditional methods. Evaluation using the ROUGE criteria showed that the proposed method achieved higher accuracy (98.00%), precision (98.30%), recall (98.72%), and F1-score (98.61%) than existing approaches. These findings suggest that this method can help create a reliable and effective system for large-scale social media data processing, enhancing data dissemination and decision-making.

Scalable quantum detector tomography by high-performance computing

Abstract At large scales, quantum systems may become advantageous over their classical counterparts at performing certain tasks. Developing tools to analyze these systems at the relevant scales, in a manner consistent with quantum mechanics, is therefore critical to benchmarking performance and characterizing their operation. While classical computational approaches cannot perform like-for-like computations of quantum systems beyond a certain scale, classical high-performance computing (HPC) may nevertheless be useful for precisely these characterization and certification tasks. By developing open-source customized algorithms using high-performance computing, we perform quantum tomography on a megascale quantum photonic detector covering a Hilbert space of 106. This requires finding 108 elements of the matrix corresponding to the positive operator valued measure (POVM), the quantum description of the detector, and is achieved in minutes of computation time. Moreover, by exploiting the structure of the problem, we achieve highly efficient parallel scaling, paving the way for quantum objects up to a system size of 1012 elements to be reconstructed using this method. In general, this shows that a consistent quantum mechanical description of quantum phenomena is applicable at everyday scales. More concretely, this enables the reconstruction of large-scale quantum sources, processes and detectors used in computation and sampling tasks, which may be necessary to prove their nonclassical character or quantum computational advantage.

Nanorobot Swarms Made with Laser-Induced Graphene@Fe3O4 Nanoparticles with Controllable Morphology for Targeted Drug Delivery.

Magnetic nanorobot swarms can mimic group behaviors in nature and can be flexibly controlled by programmable magnetic fields, thereby having great potential in various applications. This paper presents a novel approach for the rapid and large-scale processing of laser-induced graphene (LIG) @Fe3O4-based-nanorobot swarms utilizing one-step UV laser processing technology. The swarm is capable of forming a variety of reversible morphologies under the magnetic field, including vortex-like and strip-like, as well as the interconversion of these, demonstrating high levels of controllability and flexibility. Moreover, the maximum forward motion speed of the nanorobot swarm is up to 2165 μm/s, and the drug loading and release ability of such a nanorobot swarm is enhanced about 50 times due to the presence of graphene, enabling the nanorobot swarm to show rapid and precise targeted drug delivery. Importantly, by controllable morphology transformation to conform to the complicated requirements for the magnetic field, the drug-loaded swarm can smoothly pass through a width-varying zigzag channel while maintaining 96% of the initial drug-loading, demonstrating that LIG @Fe3O4 NPs-based nanorobot swarm can provide effective and controllable targeted drug delivery in complex passages.

From Polar Day to Polar Night: A Comprehensive Sun and Star Photometer Study of Trends in Arctic Aerosol Properties in Ny-Ålesund, Svalbard

The climate impact of Arctic aerosols, like the Arctic Haze, and their origin are not fully understood. Therefore, long-term aerosol observations in the Arctic are performed. In this study, we present a homogenised data set from a sun and star photometer operated in the European Arctic, in Ny-Ålesund, Svalbard, of the 20 years from 2004–2023. Due to polar day and polar night, it is crucial to use observations of both instruments. Their data is evaluated in the same way and follows the cloud-screening procedure of AERONET. Additionally, an improved method for the calibration of the star photometer is presented. We found out, that autumn and winter are generally more polluted and have larger particles than summer. While the monthly median Aerosol Optical Depth (AOD) decreases in spring, the AOD increases significantly in autumn. A clear signal of large particles during the Arctic Haze can not be distinguished from large aerosols in winter. With autocorrelation analysis, we found that AOD events usually occur with a duration of several hours. We also compared AOD events with large-scale processes, like large-scale oscillation patterns, sea ice, weather conditions, or wildfires in the Northern Hemisphere but did not find one single cause that clearly determines the Arctic AOD. Therefore the observed optical depth is a superposition of different aerosol sources.

Inhibition of polyphenol oxidase for preventing browning in edible mushrooms: A review.

Edible mushrooms are rich in nutrients and bioactive compounds, but their browning affects their quality and commercial value. This article reviews various methods to inhibit polyphenol oxidase (PPO)-induced browning in mushrooms. Physical methods such as heat treatment, low temperatures, irradiation, and ultrasound effectively reduce PPO activity but may affect mushroom texture and flavor. Chemical inhibitors, including synthetic chemicals and natural plant extracts, provide effective PPO inhibition but require careful monitoring of their content. Biological methods, including gene editing and microbial fermentation, show promise in targeting PPO genes and enhancing antioxidant production. Combining these methods offers a comprehensive strategy for preserving mushroom quality, extending shelf life, and maintaining nutritional value. PRACTICAL APPLICATION: These approaches can be applied in the food industry to improve post-harvest mushroom preservation, enhance product quality, and reduce waste, benefiting both producers and consumers. Further research and innovation are needed to optimize the practical application of these methods in large-scale processing and storage conditions.

Industrially induced warming triggered synchronous intensity changes in the East Asian summer and winter monsoons

Anomalies in the East Asian monsoon system significantly disrupt the densely populated East Asian region, underscoring the importance of understanding such changes and improving current predictive abilities. On the basis of instrumental records, previous studies have shown that the East Asian winter monsoon (EAWM) and summer monsoon (EASM) interact with each other. However, owing to the lack of long-term high-resolution EAWM records, it remains unclear how and whether human activity has affected the phase relationship between the EASM and EAWM since the Industrial Revolution. In this study, we present a precisely dated high-resolution EAWM record for the last 300 years from a crater lake in northeastern China. Our results indicate that the EAWM intensity was relatively weak and fluctuated significantly between 1700 and 1850 CE. After 1850 CE, the EAWM strengthened rapidly and exceeded its intensity observed at the end of the Little Ice Age. In addition, a comparison of our reconstructed EAWM record with the published EASM record clearly shows in-phase variations during the Current Warm Period. We concluded that the climatic effects of industrially induced warming enhance the EAWM by slowing the Atlantic meridional overturning circulation and increasing the meridional temperature gradient, while also strengthening the EASM by increasing hemispheric meridional gradients and affecting other large-scale processes. Under the sustained intensification of human activity, the EASM and EAWM are likely to continue exhibiting synchronous variations in the future.

The field-enriched finite element method with gravity effects for simulating the cracking behaviors of large-scale engineering rock masses

The field-enriched finite element method uses a scalar field defined as a field variable to describe cracks and characterize their impact on the displacement field and stress field of the solution model. It is capable of avoiding remeshing and employing level set functions to describe cracks when simulating the propagation of cracks. In this work, a field-enriched finite element model with gravity effects is proposed to simulate the large-scale failure process of engineering rock masses, and several numerical cases of geotechnical engineering are successfully analyzed. First, by introducing the unified tensile fracture criterion into the numerical model, the large-scale failure process of the intact slope is simulated. Second, the sliding process of rock slopes containing en echelon joints is numerically investigated. Third, the cracking process of the concrete dam is analyzed. Finally, the effects of joint and bedding plane inclination angles on the stability of tunnel chamber in transversely isotropic rock mass are studied. The numerical results indicate that the numerical method proposed in this work can accurately solve the large-scale failure process of rock masses.

Scheme of preparing cluster states with cat qubits

Abstract Cluster states are essential quantum resources for one-way quantum computations and quantum networks. The reliable generation of cluster states in specific quantum systems is crucial for initializing complex quantum operations. In this paper, we introduce an efficient scheme for the deterministic preparation of a cluster state via circuit quantum electrodynamics (QED). Our scheme involves four individual microwave resonators, each of which is coupled to a superconducting transmon qutrit. We demonstrated that a four-cqubit cluster state can be achieved using three controlled-phase gate operations. The cluster state is prepared deterministically, eliminating the need for measurement-based feedback. Throughout these operations, the qutrit remains in its ground state, effectively minimizing decoherence from the qutrit. Numerical simulations suggest that our scheme can generate high-fidelity cluster states using current-circuit QED technology. We believe that our model will facilitate exploration of future large-scale continuous-variable quantum information processing systems.

Optimizing Hadoop Data Locality: Performance Enhancement Strategies in Heterogeneous Computing Environments

As organizations increasingly harness big data for analytics and decision-making, the efficient processing of massive datasets becomes paramount. Hadoop, a widely adopted distributed computing framework, excels in processing large-scale data. However, its performance is contingent on effective data locality, which becomes challenging in heterogeneous computing environments comprising diverse hardware resources. This research addresses the imperative of enhancing Hadoop’s data locality performance in heterogeneous computing environments. The study explores strategies to optimize data placement and task scheduling, considering the diverse characteristics of nodes within the infrastructure. Through a comprehensive analysis of Hadoop’s data locality algorithms and their impact on performance, this work proposes novel approaches to mitigate challenges associated with disparate hardware capabilities. Weighted Extreme Learning Machine Technique (Weighted ELM) with the Firefly Algorithm (WELM-FF) is used in the proposed work. The integration of Weighted Extreme Learning Machine (WELM) with the Firefly Algorithm holds promise for enhancing machine learning models in the context of large-scale data processing. The research employs a combination of theoretical analysis and practical experiments to evaluate the effectiveness of the proposed enhancements. Factors such as network latency, disk I/O, and CPU capabilities are taken into account to develop a holistic framework for improving data locality and, consequently, overall Hadoop performance. The findings presented in this study contribute valuable insights to the field of distributed computing, offering practical recommendations for organizations seeking to maximize the efficiency of their Hadoop deployments in heterogeneous computing environments. By addressing the intricacies of data locality, this research strives to enhance the scalability and performance of Hadoop clusters, thereby facilitating more effective utilization of big data resources.

Data-Driven Fault Classification in Large-Scale Industrial Processes Using Reduced Number of Process Variables

Data-Driven Fault Classification in Large-Scale Industrial Processes Using Reduced Number of Process Variables

Differences in the impact of intense ENSO+ in Bahia Magdalena (SW of Baja California, Mexico) in the context of climate change

The Bahia Magdalena-Bahia Almejas (BM-BA) complex is the largest coastal lagoon system in the Mexican Pacific; it is in a transitional zone between temperate and tropical regions. BM-BA has a high biodiversity and is an important source of fishing resources; however, its productivity diminishes during ENSO+ events. This work aimed to compare the impact of the three most intense ENSOs (1982-1983, 1997-1998, and 2015-2016) on their environmental and biological characteristics in the context of global warming. Since 1970, this region has been warming, and the chlorophyll-a is diminishing; however, short-term environmental variations associated with basin-scale processes have modified these trends. The comparison showed that the equatorial influence on the 1997-1998 ENSO was more evident, while the North Pacific conditions and other large-scale environmental processes influenced the other ENSOs: in 1982-1983, the warm signal continued after the ENSO+ ended, and in 2015-2016, it was present before the ENSO+ onset due to the MWH The Blob and probably global warming. High temperatures at the basin scale altered large-scale wind patterns, weakening local upwelling and diminishing the nutrient input. The Equatorial and North Pacific influence on BM varies over time (as expected in a transition zone like BM); however, in recent years, North Pacific warming associated with climate change has probably gained relevance and modulated the effect of ENSO+.

Estimation of aerodynamic entrainment in developing wind-blown sand flow.

Understanding aerodynamic entrainment, a critical process in wind-blown sand dynamics, remains challenging due to the difficulty of isolating it from other mechanisms, such as impact entrainment. Aerodynamic entrainment initiates the movement of surface particles, influencing large-scale processes like sediment transport and dune formation. Previous studies focused on average aerodynamic shear stress to estimate entrainment, but the role of impulse events, which cause significant shear stress fluctuations, remains under-explored. We used 12 hot-film shear sensors to measure the spatiotemporal distribution of aerodynamic shear stress during wind-blown sand flow development. We identified impulse events exceeding the entrainment threshold and analyzed their intensity, classifying particle movement as rocking, rolling, or saltation. Results indicate that after a 2-m fetch, sediment mass flux stabilizes, with aerodynamic shear stress decreasing to 78% of the entrainment threshold. We identified key trends, including the stabilization of rocking events beyond x = 4.5 m and a significant decrease in saltation frequency, indicating fully developed wind-blown sand flow. Impulse characteristics stabilize at a greater distance (4.5 m) than sediment transport (2 m) because turbulent airflow evolves more slowly. Our findings show that impulse events significantly influence aerodynamic entrainment. These insights enhance understanding of sediment transport dynamics and improve modeling of sand dune movement.

Enhancing deep stochastic configuration networks: Efficient training via low-rank matrix approximation

Deep Stochastic Configuration Networks (DeepSCN) represent a class of randomized learner models distinguished by their universal approximation capabilities, rapid learning processes, and ease of implementation. These models surpass other randomized learners due to an embedded supervisory mechanism. Although the feature matrix is meticulously constructed, the predictive accuracy of DeepSCN also critically depends on the quality of the model parameters obtained from solving the corresponding system of linear equations. As the number of hidden neurons increases, the resulting expansion of the feature matrix renders matrix inversion computationally expensive, potentially unstable, and memory-intensive. This study addresses these challenges by integrating randomized algorithm-based low-rank matrix approximation to compute model parameters efficiently. A theoretical error bound for the obtained model parameters is derived, ensuring precision and accuracy. The effectiveness of the algorithm is validated on benchmark datasets for both classification and regression tasks. The results demonstrate that this integration significantly enhances the reliability and stability of DeepSCN, improving performance and scalability while greatly reducing computational demands. The method maintains stability despite substantial variations in the number of hidden neurons and mitigates overfitting caused by matrix inversion, ensuring stable and reliable training outcomes. This study advances DeepSCN training methodologies and large-scale data processing.

Efficient Task Scheduling for Large-scale Graph Data Processing in Cloud Computing: A Particle Swarm Optimization Approach

To fulfil the requirements of task scheduling for processing massive amounts of graph data in cloud computing environments, this thesis offers the LGPPSO method, which is based on Particle Swarm Optimisation. The LGPPSO algorithm considers the task’s overall structure when initialising numerous particles in order to broaden the search range and raise the likelihood of finding an approximation optimal solution. We evaluated it in large-scale simulation trials with 100 performance-heterogeneous virtual machines (VMs) using both randomly generated and real application graphs, and evaluated its effectiveness against the CCSH and HEFT algorithms. The experimental findings demonstrate that, in both randomly generated graphs and real graph structure applications, significantly reducing the scheduling duration of large-scale graph data is the LGPPSO algorithm. For randomly generated 200 and 400 node tasks, respectively, the scheduling length is shortened by approximately 8.3% and 9.7% on average when compared to the HEFT algorithm. The LGPPSO technique minimises the scheduling length for actual graphical structure applications by an average of 14.6% and 16.9% for the Gaussian 100 and 200 node examples, respectively.

Roll-to-Roll Manufactured Polyetherimide Nanocomposite With Different Diameters of SiO2 Nanoparticles Exhibiting Improved High-Temperature Dielectric Energy Storage Performance.

To enhance the high-temperature energy storage performance of the polymer-based dielectric film, inorganic nanofillers with large band gaps are much more effective and have been widely adopted. However, the impact of nanoparticle diameters on the dielectric properties of polymer nanocomposites has been less studied. Herein, silicon dioxide nanoparticles (SiO2-NPs)with varying diameters (20, 60, 120, 200nm) prepared by the sol-gel method are incorporated in the PEI matrix to form PEI/SiO2 nanocomposites. The characterization results reveal a distinct correlation between the dielectric properties of polyetherimide (PEI) composites and the diameters of SiO2-NPs. Leakage current density analysis and breakdown strength simulations indicate that SiO2-NPs with smaller diameters generate more deep traps that impede the transport of charge carriers, especially under high temperatures. Notably, PEI/20nm-SiO2 exhibits a high discharged energy density of 4.4 J cm-3 with an efficiency of 90% at 150°C. Furthermore, PEI/SiO2 films with 10µm in thickness are manufactured by a large-scale solution casting process. The continuously prepared PEI/20nm-SiO2 film exhibits a discharged energy density of 3.2 J cm-3 with an efficiency of 90% at 150°C. This study not only provides a strategy for the design of high-performance dielectric polymer composites but also offers a large-scale high-temperature dielectric film for practical use.

RNF20-mediated transcriptional pausing and VEGFA splicing orchestrate vessel growth

Signal-responsive gene expression is essential for vascular development, yet the mechanisms integrating signaling inputs with transcriptional activities are largely unknown. Here we show that RNF20, the primary E3 ubiquitin ligase for histone H2B, plays a multifaceted role in sprouting angiogenesis. RNF20 mediates RNA polymerase (Pol II) promoter-proximal pausing at genes highly paused in endothelial cells, involved in VEGFA signaling, stress response, cell cycle control and mRNA splicing. It also orchestrates large-scale mRNA processing events that alter the bioavailability and function of critical pro-angiogenic factors, such as VEGFA. Mechanistically, RNF20 restricts ERG-dependent Pol II pause release at highly paused genes while binding to Notch1 to promote H2B monoubiquitination at Notch target genes and Notch-dependent gene expression. This balance is crucial, as loss of Rnf20 leads to uncontrolled tip cell specification. Our findings highlight the pivotal role of RNF20 in regulating VEGF–Notch signaling circuits during vessel growth, underscoring its potential for therapeutic modulation of angiogenesis.