Massive Data Processing Research Articles

Massive Point Cloud Processing for Efficient Construction Quality Inspection and Control

The construction of large-scale civil infrastructures requires massive spatiotemporal data to support the management and control of scheduling, quality control, and safety monitoring. Existing artificial-intelligence-based data processing algorithms rely heavily on experienced engineers to adjust the parameters of data processing, which is inefficient and time-consuming when dealing with huge datasets. Limited studies have compared the performance of different algorithms on a unified dataset. This study proposes a framework and evaluation system for comparing different data processing policies for processing huge spatiotemporal data in construction quality control. The proposed method compares the combination of multiple types of algorithms involved in the processing of massive point cloud data. The performance of data processing strategies is evaluated through this framework, and the optimal point cloud processing strategies are explored based on registration accuracy and data fidelity. Results show that a reasonable choice of combinations of point cloud sampling, filtering, and registration algorithms can significantly improve the efficiency of point cloud data processing and satisfy engineering demands for data accuracy and completeness. The proposed method can be applied to the civil engineering problem of processing a large amount of point cloud data and selecting the optimal processing method.

Design of Variation Tolerant Near Threshold Processor Using Artificial Ecosystem Optimizer with Hybrid Deep Learning

Recently, several applications of data mining and pattern recognition require statistical signal processing (SP) to be used and machine learning (ML) techniques for processing massive data volumes in energy-constrained contexts. It is developing interest in executing difficult ML techniques like convolutional neural network (CNN) on lesser power embedding environments to allow on-device learning and inference. Several of these platforms are that utilized as lower power sensor nodes with lower to medium throughput conditions. Near threshold processor (NT) proposals are appropriate for these applications where as affected by a vital enhancement in variants. This research offers an Artificial Ecosystem Optimizer with Hybrid Deep Learning for Variation-Tolerant Near-Threshold Processor (AEOHDL-VTNT). The inference of embedded systems at the network edge serves as the foundation for the AEOHDL-VTNT approach that is being discussed. In the described AEOHDL-VTNT approach involves two primary processes: HDL-based VTNT design and hyper parameter tweaking. Initially, the HDL model is used to develop the VTNT. Next, in the second step, the AEO method is used for hyper parameter tweaking of the HDL model, which improves the HDL's overall performance. A number of simulations were carried out to show how the AEOHDL-VTNT approach improved performance. The simulation results showed that the AEOHDL-VTNT approach outperformed other models.

Optimization and performance improvement of distributed data storage in hybrid storage systems

With the rapid development of information technology, the storage and processing of massive data has become one of the important challenges facing the current computing field. Traditional centralized storage systems have been unable to meet the needs of big data applications, while distributed storage systems have become the infrastructure of modern data centers because of their high scalability and high availability. However, in practical applications, a single distributed storage model is often difficult to balance cost-effectiveness and performance requirements. Therefore, this paper proposes a hybrid storage system model combining local storage and cloud storage resources and discusses a series of optimization strategies for this model. The focus of the research is to improve the overall performance of the system through the design of intelligent data sharding, redundancy and fault tolerance mechanisms, and the application of effective load balancing technology. The effectiveness and superiority of the proposed method are verified by testing and analyzing several typical application scenarios on the experimental platform. The experimental results show that the optimized hybrid storage system not only significantly improves the speed of data access, but also effectively reduces the storage cost, demonstrating its potential in future large-scale data management.

Industrial IoT-Based Energy Monitoring System: Using Data Processing at Edge

Edge-assisted IoT technologies combined with conventional industrial processes help evolve diverse applications under the Industrial IoT (IIoT) and Industry 4.0 era by bringing cloud computing technologies near the hardware. The resulting innovations offer intelligent management of the industrial ecosystems, focusing on increasing productivity and reducing running costs by processing massive data locally. In this research, we design, develop, and implement an IIoT and edge-based system to monitor the energy consumption of a factory floor’s stationary and mobile assets using wireless and wired energy meters. Once the edge receives the meter’s data, it stores the information in the database server, followed by the data processing method to find nine additional analytical parameters. The edge also provides a master user interface (UI) for comparative analysis and individual UI for in-depth energy usage insights, followed by activity and inactivity alarms and daily reporting features via email. Moreover, the edge uses a data-filtering technique to send a single wireless meter’s data to the cloud for remote energy and alarm monitoring per project scope. Based on the evaluation, the edge server efficiently processes the data with an average CPU utilization of up to 5.58% while avoiding measurement errors due to random power failures throughout the day.

Reconfigurable Multilevel Storage and Neuromorphic Computing Based on Multilayer Phase-Change Memory.

In the era of big data, the amount of global data is increasing exponentially, and the storage and processing of massive data put forward higher requirements for memory. To deal with this challenge, high-density memory and neuromorphic computing have been widely investigated. Here, a gradient-doped multilayer phase-change memory with two-level states, four-level states, and linear conductance evolution using different pulse operations is proposed. The mechanism of multilevel states is revealed through high-resolution transmission electron microscopy (HRTEM) and finite-element analysis (FEA), which show that the sequential phase change among different sublayers is realized due to the different physical properties of the sublayers with different doping concentrations. Taking advantage of the devices' linear conductance evolution characteristic, a handwritten digit (28 × 28 pixel) recognition task is implemented with a high learning accuracy of 93.46% by building a simulated artificial neural network made up of this gradient-doped multilayer phase-change memory. It is proved that this gradient-doped multilayer phase-change memory is capable of both binary multilevel digital storage and brain-inspired analog in-memory computing in the same device, enabling reconfigurable applications in the future.

Research on monitoring and energy management systems for energy storage stations on the power generation side

Abstract Addressing the challenges of limited operational efficiency, inadequate battery safety, inflexible system scalability, inadequate AGC/AVC functionality, and low levels of intelligent operation and maintenance encountered by energy storage power stations integrated with new energy generation sources, this article delves into the pivotal role of these power stations on the generation side. It meticulously examines the network architecture, functional framework, and pivotal technologies underpinning the system. Furthermore, the article proposes a comprehensive monitoring and energy management system tailored specifically for energy storage on the generation side. The system uses micro-service architecture and container technology to realize the processing and storage of massive data. Through complete isolation of the information network and control network and millisecond communication with GOOSE, the charging and discharging response speed of the energy storage station is greatly improved. The power allocation algorithm based on device status recognition completes the system’s high-precision power control of PCS, and this system uses big data analysis and data mining technology to conduct a comprehensive statistical analysis of battery health status, power station operation efficiency, and equipment operation status and energy consumption. The design of the system has favorable technical and application guidance for the subsequent energy management design and engineering application of the energy storage power station.

Dual-polarized transmissive metasurfaces for near-unitary-NA analog spatial computing empowered by impedance matching and mismatching.

In recent years, due to the increasing requirement for real-time and massive data processing, optical analog computation has arisen as a promising alternative to digital computation. Optical spatial differentiation plays a fundamentally important role in various emerging technologies, including augmented reality, autonomous driving, and object recognition. However, previous demonstrations encountered several limitations, such as the dependence on polarization states and a typically limited numerical aperture (NA) of about 0.5, especially in the transmission mode. Here, a new, to our knowledge, design strategy based on the evolution between impedance matching and mismatching in a metasurface is proposed to fill this gap, which can perform dual-polarized second-order derivative for image processing. Our scheme enables high transmission under dual polarization over an 85° incident angle range (NA = 0.996), resulting in more than twofold spatial resolution. Our work paves the way for polarization-insensitive high-resolution signal and image processing in the terahertz region.

3D modelling method and application to a digital campus by fusing point cloud data and image data

ObjectiveThe use of single-source data for real-world 3D modelling currently faces problems such as deformation, pulling and fuzzy texture at the bottom of buildings in some feature models because of the lack of images. Moreover, LIDAR generates a huge amount of data, and the massive raw data processing and point cloud parsing puts high demands on the hardware arithmetic and algorithms. Aiming at the deficiencies and defects of the two data sources of inclined photogrammetry and airborne laser point cloud in the construction of high-quality and high-precision city-level 3D models. Methodsthis study uses a university library building as an example and proposes the main technical process and method of modelling after fusing the point cloud data acquired by inclined photogrammetry and 3D laser scanning technology. This is accomplished in the reconstruction stage of multi-source data fusion through data spatial alignment, coordinate system unification and data spatial integration. At the stage of multi-source data fusion and reconstruction, through data spatial alignment, coordinate system unification, point cloud coarse alignment and the iterative closest point (ICP) algorithm, a realistic 3D model of a building is constructed to verify the effectiveness of the modelling method. ResultsThe method can effectively improve the accuracy of the real-life 3D model, repair the deficiencies in the model and optimise the details of the model. It can also significantly improve the fineness of the tilt photography model and perfectly present the geometric and texture information of the building, making it a superior method for fine 3D reconstruction. ConclusionThis 3D reconstruction method of buildings, which integrates low-altitude inclined photogrammetry and airborne light detection and ranging (LiDAR), has high positional accuracy and can provide new methods and new ideas for the construction of digital campuses as well as for other engineering applications.

Diagnose Breast Cancer Using Two Machine Learning Methods andComparing and Analyzing Them

The breast cancer is currently the number one cancer in the world and has a high early treatment rate. Therefore, early screening and diagnosis of breast cancer is very important. Machine learning is very strong in analyzing and processing massive data. In this paper, we train and optimize decision tree and SVM models to identify malignant breast cancer and compare and analyze the performance of the two models. The accuracy of the earliest optimized models reaches about 98%, and it is found that the two models focus on different classification effects for the two samples, which can be further optimized by stacking and other ways. The dataset used to train the models in this paper is from the Wisconsin breast cancer database (WBCD), a benchmark dataset commonly used to compare different algorithms, which is of some significance for exploring the use of machine learning in the field of medical diagnosis.

A Deep Learning Estimation for Probing Depth of Transient Electromagnetic Observation

The probing depth of the transient electromagnetic method (TEM) refers to the depth range at which the underground conductivity changes can be effectively detected. It typically ranges from tens of meters to several kilometers and is influenced by factors such as instrument parameters and the conductivity of the subsurface structure. Rapid and accurate probing depth is useful for the selection of appropriate inversion parameters and improving survey accuracy. However, mainstream methods suffer from issues such as low computational precision, large uncertainties, or high computational requirements, making them unsuitable for processing massive airborne electromagnetic data. In this study, we propose a prediction model based on deep learning that can directly compute the probing depth from the TEM responses, and its effectiveness and accuracy are validated through synthetic models and field measurements. We compared the performance of classic deep learning models, including CNN, RESNET, and RNN, and found that RNN performed the best overall on both synthetic and field data. Furthermore, we apply this algorithm to deep learning-based ATEM inversion by constraining the one-dimensional resistivity model depths in the training set, to reduce the non-uniqueness of the inversion, accelerate the convergence, and improve its prediction accuracy.

Examining air pollution exposure dynamics in disadvantaged communities through high-resolution mapping.

This study bridges gaps in air pollution research by examining exposure dynamics in disadvantaged communities. Using cutting-edge machine learning and massive data processing, we produced high-resolution (100 meters) daily air pollution maps for nitrogen dioxide (NO2), fine particulate matter (PM2.5), and ozone (O3) across California for 2012-2019. Our findings revealed opposite spatial patterns of NO2 and PM2.5 to that of O3. We also identified consistent, higher pollutant exposure for disadvantaged communities from 2012 to 2019, although the most disadvantaged communities saw the largest NO2 and PM2.5 reductions and the advantaged neighborhoods experienced greatest rising O3 concentrations. Further, day-to-day exposure variations decreased for NO2 and O3. The disparity in NO2 exposure decreased, while it persisted for O3. In addition, PM2.5 showed increased day-to-day variations across all communities due to the increase in wildfire frequency and intensity, particularly affecting advantaged suburban and rural communities.

A pragmatic data processing system for large resistive sensor arrays.

Large resistive sensor arrays (RSAs) show great potential in tactile perception. However, the large number of sensors can result in great hardware overhead and bring difficulties for acquiring and processing mass data timely in transient measurement applications. This paper implements a field programmable gate array (FPGA)-based data processing system for a large RSA of 96 × 96, which shows good power consumption and high-speed wireless data update. For crosstalk-free measure, the zero potential method is improved with bus switches, leading to fewer operational amplifiers required and less negative power consumption. A real-time embedded data processing system is realized by FPGA for excellent parallel processing ability. A high-speed wireless transfer scheme with automatic regulated transfer size is proposed and realized by a wireless fidelity module, which allows timely data analysis at the remote end. Moreover, fault identification of RSAs fabricated by micro-electromechanical system technology is achieved. Tests carried out on a 32 × 32 RSA show that the total power consumption is 2209 mW, including 1261 mW of processors and 948 mW of readout circuits, corresponding to 2.15 mW/pixel. The total negative power consumption of 549 mW has been reduced by 50% compared with the zero potential method. The scanning speed is 400fps, and the wireless transfer speed is up to 120fps when the transceiver and receiver are 5m apart.

Reinforcement-Learning-Assisted Service Function Chain Embedding Algorithm in Edge Computing Networks

Edge computing networks are critical infrastructures for processing massive data and providing instantaneous services. However, how to efficiently allocate resources in edge computing networks to meet the embedding requirements of service function chains has become an urgent problem. In this paper, we model the resource allocation problem in edge computing networks as a service function chain embedding problem model, aiming to optimize the resource allocation through reinforcement learning algorithms to achieve the goals of low cost, high revenue, and high embedding rate. In this paper, the basic concepts of edge computing network and service function chain are elaborated, and the resource allocation problem is transformed into a service function chain embedding problem by establishing a mathematical model, which provides a foundation for the subsequent algorithm design. In this paper, a service function chain embedding algorithm based on reinforcement learning is designed to gradually optimize the resource allocation decision by simulating the learning process. In order to verify the effectiveness of the algorithm, a series of simulation experiments are conducted in this paper and compared with other algorithms. The experimental results show that the service function chain embedding algorithm based on reinforcement learning proposed in this paper exhibits superior performance in resource allocation. Compared with traditional resource allocation methods, the algorithm achieves significant improvement in terms of low cost, high revenue, and high embedding rate.

A New Reversible Data Hiding Method Using a Proportional Relation between Peak Signal-to-Noise Ratio and Embedding Capacity on Convolutional Neural Network

In the rapidly evolving fields of artificial intelligence and various industries, the secure processing and management of massive data have become paramount. This paper introduces an innovative reversible data hiding (RDH) method that leverages a Convolutional Neural Network (CNN)-based predictor to generate a predicted image from a given cover image. The secret data are ingeniously embedded within the differences in pixel values between the cover and predicted images. Our experimental analysis reveals a notable reduction in image distortion with increasing secret data size, showcasing the method’s potential for diverse applications. The unique aspect of our approach lies in the proportional relation between the Peak Signal-to-Noise Ratio (PSNR) and Embedding Capacity, highlighting its efficacy and efficiency in reversible data hiding.

Emergence of high-mass stars in complex fiber networks (EMERGE)

Context. Recent molecular surveys have revealed the rich gas organization of sonic-like filaments at small scales (so-called fibers) in all types of environments prior to the formation of low- and high-mass stars. These fibers form at the end of the turbulent cascade and are identified as the fine substructure within the hierarchical nature of the gas in the interstellar medium (ISM). Aims. Isolated fibers provide the subsonic conditions for the formation of low-mass stars. This paper introduces the Emergence of high-mass stars in complex fiber networks (EMERGE) project, which investigates whether complex fiber arrangements (networks) can also explain the origin of high-mass stars and clusters. Methods. We analyzed the EMERGE Early ALMA Survey including seven star-forming regions in Orion (OMC-1,2,3, and 4 South, LDN 1641N, NGC 2023, and the Flame Nebula) that were homogeneously surveyed in three molecular lines (N2H+ J = 1–0, HNC J = 1–0, and HC3N J = 10–9) and in the 3 mm continuum using a combination of interferometric ALMA mosaics and IRAM-30 m single-dish (SD) maps, together with a series of Herschel, Spitzer, and WISE archival data. We also developed a systematic data reduction framework allowing the massive data processing of ALMA observations. Results. We obtained independent continuum maps and spectral cubes for all our targets and molecular lines at different (SD and interferometric) resolutions, and we explored multiple data combination techniques. Based on our low-resolution (SD) observations (30″ or ~12 000 au), we describe the global properties of our sample, which covers a wide range of physical conditions, including low-(OMC-4 South and NGC 2023), intermediate (OMC-2, OMC-3, and LDN 1641N), and high-mass (OMC-1 and Flame Nebula) star-forming regions in different evolutionary stages. The comparison between our single-dish maps and ancillary YSO catalogs denotes N2H+ (1–0) as the best proxy for the dense, star-forming gas in our targets, which show a constant star formation efficiency and a fast time evolution of ≲1 Myr. While apparently clumpy and filamentary in our SD data, all targets show a much more complex fibrous substructure at the enhanced resolution of our combined ALMA+IRAM-30 m maps (4″.5 or ~2000 au). A large number of filamentary features at subparsec scales are clearly recognized in the high-density gas (≳ 105 cm−3) that is traced by N2H+ (1–0) directly connected to the formation of individual protostars. Surprisingly, this complex gas organization appears to extend farther into the more diffuse gas (~103−104 cm−3) traced by HNC (1–0). Conclusions. This paper presents the EMERGE Early ALMA Survey, which includes a first data release of continuum maps and spectral products for this project that are to be analysed in future papers of this series. A first look at these results illustrates the need of advanced data combination techniques between high-resolution interferometric (ALMA) and high-sensitivity, low-resolution single-dish (IRAM-30 m) datasets to investigate the intrinsic multiscale, gas structure of the ISM.

A Real-Time Resource Dispatch Approach for Edge Computing Devices in Digital Distribution Networks Considering Burst Tasks

Edge computing technology can effectively solve huge challenges posed by the large number of terminal devices accessing and massive data processing in digital distribution networks. Burst tasks, such as faults and data requests from the cloud, can occur at any time for edge computing devices in distribution networks. These tasks are unpredictable and usually hold the highest priority and must be completed as soon as possible. Although resources can be reserved partially at each period in the pre-scheduled operation plan, they may still be insufficient to handle burst tasks adequately. A real-time resource dispatch approach for burst tasks is developed in this study to address the above problems. The concept of flexibility for edge computing devices is presented, determining the real-time dispatch duration. Real-time resource dispatch and task handling processing are analyzed in detail, considered as task real-time dispatch models, computation process real-time dispatch constraints, and resource limitation constraints. The proposed real-time resource dispatch approach takes full advantage of the transferable characteristics for partial original plan tasks to adjust the pre-scheduled operation plan and release flexible resources for immediate processing of the burst task, completing burst tasks quickly and minimizing the impact for previous planned tasks on the edge computing device. The capability of the proposed method to efficiently deal with the burst tasks is also verified by the case study.

Retrospective Analysis of Glacial Lake Outburst Flood (GLOF) Using AI Earth InSAR and Optical Images: A Case Study of South Lhonak Lake, Sikkim

On 4 October 2023, a glacier lake outburst flood (GLOF) occurred at South Lhonak Lake in the northwest of Sikkim, India, posing a severe threat to downstream lives and property. Given the serious consequences of GLOFs, understanding their triggering factors is urgent. This paper conducts a comprehensive analysis of optical imagery and InSAR deformation results to study changes in the surrounding surface of the glacial lake before and after the GLOF event. To expedite the processing of massive InSAR data, an InSAR processing system based on the SBAS-InSAR data processing flow and the AI Earth cloud platform was developed. Sentinel-1 SAR images spanning from January 2021 to March 2024 were used to calculate surface deformation velocity. The evolution of the lake area and surface variations in the landslide area were observed using optical images. The results reveal a significant deformation area within the moraine encircling the lake before the GLOF, aligning with the area where the landslide ultimately occurred. Further research suggests a certain correlation between InSAR deformation results and multiple factors, such as rainfall, lake area, and slope. We speculate that heavy rainfall triggering landslides in the moraine may have contributed to breaching the moraine dam and causing the GLOF. Although the landslide region is relatively stable overall, the presence of a crack in the toparea of landslide raises concerns about potential secondary landslides. Our study may improve GLOF risk assessment and management, thereby mitigating or preventing their hazards.

Management information platform based on Internet of Things

In order to study the financial management mode and innovation of electric power enterprises under the current situation, a research method of electric power financial management innovation in the era of Internet of Things is proposed. According to the network topology of the system and the analysis of the Android system, the client of the financial management system, the main functional modules of the system and the data warehouse are designed. The client of the financial management system will divide the client into an information interaction unit and a data calculation unit according to the actual needs of the client; the main functional modules of the system are divided into account management unit, salary management unit, project management unit, system management unit, and fixed asset management unit. Unit, revenue and expenditure management unit and report management unit, the sub-units of each unit are divided into brief analysis, and the outpatient registration fee management module and inpatient fee management module of the revenue and expenditure management unit are introduced in detail; in the hospital financial data warehouse, in the application layer An interactive platform is built between it and the data warehouse, and the module is integrated into the existing data warehouse to realize the processing, storage and operation of massive data. The proposed system has a large operating safety factor, high user satisfaction, and an average system flexibility factor of 0.94. The power financial management information platform based on the Internet of Things has a very good performance in the application. Therefore, the method in this paper can enhance the work efficiency of electric power enterprises and enhance the core competitiveness of electric power enterprises.

Spatiotemporal self-supervised predictive learning for atmospheric variable prediction via multi-group multi-attention

Atmospheric variable prediction (AVP) is crucial for atmospheric and environmental science, as well as practical applications related to production and daily life. Traditional AVP commonly relies on numerical modeling methods. In recent years, deep learning (DL) technologies have introduced new approaches for AVP. With the support of massive high-quality data products, DL methods have achieved remarkable progress. Simultaneously, studying and annotating complex phenomena and processes within vast datasets require specialized researchers and come with substantial costs. The self-supervised learning (SSL) mechanism in DL can facilitate representation learning from unlabeled data, thereby significantly reducing the expenses associated with manual annotation. However, there has been limited exploration of the SSL mechanism in atmospheric variable prediction and analysis. Furthermore, the prevailing large-scale predicting applications based on AVP mainly focus on massive data processing, multi-modal approaches, and foundation model technologies, often neglecting the critical aspects of spatiotemporal representation learning. In light of the above, based on the cutting-edge DL techniques but diverges from the current popular contrastive or generative SSL frameworks, we devise a spatiotemporal (ST) self-supervised predictive learning (SSPL) method (dubbed ST-SSPL) for AVP, which enables the learning of predictive characteristics from unlabeled atmospheric data. Additionally, we improve a multi-group multi-attention (MGMA) method, integrating it into our predictive framework to enhance spatiotemporal representation learning. In experiments conducted on the widely used ERA5 dataset, the proposed ST-SSPL method has achieved superior performance and high efficiency using a simple convolutional neural network (CNN) architecture. These advancements offer crucial methods and references for relevant large-scale applications.

Design and Implementation of AI-based Judicial Big Data Intelligent Service Platform

Based on big data and artificial intelligence technology, the judicial big data intelligent service platform provides users with efficient and accurate legal consulting services. It has many functions such as intelligent question and answer, case analysis, and law inquiry. It also analyzes the application value of the intelligent legal consulting service platform in the field of judicial big data. By building an intelligent service platform, the rapid processing and analysis of massive judicial data has been realized, and the efficiency and quality of judicial work have been significantly improved. In addition, it emphasizes the importance of data management and quality control in the application of judicial big data, and puts forward specific measures to strengthen data management and quality control. With the continuous advancement of technology and the expansion of application fields, the platform is expected to achieve deep integration with other legal services, expand new application fields, and provide users with more comprehensive and efficient legal services. At the same time, it points out the problems that need to be paid attention to in the future development, such as data security, privacy protection and so on. Through in-depth analysis and research on judicial big data, the platform can provide strong support for judicial decision-making and promote the improvement of judicial justice and efficiency