Total Data Volume Research Articles

Research and Implementation of a Classification Method of Industrial Big Data for Security Management

ABSTRACTPurpose/SignificanceWith the extensive adoption of cloud computing, big data, artificial intelligence, the Internet of Things, and other novel information technologies in the industrial field, the data flow in industrial companies is rapidly increasing, leading to an explosion in the total volume of data. Ensuring effective data security has become a critical concern for both national and industrial entities.Method/ProcessTo tackle the challenges of classification management of industrial big data, this study proposed an Information Security Triad Assessment‐Support Vector Machine (AIC‐ASVM) model according to information security principles. Building on national policy requirements, FIPS 199 standards, and the ABC grading method, a comprehensive classification framework for industrial data, termed “two‐layer classification, three‐dimensional grading,” was developed. By integrating the concept of Data Protection Impact Assessment (DPIA) from the GDPR, the classification of large industrial data sets was accomplished using a Support Vector Machine (SVM) algorithm.Result/ConclusionSimulations conducted using MATLAB yielded a classification accuracy of 96.67%. Furthermore, comparisons with decision tree and random forest models demonstrated that AIC‐ASVM outperforms these alternatives, significantly improving the efficiency of big data classification and the quality of security management.

Fog Computing Enabled Adaptive Prognosis of Cutting Tool Remaining Life through Multi-source Data

Abstract Cutting tool remaining life during machining processes is essential for achieving sustainable production. Traditional prognosis methods are often crippled by the inability to adapt to diverse working conditions across the machining process lifecycle. This paper introduces a fog computing-enabled adaptive prognosis framework utilising multi-source data to address these challenges effectively. The key innovations include: (1) The proposed system integrates power and vibration data collected from LGMazak VTC-16A and IRON MAN QM200 machines. A standardised data fusion method combines multi-source data to enhance robustness and accuracy. (2) The transformer model is employed to improve prognosis accuracy of cutting tool remaining life, best accuracy of 98.24% and an average accuracy of 97.63% are achieved. (3) Finite Element Analysis (FEA) is incorporated to validate the model's predictions to validate reliability of deep learning model. (4) The fog computing optimisation mechanism based on the bees algorithm, which shows fitness value of 0.92 and convergence within 15 iterations. The proposed method reduces total data volume in cloud by 54.12%, prediction time by 33.64% and time complexity in the cloud layer by 4.62%. The effectiveness of fog computing in enhancing the operational efficiency and reliability of manufacturing systems is validated through advanced data integration and deep learning techniques.

텍스트 마이닝을 이용한 코로나19 전후 청소년과 식생활 관련 키워드 검색 경향 분석

This study analyzed Naver, Daum, Google, YouTube, and Twitter using TEXTOM for two years and four years as of January 18, 2020. The results are as follows. First, the total number and volume of keyword search data related to youth and diet were slightly higher after COVID-19, showing that interest increased due to COVID-19. Second, as a result of frequency analysis, ‘education’ was the highest before COVID-19, and ‘health’ was the highest after COVID-19, showing that interest in health is increasing due to the increased importance of health and immunity due to COVID-19. Third, as a result of frequency weight analysis of the top 50 keywords, ‘education’ showed the highest frequency before COVID-19, and ‘acne’ after COVID-19. Fourth, the results visualized using word cloud showed that the keywords ‘education’ before COVID-19 and ‘health’ after COVID-19 appeared the largest and boldest, showing the highest frequency and importance. As a result of the above results, we were able to use the text mining method to apply it to eating habits, and we used materials visualized as a highly readable word cloud in units such as eating problems in adolescence and balanced meal planning and selection in the home economics curriculum to improve the teaching of the class. The direction of proper eating habits education, including using it as a medium, was presented.

Understanding Data Access Patterns for dCache System

Understanding Data Access Patterns for dCache System

Construction and Practice of Enterprise Information Security Management System of Intrusion Detection Technology

This article focuses on the construction and practice of enterprise information security management system, and conducts in-depth research and practice on the information security needs of enterprises. For the information security management system of enterprises, this study designs a comprehensive information security management system, aiming to enhance the information security protection capabilities of enterprises. At the same time, to ensure the stable operation of the system, this study has established a corresponding security guarantee system. This study conducted a comprehensive inspection of the information security management system. With a network information volume of 5TB, only 103MB of data was lost under the security supervision of this system, accounting for 5.66% of the total data volume. This data is much lower than the data loss rate of the original system. The amount of information stolen by the original system under the same conditions was as high as 1819MB, accounting for 35.53% of the total information. This result fully proves that the information security management system designed in this article has high security and practicality, and can effectively protect the internal network information security of enterprises.

Distributed intelligence for pervasive optical network telemetry

Optical network automation and failure management require measuring the status and the performance of the different network devices to anticipate any degradation and ensure the quality of the provided services, i.e., optical connectivity. Such pervasive network telemetry entails collecting large amounts of measurements and events from different sources and with very fine granularity, which given the amount and variety of telemetry sources and the size of each measurement and event, imposes requirements that are hard to achieve without large investments. In this paper, we analyze the main limitations of telemetry architectures relying exclusively on centralized systems for data analysis and propose an architecture with distributed intelligence. Data aggregation techniques, especially conceived for optical network telemetry, are presented with the objective of reducing data dimensionality. Illustrative results from our experimental telemetry system reveal a reduction of 3 orders of magnitude in terms of total data volume without introducing significant error and processing delay and, more importantly, helping network automation algorithms to identify meaningful changes in the network status.

Maximizing data gathering and energy efficiency in UAV-assisted IoT: A multi-objective optimization approach

The data volume and energy consumption are important design aspects of data collection in the Internet of Things (IoT). In this paper, the unmanned aerial vehicles (UAVs)-assisted data gathering (DG) in IoT is investigated, in which multiple rotary-wing UAVs are dispatched to gather data from multiple terrestrial IoT devices within specified coverage range. In order to ensure the freshness of the collected data, we must consider that each UAV should spend as little time as possible to collect data. Moreover, it is assumed that UAVs collect data from the devices only when they are hovering, and the transmission distance and on-board energy of the UAVs are limited. Under these practical hypotheses, we formulate a UAV-assisted data gathering multi-objective optimization problem (UAVDGMOP) to simultaneously maximize the total data volume collected by UAVs, minimize the maximum-time for UAVs collecting data and approaching the optimal positions, and minimize the total energy consumptions of UAVs via jointly optimizing the locations of UAVs, the transmission power of UAVs, and the scheduling of UAV–device pairs. Moreover, the formulated UAVDGMOP consists of the hybrid and complex solutions, which is difficult to be solved by using the conventional methods, and thus we solve it by proposing a hybrid update multi-objective evolutionary algorithm based on decomposition (HUMOEA/D). Simulation results demonstrate that the proposed approach achieves higher data amount and energy efficiency of the network, and it is more suitable for solving the formulated UAVDGMOP comparing to other comparative approaches, including the random, uniform and layering deployment approaches, as well as other multi-objective evolutionary algorithms (MOEAs). Moreover, the proposed HUMOEA/D still achieves the overall excellent performance in more use cases.

Knowledge-Aided Federated Learning for Energy-Limited Wireless Networks

The conventional model aggregation-based federated learning (FL) approach requires all local models to have the same architecture, which fails to support practical scenarios with heterogeneous local models. Moreover, the frequent model exchange is costly for resource-limited wireless networks since modern deep neural networks usually have over a million parameters. To tackle these challenges, we first propose a novel knowledge-aided FL (KFL) framework, which aggregates light high-level data features, namely knowledge, in the per-round learning process. This framework allows devices to design their machine-learning models independently and reduces the communication overhead in the training process. We then theoretically analyze the convergence bound of the proposed framework under a non-convex loss function setting, revealing that scheduling more data volume in each round helps to improve the learning performance. In addition, large data volume should be scheduled in early rounds if the total scheduled data volume during the entire learning course is fixed. Inspired by this, we define a new objective function, i.e., the weighted scheduled data sample volume, to transform the inexplicit global loss minimization problem into a tractable one for device scheduling, bandwidth allocation, and power control. To deal with unknown time-varying wireless channels, we transform the considered problem into a deterministic problem for each round with the assistance of the Lyapunov optimization framework. Then, we derive the optimal bandwidth allocation and power control solution by convex optimization techniques. We also develop an efficient online device scheduling algorithm to achieve an energy-learning trade-off in the learning process. Experimental results on two typical datasets (i.e., MNIST and CIFAR-10) under highly heterogeneous local data distributions show that the proposed KFL is capable of reducing over 99% communication overhead while achieving better learning performance than the conventional model aggregation-based algorithms. In addition, the proposed device scheduling algorithm converges faster than the benchmark scheduling schemes.

The first high spatial resolution multi-scale daily SPI and SPEI raster dataset for drought monitoring and evaluating over China from 1979 to 2018

ABSTRACT Standardized Precipitation Index (SPI) and Standardized Precipitation Evapotranspiration Index (SPEI), traditionally derived at a monthly scale, are widely used drought indices. To overcome temporal-resolution limitations, we have previously developed and published a well-validated daily SPI/SPEI in situ dataset. Although having a high temporal resolution, this in situ dataset presents low spatial resolution due to the scarcity of stations. Therefore, based on the China Meteorological Forcing Dataset, which is composed of data from more than 1,000 ground-based observation sites and multiple remote sensing grid meteorological dataset, we present the first high spatiotemporal-resolution daily SPI/SPEI raster datasets over China. It spans from 1979 to 2018, with a spatial resolution of 0.1° × 0.1°, a temporal resolution of 1-day, and the timescales of 30-, 90-, and 360-days. Results show that the spatial distributions of drought event characteristics detected by the daily SPI/SPEI are consistent with the monthly SPI/SPEI. The correlation between the daily value of the 12-month scale and the monthly value of SPI/SPEI is the strongest, with linear correlation, Nash-Sutcliffe coefficient, and normalized root mean square error of 0.98, 0.97, and 0.04, respectively. The daily SPI/SPEI is shown to be more sensitive to flash drought than the monthly SPI/SPEI. Our improved SPI/SPEI shows high accuracy and credibility, presenting enhanced results when compared to the monthly SPI/SPEI. The total data volume is up to 150 GB, compressed to 91 GB in Network Common Data Form (NetCDF). It can be available from Figshare (https://doi.org/10.6084/m9.figshare.c.5823533) and ScienceDB (https://doi.org/10.57760/sciencedb.j00076.00103).

Memory-aware Partitioning, Scheduling, and Floorplanning for Partially Dynamically Reconfigurable Systems

Partially dynamic reconfiguration (PDR) technology can accelerate the reconfiguration process and overcome hardware resource constraints when facing the challenge of high performance with respect to applications and resources constraints on field-programmable gate arrays (FPGAs). On FPGAs with PDR technology, the available on-chip Block RAM (BRAM) resources may not satisfy the memory requirements for all data. If we reserve more BRAM resources, then the total area of the dynamically reconfigurable region (DRR) that is used for calculation will decrease, with a reduction in system performance. We propose a memory-aware optimization framework to search for the optimal solution considering partitioning, scheduling, and floorplanning, where we make a tradeoff between performance and on-chip memory resources utilization. We then propose methods for memory allocation: An ILP model and a heuristic algorithm are provided to determine the minimum memory requirements and the number of corresponding memory blocks for data, as well as to determine whether the memory block with its stored data is assigned on-chip or off-chip by formulating the problem into a 0-1 knapsack problem and solving it using dynamic programming. Experimental results show that the memory-aware optimization framework and methods of memory allocation can increase the amount of on-chip data access to 29.65% of the total data volume with guaranteed performance.

Global Surface Water Density Masks to Refine Sentinel 3 Data Acquisitions

The Copernicus Sentinel-3 mission is part of the first generation of Copernicus satellites and was set out to deliver operationally high quality measurements over ocean, land, and atmosphere. The European Space Agency (ESA) is now studying concepts for the Next Generation Sentinel-3 Topography (S3NG-T) mission that would launch in the 2032+ time period. In order to meet the primary objectives of the S3NG-T mission requirement document a complex analysis of river and lake targets is required to size the satellite mass memory and downlink system. We have adopted a different approach that derives a non-conservative mask of water density that defines a more useful user friendly approach to specifying where river and lake targets are located. Within this work, the design and construction of our water density maps as well as application examples are discussed. The results prove an easy-to-apply methodology to create global surface water density maps using high-resolution water body maps. The work further illustrates different use cases to optimize data sizing considering constraints such as global coverage, river width, and total data volume.

A machine learning framework to classify Southeast Asian echolocating bats

Bats comprise a quarter of all mammal species, provide key ecosystem services and serve as effective bioindicators. Automated methods for classifying echolocation calls of free-flying bats are useful for monitoring but are not widely used in the tropics. This is particularly problematic in Southeast Asia, which supports more than 388 bat species. Here, sparse reference call databases and significant overlap among species call characteristics makes the development of automated processing methods complex. To address this, we outline a semi-automated framework for classifying bat calls in Southeast Asia and demonstrate how this can reliably speed up manual data processing. We implemented the framework to develop a classifier for the bats of Borneo and tested this at a landscape in Sabah. Borneo has a relatively well-described bat fauna, including reference calls for 52% of all 81 known echolocating species on the island. We applied machine learning to classify calls into one of four call types that serve as indicators of dominant ecological ensembles: frequency-modulated (FM; forest-specialists), constant frequency (CF; forest-specialists and edge/gap foragers), quasi-constant frequency (QCF; edge/gap foragers), and frequency-modulated quasi constant frequency (FMqCF; edge/gap and open-space foragers) calls. Where possible, we further identified calls to species/sonotype. Each classification is provided with a confidence value and a recommended threshold for manual verification. Of the 245,991 calls recorded in our test landscape, 85% were correctly identified to call type and only 10% needed manual verification for three of the call types. The classifier was most successful at classifying CF calls, reducing the volume of calls to be manually verified by over 95% for three common species. The most difficult bats to classify were those with FMqCF calls, with only a 52% reduction in files. Our framework allows users to rapidly filter acoustic files for common species and isolate files of interest, cutting the total volume of data to be processed by 86%. This provides an alternative method where species-specific classifiers are not yet feasible and enables researchers to expand non-invasive monitoring of bat species. Notably, this approach incorporates aerial insectivorous ensembles that are regularly absent from field datasets despite being important components of the bat community, thus improving our capacity to monitor bats remotely in tropical landscapes.

Developing Metrics for NASA Earth Science Interdisciplinary Data Products and Services

Metrics are measures that are able to produce quantifiable information. There are many applications of metrics in Earth science data and services; for example, metrics are frequently used to track service performance and progress. In short, developing, collecting and analyzing metrics are essential activities to better support Earth science research, applications, and education. As one of the largest repositories of Earth science data in the world, NASA’s Earth Science Data and Information System (ESDIS) Project supports twelve Distributed Active Archive Centers (DAACs). Standard metrics have been developed by the ESDIS Metrics System (EMS). These metrics are collected and analyzed routinely at each DAAC. As it is expected that the total data volume will continue to grow rapidly, and the timely developed technologies (e.g., cloud computing, AI/ML) will continue to improve data discovery and accessibility, opportunities for developing new data services for the Earth science community will also arise, especially in interdisciplinary research and applications. However, developing such metrics has become a challenge because multiple datasets are often needed. Current metrics are designed for a single predefined dataset or service, a disadvantage for collecting metrics for interdisciplinary data services. In this paper, we assess current metrics using one of the NASA DAACs, the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), as an example, to discuss challenges and opportunities, along with recommendations for developing metrics addressing interdisciplinary satellite data products and services. <strong>Highlights</strong> <ul><li>Overview of NASA GES DISC Earth science datasets and services</li><li>Overview of existing metrics collection methods and analysis tools with examples</li><li>Discuss challenges and opportunities in collecting metrics for Earth science interdisciplinary data and services</li></ul>

IProX in 2021: connecting proteomics data sharing with big data.

The rapid development of proteomics studies has resulted in large volumes of experimental data. The emergence of big data platform provides the opportunity to handle these large amounts of data. The integrated proteome resource, iProX (https://www.iprox.cn), which was initiated in 2017, has been greatly improved with an up-to-date big data platform implemented in 2021. Here, we describe the main iProX developments since its first publication in Nucleic Acids Research in 2019. First, a hyper-converged architecture with high scalability supports the submission process. A hadoop cluster can store large amounts of proteomics datasets, and a distributed, RESTful-styled Elastic Search engine can query millions of records within one second. Also, several new features, including the Universal Spectrum Identifier (USI) mechanism proposed by ProteomeXchange, RESTful Web Service API, and a high-efficiency reanalysis pipeline, have been added to iProX for better open data sharing. By the end of August 2021, 1526 datasets had been submitted to iProX, reaching a total data volume of 92.42TB. With the implementation of the big data platform, iProX can support PB-level data storage, hundreds of billions of spectra records, and second-level latency service capabilities that meet the requirements of the fast growing field of proteomics.

Fusion of visible-to-near-infrared and mid-infrared spectroscopy to estimate soil organic carbon

Spectral techniques such as visible-to-near-infrared (VIS–NIR) and mid-infrared (MIR) spectroscopies have been regarded as effective alternatives to laboratory-based methods for determining soil organic carbon (SOC). Research to explore the potential of the fusion of VIS–NIR and MIR absorbance for improving SOC prediction is needed, since each individual spectral range may not contain sufficient information to yield reasonable estimation accuracy. Here, we investigated two data fusion strategies that differed in input data, including direct concatenation of full-spectral absorbance and concatenation of selected predictors by optimal band combination (OBC) algorithm. Specifically, continuous wavelet transform (CWT) was adopted to optimize the spectral data before and after data fusion. Prediction models for SOC were developed using partial least squares regression. Results demonstrated that estimations for SOC using MIR absorbance (i.e., validation R2 = 0.45–0.64) generally outperformed those using VIS–NIR (i.e., validation R2 = 0.20–0.44). Compared to the raw absorbance counterparts, CWT decomposing could improve the prediction accuracy for SOC, for both the individual absorbance and the fusion of VIS–NIR and MIR absorbance. Among all the models investigated, the combinational use of VIS–NIR and MIR using OBC fusion at CWT scale of 1 yielded the optimal prediction, providing the highest validation R2 of 0.66. This model with 10 selected spectral parameters as input is of small total data volume, large processing speed and efficiency, confirming the potential of OBC in fusing both types of spectral data. In summary, CWT decomposing and OBC strategy are powerful algorithms in analyzing the spectral data, and allow the VIS–NIR and MIR spectral fusion models to improve the SOC estimation.

UbiBrowser 2.0: a comprehensive resource for proteome-wide known and predicted ubiquitin ligase/deubiquitinase-substrate interactions in eukaryotic species.

As an important post-translational modification, ubiquitination mediates ∼80% of protein degradation in eukaryotes. The degree of protein ubiquitination is tightly determined by the delicate balance between specific ubiquitin ligase (E3)-mediated ubiquitination and deubiquitinase-mediated deubiquitination. In 2017, we developed UbiBrowser 1.0, which is an integrated database for predicted human proteome-wide E3–substrate interactions. Here, to meet the urgent requirement of proteome-wide E3/deubiquitinase–substrate interactions (ESIs/DSIs) in multiple organisms, we updated UbiBrowser to version 2.0 (http://ubibrowser.ncpsb.org.cn). Using an improved protocol, we collected 4068/967 known ESIs/DSIs by manual curation, and we predicted about 2.2 million highly confident ESIs/DSIs in 39 organisms, with >210-fold increase in total data volume. In addition, we made several new features in the updated version: (i) it allows exploring proteins’ upstream E3 ligases and deubiquitinases simultaneously; (ii) it has significantly increased species coverage; (iii) it presents a uniform confidence scoring system to rank predicted ESIs/DSIs. To facilitate the usage of UbiBrowser 2.0, we also redesigned the web interface for exploring these known and predicted ESIs/DSIs, and added functions of ‘Browse’, ‘Download’ and ‘Application Programming Interface’. We believe that UbiBrowser 2.0, as a discovery tool, will contribute to the study of protein ubiquitination and the development of drug targets for complex diseases.

C45 Algorithm for Motorcycle Sales Prediction On CV Mokas Rawajitu

CV Mokas Rawajitu is a company that sells various types of used motorbikes both in cash and on credit. In sales, the problem that occurs is the frequent occurrence of ups and downs in motorcycle sales due to the mismatch of the available motorcycle variants with consumer interests so that motorcycle sales often do not reach the target. The role of data mining is needed to analyze consumer purchasing patterns at CV Mokas Rawajitu which can produce information, namely knowing what types of motorbikes most in-demand by consumers are and which are most in-demand in the market by predicting using the C4.5 algorithm based on the sales transaction data they have. from previous periods. The study used a dataset of motorcycle sales at CV Mokas Rawajitu from 2017-2019 with a total data volume of 1,411 data. The attributes used are the motorbike category, the motorbike brand, the motorbike price, and the year of production. The tools used in this research are Rapid Miner. The results of the application of the C4.5 Algorithm can be used as a prediction of sales at CV Mokas Rawajitu because the results of the accuracy of testing data and models using 9-Fold Cross Validation reach a value of 87.95% where the 9th fold reaches the highest accuracy value with a Sensitivity level of 97, 15%, 69.05% Specificity, 86.57% Precision, 12.05% Error (Error Rate) and 30.95% False Positive Rate.

Parameterized data handling for forming tool tryout: reverse engineering, data consolidation and springback compensation

The forming tool design process generates large amounts of data up to the first falling parts. On the one hand, simulation results, geometric measurements and design models originate from different software tools, which leads to a non-consolidated set of data inventory. On the other hand, the total data volume is hard to handle economically. A stable and user-friendly data structure for overarching tool tryout is missing. Often several experience-based iterations are necessary to derive the tool’s working surfaces, which is both time- and resource consuming and even may lead to postponed start of production. Meanwhile, early-generated data does not involve into the manual optimization process. Therefore, in this paper a parameterized data handling methodology is introduced, which enables systematic reverse engineering, data consolidation, and springback compensation. Each generated dataset during tryout is traced back to a mathematical description of geometry using so called control points (B-Spline model). Through, the parameterized description, the different data formats interact straightforward and need minimum storage. The developed concept is demonstrated for the springback analysis of a forming component using design models and numerical data.

Socioeconomic Relation with Plastic Consumption on 61 Countries Classified by Continent, Income Status and Coastal Regions.

In this study, World Bank socioeconomic data of 61 countries during 2009-2015 combine with a semi-systematic literature review was applied to analyze the relationship between plastic consumptions and socioeconomic levels statistically. Based on a total data volume of 5551, we found that all countries can be divided into different groups like the continent, income status, and coastal countries. It is showed that most plastics were consumed by North America, Europe, high-income countries, and coastal countries in the Pacific and the Atlantic Ocean. A potential relation occurred in plastic consumptions with different socioeconomic parameters like Gross National Income (GNI) per capita, Gross Domestic Product (GDP) per capita, life expectancy, agriculture land use, etc. The results indicated the need should continue to support initiatives to cover identified gaps, either geographic or thematic, to address plastic consumptions.

Regionally aggregated, stitched and de‐drifted CMIP‐climate data, processed with netCDF‐SCM v2.0.0

AbstractThe world's most complex climate models are currently running a range of experiments as part of the Sixth Coupled Model Intercomparison Project (CMIP6). Added to the output from the Fifth Coupled Model Intercomparison Project (CMIP5), the total data volume will be in the order of 20PB. Here, we present a dataset of annual, monthly, global, hemispheric and land/ocean means derived from a selection of experiments of key interest to climate data analysts and reduced complexity climate modellers. The derived dataset is a key part of validating, calibrating and developing reduced complexity climate models against the behaviour of more physically complete models. In addition to its use for reduced complexity climate modellers, we aim to make our data accessible to other research communities. We facilitate this in a number of ways. Firstly, given the focus on annual, monthly, global, hemispheric and land/ocean mean quantities, our dataset is orders of magnitude smaller than the source data and hence does not require specialized ‘big data’ expertise. Secondly, again because of its smaller size, we are able to offer our dataset in a text‐based format, greatly reducing the computational expertise required to work with CMIP output. Thirdly, we enable data provenance and integrity control by tracking all source metadata and providing tools which check whether a dataset has been retracted, that is identified as erroneous. The resulting dataset is updated as new CMIP6 results become available and we provide a stable access point to allow automated downloads. Along with our accompanying website (cmip6.science.unimelb.edu.au), we believe this dataset provides a unique community resource, as well as allowing non‐specialists to access CMIP data in a new, user‐friendly way.