Data Processing Requirements Research Articles

Using cross-recurrence quantification analysis to compute similarity measures for time series of unequal length with applications to sleep stage analysis

Comparing time series of unequal length requires data processing procedures that may introduce biases. This article describes, validates, and applies Cross-Recurrence Quantification Analysis (CRQA) to detect and quantify correlation and coupling among time series of unequal length without prior data processing. We illustrate and validate this application using continuous and discrete data from a model system (study 1). Then we use the method to re-analyze the Sleep Heart Health Study (SHHS), a rare large dataset comprising detailed physiological sleep measurements acquired by in-home polysomnography. We investigate whether recurrence patterns of ultradian NREM/REM sleep cycles (USC) predict mortality (study 2). CRQA exhibits better performance compared with traditional approaches that require trimming, stretching or compression to bring two time series to the same length. Application to the SHHS indicates that recurrence patterns linked to stability of USCs are associated with all-cause mortality even after controlling for other sleep parameters, health, and sociodemographics. We suggest that CRQA is a useful tool for analyzing categorical time series, where the underlying structure of the data is unlikely to result in matching data points—such as ultradian sleep cycles.

The RAS Paradigm in Mainframe Systems: A Comprehensive Analysis of Design Principles and Operational Benefits

This article examines the critical role of Reliability, Availability, and Serviceability (RAS) in mainframe computing systems. We explore how RAS principles have become fundamental design features in modern mainframes, enhancing their ability to meet the demanding requirements of enterprise-level data processing. The article provides an in-depth analysis of each RAS component, detailing how reliability is achieved through robust hardware self-checking and extensive software testing, availability is maintained via seamless component failover and layered error recovery, and serviceability is ensured through advanced diagnostic capabilities and modular replacement units. We argue that the holistic integration of RAS principles in mainframe architecture not only maximizes system uptime and operational continuity but also significantly impacts application design and overall system efficiency. Through a comprehensive review of current mainframe technologies and industry practices, this article highlights the enduring importance of RAS in an era of increasing computational complexity and datadriven decision making. Our findings suggest that RAS principles will continue to evolve, playing a crucial role in shaping the future of high-performance, mission-critical computing systems.

Dual-polarized transmissive metasurfaces for near-unitary-NA analog spatial computing empowered by impedance matching and mismatching.

In recent years, due to the increasing requirement for real-time and massive data processing, optical analog computation has arisen as a promising alternative to digital computation. Optical spatial differentiation plays a fundamentally important role in various emerging technologies, including augmented reality, autonomous driving, and object recognition. However, previous demonstrations encountered several limitations, such as the dependence on polarization states and a typically limited numerical aperture (NA) of about 0.5, especially in the transmission mode. Here, a new, to our knowledge, design strategy based on the evolution between impedance matching and mismatching in a metasurface is proposed to fill this gap, which can perform dual-polarized second-order derivative for image processing. Our scheme enables high transmission under dual polarization over an 85° incident angle range (NA = 0.996), resulting in more than twofold spatial resolution. Our work paves the way for polarization-insensitive high-resolution signal and image processing in the terahertz region.

LLM-Twin: mini-giant model-driven beyond 5G digital twin networking framework with semantic secure communication and computation

Beyond 5G networks provide solutions for next-generation communications, especially digital twins networks (DTNs) have gained increasing popularity for bridging physical and digital space. However, current DTNs pose some challenges, especially when applied to scenarios that require efficient and multimodal data processing. Firstly, current DTNs are limited in communication and computational efficiency, since they require to transmit large amounts of raw data collected from physical sensors, as well as to ensure model synchronization through high-frequency computation. Second, current models of DTNs are domain-specific (e.g. E-health), making it difficult to handle DT scenarios with multimodal data processing requirements. Finally, current security schemes for DTNs introduce additional overheads that impair the efficiency. Against the above challenges, we propose a large language model (LLM) empowered DTNs framework, LLM-Twin. First, based on LLM, we propose digital twin semantic networks (DTSNs), which enable more efficient communication and computation. Second, we design a mini-giant model collaboration scheme, which enables efficient deployment of LLM in DTNs and is adapted to handle multimodal data. Then, we designed a native security policy for LLM-twin without compromising efficiency. Numerical experiments and case studies demonstrate the feasibility of LLM-Twin. To our knowledge, this is the first to propose an LLM-based semantic-level DTNs.

TOMOMAN: a software package for large scale cryo-electron tomography data preprocessing, community data sharing, and collaborative computing.

Cryo-electron tomography (cryo-ET) and subtomogram averaging (STA) are becoming the preferred methodologies for investigating subcellular and macromolecular structures in native or near-native environments. While cryo-ET is amenable to a wide range of biological problems, these problems often have data processing requirements that need to be individually optimized, precluding the notion of a one-size-fits-all processing pipeline. Cryo-ET data processing is also becoming progressively more complex due to an increasing number of packages for each processing step. Though each package has its own strengths and weaknesses, independent development and different data formats makes them difficult to interface with one another. TOMOMAN (TOMOgram MANager) is an extensible package for streamlining the interoperability of packages, enabling users to develop project-specific processing workflows. TOMOMAN does this by maintaining an internal metadata format and wrapping external packages to manage and perform preprocessing, from raw tilt-series data to reconstructed tomograms. TOMOMAN can also export this metadata between various STA packages. TOMOMAN also includes tools for archiving projects to data repositories; allowing subsequent users to download TOMOMAN projects and directly resume processing where it was previously left off. By tracking essential metadata, TOMOMAN streamlines data sharing, which improves reproducibility of published results, reduces computational costs by minimizing reprocessing, and enables distributed cryo-ET projects between multiple groups and institutions. TOMOMAN provides a way for users to test different software packages to develop processing workflows that meet the specific needs of their biological questions and to distribute their results with the broader scientific community.

Multi source data security protection of smart grid based on edge computing

In order to cope with the continuous development of information technology, the data volume of edge devices in the power supply network is increasing rapidly, and the higher requirements for real-time data processing and transmission bandwidth are put forward, the author proposes the research on security protection of multi-source data in smart grid based on edge computing. Established a distribution network safety risk map, calculated and analyzed safety risks through potential functions, and obtained node potential values and key node data; The intelligent grid multi-source heterogeneous data monitoring model based on edge computing can realize the timely perception and real-time response of distribution network faults, shorten the outage time, and improve the power supply reliability and user satisfaction of the distribution network. The results indicate that: Taking a distribution network line in a certain area as an example, the highest potential value is node 9, followed by nodes 10 and 3. This indicates that in order to ensure the reliable operation of the distribution network and avoid power outages, it is necessary to improve the repair capacity, the proportion of old equipment, and the power supply radius exceeding the standard. ConclusionThe effectiveness of potential function in risk assessment of edge computing is verified, which can provide theoretical guidance for distribution network fault diagnosis.

Unsupervised method for detecting surface defects in steel based on joint optimization of pseudo-labeling and clustering

Abstract Advances in the field of measurement science and technology have improved the detection of defects in industrial production. One of the key challenges in steel plate surface defect detection is the need to quickly detect a small number of defects in an overwhelmingly defect-free sample. Unlike supervised learning, which relies heavily on precise sample labeling, unsupervised learning leverages its inherent learning capabilities for detection. This paper introduces an innovative method for smart steel diagnosis, integrating joint optimization of feature extraction and clustering. The proposed approach merges mini-batch K-Means clustering with a feature extraction network to acquire pseudo-label information for current images. It employs a multi-view transformation strategy, enabling classification through the optimized feedback from pseudo-labels. This method allows the network to self-optimize the distinction of image features through backpropagation. The method exhibits a mere 4% classification failure rate for steel surface images. This significant reduction in additional data processing requirements enhances the inspection system's efficiency and accuracy. Furthermore, the versatility of this method extends beyond steel defect diagnosis. It holds potential for application in various engineering domains, particularly in scenarios characterized by data imbalance.

Data and Energy Impacts of Intelligent Transportation—A Review

The deployment of intelligent transportation is still in its early stages and there are many challenges that need to be addressed before it can be widely adopted. Autonomous vehicles are a class of intelligent transportation that is rapidly developing, and they are being deployed in selected cities. A combination of advanced sensors, machine learning algorithms, and artificial intelligence are being used in these vehicles to perceive their environment, navigate, and make the right decisions. These vehicles leverage extensive data sourced from various sensors and computers integrated into the vehicle. Hence, massive computational power is required to process the information from various built-in sensors in milliseconds to make the right decision. The power required by the sensors and the use of additional computational power increases the energy consumption, and, hence, could reduce the range of the autonomous electric vehicle relative to a standard electric car and lead to additional emissions. A number of review papers have highlighted the environmental benefits of autonomous vehicles, focusing on aspects like optimized driving, improved route selection, fewer stops, and platooning. However, these reviews often overlook the significant energy demands of the hardware systems—such as sensors, computers, and cameras—necessary for full autonomy, which can decrease the driving range of electric autonomous vehicles. Additionally, previous studies have not thoroughly examined the data processing requirements in these vehicles. This paper provides a more detailed review of the volume of data and energy usage by various sensors and computers integral to autonomous features in electric vehicles. It also discusses the effects of these factors on vehicle range and emissions. Furthermore, the paper explores advanced technologies currently being developed by various industries to enhance processing speeds and reduce energy consumption in autonomous vehicles.

An Efficient Approach to Extract and Store Big Semantic Web Data Using Hadoop and Apache Spark GraphX

The volume of data is growing at an astonishingly high speed. Traditional techniques for storing and processing data, such as relational and centralized databases, have become inefficient and time-consuming. Linked data and the Semantic Web make internet data machine-readable. Because of the increasing volume of linked data and Semantic Web data, storing and working with them using traditional approaches is not enough, and this causes limited hardware resources. To solve this problem, storing datasets using distributed and clustered methods is essential. Hadoop can store datasets because it can use many hard disks for distributed data clustering; Apache Spark can be used for parallel data processing more efficiently than Hadoop MapReduce because Spark uses memory instead of the hard disk. Semantic Web data has been stored and processed in this paper using Apache Spark GraphX and the Hadoop Distributed File System (HDFS). Spark's in-memory processing and distributed computing enable efficient data analysis of massive datasets stored in HDFS. Spark GraphX allows graph-based semantic web data processing. The fundamental objective of this work is to provide a way for efficiently combining Semantic Web and big data technologies to utilize their combined strengths in data analysis and processing. First, the proposed approach uses the SPARQL query language to extract Semantic Web data from DBpedia datasets. DBpedia is a hugely available Semantic Web dataset built on Wikipedia. Secondly, the extracted Semantic Web data was converted to the GraphX data format; vertices and edges files were generated. The conversion process is implemented using Apache Spark GraphX. Third, both vertices and edge tables are stored in HDFS and are available for visualization and analysis operations. Furthermore, the proposed techniques improve the data storage efficiency by reducing the amount of storage space by half when converting from Semantic Web Data to a GraphX file, meaning the RDF size is around 133.8 and GraphX is 75.3. Adopting parallel data processing provided by Apache Spark in the proposed technique reduces the required data processing and analysis time. This article concludes that Apache Spark GraphX can enhance Semantic Web and Big Data technologies. We minimize data size and processing time by converting Semantic Web data to GraphX format, enabling efficient data management and seamless integration.

Dirbtinio intelekto taikymo įmonių vidaus tyrimuose teisinės ribos ir baudžiamieji procesiniai padariniai

The paper addressed the following questions: how can AI be applied in corporate internal investigations and what are the legal limits for its use? In case of effective application, how can it affect the admissibility and valuation of the information collected from these procedures in an eventual criminal lawsuit? It was demonstrated that, with the expectation of increasing the effectiveness and efficiency of compliance programs, AI has been used in real-time – and often predictive – supervision of employees, as well as in carrying out internal investigations, through, for example, lie detection systems. However, due to the limitations and risks of this technology, its employment must comply with legal parameters and restrictions. Therefore, three categories of restrictions were analyzed: with respect to (i) legal requirements for data processing, we noted that the processing of data to be used as input in those systems must find legal support in one of the hypotheses provided for in Articles 6 and 9(2) of the GDPR. Furthermore, the fundamental principles that substantiate the test of proportionality between the intended purpose and the intervention to be carried out, provided for in Article 5, must be observed. Regarding (ii) possible prohibitions on the use of AI systems in the context of internal investigations, we observed that the use of technologies to control and supervise the work environment must take into account the requirements of legality, shall not affect areas in which employees’ expectations of privacy must prevail, and, lastly, must observe a second test of proportionality, aimed at assessing whether the technology in question is adequate and necessary to achieve the purpose intended with it and whether the rights possibly affected by it should not prevail. About the (iii) limits to the admissibility and valuation of elements of information from internal investigations, we concluded that their admission in criminal proceedings will depend on a new judgment of legality and proportionality. In addition, they may be admitted when presented by the company, in its defence, or by Public Prosecution, provided that a third test of proportionality is observed and that these elements of information are not considered sufficient to substantiate the conviction of any defendant, therefore having probative value similar to elements from state investigation acts.

Asteroid spin and shape properties from Gaia DR3 photometry

The third data release of in June 2022, included the first large sample of sparse photometric data for more than 150\,000 Solar System objects (SSOs), mainly asteroids. The SSO photometric data can be processed to derive information on the physical properties for a large number of objects, including spin properties, surface photometric behaviour in a variety of illumination conditions, and overall shape. After selecting a set of 22\,815 objects for which an adequate number of accurate photometric measurements had been obtained by we applied the 'genetic' algorithm of photometric inversion developed by the Data Processing and Analysis Consortium to process SSO photometric data. Given the need to minimise the required data processing time, the algorithm was set to adopt a simple triaxial ellipsoid shape model. Our results show that in spite of the limited variety of observing circumstances and the limited numbers of measurements per object at present (in the majority of cases no greater than $40$ and still far from the number expected at the end of the mission of about $60$ - $70$), the proportion of correct determinations for the spin period among the observed targets is about $85$<!PCT!>. This percentage is based on a comparison with reliable literature data following a moderate filtering procedure developed to remove dubious solutions. The analysis performed in this paper is important in the context of developing further improvements to the adopted data reduction procedure. This includes the possible development of better solution filtering procedures that take into account, for each object, the possible presence of multiple, equivalent spin period solutions that have not been systematically investigated in this preliminary application.

Memristive crossbar-based circuit design of back-propagation neural network with synchronous memristance adjustment

The performance improvement of CMOS computer fails to meet the enormous data processing requirement of artificial intelligence progressively. The memristive neural network is one of the most promising circuit hardwares to make a breakthrough. This paper proposes a novel memristive synaptic circuit that is composed of four MOS transistors and one memristor (4T1M). The 4T1M synaptic circuit provides flexible control strategies to change memristance or respond to the input signal. Applying the 4T1M synaptic circuit as the cell of memristive crossbar array, based on the structure and algorithm of the back-propagation (BP) neural network, this paper proposes circuit design of the memristive crossbar-based BP neural network. By reusing the 4T1M memristive crossbar array, the computations in the forward-propagation process and back-propagation process of BP neural network are accomplished on the memristive crossbar-based circuit to accelerate the computing speed. The 4T1M memristive crossbar array can change all the cells’ memristance at a time, accordingly, the memristive crossbar-based BP neural network can realize synchronous memristance adjustment. The proposed memristive crossbar-based BP neural network is then evaluated through experiments involving XOR logic operation, iris classification, and MNIST handwritten digit recognition. The experimental results present fewer iterations or higher classification accuracies. Further, the comprehensive comparisons with the existing memristive BP neural networks highlight the advantages of the proposed memristive crossbar-based BP neural network, which achieves the fastest memristance adjustment speed using relatively few components.

A cloud-edge cooperative scheduling model and its optimization method for regional multi-energy systems

In the process of multi-energy system optimal scheduling, due to the high data processing requirements of the multi-energy devices and loads and the complexity of the operating states of the multi-energy devices, the scheduling optimization of the system is to some extent more difficult. To address this problem, this paper proposes a regional multi-energy system optimal scheduling model based on the theory of cloud-edge collaboration. First, based on intelligent data sensors, a cloud-edge cooperative scheduling framework of the regional multi-energy system is constructed. Then, the physical model of operating state data of multi-energy system equipment and the allocation mechanism of system scheduling tasks are studied. With the cloud service application layer and the edge computing layer as the upper and lower optimization scheduling layers, the double-layer optimization scheduling model of the regional multi-energy system is established. The objectives of the model are optimal scheduling cost and minimum delay of scheduling data transmission. The multi-objective whale optimization algorithm is used to solve the model. Finally, a simulation model is built for verification. The simulation results show that the scheduling model established in this paper can effectively improve the scheduling data processing capability and improve the economy of regional multi-energy system scheduling.

Analysis and prospects of automobile intelligent assisted driving characteristics based on FPGA technology

This article provides a comprehensive exploration of the pivotal role that field-programmable gate arrays (FPGAs) play in the advancement of autonomous driving technology. FPGAs, which made their debut in the early 1980s, have emerged as a crucial component in this field, owing to their robust parallel processing capabilities, real-time data analysis capabilities, and exceptional customizability. With the ever-increasing demand for autonomous driving solutions, the adoption of FPGAs has become indispensable in meeting the requirements for high-speed data processing and instantaneous response times. Within these pages, we delve into the significant role that FPGAs assume in elevating intelligent driving systems, offering a deep dive into this subject through meticulous case studies and technical insights. This article casts a spotlight on several compelling instances where FPGAs shine, notably in adaptive cruise systems, obstacle recognition, automatic emergency braking, and intelligent parking assistance, achieved through seamless integration with YOLO technology. These real-world examples serve to underscore the pivotal role that FPGAs play in ensuring road safety and propelling technological advancement in the realm of autonomous driving.

A High-Efficiency Satellite Image Classification Method for Real-Time Application Using Augmented Incremental Transfer Learning

Researchers have put forth a range of models for processing satellite images, each with distinct data formats and processing requirements. One limitation arises from variations in the forms of modules, such as the image capturing and feature extraction modules, impacting accuracy and scalability in real-time scenarios. This passage introduces and explores a pioneering very competent temporal engine crafted specifically for classification of real-time satellite images. The proposed approach utilizes augmented incremental transfer learning, aiming to mitigate the limitations associated with diverse processing requirements. This approach involves capturing real-time satellite data through Google’s Earth Engine and subsequently processing it using a Convolutional Neural Network (CNN) based on transfer learning. The CNN employs backscatter coefficient analysis, utilizing coefficients derived from Precision Image's average intensity value across a distributed target. By integrating incremental learning and CNN for classification, the model achieves an impressive average accuracy of 98.06% in detecting crop type and severity of damage. Comparative analysis with state-of-the-art approaches reveals the superiority of the proposed model. It outperforms existing models by 5% in accuracy, showcasing its efficacy in satellite image processing and classification.

Categories of personal data of employees

The legislation on personal data has changed significantly in recent years, data processing requirements have been tightened. The trend will obviously continue, taking into account the development of technologies, the development and implementation of artificial intelligence, which will require appropriate legal regulation and strengthening the protection of personal data of certain categories. This article is devoted to highlighting and studying various categories of personal data of employees, as well as special rules for their processing.

Leveraging LiDAR-Based Simulations to Quantify the Complexity of the Static Environment for Autonomous Vehicles in Rural Settings.

This paper uses virtual simulations to examine the interaction between autonomous vehicles (AVs) and their surrounding environment. A framework was developed to estimate the environment's complexity by calculating the real-time data processing requirements for AVs to navigate effectively. The VISTA simulator was used to synthesize viewpoints to replicate the captured environment accurately. With an emphasis on static physical features, roadways were dissected into relevant road features (RRFs) and full environment (FE) to study the impact of roadside features on the scene complexity and demonstrate the gravity of wildlife-vehicle collisions (WVCs) on AVs. The results indicate that roadside features substantially increase environmental complexity by up to 400%. Increasing a single lane to the road was observed to increase the processing requirements by 12.3-16.5%. Crest vertical curves decrease data rates due to occlusion challenges, with a reported average of 4.2% data loss, while sag curves can increase the complexity by 7%. In horizontal curves, roadside occlusion contributed to severe loss in road information, leading to a decrease in data rate requirements by as much as 19%. As for weather conditions, heavy rain increased the AV's processing demands by a staggering 240% when compared to normal weather conditions. AV developers and government agencies can exploit the findings of this study to better tailor AV designs and meet the necessary infrastructure requirements.

Improving time efficiency in big data through progressive sampling-based classification model

<span>The proposed system aims to overcome challenges posed by large databases, data imbalance, heterogeneity, and multidimensionality through progressive sampling as a novel classification model. It leverages sampling techniques to enhance processing performance and overcome memory restrictions. The random forest regressor feature importance technique with the gini significance method is employed to identify important characteristics, reducing the data’s features for classification. The system utilizes diverse classifiers such as random forest, ensemble learning, support vector machine (SVM), k-nearest neighbors’ algorithm (KNN), and logistic regression, allowing flexibility in handling different data types and achieving high accuracy in classification tasks. By iteratively applying progressive sampling to the dataset with the best features, the proposed technique aims to significantly improve performance compared to using the entire dataset. This approach focuses computational resources on the most informative subsets of data, reducing time complexity. Results show that the system can achieve over 85% accuracy even with only 5-10% of the original data size, providing accurate predictions while reducing data processing requirements. In conclusion, the proposed system combines progressive sampling, feature selection using random forest regressor feature importance (RFRFI-PS), and a range of classifiers to address challenges in large databases and improve classification accuracy. It demonstrates promising results in accuracy and time complexity reduction.</span>

Heat flux measurement approach for an enhanced thermometric method: preliminary tests

In situ tests are suitable to confirm the real thermal performance of building components, and several significant on-site measurement techniques have been studied in literature. However, among them the Thermometric (THM) method has been poorly examined by the scientific community, thus having opportunities for improvement, being a quite a new and non-standardized technique. The theory behind this technique is the Newton’s law of cooling and the main issue is associated to the heat transfer coefficient for which there is no agreement about the value to use. Here, a simple experimental apparatus characterized by a vertical heated sample, suitably thermally insulated was realized. Sensors were installed and direct heat flux measurements through a heat flux plate were performed and compared with (i) the heat flows obtained through the THM method (test conducted using the internal heat transfer coefficient recommended by the ISO 6946) and (ii) the heat fluxes obtained through the proposal of an enhanced THM method based on dimensionless groups analysis, thus requiring data processing based on convective and radiative components.

Comparative analysis of MapReduce and Apache Tez Performance in Multinode clusters with data compression

This article conducts a thorough comparative analysis of Apache Tez and MapReduce in the context of big data processing. It focuses on key performance metrics, scalability, and ease of use. The analysis begins with an overview of the architectural distinctions between the two frameworks, emphasizing their fundamental design principles. A detailed performance evaluation follows, considering factors such as execution time, resource utilization, and throughput across diverse workloads. The study explores scalability by examining how Apache Tez and MapReduce respond to increasing data volumes and computational demands. Cluster size effects, resource allocation strategies, and adaptability to dynamic workloads are scrutinized. Additionally, the article evaluates the frameworks' ease of use for developers and administrators, incorporating aspects like programming model simplicity, debugging capabilities, and system configurability. User experiences are gathered through surveys and practical use cases. The conclusions drawn from this analysis offer valuable insights for organizations and practitioners seeking suitable distributed computing frameworks. By addressing both performance and user experience, the article aims to provide a comprehensive perspective on the strengths and weaknesses of Apache Tez and MapReduce, assisting decision-makers in making informed choices for their big data processing requirements.