Data Processing Research Articles

The weight of data: an analysis based on the impact on the environment

The carbon footprint generated by the information and communications technology (ICT) sector is increasingly significant, emitting greenhouse gases due to high energy consumption, regardless of the way in which energy is generated, the expansion and growth in data centers, as well as the impact generated by the cryptocurrency sector that in the end represents is reflected in greater consumerism, processing, storage, and transport of information that will be somewhere in the world. Current research addresses the problems and the contrast of figures in energy consumption due to the use of a computer, data processing, the role of the user as an internet consumer, the impact of data centers both in carbon footprint, water footprint and soil footprint, the impact of cryptocurrency mining and its contribution to global energy expenditure as well as the ethical debate of new technologies. And finally, the advances in seeking to optimize energy resources, sustainable and conscious for both consumers and service providers, show the trends focused on energy optimization through software and hardware based on a judicious review of research documents.

Empowering geological data analysis with specialized software GIS modules

This research is devoted to the development of a geographic information system (GIS) for the analysis of geological data. It presents two specialized software modules designed to solve complex geological problems related to potential progress to disturbed masses and magnetotelluric sounding. These modules are integrated into the QGIS environment, offering efficient data processing and analysis capabilities, contributing to a deeper understanding of geological structures. The study presents a mathematical model for the problem of magnetotelluric sounding (MTS) and the continuation of potentials towards the perturbed masses, demonstrating numerical results using the developed algorithm. To confirm the accuracy of the model, a comparative analysis was carried out with empirical data for various chemical elements, which showed high accuracy, especially at shallow depths, with an error rate of less than 2%. In addition, the study highlights the importance of powerful GIS for the analysis and interpretation of geological data, including geochemical, geophysical and remote sensing information. The advanced functionality of QGIS simplifies data processing and visualization, which makes it an invaluable tool for geologists and researchers.

A new experimental setup for the determination of drying soil–water characteristic curve and coefficient of permeability using the continuous evaporation method and osmotic tensiometers

Soil-water characteristic curve (SWCC) and coefficient of permeability (k) are amongst the most crucial soil properties in unsaturated soil mechanics for soil moisture conservation. The direct measurement and prediction of such properties are important to engineering applications, but they are complicated. This study developed a new experimental setup (the OT permeameter) using osmotic tensiometers (OTs) and the continuous evaporation method. The proposed OT permeameter could directly measure SWCC and k up to 1000 kPa suction within one to two weeks. Moreover, the proposed setup is fully automated and data processing was minimal. In addition, a new general equation has been developed to predict and best fit the unsaturated k measurement. The proposed equation uses a minimal number of parameters that carry physical meaning and provides an intuitive and easy way to predict and best-fit k measurement. The accuracy and robustness of both the proposed OT permeameter and prediction equation were evaluated using published data and experimental data from this study.

Infrared and Visual Fusion Detection Framework for Oil and Gas Pipeline Environmental Inspection

The inspection of oil and gas pipeline retains important. These newly constructed pipelines may have defects near the circumferential welds and potential issues with water accumulation. Improper management of these issues can lead to severe consequences. Thus, the inner pipe structure detection prior to their operation is critically essential. Visual camera provides a direct view of the pipeline environment such as circumferential weld seams and water accumulation. In contrast, infrared thermal camera capture infrared radiation distribution from varied shapes and materials, offering an indirect insight into the pipeline environment. This paper designs an integrated infrared and visual NDT system for pipeline detection, capable of capturing time-synchronized infrared and visible image pairs along with corresponding odometer and gyroscope data for further processing. Moreover, an infrared and visual fusion detection algorithm is proposed, comprising coarse registration, base detectors, and probability-based late fusion strategy. The late fusion strategy enhances detection accuracy by leveraging the complementary information from both infrared and visual modalities. Furthermore, as the focus is primarily on the defects near the circumferential welds, the size of the circumferential welds in forward images can be used to predict their position in the circumferential visual detection system, achieving a 96.7% reduction in computational complexity compared to processing all images. The proposed system and algorithm were actually tested in a 26.8 km 1219 pipeline. The results verified the reliability of the system and the robustness and accuracy of the algorithm. The demo code is available at: [https://github.com/ljcuestc/Probability-based-late-fusion.git].

Enhancing data retrieval efficiency in large-scale JavaScript object notation datasets by using indexing techniques

<div class="translate-tooltip-mtz green sm-root translate hidden_translate"><div class="header-wrapper"><div class="header-controls"><span>The use of JavaScript Object Notation (JSON) format as a Not only Structured Query Language (NoSQL) storage solution has grown in popularity, but has presented technical challenges, particularly in indexing large-scale JSON files. This has resulted in slow data retrieval, especially for larger datasets. In this study, we propose the use of JSON datasets to preserve data in resource survey processes. We conducted experiments on a 32-gigabyte dataset containing 1,000,000 transactions in JSON format and implemented two indexing methods, dense and sparse, to improve retrieval efficiency. Additionally, we determined the optimal range of segment sizes for the indexing methods. Our findings revealed that adopting dense indexing reduced data retrieval time from 15,635 milliseconds to 55 milliseconds in one-to-one data retrieval, and from 38,300 milliseconds to 1 millisecond in the absence of keywords. In contrast, using sparse indexing reduced data retrieval time from 33,726 milliseconds to 36 milliseconds in one-to-many data retrieval and from 47,203 milliseconds to 0.17 milliseconds when keywords were not found. Furthermore, we discovered that the optimal segment size range was between 20,000 and 200,000 transactions for both dense and sparse indexing.</span></div></div></div>

Blockchain and differential privacy-based data processing system for data security and privacy in urban computing

Recently, big data related to human movement, air quality, and meteorology have been generated in urban computing through sensing technology and the computing infrastructure. However, security problems arise as data utilization increases. If the sensing data from internet of things devices are constantly exposed, the users’ private information can be determined, a critical security risk that could result in privacy breaches. This paper proposes a secure data processing system using the blockchain and differential privacy for data security and privacy protection in urban computing. When a service provider requests information, the system generates it from urban computing data using machine learning. We apply differential privacy to these data to protect privacy. However, if a query repeats, differential privacy may provide insufficient privacy protection. Therefore, we reduce the total privacy cost by reusing noise for the same data and privacy parameters using the blockchain. Machine learning accuracy may decrease when noisy data are used for training. Thus, we increase accuracy by storing and appropriately using the model parameters generated by the same data in the blockchain. We design, simulate, and analyze the results of an experimental environment for reusing noise for differential privacy and parameter utilization of machine learning using the blockchain. The proposed approach reduces privacy costs compared to the existing mechanism while protecting data privacy. We demonstrate that, through parameter utilization, the accuracy improves compared to conventional mechanisms.

A method for calculating fall risk parameters from discrete stride time series regardless of sensor placement

BackgroundTo complement traditional clinical fall risk assessments, research is oriented towards adding real-life gait-related fall risk parameters (FRP) using inertial sensors fixed to a specific body position. While fixing the sensor position can facilitate data processing, it can reduce user compliance. A newly proposed step detection method, Smartstep, has been proven to be robust against sensor position and real-life challenges. Moreover, FRP based on step variability calculated from stride times (Standard deviation (SD), Coefficient of Variance (Cov), fractal exponent, and sample entropy of stride duration) proved to be useful to prospectively predict the fall risk. Research questionsTo evaluate whether Smartstep is convenient for calculating FRP from different sensor placements. Methods29 elderly performed a 6-minute walking test with IMU placed on the waist and the wrist. FRP were computed from step-time estimated from Smartstep and compared to those obtained from foot-mounted inertial sensors: precision and recall of the step detection, Root mean square error (RMSE) and Intraclass Correlation Coefficient (ICC) of stride durations, and limits of agreement of FRP. ResultsThe step detection precision and recall were respectively 99.5% and 95.9% for the waist position, and 99.4% and 95.7% for the wrist position. The ICC and RMSE of stride duration were 0.91 and 54 ms respectively for both the waist and the hand position. The limits of agreement of Cov, SD, fractal exponent, and sample entropy of stride duration are respectively 2.15%, 25 ms, 0.3, 0.5 for the waist and 1.6%, 16 ms, 0.23, 0.4 for the hand. SignificanceRobust against the elderly's gait and different body locations, especially the wrist, this method can open doors toward ambulatory measurements of steps, and calculation of different discrete stride-related falling risk indicators.

Energy efficient reliable data transmission for optimizing IoT data transmission in smart city

The rapid proliferation of the internet of things (IoT) technology has significantly transformed urban landscapes, giving rise to smart city frameworks that leverage interconnected devices for enhanced efficiency and functionality. In these environments, vast amounts of data are generated by diverse sensors and devices, necessitating advanced strategies for effective data collection and transmission. This paper introduces a novel approach to address data collection and transmission challenges in IoT-enabled smart city frameworks. The proposed design integrates IoT-Cloud for efficient data collection and employs the energy efficient reliable data transmission (EERDT) model, optimizing IoT data transmission. The enhanced dragonfly routing algorithm, incorporating the firefly algorithm, enhances data routing efficiency. Experimental results demonstrate EERDT's superiority over energy-aware iot-routing (EAIR) and location-centric energy-harvesting aware-routing (LCEHAR), revealing significant improvements in communication overhead, data processing latency, and network lifetime. The EERDT exhibits substantial reductions in communication overhead, enhancing overall network performance. The EERDT model showcases lower data processing latency and energy consumption, highlighting its potential for resource-efficient IoT data transmission. This work contributes an innovative solution for smart city IoT networks, emphasizing performance enhancements and resource efficiency.

Developing psychologist attributes in higher educational institutions: case of Kazakhstan

This study’s purpose is to identify the most necessary attributes of a modern psychologist through a survey of practical psychologists, teachers, and students from the Faculty of Psychology, Toraigyrov University. The survey allowed us to develop an effective training program for psychologists. The study involved 115 respondents from Kazakhstan, 60 of them were future psychologists. The respondents took the survey to determine the key attributes of a modern psychologist. The research was also based on literature analysis and generalization, questionaries, and ranking. The Cronbach’s alpha coefficient was utilized to assess the reliability of the authors’ questionnaire items. SPSS Statistics program was applied for statistical data processing. According to the results, 80% of practical psychologists wanted to improve the curriculum for their students. At the same time, 72% of students believe that the program should primarily be practice-oriented. The students unanimously noted the key quality of the psychologist–empathy. The vast majority (94.1%) mentioned moral principles, and 85.7% indicated authenticity. However, teachers and psychologists did not emphasize the above criteria. The current study aims to develop a basis for an effective training program for future psychologists studying at the universities of Kazakhstan according to their modern needs.

Feature selection based on chi-square and ant colony optimization for multi-label classification

Text classification is widely used in organizations with large databases and digital documents. In text classification, there are many features, most of which are redundant. High-dimensional features impact multi-label classification performance. Feature selection is a data processing technique that can overcome this problem. Feature selection techniques have two major approaches: filter and wrapper. This paper proposes a hybrid filter-wrapper technique combining two algorithms: Chi-square (CS) and ant colony optimization (ACO). In the first stage, CS is used to reduce the number of irrelevant features. The ACO method is in the second stage. The ACO is applied to select the efficient features and improve classifier performance. The experiment results show that CS-ACO, CS-grey wolf optimizer (GWO), CS, and without feature selection (FS) have a micro F1-score based multinomial naïve Bayes classifier including 80%, 79.75%, 79.64% and 77.78%. The result indicates that the CS-ACO algorithm is suitable for solving multi-label classification problems.

Optimizing gula apong production with an IoT-based temperature monitoring system

<p>Determining the quality of <em>gula apong</em> is crucial to optimizing its production, with cooking temperature being a key factor affecting both taste and shelf life. The <em>gula apong</em> industry faced challenges due to the lack of reliable real-time temperature monitoring methods during the cooking process. Traditional approaches were inefficient and inaccurate, leading to difficulties in maintaining consistent product quality and meeting market demands. This highlights the necessity of monitoring the temperature throughout each cooking process. This research aims to develop an internet of things (IoT)- based cooking temperature monitoring system to enhance quality control measures in the production of <em>gula apong</em>. The IoT prototype collects temperature data from the thermocouple sensor, then transmits it to cloud storage through a Wi-Fi communication network, utilizing the Node-RED platform for data processing and analysis. Data obtained from the on-site measurement shows that the optimal temperature for producing standard-quality <em>gula apong</em> is approximately around 165 °C. The recommended boiling temperature for Nipah sap is 140 °C. This IoT system can reduce the cooking time of <em>gula apong</em> to 3 hours from the traditional 4 to 6 hours. Utilizing the data acquired from this study helps the producers not only maintaining the quality of <em>gula apong</em> but also reduce the cooking time and cost.</p>

Development of an energy-autonomous eddy current sensor system for in-situ component monitoring

Energy-autonomous non-destructive testing (NDT) systems provide new opportunities for structural health monitoring. An NDT system that is energy-autonomous and transmits the recorded data wirelessly can be used in places that are not easily accessible for inspection. In addition, it can provide for continuous monitoring, which can improve the operational safety of various components and structures. This study presents a concept that enables energy-autonomous and wireless structural health monitoring using a splined shaft as a demonstrator. To monitor mechanical overloads, a material sensor is applied to critical areas of these components by a laser heat treatment. The structural change that occurs as a result of an overload can be monitored by the developed smart sensor system. This system consists of a material sensor, a low-cost eddy current sensor, an energy-efficient evaluation unit with an ultra-low-power microcontroller unit (MCU) and a Long-Range Wide Area Network (LoRaWAN) data transmission unit. Wireless data transmission via LoRaWAN provides access to the Industrial Internet of Things network, enabling the collection of NDT data from multiple smart sensors in a cloud and the processing of these data to assess the condition of the component from any location. It is shown that the MCU-based eddy current testing system is able to generate a similar signal quality for material characterisation compared to an industrial eddy current testing system, despite a much more compact design and a significantly higher energy efficiency. The energy autonomy of the smart sensor system is achieved by an energy harvesting system integrated into the splined shaft. In addition, three different testing scenarios are presented, which differ mainly in the scope of data processing and evaluation on the MCU, and thus also in the energy required.

Cost-aware optimal resource provisioning Map-Reduce scheduler for hadoop framework

<p><span lang="EN-US">Distributed data processing model has been one of the primary components in the case of data-intensive applications; furthermore, due to advancements in technologies, there has been a huge volume of data generation of diverse nature. Hadoop map reduce framework is responsible for adopting the ease of deployment mechanism in an open-source framework. The existing Hadoop MapReduce framework possesses high makespan time and high Input/Output overhead and it mainly affects the cost of a model. Thus, this research work presents an optimized cost aware resource provisioning MapReduce model also known as the cost-effective resource provisioning MapReduce (CRP-MR) model. CRP-MR model introduces the two integrated approaches to minimize the cost; at first, this model presents the optimal resource optimization and optimal Input/Output optimization cleansing in the Hadoop MapReduce (HMR) scheduler. CRP-MR is evaluated considering the bioinformatics dataset and CRP-MR performs better than the existing model. </span></p>

Orbital mechanics with the Global Positioning System

The recent availability of relatively inexpensive dual-frequency receivers for signals from Global Navigation Satellite Systems (GNSS) provides access to ultra-precise, real-time data such as positions and velocities of dozens of satellites orbiting the Earth. We discuss how these data can be obtained, processed, and analyzed either with or without the actual purchase of a GNSS receiver. The positional information can be used to verify Kepler's three laws at lowest order as well as to reveal the presence of higher-order perturbations such as the oblateness of the Earth and the gravitational influences of the Sun and Moon on these satellites. The supplementary material includes both introductory laboratory exercises and Python scripts used to gather and process data suitable for intermediate courses.

Blockchian Based Hybrid Consensus Algorithm for Data Security in Edge Computing

Abstract: Edge computing has emerged as a paradigm to process data closer to its source, enabling low-latency and real-time applications. However, the distributed and heterogeneous nature of edge environments poses significant challenges for ensuring data security and integrity. Blockchain technology offers a promising solution by providing a decentralized and immutable ledger for recording transactions. This abstract proposes a consensus algorithm tailored for securing data in edge computing environments using blockchain technology. The proposed consensus algorithm leverages a hybrid approach combining Proof of Authority (PoA) and Practical Byzantine Fault Tolerance (PBFT). PoA ensures that only trusted nodes within the edge network are eligible to participate in the consensus process, mitigating the risk of malicious attacks. PBFT enhances the efficiency and scalability of the consensus algorithm by allowing fast decision-making among a subset of nodes, thereby reducing latency in data validation and verification. Furthermore, smart contracts are employed to enforce data access control and execute predefined rules for data processing and sharing. These smart contracts facilitate secure and transparent interactions among edge devices, ensuring that only authorized parties can access and manipulate sensitive data. The integration of blockchain technology with edge computing offers several benefits, including enhanced data security, traceability, and accountability. By employing a tailored consensus algorithm, edge computing environments can effectively mitigate security threats while maintaining low-latency data processing capabilities. Future research directions may explore optimization techniques to further improve the performance and scalability of the proposed solution in large-scale edge networks.

Crack Detection of Structures using Artificial Intelligence System

Abstract: Structural Health Monitoring (SHM) having wider scope of operations such as structural audit, rehabilitation of structures and along with it a major scope of crack detection is also included. Effective crack detection is vital for ensuring the safety and longevity of civil engineering structures. Crack Detection in existing methods is tedious process, it requires costly equipment’s as well as time consuming, and inaccuracy in mechanical equipment’s. The technological advancements especially in the field of Artificial Intelligence (AI) maybe an acceptable solution for crack identification through mobile app using Image processing techniques. The mobile app can be used to accurately identify crack pattern and measure the size of cracks in structures (concrete and steel). The app needed to capture and process data from diverse sources, such as images, in real time. It allows for real-time monitoring of structures and ensures safety and integrity of it. The image processing algorithm is used for to accurately identifying various types of cracks in different lighting, environmental conditions, effective and optimized solution. This project explained about real-time data processing for crack detection and its challenges. This demanded robust backend infrastructure and efficient data processing techniques to deliver prompt and accurate results. User Interface and Experience developing an intuitive and user-friendly interface for the app while maintaining high performance.

Novel Approach for Vehicle Accident Tracking using IoT

Abstract: In recent years, there has been significant interestin integrating Internet of Things (IoT) technology into vehicle safety systems to enhance emergency response capabilities and road safety. This paper introduces a comprehensive Vehicle Accident Tracking System (VATS) that utilizes IoT principles. The proposed system incorporates various sensors and communication devices installed within vehicles to monitor theirstatus and detect potential accidents in real-time. When anaccident is detected, the system automatically triggers an alert mechanism, notifying relevant stakeholders such as emergency services and fleet managers. Key components of the VATSinclude accelerometers, GPS modules, and onboard communication units, which work together to provide precise accident geospatial data and vehicle condition data. The systemutilizes cloud-based platforms for data storage and processing, ensuring seamless integration with existing infrastructure and scalability. By continuously monitoring and analyzing vehicle telemetry data, the VATS facilitates proactive accident prevention measures and post-accident analysis. Additionally, it offers benefits such as reduced emergency response times, improved accident documentation for insurance claims, and enhanced overall road safety. Privacy and security concerns areaddressed through encryption protocols and access control mechanisms to protect sensitive information. Overall, the Vehicle Accident Tracking System presented in this paper represents a significant advancement in leveraging IoT technology to mitigate the impact of accidents on roadways, thereby contributing to a safer and more efficient transportation ecosystem.

Privacy Challenges and Solutions in Big Data Analytics: A Comprehensive Review

Abstract: Large data collections with a more intricate and diversified structure are referred to as "big data." These traits are typically associated with more challenges when it comes to storing, analyzing, and applying additional procedures or getting results. The technique of analyzing vast quantities of intricate data to uncover patterns or connections that are concealed is known as "big data analytics." Nonetheless, there is a clear conflict between big data's extensive use and its security and privacy. This paper examines privacy needs in big data, distinguishes between privacy and security, and focuses on privacy and security issues in large data. This study addresses the applications of privacy in business by utilizing established techniques like HybrEx, k-anonymity, T-closeness, and L-diversity. Many privacy-preserving techniques have been created for protection of privacy at various phases of the big data life cycle (such as data generation, storage, and processing). This work aims to explain the issues facing current privacy preservation strategies in big data and to give a comprehensive evaluation of them. This study also introduces new approaches to privacy preservation in big data, such as quick anonymization of massive data streams, identity-based anonymization, differential privacy, privacy preserving big data publishing, and hiding a needle in a haystack. This study discusses big data privacy and security issues in healthcare. A comparative analysis of some contemporary large data privacy approaches is also conducted. A comparative analysis of the latest big data privacy approaches is also conducted.

Desktop Voice Assistant

Abstract: Desktop voice assistants have become a popular interface for human-computer interaction because they allow users to speak with their computers in natural language. However, modern desktop voice assistants sometimes struggle to understand and respond to complex queries and context. Privacy concerns concerning the collection and use of user data also remain significant barriers. In order to enhance desktop voice assistants, we present a novel personalization and multimodal interaction technique in this research. By combining text, voice, and visual inputs, our approach improves the accuracy and relevance of the responses. By combining several modalities, our system is able to better understand context and user intent, leading to more meaningful interactions. Furthermore, we propose a user-centric personalization strategy whereby the voice assistant progressively learns each user's preferences and usage patterns. Because it depends less on centralized data collection and processing, this customized approach improves user experience while simultaneously addressing privacy concerns. Through a series of experiments and user surveys, we show how effective our approach is in improving the overall performance and user experience of desktop voice assistants. Our results show that the combination of multimodal interaction with personalization produces more accurate and contextually relevant responses, improving the user experience. All things considered, our research advances desktop voice assistant technology by addressing significant interface and personalization issues. Our approach has the potential to significantly improve desktop voice assistant usability and user acceptance, which could result in considerably more intuitive and natural human-computer interaction.

질적 메타분석을 통해 살펴본 교사-부모 관계에 관한 영유아교사의 경험

Objectives This study aims to identify factors that influence the formation of teacher-parent relationships as early childhood educators experience interactions with parents, and to explore how these variables affect changes in early childhood educators. Methods To achieve this, a qualitative meta-analysis was conducted on qualitative research materials regarding early childhood teacher-parent relationship experiences. The analysis included 18 qualitative research papers published in domestic journals (KCI listed) from 2000 to 2023. Through processes of data coding between in-dividual studies, case categorization and classification across studies, comparative analysis between studies, and deriving and interpreting results, the changes in teachers during the early childhood teacher-parent relationship process were identified. Results First, the factors of negative experience in the teacher-parent relationship were parental disposition, ex-cessive demands of parents, imbalance in the relationship, teachers' own factors, problems in the teacher training process, and emotional dissonance. Second, the teacher-parent relationship was shown to be a change stage in three stages: teacher-parent relationship experience, coping, and growth into a professional early childhood teacher. Conclusions This study attempted to clarify in depth by comprehensively comparing and interpreting individual qualitative studies on the experience of teacher-parent relationships. We identified relevant factors linked to the teacher-parent relationship and derived comprehensive results.