Volume Of Data Research Articles

Optimizing Top-K query processing on GPU using data compression and pre-filtering

In large-scale data analysis, efficient Top-K query processing is critical for numerous applications in science, industry, and society. Traditional approaches often involve substantial data transfer and computational overhead, making it difficult to meet the scalability and efficiency demands of modern datasets. This paper proposes a GPU-accelerated Top-K query processing method that integrates data compression and pre-filtering techniques to address these challenges. By partitioning and compressing data on the host side, it alleviates common PCIe bottlenecks in heterogeneous computing environments. A metadata-driven pre-filtering technique further reduces the data volume processed on the GPU, significantly improving query performance, particularly when handling anti-correlated datasets. Experimental results demonstrate that this method markedly reduces data transfer and processing time, confirming its effectiveness in enhancing the efficiency and scalability of Top-K query processing compared to existing methods.

Spectropolarimetric Inversion in Four Dimensions with Deep Learning (SPIn4D). I. Overview, Magnetohydrodynamic Modeling, and Stokes Profile Synthesis

The National Science Foundation’s Daniel K. Inouye Solar Telescope (DKIST) will provide high-resolution, multiline spectropolarimetric observations that are poised to revolutionize our understanding of the Sun. Given the massive data volume, novel inference techniques are required to unlock its full potential. Here, we provide an overview of our “SPIn4D” project, which aims to develop deep convolutional neural networks (CNNs) for estimating the physical properties of the solar photosphere from DKIST spectropolarimetric observations. We describe the magnetohydrodynamic (MHD) modeling and the Stokes profile synthesis pipeline that produce the simulated output and input data, respectively. These data will be used to train a set of CNNs that can rapidly infer the four-dimensional MHD state vectors by exploiting the spatiotemporally coherent patterns in the Stokes profile time series. Specifically, our radiative MHD model simulates the small-scale dynamo actions that are prevalent in quiet-Sun and plage regions. Six cases with different mean magnetic fields have been explored; each case covers six solar-hours, totaling 109 TB in data volume. The simulation domain covers at least 25 × 25 × 8 Mm, with 16 × 16 × 12 km spatial resolution, extending from the upper convection zone up to the temperature minimum region. The outputs are stored at a 40 s cadence. We forward model the Stokes profile of two sets of Fe i lines at 630 and 1565 nm, which will be simultaneously observed by DKIST and can better constrain the parameter variations along the line of sight. The MHD model output and the synthetic Stokes profiles are publicly available, with 13.7 TB in the initial release.

A New Compressed Data Acquisition Method for Power System Based on Chaotic Compressive Measurement

The digitalization level of the new power system driven by “dual carbon” is increasing, leading to a growth in the amount of data that need to be acquired. This has intensified the contradiction between data volume and acquisition capacity. Therefore, it is urgent to study compressed data acquisition methods for power systems based on data compression. In this regard, a novel compressed data acquisition method based on chaotic compressive measurement with the compressed sensing principle is proposed. Firstly, the advantages of applying compressed sensing are analyzed for data acquisition in power systems, and the key issues that need to be addressed are identified. Subsequently, a chaotic map is sampled based on the basic requirements of the measurement matrix in compressed sensing, and the chaotic compressive measurement matrix is constructed and optimized based on the sampling results. Next, the sparse data difference of the power system is used as the compression target for the optimized chaotic measurement matrix, and an acquisition process is designed to recover the complete power data from a small amount of compressed data. Finally, the proposed method is validated in a case study, and the results demonstrate that the method is correct and effective.

Navigating uncertainty with cybernetics principles: A scoping review of interdisciplinary resilience strategies for rail systems

AbstractCommon difficulties across industries are discovered in data management, where handling the volume, variety, and quality of data is crucial for informed decisions in uncertain environments. In this context, rail management must navigate complex decision‐making to ensure safety, service continuity, and cost‐effectiveness. The 2020 Stonehaven derailment is an example of the increasing vulnerability of rail infrastructure to environmental factors and systemic failures. It emphasizes the need for resilient systems, proficient at preventative maintenance and adaptable to escalating challenges. These matters further accentuate the need for context‐dependent strategies that bridge theoretical insights and practical applications. This scoping review explores strategies for decision‐making under uncertainty across sectors such as civil infrastructure, agriculture, water management, and emergency response. It unfolds a selection of procedures addressing the impacts of extreme weather and other unexpected disruptions. It also sets a foundation for future research to support rail infrastructure adaptation to climate change by advocating the use of cybernetic principles and artificial intelligence (AI) to enhance decision‐making processes. Cybernetics enables collaborative human‐AI methods, improving adaptability and resilience. However, balancing and incorporating diverse stakeholder viewpoints into decision chains remains difficult. While promising, substantial research and system improvements are needed to fully harness the potential of AI.

Selection of an Appropriate Extrinsic Camera Calibration Method for Handheld Mobile Mapping Systems

Mobile mapping systems integrate multiple sensors to collect large volumes of geospatial data in motion. With the growing demand for mapping enclosed and hard-to-reach areas, there has been significant advancement in handheld mobile mapping systems utilizing SLAM (Simultaneous Localization and Mapping) technology. To ensure theirdata is efficiently usable, these systems should produce oriented images, which are essential for visualization, point cloud colorization, and monoplotting. Achieving so requiresprecise extrinsic camera calibration. This research provides a comprehensive overview of existing extrinsic camera calibration methods and evaluates their suitability for applicationin handheld mobile mapping systems. The goal is to identify a method that meets the accuracy and practical needs of these systems, facilitating more effective data processingand utilization in challenging environments.

A Comparison of Open Data Observatories

Open Data Observatories refer to online platforms that provide real-time and historical data for a particular application context, e.g., urban/non-urban environments or a specific application domain. They are generally developed to facilitate collaboration within one or more communities through reusable datasets, analysis tools, and interactive visualizations. Open Data Observatories collect and integrate various data from multiple disparate data sources—some providing mechanisms to support real-time data capture and ingest. Data types can include sensor data (soil, weather, traffic, pollution levels) and satellite imagery. Data sources can include Open Data providers, interconnected devices, and services offered through the Internet of Things. The continually increasing volume and variety of such data require timely integration, management, and analysis, yet presented in a way that end-users can easily understand. Data released for open access preserve their value and enable a more in-depth understanding of real-world choices. This survey compares thirteen Open Data Observatories and their data management approaches - investigating their aims, design, and types of data. We conclude with research challenges that influence the implementation of these observatories, outlining some strengths and limitations for each one and recommending areas for improvement. Our goal is to identify best practices learned from the selected observatories to aid the development of new Open Data Observatories.

Advanced Deep Learning Approach for Smart Home Appliance Identification Using Recurrent Neural Networks with LSTM

In the Internet of Things (IoT) domain, vast numbers of smart devices are interconnected, generating large volumes of data requiring advanced management mechanisms. One major challenge in smart environments is the ability to accurately distinguish and categorize the various types of objects within these systems. To address this issue, the study introduces a recurrent neural network (RNN) model designed for classifying data from smart home devices. Using a dataset from Kaggle, the research outlines the processes of data collection, loading, normalization, and model development. The RNN, enhanced with long short-term memory (LSTM) layers, was trained and evaluated, showing notable improvements in training and validation accuracy over ten epochs. The model achieved a test accuracy of 83.25%, a loss of 35.4%, a precision of 85%, and a recall of 81%. The evaluation of the model on the test set includes a detailed analysis using ROC curves, area under the curve (AUC) scores for multi-class classification, and a confusion matrix. With an AUC score of 0.9896, the model demonstrated exceptional performance in accurately classifying IoT device categories. These results suggest that the LSTM-equipped RNN offers strong learning efficiency and generalization, making it a highly suitable approach for IoT device classification. Additionally, the article explores the concept of IoT and reviews recent advancements in using deep learning models across various IoT sectors, including smart homes, industrial systems, and healthcare. Future research could aim to improve the model’s real-time processing abilities and scalability and incorporate a wider variety of IoT data types to enhance its practical applications and expand its utility across more IoT environments.

Compression of Hyperspectral Image using JPEG Compression Algorithm

This paper delves into the considerable challenges of working with hyperspectral images, which are notably large and multidimensional, with file sizes often surpassing hundreds of megabytes. Hyperspectral imaging captures light across a continuous range of wavelengths, providing detailed spectral information for each pixel. This rich dataset is invaluable for applications such as environmental monitoring, precision agriculture, mineral exploration, and medical diagnostics, where accurate spectral data aids in identifying materials and detecting subtle variations. However, the immense data volume not only strains storage and transmission resources but also requires efficient processing and analysistechniques to handle the high-dimensional data without compromising quality. Additionally, compression methods are essential to manage storage constraints and improve real-time usability, but they must balance data reduction with the preservation of spectral integrity for effective analysis and application.

Theoretical Analysis and Experimental Evaluation of Wide-Lane Combination for Single-Epoch Positioning with BeiDou-3 Observations

Multi-frequency signals can enable some wide-lane (WL) observations to achieve instantaneous ambiguity resolution (AR) in complex scenarios, but simply adding WL observations will also place additional pressure on real-time kinematic data transmission. With the official service of the third-generation Beidou Navigation Satellite System, which broadcasts five-frequency signals, this dilemma has become increasingly evident. It is significant to explore multi-frequency observation combination methods that take into account both positioning precision and data transmission burden. In this work, we use the least squares method to derive the theoretical precision of the single-epoch WL combination of 16 schemes with varying frequency numbers (three or more) under the ionosphere-fixed model and the ionosphere-float model. The baseline solutions of 4.3 km and 93.56 km confirm that the positioning results are broadly consistent with the theoretical derivations under both models. In the ionosphere-fixed mode, the five-frequency scheme (B1C, B1I, B3I, B2b, B2a) yields the best positioning performance, improving the 3-dimensional positioning error standard deviation, circle error probable (CEP), and spherical error probable at 75% probability by 7.8%, 11.5%, and 6.7%, respectively, compared with the optimal triple-frequency scheme (B1C, B3I, B2a). Under the ionosphere-float model, the quad-frequency scheme (B1C, B3I, B2b, B2a) provides the best positioning performance, with only the CEP at 75% improving by 1.3% over the triple-frequency scheme. Given that the optimal triple-frequency scheme has a lower data volume, this work recommends it as the preferred scheme.

HyperGAN: A Hyperspectral Image Fusion Approach Based on Generative Adversarial Networks

The objective of hyperspectral pansharpening is to fuse low-resolution hyperspectral images (LR-HSI) with corresponding panchromatic (PAN) images to generate high-resolution hyperspectral images (HR-HSI). Despite advancements in hyperspectral (HS) pansharpening using deep learning, the rich spectral details and large data volume of HS images place higher demands on models for effective spectral extraction and processing. In this paper, we present HyperGAN, a hyperspectral image fusion approach based on Generative Adversarial Networks. Unlike previous methods that deepen the network to capture spectral information, HyperGAN widens the structure with a Wide Block for multi-scale learning, effectively capturing global and local details from upsampled HSI and PAN images. While LR-HSI provides rich spectral data, PAN images offer spatial information. We introduce the Efficient Spatial and Channel Attention Module (ESCA) to integrate these features and add an energy-based discriminator to enhance model performance by learning directly from the Ground Truth (GT), improving fused image quality. We validated our method on various scenes, including the Pavia Center, Eastern Tianshan, and Chikusei. Results show that HyperGAN outperforms state-of-the-art methods in visual and quantitative evaluations.

Role of Social Media in Supporting Information and Intelligence Gathering by Densus 88

Terrorism in the digital era has rapidly evolved, with terrorist groups increasingly utilizing social media platforms to spread radical ideologies, recruit new members, and plan attacks. Indonesia, having experienced various terrorism incidents, continues to face this ongoing threat. In this context, the Special Detachment 88 Anti-Terrorism Unit (Densus 88) leverages social media as a critical tool for intelligence gathering and preventing terrorist activities. This study aims to explore how Densus 88 utilizes platforms such as Facebook, Twitter, Instagram, and YouTube to identify, monitor, and analyze terrorism-related activities. Through a qualitative approach, this research investigates the perspectives and experiences of Densus 88 members involved in using social media as an intelligence tool. The findings indicate that social media provides a continuous data stream that can be used to monitor activities, track trends, and identify potential threats at an early stage. However, significant challenges remain, including the vast volume and complexity of data and difficulties in distinguishing genuinely dangerous activities from unrelated content. This study provides valuable insights into the strategic use of social media in counter-terrorism operations and suggests measures to enhance the effectiveness of intelligence gathering in the digital era.

A Noise-Oriented and Redundancy-Aware Instance Selection Framework

Fine-tuning transformer-based deep learning models is currently at the forefront of natural language processing (NLP) and information retrieval (IR) tasks. However, fine-tuning these transformers for specific tasks, especially when dealing with ever-expanding volumes of data, constant retraining requirements, and budget constraints, can be computationally and financially costly, requiring substantial energy consumption and contributing to carbon dioxide emissions. This article focuses on advancing the state-of-the-art (SOTA) on instance selection (IS) – a range of document filtering techniques designed to select the most representative documents for the sake of training. The objective is to either maintain or enhance classification effectiveness while reducing the overall training (fine-tuning) total processing time. In our prior research, we introduced the E2SC framework, a redundancy-oriented IS method focused on transformers and large datasets – currently the state-of-the-art in IS. Nonetheless, important research questions remained unanswered in our previous work, mostly due to E2SC's sole emphasis on redundancy. In this article, we take our research a step further by proposing biO-IS – an extended bi - ob jective i nstance s election solution, a novel IS framework aimed at simultaneously removing redundant and noisy instances from the training. biO-IS estimates redundancy based on scalable, fast, and calibrated weak classifiers and captures noise with the support of a new entropy-based step. We also propose a novel iterative process to estimate near-optimum reduction rates for both steps. Our extended solution is able to reduce the training sets by 41% on average (up to 60%) while maintaining the effectiveness in all tested datasets, with speedup gains of 1.67 on average (up to 2.46x). No other baseline, not even our previous SOTA solution, was capable of achieving results with this level of quality, considering the tradeoff among training reduction, effectiveness, and speedup. To ensure reproducibility, our documentation, code, and datasets can be accessed on GitHub – https://github.com/waashk/bio-is .

Photon-Counting Detector CT: Advances and Clinical Applications in Cardiovascular Imaging.

Since the approval of the first dual-source photon-counting detector CT (PCD-CT) in the fall of 2021, significant insights have been gained in its application for cardiovascular imaging. This review aims to provide a comprehensive overview of the current state of knowledge and the growing body of research literature, illustrating innovative applications and perspectives through case examples.We conducted a structured literature review, identifying relevant studies via Google Scholar and PubMed, using the keywords "photon-counting detector", "cardiovascular CT", "cardiac CT", and "ultra-high-resolution CT". We analyzed studies published since January 2015. Additionally, we integrated our own clinical experiences and case examples.In addition to the well-known benefit of increased temporal resolution offered by dual-source scanners, dual-source PCD-CT provides three key advantages: 1) Optimized geometric dose efficiency with an improved contrast-to-noise ratio, 2) intrinsic spectral sensitivity, and 3) the ability for ultrahigh-resolution CT. This technology enables improved image quality or radiation dose reduction in established cardiovascular protocols. Its use in non-invasive cardiac diagnostics for obese patients, those with a high plaque burden, or after stent implantation appears technically feasible, potentially expanding the scope of CT. The spectral sensitivity also allows tailored image acquisition, reducing metallic artifacts and contrast agent doses in patients with renal impairment. Early studies and clinical experience support these potential applications of PCD-CT in cardiovascular diagnostics, suggesting workflow optimization and improved patient management.However, challenges remain, including high costs, large data volumes, somewhat longer reconstruction times, and technical difficulties in combining spectral sensitivity with ultra-high resolution. Prospective randomized studies with clinical endpoints are lacking to confirm the clear advantage over conventional scanners. Future research should focus on endpoint-based studies and robust cost-benefit analyses to evaluate the potential of this technology and facilitate its evidence-based integration in clinical practice. · Photon-counting detector CT represents a technological advancement in computed tomography.. · Spectral sensitivity enhances iodine signal and minimizes artifacts.. · Ultra-high-resolution CT allows precise imaging, even in stents and advanced sclerosis.. · This technology must be validated through endpoint-based, randomized studies.. · Hagar MT, Schlett CL, Oechsner T et al. Photon-Counting Detector CT: Advances and Clinical Applications in Cardiovascular Imaging. Fortschr Röntgenstr 2024; DOI 10.1055/a-2452-0288.

Synthetizing Virtual Construction Images to Strengthen Real Data Volume and Variety in Real-world Application Scenarios

Despite the potential of synthetic construction images, it remains unknown whether they can strengthen real-data volume and variety in real-world scenarios, wherein a given, real training dataset is small and biased, or large but biased. To address this, we synthetize artificial images in a computer environment to strengthen a real training dataset and test its supplementary effects in both scenarios. Specifically, we simulate a worker’s physical behaviors, capture 2D synthetic images, and annotate its bounding box using a 3D-2D projection algorithm. After combining these synthetic images with a real dataset, we train a vision-based worker detection model and evaluate its performance in each scenario. Results show that the model’s performance is improved by up to 59.1% and 12.8% in each scenario, respectively, comparing to only adopting real images. This indicates that synthetic images can enrich the restricted volume and variety of a given, real training dataset in field application scenarios.

Multi-Model Fusion Demand Forecasting Framework Based on Attention Mechanism

The accuracy of demand forecasting is critical for supply chain management and strategic business decisions. However, as data volumes grow and demand patterns become increasingly complex, traditional forecasting methods encounter significant challenges in processing intricate multi-dimensional data and achieving a satisfactory predictive accuracy. To address these challenges, this paper proposed an end-to-end multi-model demand forecasting framework based on attention mechanisms. The framework employs a dual attention mechanism to dynamically extract features from both the temporal and product dimensions, while integrating conditional information captured through convolutional neural networks, thereby enhancing its ability to model complex demand patterns. Additionally, a channel attention mechanism is introduced to perform the weighted fusion of outputs from multiple predictive models, thereby overcoming the limitations of single-model approaches and improving adaptability to varying demand patterns across diverse scenarios. The experimental results demonstrate that the proposed method outperforms conventional approaches across several evaluation metrics, achieving a 42% reduction in Mean Squared Error (MSE) compared to the baseline model. This notable improvement enhances both the accuracy and stability of demand forecasting. The framework offers valuable insights for addressing large-scale and complex demand patterns, providing guidance for precise decision-making and resource optimization within supply chain management. Future research will concentrate on further enhancing the model’s generalization capability to manage missing data and demand fluctuations. Additionally, efforts will focus on integrating diverse heterogeneous data sources to assess its performance in various practical scenarios, ultimately improving the model’s accuracy and flexibility.

Efficient Video Compression Using Afterimage Representation.

Recent advancements in large-scale video data have highlighted the growing need for efficient data compression techniques to enhance video processing performance. In this paper, we propose an afterimage-based video compression method that significantly reduces video data volume while maintaining analytical performance. The proposed approach utilizes optical flow to adaptively select the number of keyframes based on scene complexity, optimizing compression efficiency. Additionally, object movement masks extracted from keyframes are accumulated over time using alpha blending to generate the final afterimage. Experiments on the UCF-Crime dataset demonstrated that the proposed method achieved a 95.97% compression ratio. In binary classification experiments on normal/abnormal behaviors, the compressed videos maintained performance comparable to the original videos, while in multi-class classification, they outperformed the originals. Notably, classification experiments focused exclusively on abnormal behaviors exhibited a significant 4.25% improvement in performance. Moreover, further experiments showed that large language models (LLMs) can interpret the temporal context of original videos from single afterimages. These findings confirm that the proposed afterimage-based compression technique effectively preserves spatiotemporal information while significantly reducing data size.

Оптимізація обробки великих даних за допомогою «лінивих обчислень»: систематичний огляд технологій та застосування

The article examines the concept of lazy operations and its application for efficient processing of large volumes of data. The main principles of lazy computations, their implementation in various programming languages, and strategies for effective use in Big Data processing are analyzed. The advantages and limitations of the lazy approach are investigated, particularly regarding memory savings, performance improvement, and the ability to work with infinite data streams. A concept for selecting computation strategies based on data size and computational complexity is proposed.

A personalized query method for spatial keywords in indoor environments

AbstractWith the rapid development of indoor Location Based Services (LBS), a growing volume of textual data is being generated in indoor environments. Consequently, indoor spatial keyword query holds significant potential for development in the coming years. However, existing methods for indoor spatial keyword queries often neglect the personalized needs of users. To solve this problem, we propose an Indoor Spatial Keyword Personalized Query (ISKPQ) method. First, a novel index structure called ISKIR-tree has been designed. This index integrates Hilbert encoding techniques and introduces Bloom filters and distance matrices to enhance the efficiency of processing indoor spatial keyword problems. Subsequently, an efficient pruning algorithm based on the ISKIR-tree index is proposed to refine the dataset effectively. Finally, a comprehensive scoring function that considers text similarity, spatial proximity, and user preferences is introduced to score and rank the pruned data points, thereby filtering out the optimal query results that meet users’ personalized requirements. Theoretical analysis and experimental studies demonstrate the outstanding performance of the proposed method in terms of both efficiency and accuracy.

Categorizing and Determining of BI-RAD Breast Cancer Score by Using Deep Learning Techniques

Breast cancer is second only to skin cancer in prevalence. Failure to detect and treat breast cancer early could be fatal. Breast cancer is more common in underdeveloped countries due to low health awareness and limited early detection methods. However, detecting equipment and radiography advances have helped solve this difficulty and suggest a drop in mortality. Along with ultrasounds, mammography, Magnetic Resonance Imaging (MRI), Computed tomography (CT) scan, and Positron emission tomography (PET) scans are used for diagnosis. These devices add data to test findings for greater accuracy. Diagnosticians determine breast cancer stage by examining tumor size, shape, and spread. Artificial intelligence users employ these factors as fixed values to classify the illness using region of interest (RIO), Gray Level Co-Occurrence Matrix (GLCAM), clustering, or mass constraints like diameter and volume. Artificial intelligence algorithms have created many techniques to extract features from massive volumes of data through training, such as deep learning. The study investigated DL algorithms (Attention mechanism and efficintNetB7 ) to determine the breast imaging-reporting and data system (BI-RADS) score by categorizing mammography images. The dataset Digital Database for Screening Mammography (DDSM) used in this study, acquired from Kaggle, contains 55,890 training samples, of which 86% are negative, and 14% are positive, categorized into classes C1, C2, C3, C4, and C5. This work utilized deep learning techniques and algorithms to accurately categorize breast cancer at 93%, 92%, 90%, and 93% for each category (C2, C3, C4, and C5). We developed the BI-RADS classifier model using the bycharm framework, an integrated development environment (IDE) for Python programming, and using multiple libraries, namely NumPy, Pandas, TensorFlow, Keras, Matplotlib, and Seaborn. This study used a laptop with the following specifications: an Intel Core i7 CPU, 16GB of RAM, and Intel RTX integrated graphics.

Building an AI Portfolio: A Technical Guide for Modern Logistics Specialists

This technical article provides a comprehensive framework for logistics specialists to build an effective AI portfolio in modern supply chain management. As the industry processes massive volumes of data annually, the integration of artificial intelligence has revolutionized logistics operations. The article addresses critical aspects, including demand forecasting, route optimization, inventory management, and security implementations. With AI implementations demonstrating significant performance improvements across cycle time reduction, transportation expense optimization, and inventory accuracy enhancement, the need for structured portfolio development has become increasingly important. The article outlines detailed technical requirements, implementation frameworks, and best practices for developing AI solutions in logistics, supported by real-world performance metrics and industry standards. Special attention is given to security considerations, scalability architecture, and professional development strategies, providing logistics professionals with a holistic approach to AI integration. The comprehensive coverage encompasses technical depth and practical implementation guidelines, making it a valuable resource for professionals seeking to advance their careers in AI-driven logistics operations.