Data Chunks Research Articles

Speed-Dedup: A New Deduplication Framework for Enhanced Performance and Reduced Overhead in Scale-Out Storage

Conventional deduplication systems face critical challenges such as excessive write amplification, high read/write latency, and sub-optimal storage utilization. These limitations often undermine the performance benefits of deduplication by slowing down I/O acknowledgements due to amplified deduplication I/Os, excessive data chunk replication, and strict consistency requirements. To address these issues, we present Speed-Dedup, a novel deduplication framework that employs a deduplicated primary–semi-deduplicated replica object approach. This strategy reduces write amplification by restricting deduplication to the primary object while maintaining a semi-deduplicated replica object used for immediate read/write acknowledgements, thus enhancing I/O latency and storage efficiency. Speed-Dedup also replaces traditional strong consistency models with eventual consistency, allowing for non-blocking read operations and improving overall system throughput. Experimental results demonstrate that Speed-Dedup significantly outperforms traditional methods like GRATE and CAO, showing up to 21% improvement in I/O performance under low deduplication ratios and maintaining 14% or more gains under higher ratios. Additionally, write amplification is substantially reduced and latency improves by over 100% with faster recovery times during system failures. These findings highlight the effectiveness of Speed-Dedup as a scalable and efficient solution.

Fault3DNnet: A lightweight 3D seismic fault detection network with bidirectional decoding

Fault detection from seismic data is a critical component of hydrocarbon exploration and production. In recent years, deep-learning techniques have made tremendous achievements and breakthroughs in seismic fault detection. Especially U-nets, represented by FaultSeg3D and its variants, are a dominant method for 3D deep-learning-based seismic fault interpretation. The incorporation of state-of-the-art network modules or training strategies with U-nets continuously improves the performance of fault detection for field seismic data, usually at the cost of increased computational intensity and hardware consumption. We design a lightweight bidirectional decoding network, Fault3DNnet, for fault detection from 3D seismic data volume. To make it simple and concise, our Fault3DNnet is equipped with basic conventional convolution and regular operation modules. The two decoding branches in Fault3DNnet could provide complementary information and improve fault detection performance. The superiority of Fault3DNnet is verified using publicly available synthetic and four field seismic data sets. The experimental results indicate that the method predicts fault structures with improved completeness and continuity compared with FaultSeg3D. Meanwhile, the parameter and computational quantity of the network are only 1/7 and 1/5 of FaultSeg3D, respectively. With the same graphics processing unit memory consumption, Fault3DNnet can handle the size of approximately six times the volume of 3D seismic data during inferencing as that of FaultSeg3D. It helps to improve prediction stability and decrease discontinuous artifacts at the stitching boundaries when predicting the large field data in chunks. In general, our Fault3DNnet delivers superior fault detection results while greatly reducing computational intensity and memory usage.

TS-DM: A Time Segmentation-Based Data Stream Learning Method for Concept Drift Adaptation.

Concept drift arises from the uncertainty of data distribution over time and is common in data stream. While numerous methods have been developed to assist machine learning models in adapting to such changeable data, the problem of improperly keeping or discarding data samples remains. This may results in the loss of valuable knowledge that could be utilized in subsequent time points, ultimately affecting the model's accuracy. To address this issue, a novel method called time segmentation-based data stream learning method (TS-DM) is developed to help segment and learn the streaming data for concept drift adaptation. First, a chunk-based segmentation strategy is given to segment normal and drift chunks. Building upon this, a chunk-based evolving segmentation (CES) strategy is proposed to mine and segment the data chunk when both old and new concepts coexist. Furthermore, a warning level data segmentation process (CES-W) and a high-low-drift tradeoff handling process are developed to enhance the generalization and robustness. To evaluate the performance and efficiency of our proposed method, we conduct experiments on both synthetic and real-world datasets. By comparing the results with several state-of-the-art data stream learning methods, the experimental findings demonstrate the efficiency of the proposed method.

Deduplication-Aware Healthcare Data Distribution in IoMT

As medical sensors undergo expeditious advancements, there is rising interest in the realm of healthcare applications within the Internet of Medical Things (IoMT) because of its broad applicability in monitoring the health of patients. IoMT proves beneficial in monitoring, disease diagnosis, and better treatment recommendations. This emerging technology aggregates real-time patient health data from sensors deployed on their bodies. This data collection mechanism consumes excessive power due to the transmission of data of similar types. It necessitates a deduplication mechanism, but this is complicated by the variable sizes of the data chunks, which may be either very small or larger in size. This reduces the likelihood of efficient chunking and, hence, deduplication. In this study, a deduplication-based data aggregation scheme was presented. It includes a Delimiter-Based Incremental Chunking Algorithm (DICA), which recognizes the breakpoint among two frames. The scheme includes static as well as variable-length windows. The proposed algorithm identifies a variable-length chunk using a terminator that optimizes the windows that are variable in size, with a threshold limit for the window size. To validate the scheme, a simulation was performed by utilizing NS-2.35 with the C language in the Ubuntu operating system. The TCL language was employed to set up networks, as well as for messaging purposes. The results demonstrate that the rise in the number of windows of variable size amounts to 62%, 66.7%, 68%, and 72.1% for DSW, RAM, CWCA, and DICA, respectively. The proposed scheme exhibits superior performance in terms of the probability of the false recognition of breakpoints, the static and dynamic sizes of chunks, the average sizes of chunks, the total attained chunks, and energy utilization.

Adaptive bagging-based dynamic ensemble selection in nonstationary environments

The classification of nonstationary data streams presents a substantial challenge mainly due to the simultaneous presence of class imbalance and concept drift. Ensemble learning is a widely acknowledged approach to address these two problems. Nevertheless, most of the existing methods establish independent strategies for each, ignoring their relationship and interactions, which prevents the corresponding strategies from performing as expected. This paper presents a novel algorithm AB-DES (Adaptive Bagging-based Dynamic Ensemble Selection), which employs bagging technique and dynamic ensemble selection system. In AB-DES, instances are stored to balance subsequent data chunks instead of relying solely on resampling methods. Then, all base classifiers are updated using a new dual adaptive sampling bagging strategy, which can help to reduce the classification inaccuracies involving by class imbalance. In the prediction phase, DES system constructs different ensemble models for different instances. The inclusion of a sliding window-based drift detector makes the above two strategies adapt to concept drift. Experimental results show that AB-DES achieves higher classification accuracy and exhibits enhanced robustness compared to 8 state-of-the-art methods on 10 synthetic datasets and 5 real world datasets with different types of concept drift.

A STUDY ON THE EMERGING TRENDS OF DATA JOURNALISM AND ENVIRONMENT IN INDIA

There is a belief that Data journalism is relatively a new term for environmental practices, but in fact, the relation of environment and data goes way long back than one can think. Now more advanced and visual methods have replaced the older one and due to which the relation between the two variables is more visible. Environment and data were always related. But with the advances in computer-based reporting, the use of data in environmental reporting has excelled. With the increasing population and human penetration in nature, has resulted in many issues that for which mankind can be blamed. Now whenever any environment is affected the numbers of casualties are huge. Not only casualties but everything it touches is huge, be the damage, the geography it covered, anything. It becomes a huge task for the journalist to compile that whole data and show it in understandable manner. Because, until and unless the proper message is conveyed, people won’t be able to understand the gravity of the disaster. And to combat this problem of dealing huge chunk of data is data journalism. In this chapter we will see how data journalism was used to show the environmental issues.

Bounded-memory adjusted scores estimation in generalized linear models with large data sets

The widespread use of maximum Jeffreys’-prior penalized likelihood in binomial-response generalized linear models, and in logistic regression, in particular, are supported by the results of Kosmidis and Firth (Biometrika 108:71–82, 2021. https://doi.org/10.1093/biomet/asaa052), who show that the resulting estimates are always finite-valued, even in cases where the maximum likelihood estimates are not, which is a practical issue regardless of the size of the data set. In logistic regression, the implied adjusted score equations are formally bias-reducing in asymptotic frameworks with a fixed number of parameters and appear to deliver a substantial reduction in the persistent bias of the maximum likelihood estimator in high-dimensional settings where the number of parameters grows asymptotically as a proportion of the number of observations. In this work, we develop and present two new variants of iteratively reweighted least squares for estimating generalized linear models with adjusted score equations for mean bias reduction and maximization of the likelihood penalized by a positive power of the Jeffreys-prior penalty, which eliminate the requirement of storing O(n) quantities in memory, and can operate with data sets that exceed computer memory or even hard drive capacity. We achieve that through incremental QR decompositions, which enable IWLS iterations to have access only to data chunks of predetermined size. Both procedures can also be readily adapted to fit generalized linear models when distinct parts of the data is stored across different sites and, due to privacy concerns, cannot be fully transferred across sites. We assess the procedures through a real-data application with millions of observations.

Discrete online cross-modal hashing with consistency preservation

Online cross-modal hashing has attracted widespread attention with the rapid expansion of large-scale streaming data, which can reduce storage requirements and enhance efficiency for online cross-modal retrieval. However, despite promising progress, existing methods still suffer from defective accuracy in a way, primarily attributed to two issues: insufficient semantic information exploitation and mismatched training-retrieval process. To address these challenges, we propose a novel supervised hashing method with dual consistency preservation, called Discrete Online Cross-Modal Hashing (DOCMH). On the one hand, we design more informative continuous semantic labels and fine-grained similarity graphs to preserve semantic consistency across different streaming data chunks and modality representations. On the other hand, we propose an effective modality deviation calibration mechanism for preserving learning process consistency between the training and retrieval phases. Extensive experiments on three widely used benchmark datasets demonstrate the superior performance of the proposed DOCMH under various scenarios.

Online Cross-modal Hashing With Dynamic Prototype

Online cross-modal hashing has received increasing attention due to its efficiency and effectiveness in handling cross-modal streaming data retrieval. Despite the promising performance, these methods mainly focus on the supervised learning paradigm, demanding expensive and laborious work to obtain clean annotated data. Existing unsupervised online hashing methods mostly struggle to construct instructive semantic correlations among data chunks, resulting in the forgetting of accumulated data distribution. To this end, we propose a Dynamic Prototype-based Online Cross-modal Hashing method, called DPOCH. Based on the pre-learned reliable common representations, DPOCH generates prototypes incrementally as sketches of accumulated data and updates them dynamically for adapting streaming data. Thereafter, the prototype-based semantic embedding and similarity graphs are designed to promote stability and generalization of the hashing process, thereby obtaining globally adaptive hash codes and hash functions. Experimental results on benchmarked datasets demonstrate that the proposed DPOCH outperforms state-of-the-art unsupervised online cross-modal hashing methods.

GPU optimization techniques to accelerate optiGAN—a particle simulation GAN

The demand for specialized hardware to train AI models has increased in tandem with the increase in the model complexity over the recent years. Graphics processing unit (GPU) is one such hardware that is capable of parallelizing operations performed on a large chunk of data. Companies like Nvidia, AMD, and Google have been constantly scaling-up the hardware performance as fast as they can. Nevertheless, there is still a gap between the required processing power and processing capacity of the hardware. To increase the hardware utilization, the software has to be optimized too. In this paper, we present some general GPU optimization techniques we used to efficiently train the optiGAN model, a Generative Adversarial Network that is capable of generating multidimensional probability distributions of optical photons at the photodetector face in radiation detectors, on an 8GB Nvidia Quadro RTX 4000 GPU. We analyze and compare the performances of all the optimizations based on the execution time and the memory consumed using the Nvidia Nsight Systems profiler tool. The optimizations gave approximately a 4.5x increase in the runtime performance when compared to a naive training on the GPU, without compromising the model performance. Finally we discuss optiGANs future work and how we are planning to scale the model on GPUs.

Task Scheduling with Dynamic Load Balancing for Cloud Computing

Using powerful cryptographic techniques and complex machine learning algorithms, this study explores an enhanced way for dynamic job scheduling, data recovery, and workflow management in cloud computing. To improve data recovery procedures and optimize job scheduling, we investigate the combination of the Random Forest, Q-learning from artificial neural networks (ANNs), and the J48 decision tree algorithms. To ensure data redundancy and integrity, we also use the Kochi matrix for effective data chunk generation and backup. The Advanced Encryption Standard (AES) is used to protect data. Our all-encompassing approach is to increase the effectiveness of resource allocation, hasten data recovery, and offer strong security in cloud environments. The efficiency of the suggested methodologies in improving overall system performance, security, and reliability is highlighted by the experimental findings, which show notable advancements over conventional approaches. Key Words: Load Balancing, Machine Learning, ANN, Random Forest, AES, Q-learning

Dynamic Model Interpretation-Guided Online Active Learning Scheme for Real-Time Safety Assessment.

Chunk-level real-time safety assessment of dynamic systems is a critical component of industrial processes, which is essential to prevent hazards and reduce the risk of injury or damage to equipment and facilities, especially in nonstationary environments. In this context, multiple real and complex concept drifts are inevitable in industrial settings, making it crucial to understand their detection and adaptation processes. The incremental learning scheme should also be well considered. However, existing methods have certain limitations in dealing with such issues. In this article, a dynamic model interpretation-guided online active learning scheme, termed a dynamic model interpretation-guided learning scheme (DMI-LS), is proposed. Specifically, the model update strategy with chunk data is designed based on the implementation of the broad learning system. A novel query strategy is then investigated to consider the ranking preference difference, which relies on the interpretation generated by the explainable artificial intelligence method. Several experiments based on the JiaoLong deep-sea manned submersible data are conducted to verify the effects of the proposed DMI-LS. The results show that it outperforms the other advanced existing approaches with different settings in most scenarios.

Impact of Design Decisions on Performance of Embarrassingly Parallel .NET Database Application

The implementation of parallel applications is always a challenge. It embraces many distinctive design decisions that are to be taken. The paper presents issues of parallel processing with use of .NET applications and popular Database Management Systems (DBMSes). In the paper, four design dilemmas are addressed: how efficient is the auto-parallelism implemented in the .NET TPL library, how do popular DBMSes differ in serving parallel requests, what is the optimal size of data chunks in the data parallelism scheme, and how the TPL auto-parallelism behaves in the public clouds. They are analyzed in the context of the typical and practical business case originated from IT solutions which are dedicated for the energy market participants. The paper presents the results of experiments conducted in a controlled, on-premises and cloud environments. The experiments allowed to compare the performance of the TPL auto-parallelism with a wide range of manually set numbers of worker threads. They also helped to evaluate four DBMSes: Oracle, MySQL, PostgreSQL, and MSSQL in the scenario of serving parallel queries. Finally, they showed the impact of data chunk sizes on the overall performance.

A randomized encryption deduplication method against frequency attack

Encryption deduplication technology first encrypts the file or chunks of data, then detects and removes duplicate ciphertexts, storing only one copy on the cloud storage server (CSP). Although traditional encryption deduplication schemes have the function of ciphertext deduplication, they also have a serious problem. Since the key generation algorithm and the encryption algorithm are deterministic, the frequency of plaintext chunking will be leaked. It is possible for the attacker to infer the information of the plaintext. We propose a Randomized Encryption deduplication method against Frequency Attack (REFA) that provides high randomness in ciphertexts. The core idea of REFA is that after obtaining a convergence key generated with the aid of a key server, the user randomizes the convergence key with a random value of the same length. Then, REFA encrypts the plaintext chunk with a randomized key. Our scheme hides the random value in the ciphertext and encrypts the convergence key for user ownership verification. REFA perfectly masks the frequency of plaintext chunks and has high storage efficiency and data security. Furthermore, the CSP performs ciphertext updates with ciphertexts uploaded by subsequent users. REFA can effectively protect against frequency analysis attacks.

The Role of Emotions in Qualitative Analysis: Researchers’ Perspectives

Qualitative research is an inherently social and relational endeavor that relies on and engages our emotions. Yet, researchers receive little guidance on how to engage emotions without being swayed by personal biases. Lustick (2021) developed a framework called “emotion coding” for systematically engaging thoughts and emotions in qualitative data analysis by asking what a chunk of data can teach us about ourselves, our participants, and our study. In this study, we interviewed 15 researchers who had tried using the emotion coding technique, about their impressions of this technique and the role of emotion in qualitative research overall. Framed by Goffman and Hochschild’s theories of emotion work, we find that emotions play a key role in researchers’ decisions throughout the research process, but that researchers are more likely to engage in emotion work (adjusting their feelings) when they perceive their participants hold a different level of privilege than they do. Emotion coding was a useful way to be more aware of their emotions and more intentional about how these emotions influence their findings and design decisions. We discuss implications for qualitative methodology.

DATA FINDING, SHARING AND DUPLICATION REMOVAL IN THE CLOUD

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

CLOUD BASED DUPLICATION REMOVAL SYSTEM

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Deep temporal clustering features for speech emotion recognition

Deep clustering is a popular unsupervised technique for feature representation learning. We recently proposed the chunk-based DeepEmoCluster framework for speech emotion recognition (SER) to adopt the concept of deep clustering as a novel semi-supervised learning (SSL) framework, which achieved improved recognition performances over conventional reconstruction-based approaches. However, the vanilla DeepEmoCluster lacks critical sentence-level temporal information that is useful for SER tasks. This study builds upon the DeepEmoCluster framework, creating a powerful SSL approach that leverages temporal information within a sentence. We propose two sentence-level temporal modeling alternatives using either the temporal-net or the triplet loss function, resulting in a novel temporal-enhanced DeepEmoCluster framework to capture essential temporal information. The key contribution to achieving this goal is the proposed sentence-level uniform sampling strategy, which preserves the original temporal order of the data for the clustering process. An extra network module (e.g., gated recurrent unit) is utilized for the temporal-net option to encode temporal information across the data chunks. Alternatively, we can impose additional temporal constraints by using the triplet loss function while training the DeepEmoCluster framework, which does not increase model complexity. Our experimental results based on the MSP-Podcast corpus demonstrate that the proposed temporal-enhanced framework significantly outperforms the vanilla DeepEmoCluster framework and other existing SSL approaches in regression tasks for the emotional attributes arousal, dominance, and valence. The improvements are observed in fully-supervised learning or SSL implementations. Further analyses validate the effectiveness of the proposed temporal modeling, showing (1) high temporal consistency in the cluster assignment, and (2) well-separated emotional patterns in the generated clusters.

Incremental Weighted Ensemble for Data Streams With Concept Drift

As a popular strategy to tackle concept drift, chunk-based ensemble method adapts a new concept by adjusting the weights of historical classifiers. However, most previous approaches normally evaluate the historical classifier based on an entire chunk newly arrived, which may cause delayed adaptation. To address the issue, two novel ensemble models, named incremental weighted ensemble (IWE) and incremental weighted ensemble for multi-classification (IWE-M), are proposed. At each time step, all base classifiers are incrementally updated on a newly arrived instance. Following that, the instance is collected into a cache array. Once a data chunk is formed, a new base classifier is created. More specially, a forgetting mechanism based on variable-size window is designed to adjust the weight of each base classifier in IWE in terms of its classification accuracy on the latest instances in an online manner. IWE-M, an extension of IWE, aims to solve multi-class problems with local concept drifts. In IWE-M, the weight of a base classifier is expanded to a weight vector. In this way, this ensemble model can retain specific historical information about non-drift regions from a local drift. Experimental results show that the proposed ensemble frameworks outperform six competitive approaches on Accuracy and G-mean.

An Incremental Naive Bayes Learner for Real-time Health Prediction

Healthcare monitoring systems have improved with the Internet of Things and machine learning prediction models. Traditional batch machine-learning approaches cannot generate an effective model since most data will be continuous and real-time. Real-time medical data processing is challenging since the entire data is unavailable during prediction. Here, a continuous model adaptation based on incremental learning is demanded. The development of such a system can significantly improve patient outcomes and reduce healthcare costs. This paper proposes an Incremental Naive Bayes Learner that can handle concept drifts in data. The algorithm keeps a sliding window of data constantly updated as new data is added. The adaptive window size determines how much data is used to train the model at any given time. After processing each chunk of data, the algorithm calculates the model’s accuracy. The system detects a concept drift and dynamically updates the training set if the accuracy drops below a predefined threshold. We conducted a comparative evaluation of state-of-the-art batch and incremental learning algorithms on different medical datasets, demonstrating the impact of incremental learning. The results demonstrate the effectiveness of our approach: the Agrawal dataset achieved the highest accuracy at 66.6%, followed by Dialysis at 61.6%, while Liver and HCC achieved 50% and 58.3% accuracy, respectively. This approach ensures a sustained high level of accuracy in healthcare monitoring systems over time, even amidst concept drift.