Reduce Data Size Research Articles

A comparative study of clustering techniques for electrical load pattern segmentation

Smart meters have been widely deployed in power networks since the last decade. This trend has resulted in an enormous volume of data being collected from the electricity customers. To gain benefits for various stakeholders in power systems, proper data mining techniques, such as clustering, need to be employed to extract the underlying patterns from energy consumptions. In this paper, a comparative study of different techniques for load pattern clustering is carried out. Different parameters of the methods that affect the clustering results are evaluated and the clustering algorithms are compared for two data sets. In addition, the two suitable and commonly used data size reduction techniques and feature definition/extraction methods for load pattern clustering are analysed. Furthermore, the existing studies on clustering of electricity customers are reviewed and the main results are highlighted. Finally, the future trends and major applications of clustering consumption patterns are outlined to inform industry practitioners and academic researchers to optimize smart meter operational use and effectiveness.

Sequential PCA-based Classification of Mediterranean Forest Plants using Airborne Hyperspectral Remote Sensing

In recent years, hyperspectral remote sensing (HRS) has become common practice for remote analyses of the physiognomy and composition of forests. Supervised classification is often used for this purpose, but demands intensive sampling and analyses, whereas unsupervised classification often requires information retrieval out of the large HRS datasets, thereby not realizing the full potential of the technology. An improved principal component analysis-based classification (PCABC) scheme is presented and intended to provide accurate and sequential image-based unsupervised classification of Mediterranean forest species. In this study, unsupervised classification and reduction of data size are performed simultaneously by applying binary sequential thresholding to principal components, each time on a spatially reduced subscene that includes the entire spectral range. The methodology was tested on HRS data acquired by the airborne AisaFENIX HRS sensor over a Mediterranean forest in Mount Horshan, Israel. A comprehensive field-validation survey was performed, sampling 257 randomly selected individual plants. The PCABC provided highly improved results compared to the traditional unsupervised classification methodologies, reaching an overall accuracy of 91%. The presented approach may contribute to improved monitoring, management, and conservation of Mediterranean and similar forests.

Personal Recognition Method using Coupling Image of ECG Signal

심전도 신호는 위조가 불가능하며 양쪽 손목에서 신호를 간편히 취득할 수 있는 장점이 있다. 본 논문에서는 심전도 신호의 방향 정보를 이용해 커플링 이미지를 생성하고, 이를 이용한 개인 인식 방법을 제안한다. 제안하는 커플링 이미지는 정방향 심전도 신호와 R-peak를 기준으로 회전된 역방향 심전도 신호를 이용해 생성하며, 생성한 커플링 이미지는 개인별로 고유한 패턴과 명암을 나타낸다. 또한 같은 주기의 심전도 신호 연산을 통해 R-peak 영역 데이터가 증가하여 개인 인식 성능 향상이 가능하다. 생성한 커플링 이미지는 제안한 합성곱 신경망을 이용해 패턴 및 명암에 대한 특징을 추출하며, 네트워크 속도 향상을 위해 다수의 풀링층을 사용해 데이터 크기를 축소한다. 실험은 47명의 공개된 심전도 데이터를 이용하며, 공개된 네트워크 중 top-5 성능이 상위권인 5개 네트워크와 제안한 네트워크를 이용해 비교 실험을 진행한다. 실험 결과 제안한 네트워크의 개인 인식 성능이 99.28%로 가장 높게 나타남에 따라, 제안한 커플링 이미지를 이용한 개인 인식 방법이 유효함을 확인하였다.Electrocardiogram (ECG) signals cannot be counterfeited and can easily acquire signals from both wrists. In this paper, we propose a method of generating a coupling image using direction information of ECG signals as well as its usage in a personal recognition method. The proposed coupling image is generated by using forward ECG signal and rotated inverse ECG signal based on R-peak, and the generated coupling image shows a unique pattern and brightness. In addition, R-peak data is increased through the ECG signal calculation of the same beat, and it is thus possible to improve the recognition performance of the individual. The generated coupling image extracts characteristics of pattern and brightness by using the proposed convolutional neural network and reduces data size by using multiple pooling layers to improve network speed. The experiment uses public ECG data of 47 people and conducts comparative experiments using five networks with top 5 performance data among the public and the proposed networks. Experimental results show that the recognition performance of the proposed network is the highest with 99.28%, confirming potential of the personal recognition.

A pattern recognition methodology for analyzing residential customers load data and targeting demand response applications

The availability of smart meter data allows defining innovative applications such as demand response (DR) programs for households. However, the dimensionality of data imposes challenges for the data mining of load patterns. In addition, the inherent variability of residential consumption patterns is a major problem for deciding on the characteristic consumption patterns and implementing proper DR settlements. In this regard, this paper utilizes a data size reduction and clustering methodology to analyze residential consumption behavior. Firstly, the distinctive time periods of household activity during the day are identified. Then, using these time periods, a modified symbolic aggregate approximation (SAX) technique is utilized to transform the load patterns into symbolic representations. In the next step, by applying a clustering method, the major consumption patterns are extracted and analyzed. Finally, the customers are ranked based on their stability over time. The proposed approach is applied on a large dataset of residential customers’ smart meter data and can achieve three main goals: 1) it reduces the dimensionality of data by utilizing the data size reduction, 2) it alleviates the problems associated with the clustering of residential customers, 3) its results are in accordance with the needs of systems operators or demand response aggregators and can be used for demand response targeting. The paper also provides a thorough analysis of different aspects of residential electricity consumption and various approaches to the clustering of households which can inform industry and research activity to optimize smart meter operational use.

Approximate TMR based on successive approximation and loop perforation in microprocessors

This work presents an approximate triple modular redundancy (ATMR) method based on successive approximation and loop perforation. Successive approximation consists of loop-based algorithms that improve their output quality on every iteration execution. By manipulating the number of executed iterations, a designer can balance output quality and execution time performance. The proposed technique exploits this approximate computing paradigm combined with data size and precision reduction to minimize traditional triple modular redundancy (TMR) costs while maintaining a good fault masking capability. A number of ATMR configurations are presented and tested under laser fault injection to evaluate their fault masking capability. The benchmarks are implemented as embedded software in the ARM Cortex A9 processor of a Xilinx Zynq-7000 series board. The proposed technique was able to improve the execution time overhead with the usage of approximations, with an acceptable fault masking rate.

Subsampled Acquisition to Increase Speed and Reduce Data Size for In Situ TEM

Subsampled Acquisition to Increase Speed and Reduce Data Size for In Situ TEM

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

With ever-increasing volumes of scientific data produced by HPC applications, significantly reducing data size is critical because of limited capacity of storage space and potential bottlenecks on I/O or networks in writing/reading or transferring data. SZ and ZFP are the two leading lossy compressors available to compress scientific data sets. However, their performance is not consistent across different data sets and across different fields of some data sets: for some fields SZ provides better compression performance, while other fields are better compressed with ZFP. This situation raises the need for an automatic online (during compression) selection between SZ and ZFP, with a minimal overhead. In this paper, the automatic selection optimizes the rate-distortion, an important statistical quality metric based on the signal-to-noise ratio. To optimize for rate-distortion, we investigate the principles of SZ and ZFP. We then propose an efficient online, low-overhead selection algorithm that predicts the compression quality accurately for two compressors in early processing stages and selects the best-fit compressor for each data field. We implement the selection algorithm into an open-source library, and we evaluate the effectiveness of our proposed solution against plain SZ and ZFP in a parallel environment with 1,024 cores. Evaluation results on three data sets representing about 100 fields show that our selection algorithm improves the compression ratio up to 70% with the same level of data distortion because of very accurate selection (around 99%) of the best-fit compressor, with little overhead (less than 7% in the experiments).

Fuzzy-Rough Set Bireducts for Data Reduction

Data reduction is an important step that helps ease the computational intractability for learning techniques when data are large. This is particularly true for the huge datasets that have become commonplace in recent times. The main problem facing both data preprocessors and learning techniques is that data are expanding both in terms of dimensionality and also in terms of the number of data instances. Approaches based on fuzzy-rough sets offer many advantages for both feature selection and classification, particularly for real-valued and noisy data; however, the majority of recent approaches tend to address the task of data reduction in terms of either dimensionality or training data size in isolation. This paper demonstrates how the notion of fuzzy-rough bireducts can be used for the simultaneous reduction of data size and dimensionality. It also shows how bireducts and, therefore, reduced subtables of data can be used not only as a preprocessing tool but also for the learning of compact and robust classifiers. Furthermore, the ideas can also be extended to the unsupervised domain when dealing with unlabeled data. Experimental evaluation of various techniques demonstrate that high levels of simultaneous reduction of both dimensionality and data size can be achieved whilst maintaining robust performance.

Efficient and realistic character animation through analytical physics-based skin deformation

Physics-based skin deformation methods can greatly improve the realism of character animation, but require non-trivial training, intensive manual intervention, and heavy numerical calculations. Due to these limitations, it is generally time-consuming to implement them, and difficult to achieve a high runtime efficiency. In order to tackle the above limitations caused by numerical calculations of physics-based skin deformation, we propose a simple and efficient analytical approach for physics-based skin deformations. Specifically, we (1) employ Fourier series to convert 3D mesh models into continuous parametric representations through a conversion algorithm, which largely reduces data size and computing time but still keeps high realism, (2) introduce a partial differential equation (PDE)-based skin deformation model and successfully obtain the first analytical solution to physics-based skin deformations which overcomes the limitations of numerical calculations. Our approach is easy to use, highly efficient, and capable to create physically realistic skin deformations.

A Lightweight, Robust Exploitation System for Temporal Stacks of UAS Data: Use Case for Forward-Deployed Military or Emergency Responders

The availability and precision of unmanned aerial systems (UAS) permit the repeated collection of very-high quality three-dimensional (3D) data to monitor high-interest areas, such as dams, urban areas, or erosion-prone coastlines. However, challenges exist in the temporal analysis of this data, specifically in conducting change-detection analysis on the high-quality point cloud data. These files are very large in size and contain points in varying locations that do not align between scenes. These large file sizes also limit the use of this data for individuals with low computational resources, such as first responders or forward-deployed soldiers. In response, this manuscript presents an approach that aggregates data spatially into voxels to provide the user with a lightweight, web-based exploitation system coupled with a flexible backend database. The system creates a robust set of tools to analyze large temporal stacks of 3D data and reduces data size by 78%, all while being able to query the original point cloud data. This approach offers a solution for organizations analyzing high-resolution, temporal point-clouds, as well as a possible solution for operations in areas with poor computational and connectivity resources requiring high-quality, 3D data for decision support and planning.

Is Blockchain Technology Suitable for Managing Personal Health Records? Mixed-Methods Study to Test Feasibility

BackgroundThere are many perspectives on the advantages of introducing blockchain in the medical field, but there are no published feasibility studies regarding the storage, propagation, and management of personal health records (PHRs) using blockchain technology.ObjectiveThe purpose of this study was to investigate the usefulness of blockchains in the medical field in relation to transactions with and propagation of PHRs in a private blockchain.MethodsWe constructed a private blockchain network using Ethereum version 1.8.4 and conducted verification using the de-identified PHRs of 300 patients. The private blockchain network consisted of one hospital node and 300 patient nodes. In order to verify the effectiveness of blockchain-based PHR management, PHRs at a time were loaded in a transaction between the hospital and patient nodes and propagated to the whole network. We obtained and analyzed the time and gas required for data transaction and propagation on the blockchain network. For reproducibility, these processes were repeated 100 times.ResultsOf 300 patient records, 74 (24.7%) were not loaded in the private blockchain due to the data block size of the transaction block. The remaining 226 individual health records were classified into groups A (80 patients with outpatient visit data less than 1 year old), B (84 patients with outpatient data from between 1 and 3 years before data collection), and C (62 patients with outpatient data 3 to 5 years old). With respect to mean transaction time in the blockchain, C (128.7 seconds) had the shortest time, followed by A (132.2 seconds) and then B (159.0 seconds). The mean propagation times for groups A, B, and C were 1494.2 seconds, 2138.9 seconds, and 4111.4 seconds, respectively; mean file sizes were 5.6 KB, 18.6 KB, and 45.38 KB, respectively. The mean gas consumption values were 1,900,767; 4,224,341; and 4,112,784 for groups A, B, and C, respectively.ConclusionsThis study confirms that it is possible to exchange PHR data in a private blockchain network. However, to develop a blockchain-based PHR platform that can be used in practice, many improvements are required, including reductions in data size, improved personal information protection, and reduced operating costs.

Joint medical image compression–encryption in the cloud using multiscale transform-based image compression encoding techniques

The recent years have witnessed rapid strides in the use of cloud computing and its countless applications. A cloud can contain massive volumes of multimedia data in the form of images, video and audio. Cloud computing platforms confront challenges in terms of data confidentiality, message integrity, user authentication and compression. Multimedia data needs plenty of storage capacity. Consequently, there is a need for multimedia data compression to reduce data size. Compression techniques are quite reliable, offering benefits to organizations dealing with metasized data in the cloud. Compressing large quanta of data leads to superior utilization of cloud storage. Compression techniques can compress data used for storage and transmission, yet compression alone is inadequate because multimedia data shared should, of necessity, be secure. Therefore, both multimedia compression and security are mandatory in the cloud. The chief goal of this paper is to propose a new framework, comprising multiscale transforms, public key cryptography and appropriate encoding techniques, that performs joint medical image compression and image encryption in the cloud. Multiscale transforms play a lead role in image compression, and the ones discussed in this paper include wavelet, bandelet, curvelet, ridgelet and contourlet transforms. Wavelet transforms offer robust localization both in terms of time and frequency domains. Bandelet transforms offer natural images geometric regularity to help improve the efficiency of representation. Curvelet transforms handle curve discontinuities well, with ridgelet transforms being the core idea behind curvelets. Contourlet transforms capture smooth contours and edges at any orientation. The Rivest-Shamir-Adleman (RSA) algorithm is used to encrypt images to provide maximum security when they are being transferred. Encoding techniques involved in this paper comprise the Embedded Zerotree Wavelet (EZW), Set Partitioning in Hierarchical Trees (SPIHT), Wavelet Difference Reduction (WDR), and Adaptively Scanned Wavelet Difference Reduction (ASWDR). Performance parameters such as peak signal to noise ratio (PSNR), mean square error (MSE), image quality index and structural similarity index (SSIM) are used for evaluation. It is justified that the proposed framework compresses images securely in the cloud.

Local-Set Based-on Instance Selection Approach for Autonomous Object Modelling

With the increasing presence of robotic agents in our daily life, computationally efficient modelling of real-world objects by autonomous systems is of prime importance for enabling these artificial agents to automatically and effectively perform tasks such as visual object recognition. For this purpose, we introduce a novel, machine-learning approach for instance selection called Approach for Selection of Border Instances (ASBI). This method adopts the notion of local sets to select the most representative instances at the boundaries of the classes, in order to reduce the set of training instances and, consequently, to reduce the computational resources that are necessary to perform the learning process of real-world objects by the artificial agents. Our new algorithm was validated on 27 standard datasets and applied on 2 challenging object-modelling datasets to test the automated object recognition task. ASBI performances were compared to those of 6 state-of-art algorithms, considering three standard metrics, namely, accuracy, reduction, and effectiveness. All the obtained results show that the proposed method is promising for the autonomous recognition task, while presenting the best trade-off between the classification accuracy and the data size reduction.

An Evaluation of Data Size Reduction Techniques for Improving the Reliability of Cloud-based CNC for a 3D Printer

Abstract Cloud-based computer numerical control (C-CNC) is an emerging paradigm of Industry 4.0 where CNC functionalities are moved to the cloud and provided to manufacturing machines as a service. Among many benefits, C-CNC allows manufacturing machines to leverage advanced control algorithms running on cloud computers to boost their performance at low cost, without need for major hardware upgrades. However, a major challenge of C-CNC is how to guarantee safe and reliable control of a manufacturing machine from the cloud, given delays and other Internet quality-of-service issues. This paper evaluates several techniques for reducing the size of control commands transmitted to a 3D printer from a cloud-based controller (with elements of C-CNC), in an effort to prevent degradation of 3D printer accuracy and speed caused by high latency Internet connection. Six of the evaluated techniques are popular statistical lossless data compression methods while one of the techniques, advocated in this paper, involves physics-based pseudo-lossless data truncation (i.e., no loss of key information content needed for control). In benchmark tests, the truncation technique is found to perform best in terms of the consistency and magnitude of data reduction ratio. Moreover, while the compression techniques require untenable amounts of memory for decompression on the microprocessor of the 3D printer, the memory requirements of the truncation technique are small. Consequently, using the truncation technique for data size reduction, frequent pauses during print that occur when controlling the 3D printer over high latency Internet connections, and cause poor surface finish of the printed part, are mitigated. As a result, high-quality prints are obtained at high-speed from the cloud-based controller over high latency Internet connections.

Electronic Scoring Scheme Based on Real Paillier Encryption Algorithms

Paillier algorithm has higher time efficiency than other homomorphic encryption algorithms due to only one power operation is required, but its encryption range is limited to integer, which cannot meet the need of electronic scoring system to encrypt real. To solve the problem, an electronic scoring scheme based on real Paillier algorithm is put forward. In the proposed scheme, compound transformation is introduced to convert real to large integer. The encryption range of Paillier algorithm is extended from integer to real by the large integer operation, which can make the scoring results more precision. In addition, a more secure and efficient two-way identity authentication strategy between clients and servers is designed by using Horner's rule. After hashing the username and password by Horner's rule, the obtained hash value is encrypted and transmitted, which can not only improve system security by avoiding disclosure of user's information during transmission, but also increase the computation efficiency due to the reduction of data size in encryption and decryption. In the experiment, compared with the existing encryption algorithms on different scoring data sets, the whole encryption and decryption time is slightly increased by 0.28% compared with the traditional Paillier, but computing real score is realized in the proposed method. The efficiency of addition and multiplication of the proposed method is increased averagely by 25.99% and 42.75%. The average shorten time of two-way identity authentication is 27.22 milliseconds due to the using of Horner's rule. When the number of clients is more than 105, the time efficiency is improved more obviously.

A Rate Control Scheme Compatible with a Legacy JPEG Decoder to Reduce Data Size Keeping Subjective Quality for Video Advertising on Mobile Devices

A Rate Control Scheme Compatible with a Legacy JPEG Decoder to Reduce Data Size Keeping Subjective Quality for Video Advertising on Mobile Devices

Self-Adaptive Online Trajectory Sampling (SAOTS) Using Spectral Domain Properties

Abstract We study the vehicle trajectory compression problem in this paper to reduce data size and transmission cost, while ensuring that the reduced data can satisfy the needs of transportation applications. We first propose an online segmentation method and then explore the spectral domain properties of vehicle trajectory segments by using the speed profile as the input signal. Based on the analysis, we introduce a spectral domain property called critical frequency that reflects the nature of the speed variation experienced by an individual vehicle trajectory. We show that, this property can be utilized to determine the proper sampling strategy of trajectory segments. We then design SAOTS, a self-adaptive online trajectory sampling method, for vehicle trajectory segmentation and compression. SAOTS utilizes vehicle flow state to segment a vehicle trajectory, and the critical frequency to sample the data records in the segment. Results show that resampling the trajectory segments based on the critical frequencies can vastly reduce the size of the GPS data while conserving most of the useful information. Tested using a small field dataset and a large, open dataset and compared with existing trajectory compression methods, SAOTS can help achieve the best balance in terms of data reduction, latency, accuracy, and application performances.

On-demand assessment of air traffic impact of blocking airspace

ABSTRACTThe Federal Aviation Administration often blocks strategically located airspace volumes to ensure safety during a variety of operations that are potentially hazardous to aircraft, such as space launches. As the frequency of these operations increases, there is a growing need to deepen collaboration and transparency between stakeholders regarding the use of airspace. This collaboration can be supported by models and capabilities to quickly assess the impact of airspace closures, up to 12 months into the future. This paper presents a technique to enable a ‘what-if’ analysis capability coupled with a prediction model, whereby changes in airspace dimension, location, and activation time are reflected instantaneously as measures of projected impact. The technique can also be used for quick post-operations analysis using historical traffic data and to develop air traffic impact assessment capabilities accessible to a broad range of users outside of the air traffic domain. This research has three key components: developing a model to predict air traffic demand up to 12 months into the future, modelling air traffic impact to the affected traffic, and reducing this information into a data structure that can support on-demand analysis. The focus of this paper is on new techniques to predict demand using a large set of historical track data and further encode these projections to support the quick assessment of the impact of blocking various airspace volumes. Initial results show that the proposed data reduction scheme accurately represented the traffic crossing an airspace and resulted in data size reduction by over 50%. The projection model performed well, the actual number of impacted flights were within the estimated range of approximately 80% of the time. Finally, the responsiveness of the web-based prototype developed to illustrate the concept demonstrated the model’s ability to support an on-demand assessment of the air traffic impact of blocking airspace. A significant limitation of the projection model is that it is based on the historical traffic pattern within the U.S. airspace; separate analysis is needed to adapt it to other geographical location.

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

A substantial amount of information in companies and on the Internet is present in the form of text. The value of this semi-structured and unstructured data has been widely acknowledged, with consequent scientific and commercial exploitation. The ever-increasing data production, however, pushes data analytic platforms to their limit. Compression as an effective means to reduce data size has been employed by many emerging data analytic platforms, whom the main purpose of data compression is to save storage space and reduce data transmission cost over the network. Since general purpose compression methods endeavour to achieve higher compression ratios by leveraging data transformation techniques and contextual data, this context-dependency forces the access to the compressed data to be sequential. Processing such compressed data in parallel, such as desirable in a distributed environment, is extremely challenging. This work proposes techniques for more efficient textual big data analysis with an emphasis on content-aware compression schemes suitable for the Hadoop analytic platform. The compression schemes have been evaluated for a number of standard MapReduce analysis tasks using a collection of public and private real-world datasets. In comparison with existing solutions, they have shown substantial improvement in performance and significant reduction in system resource requirements.

Neighborhood Attribute Reduction: A Multicriterion Strategy Based on Sample Selection

In the rough-set field, the objective of attribute reduction is to regulate the variations of measures by reducing redundant data attributes. However, most of the previous concepts of attribute reductions were designed by one and only one measure, which indicates that the obtained reduct may fail to meet the constraints given by other measures. In addition, the widely used heuristic algorithm for computing a reduct requires to scan all samples in data, and then time consumption may be too high to be accepted if the size of the data is too large. To alleviate these problems, a framework of attribute reduction based on multiple criteria with sample selection is proposed in this paper. Firstly, cluster centroids are derived from data, and then samples that are far away from the cluster centroids can be selected. This step completes the process of sample selection for reducing data size. Secondly, multiple criteria-based attribute reduction was designed, and the heuristic algorithm was used over the selected samples for computing reduct in terms of multiple criteria. Finally, the experimental results over 12 UCI datasets show that the reducts obtained by our framework not only satisfy the constraints given by multiple criteria, but also provide better classification performance and less time consumption.