Data Storage Costs Research Articles

Adoption of Surveillance Technologies: Data Openness, Privacy, and Cultural Tightness

The diffusion and adoption of surveillance technologies have increased significantly in the past few decades. In addition to reductions in the cost of sensors, data storage, and algorithms, social norms have undoubtedly affected patterns of adoption. In addition to extending human abilities to monitor and regulate behavior, shared social norms related to privacy, security, and conformity will affect the nature and extent of adoption. In this study, archival data is used to examine the associations between data openness practices, privacy norms, and conformity bias, i.e., cultural “tightness.” Results suggested that social groups that emphasized conformity and had weaker open data policies, were most likely to adopt a variety of surveillance technologies.

3D Expression-Invariant Face Verification Based on Transfer Learning and Siamese Network for Small Sample Size

Three-dimensional (3D) face recognition has become a trending research direction in both industry and academia. However, traditional facial recognition methods carry high computational costs and face data storage costs. Deep learning has led to a significant improvement in the recognition rate, but small sample sizes represent a new problem. In this paper, we present an expression-invariant 3D face recognition method based on transfer learning and Siamese networks that can resolve the small sample size issue. First, a landmark detection method utilizing the shape index was employed for facial alignment. Then, a convolutional network (CNN) was constructed with transfer learning and trained with the aligned 3D facial data for face recognition, enabling the CNN to recognize faces regardless of facial expressions. Following that, the weighted trained CNN was shared by a Siamese network to build a 3D facial recognition model that can identify faces even with a small sample size. Our experimental results showed that the proposed method reached a recognition rate of 0.977 on the FRGC database, and the network can be used for facial recognition with a single sample.

Quantification of the Information Loss Resulting from Temporal Aggregation of Wind Turbine Operating Data

SCADA operating data are more and more used across the wind energy domain, both as a basis for power output prediction and turbine health status monitoring. Current industry practice to work with this data is by aggregating the signals at coarse resolution of typically 10-min averages, in order to reduce data transmission and storage costs. However, aggregation, i.e., downsampling, induces an inevitable loss of information and is one of the main causes of skepticism towards the use of SCADA operating data to model complex systems such as wind turbines. This research aims to quantify the amount of information that is lost due to this downsampling of SCADA operating data and characterize it with respect to the external factors that might influence it. The issue of information loss is framed by three key questions addressing effects on the local and global scale as well as the influence of external conditions. Moreover, recommendations both for wind farm operators and researchers are provided with the aim to improve the information content. We present a methodology to determine the ideal signal resolution that minimized storage footprint, while guaranteeing high quality of the signal. Data related to the wind, electrical signals, and temperatures of the gearbox resulted as the critical signals that are largely affected by an information loss upon aggregation and turned out to be best recorded and stored at high resolutions. All analyses were carried out using more than one year of 1 Hz SCADA data of onshore wind farm counting 12 turbines located in the UK.

How index selection, compression, and recording schedule impact the description of ecological soundscapes.

Acoustic indices derived from environmental soundscape recordings are being used to monitor ecosystem health and vocal animal biodiversity. Soundscape data can quickly become very expensive and difficult to manage, so data compression or temporal down‐sampling are sometimes employed to reduce data storage and transmission costs. These parameters vary widely between experiments, with the consequences of this variation remaining mostly unknown.We analyse field recordings from North‐Eastern Borneo across a gradient of historical land use. We quantify the impact of experimental parameters (MP3 compression, recording length and temporal subsetting) on soundscape descriptors (Analytical Indices and a convolutional neural net derived AudioSet Fingerprint). Both descriptor types were tested for their robustness to parameter alteration and their usability in a soundscape classification task.We find that compression and recording length both drive considerable variation in calculated index values. However, we find that the effects of this variation and temporal subsetting on the performance of classification models is minor: performance is much more strongly determined by acoustic index choice, with Audioset fingerprinting offering substantially greater (12%–16%) levels of classifier accuracy, precision and recall.We advise using the AudioSet Fingerprint in soundscape analysis, finding superior and consistent performance even on small pools of data. If data storage is a bottleneck to a study, we recommend Variable Bit Rate encoded compression (quality = 0) to reduce file size to 23% file size without affecting most Analytical Index values. The AudioSet Fingerprint can be compressed further to a Constant Bit Rate encoding of 64 kb/s (8% file size) without any detectable effect. These recommendations allow the efficient use of restricted data storage whilst permitting comparability of results between different studies.

B-IoMV: Blockchain-based onion routing protocol for D2D communication in an IoMV environment beyond 5G

Blockchain technology's popularity in terms of security, privacy, traceability, and trust is being applied in various major applications concerning connected autonomous vehicles. One of the most sensitive applications is military operations. We do not compromise with either security, privacy, trust, or communication latency of connected military vehicles, i.e., Internet of military vehicles (IoMVs). Achieving anonymity along with the security and privacy of sender, receiver, and data path for IoMVS is still an open question. Efforts we have made to address the aforementioned issue by proposing a blockchain-based onion routing protocol for IoMVs, i.e., B-IoMV, to achieve secure, trusted, and anonymous D2D communication. We studied the working of onion routing and presented a blockchain and token-based solution to strengthen the security and anonymity of IoMVs. A blockchain-based solution is quite costly and we use InterPlanetary File System (IPFS) to make the proposed B-IoMV system cost-effective. Finally, results show that the proposed B-IoMV system achieved better communication latency, data storage cost, and network bandwidth utilization.

Using Google Earth Engine to Assess Temporal and Spatial Changes in River Geomorphology and Riparian Vegetation

AbstractWe developed a new approach using a cloud‐based remote sensing and geospatial analysis platform, Google Earth Engine, to quantify temporal changes in river channel location and adjacent riparian vegetation extent and fraction. Our new method uses publicly available 1 m aerial images and eliminates manual processing need by incorporating an automatic image classification algorithm. Classification of riparian vegetation is enhanced by increasing the temporal resolution to monthly and by mapping vegetation at high spatial resolution. We illustrate the application of our method by characterizing temporal and spatial trends in riparian vegetation and river channel position for the mainstem of the Genesee River, New York, from 2006 to 2015. Annual change in riparian vegetation extent along the Genesee River ranged from a loss of 17% to a gain of 13%, and 64 km (28%) of the river demonstrated channel migration across consecutive aerial images. Our method also successfully distinguished differences between sections of the river above and below a dam. The enhanced capacity to map vegetation monthly allowed us to identify that seasonal active vegetation fractions along the Genesee River peaked at 75% in the summer and remained lower than 25% in winter. Our method allows stakeholders and managers to process remotely sensed imagery and investigate trends in river channel and riparian vegetation dynamics over time, while reducing the costs of data processing and storage.

Quantifying nitrogen loss hotspots and mitigation potential for individual fields in the US Corn Belt with a metamodeling approach

The high productivity in the US Corn Belt is largely enabled by the consumption of millions of tons of manufactured fertilizer. Excessive application of nitrogen (N) fertilizer has been pervasive in this region, and the unrecovered N eventually escaped from croplands in forms of nitrous oxide (N2O) emission and N leaching. Mitigating these negative impacts is hindered by a lack of practical information on where to focus and how much mitigation potential to expect. At a large scale, process-based crop models are the primary tools for predicting variables required by decision making, but their applications are prohibited by expensive computational and data storage costs. To overcome these challenges, we built a series of metamodels to learn the key mechanisms regarding the carbon (C) and N cycle from a well-validated process-based biogeochemical model, ecosys. The trained metamodel captures over 98% of the variability of the ecosys simulated outputs for 99 randomly selected counties in Iowa, Illinois, and Indiana. To identify hotspots with high mitigation potential, we introduce net societal benefit (NSB) as an indicator for synthesizing the loss in yield and social benefits through emissions and pollutants avoided. Our results show that reducing N fertilizer by 10% leads to 9.8% less N2O emissions and 9.6% less N leaching at the cost of 4.9% more SOC depletion and 0.6% yield reduction over the study region. The estimated total annual NSB is $395 M (uncertainty ranges from $114 M to $1271 M), including $334 from social benefits (uncertainty ranges from $46 M to $1076 M), $100 M from saving fertilizer (uncertainty ranges from $13 M to $455 M), and −$40 M due to yield changes (uncertainty ranges from −$261 M to $69 M). For the median scenario, we noted that 20% of the study area accounts for nearly 50% of the NSB, and thus represent hotspot locations for targeted mitigation. Although the uncertainty range suggests that developing such a high-resolution framework is not yet settled and the scenario based estimations are not appropriate to inform the management practices for individual farmers, our efforts shed light on the new generation of analytical tools for life cycle assessment.

Research on Distributed Storage Technology of Database Big Data Based on Cloud Computing

With China entering the era of big data, the storage and management of big data has become a hot issue. The proposal and use of distributed storage technology has achieved good results. The continuous development of Internet communication technology and the popularization of computer information technology promote the evolution of traditional data model. The distributed storage system for big data has strong expansibility. It can optimize the existing storage space, realize the optimal allocation of space resources, and reduce the cost of data storage. Cloud computing as a resource sharing business computing model, it receives a huge amount of data every day, and the amount of data is still expanding. This has caused great pressure on distributed storage in cloud computing environment. This paper studies the distributed storage technology of database big data based on cloud computing, in order to strengthen the data management in the era of big data and improve the security and utilization of data.

Next-generation regional ocean projections for living marine resource management in a changing climate

AbstractEfforts to manage living marine resources (LMRs) under climate change need projections of future ocean conditions, yet most global climate models (GCMs) poorly represent critical coastal habitats. GCM utility for LMR applications will increase with higher spatial resolution but obstacles including computational and data storage costs, obstinate regional biases, and formulations prioritizing global robustness over regional skill will persist. Downscaling can help address GCM limitations, but significant improvements are needed to robustly support LMR science and management. We synthesize past ocean downscaling efforts to suggest a protocol to achieve this goal. The protocol emphasizes LMR-driven design to ensure delivery of decision-relevant information. It prioritizes ensembles of downscaled projections spanning the range of ocean futures with durations long enough to capture climate change signals. This demands judicious resolution refinement, with pragmatic consideration for LMR-essential ocean features superseding theoretical investigation. Statistical downscaling can complement dynamical approaches in building these ensembles. Inconsistent use of bias correction indicates a need for objective best practices. Application of the suggested protocol should yield regional ocean projections that, with effective dissemination and translation to decision-relevant analytics, can robustly support LMR science and management under climate change.

Data Membership Identification using Bloom Filter in Cloud Storage for Effective Resource Allocation

Cloud computing has recently been the buzz word that had changed the entire software industry with its PaaS, IaaS and SaaS architecture model. The legacy systems operational in the organizations across different industries have been using the database that had an overhead in terms of cost of data storage, runtime operation and frequent data maintenance. Cloud database concept had challenged the existing storage and operational norms of data. In the distributed environment, resources used in the cloud databases need to identify whether the requested data belongs to the data nodes of a cluster. With databases began to be ubiquitous, the data storage needed to satisfy heterogeneous data structures rather the unstructured data storage support is looked for. The Use of Bloom's filter for data membership identification is the novel approach and can effectively improve the resource organization strategy on cloud. Dynamic resource organization can further improve the query efficiency as well. The concern raised during this is the data privacy which can also be ascertain by maintaining the data access authority levels. Bloom filter uses less memory space against the large dataset to store it's information. A Bloom filter is proposed to be used to determine whether an element is part of a reference set. It is a very compact hash-based data structure with efficient look-up times and a manageable risk of giving false positives.

A secure data analytics scheme for multimedia communication in a decentralized smart grid

With the exponential increase in energy demands (commercial as well as residential), the traditional grid infrastructure significantly shifted to intelligent ICT-based Smart Grid (SG) infrastructure. In the SG environment, only efficient energy management may not be sufficient as the SG dynamics have significant impacts on multimedia communications such as video surveillance of the technical/non-technical losses of energy and many more. The inevitable energy losses can be identified by process and analyze the massive amount of heterogeneous data, i.e., Big Data (BD) generated through smart devices such as sensors, Smart Meters (SMs), and others. The key challenges in analyzing multimedia BD are computational complexity, operational integration complexity, data security, and privacy. To overcome the aforementioned issues, this paper proposes a blockchain-based data analytics scheme called ChoIce, which offers secure data collection, analysis, and decision support for the SG systems. It works in two phases; (i) secure data collection over Ethereum and (ii) BD analytics and decision-making using deep learning (DL). The robust and secure data analytics, efficient network management, and high-performance computing for BD are crucial towards the optimization of SG operation. The performance of ChoIce is evaluated considering parameters such as the data storage cost, multimedia communication latency, and prediction accuracy. Thus, the results of ChoIce shows that it outperforms in contrast to other state-of-the-art approaches.

ESR Estimation for Aluminum Electrolytic Capacitor of Power Electronic Converter Based on Compressed Sensing and Wavelet Transform

The aluminum electrolytic capacitor (AEC) is one of the most important parts of a power electronic converter. Throughout its lifespan, the equivalent series resistance (ESR) of the AEC will increase, which will affect the performance of the power electronic converter. To guarantee the reliability of a converter, it is necessary to monitor the health state of the AEC, for which one common technique is to estimate the ESR of the AEC. Equipment with a high sampling rate is usually required because converter usually works at a high frequency. This requirement will increase the cost of ESR monitoring. To address this issue, this article proposes a novel method of estimating the ESR for an AEC. The proposed method uses an innovative compressed sensing approach to acquire the ripple voltage and current. Instead of complete reconstruction of the ripple voltage and current, only the high-frequency features in the wavelet subspace are reconstructed and used to calculate the ESR. The proposed method reduces the cost of data sampling, transmission, and storage. This method is not limited by the converter circuit structure and can be applied in various environments. Simulations and experiments were carried out to verify the effectiveness of the proposed method.

Serverless distributed learning for smart grid analytics* *Project supported by the National Natural Science Foundation of China (Grant Nos. 52007173 and U19B2042), Zhejiang Provincial Natural Science Foundation of China (Grant No. LQ20E070002), and Zhejiang Lab’s Talent Fund for Young Professionals (Grant No. 2020KB0AA01).

The digitization, informatization, and intelligentization of physical systems require strong support from big data analysis. However, due to restrictions on data security and privacy and concerns about the cost of big data collection, transmission, and storage, it is difficult to do data aggregation in real-world power systems, which directly retards the effective implementation of smart grid analytics. Federated learning, an advanced distributed learning method proposed by Google, seems a promising solution to the above issues. Nevertheless, it relies on a server node to complete model aggregation and the framework is limited to scenarios where data are independent and identically distributed. Thus, we here propose a serverless distributed learning platform based on blockchain to solve the above two issues. In the proposed platform, the task of machine learning is performed according to smart contracts, and encrypted models are aggregated via a mechanism of knowledge distillation. Through this proposed method, a server node is no longer required and the learning ability is no longer limited to independent and identically distributed scenarios. Experiments on a public electrical grid dataset will verify the effectiveness of the proposed approach.

Cluster load based content distribution and speculative execution for geographically distributed cloud environment

The scale of big data has shown an explosive growth, which makes the processing of big data put forward higher requirements on data centers, and a single data center can no longer meet the needs of big data processing. To deal with this situation, a geographically distributed cloud system needs to be built. However, in the geographically distributed cloud system, each data center is distributed in different geographic locations, which makes the data placement operations in the geographically distributed cloud system lead to greater overhead. To solve this problem, this paper proposes a data placement strategy. This strategy comprehensively considers the data transmission latency, bandwidth cost, cloud server storage capacity, and load capacity during the data placement process, and formulates a data placement problem that minimizes the energy consumption of data transmission. Then the minimum set cover method based on Lagrangian relaxation is used to solve this problem and obtain the optimal data placement scheme. On the other hand, in a geographically distributed cloud data center, the execution progress of the job submitted by the user will be affected by the straggler task. To solve this problem, this paper proposes a speculative execution strategy for the geographically distributed cloud system. This strategy performs different speculative execution operations according to the state of the cluster load, and then calculates the load capacity of the nodes in the cluster. The node with the strongest load capacity in the cluster is used to perform speculative execution operations. Experimental results show that the proposed data placement strategy can effectively improve the performance of the energy consumption, the data storage cost, the network transmission cost and the data transmission time. The proposed speculative execution strategy can effectively improve the performance of the job completion time, cluster throughput and QoS satisfaction rate.

Layer Regeneration Network With Parameter Transfer and Knowledge Distillation for Intelligent Fault Diagnosis of Bearing Using Class Unbalanced Sample

In recent years, more and more researchers are using deep learning to monitor and diagnose mechanical equipment faults. When new task data not considered in the training stage are generated during the operation of the equipment, it is difficult for the model to recognize this type of data. If only new task data are used in training, it will lead to poor performance in the old task. When using all the data in training, with the accumulation of task data, the cost of data storage will increase and the speed of model update will be greatly reduced. Therefore, an intelligent fault diagnosis method based on the layer regeneration network under class imbalanced samples is proposed, which uses only new task data to update the model. The method holds that the data contain some information of other categories, but they are covered by information of their own categories, and the knowledge between classes is extracted by knowledge distillation to enhance the learning of other categories. First, the cross-domain learning method based on parameter transfer is adopted to make the layer regeneration network model (LRNM) converge quickly on the new task. Then, the implicit knowledge related to the old task in the new task data is extracted by the distillation learning method to adjust global parameters, alleviate the catastrophic forgetting problem in model updating, and realize the model continuous learning. Through experiments, the use of dark knowledge can effectively enhance the learning of other types of knowledge.

Analysis of the impact of data compression on condition monitoring algorithms for ball screws

The overall equipment effectiveness (OEE) is a management ratio to evaluate the added value of machine tools. Unplanned machine downtime reduces the operational availability and therefore, the OEE. Increased machine costs are the consequence. An important cause of unplanned machine downtimes is the total failure of ball screws of the feed axes due to wear. Therefore, monitoring of the condition of ball screws is important. Common concepts rely on high-frequency acceleration sensors from external control systems to detect a change of the condition. For trend and detailed damage analysis, large amounts of data are generated and stored over a long time period (>5 years), resulting in corresponding data storage costs. Additional axes or machine tools increase the data volume further, adding to the total storage costs. To minimize these costs, data compression or source coding has to be applied. To achieve maximum compression ratios, lossy coding algorithms have to be used, which introduce distortion in a signal. In this work, the influence of lossy coding algorithms on a condition monitoring algorithm (CMA) using acceleration signals is investigated. The CMA is based on principal component analysis and uses 17 features such as standard deviation to predict the preload condition of a ball screw. It is shown that bit rate reduction through lossy compression algorithms is possible without affecting the condition monitoring - as long as the compression algorithm is known. In contrast, an unknown compression algorithm reduces the classification accuracy of condition monitoring by about 20 % when coding with a quantizer resolution of 4 bit/sample.

Light-Weight CNN Enabled Edge-Based Framework for Machine Health Diagnosis

Condition Based Monitoring (CBM) leverages sensor measurements for measuring state of health of an asset and autonomous diagnosis of faults to trigger remedial actions. Countless deep learning architectures are available for cloud-based feature engineering, feature extraction and classification of data for CBM models. However, complex models pose memory and processing speed constraints for edge implementation, while cloud-based computing poses high latency, high cost of data transmission and storage and privacy threats. This calls for a data driven machine health diagnosis system that is effective yet edge-compatible and secure. In this work, we present a model that is light-weight and non-redundant by optimizing the model size and complexity for edge implementation onto resource constrained, low-cost hardware. The model is based on a light-weight, edge implementable Convolutional Neural Network (CNN) algorithm that utilizes vibration sensor measurements for fault event estimation of machines. The model was trained and tested on two publicly available and widely studied vibration datasets for rolling element bearing faults. This CNN based classification system using spectrograms is able to achieve near perfect accuracy for both binary as well as multi-class fault classification. Lastly, the trained model was implemented and tested on a Raspberry Pi single-board computer, thus realizing an edge-compatible implementation which is cost-saving, secure and reliable.

Power quality signal reconstruction based on multitask Bayesian compressive sensing

Compressed sensing technology breaks through the limitation of Nyquist sampling theorem, and can effectively reduce the cost of data storage and transmission. Only a few samples are needed to reconstruct the power quality signal by using the compressed sensing technology, which is of great significance for the detection and analysis of power quality. We propose an algorithm based on multitask Bayesian compressed sensing (MT-BCS) theory for power quality signals compression and reconstruction in this paper. The power quality signals are changed to sparse vectors by taking the fast Fourier transform basis as the sparse basis. The real and imaginary parts of sparse vectors are then treated as two compression and reconstruction tasks. Considering the internal correlation of data corresponding to these two tasks, the power quality signal is reconstructed using the sharing mechanism of hyperparameter estimation. The simulation results show that the algorithm is superior to the orthogonal matching pursuit algorithm and the Bayesian compressed sensing algorithm in anti-noise performance and reconstruction accuracy, and is more suitable for compressing and reconstructing power quality signals with complex disturbance.

Research on Cloud Storage Platform Technology Based on Hadoop

In order to solve the problem of high cost of traditional data calculation and storage, and difficulty in writing parallel programs, this article summarizes the core technology of Hadoop, and uses Hadoop distributed processing technology and virtualization technology to design and build a cloud computing storage platform. The experiment proves that, compared with the traditional single-machine computing method and the physical machine cluster computing method, the platform has higher performance, and the resource utilization rate has also been improved.

Low-rate DDoS attacks detection method using data compression and behavior divergence measurement

Distributed denial of service (DDoS) attacks have been a typical and extremely destructive threat to the Internet. DDoS attack detections suffer from the nonnegligible high complexity of massive traffic flow storage in the high-speed network. Besides, hidden low-rate DDoS (LDDoS) attacks evade the existing detection methods due to the similarity between LDDoS attack traffic and normal traffic. Focusing on these problems, this paper proposes a new low-rate DDoS attack detection method (LDDM) by designing the multidimensional sketch structure and novel measurement methods on network flows. First, the multidimensional sketch structure is designed to aggregate and compress network flows, which contributes to reduce the cost of data storage and enhance detection performance. Then, the improved behavior divergence measurement method based on daub 4 wavelet transform is proposed to calculate the energy percentage of each sketch divergence. This method obtains effective results in distinguishing the normal traffic and attack traffic. Furthermore, a modified weighted exponential moving average method is designed to construct the dynamic threshold of normal network. Meanwhile, a traffic freezing mechanism is proposed to ensure the standardization of the dynamic threshold. Finally, the effectiveness of the LDDM is evaluated using several real low-rate DDoS attack datasets. The comparisons with other methods illustrate our method has a lower false positive rate and false negative rate, as well as higher accuracy in the detection of stealthy low-rate DDoS attacks.