Storage Management Problem Research Articles

Distributed storage optimization using multi-agent systems in Hadoop

Understanding data and extracting information from it are the main objectives of data science, especially when it comes to big data. To achieve these goals, it is necessary to collect and process massive data sets, arriving at the system in different formats at great velocity. The Big Data era has brought us new challenges in data storage and management, and existing state-ofthe-art data storage and processing tools are poised to meet the challenges while posing challenges to the next generation of data. Big Data storage optimization is essential for improving the overall efficiency of Big Data systems by maximizing the use of storage resources. It also reduces the energy consumption of Big Data systems, resulting in financial savings, environmental protection, and improved system performance. Hadoop provides a solution for storing and analysing large quantities of data. However, Hadoop can encounter storage management problems due to its distributed nature and the management of large volumes of data. In order to meet future challenges, the system needs to intelligently manage its storage system. The use of a multi-agent system presents a promising approach for efficiently managing hot and cold data in HDFS. These systems offer a flexible, distributed solution for solving complex problems. This work proposes an approach based on a multi-agent system capable of gathering information on data access activity in the HDFS cluster. Using this information, it classifies data according to its temperature (hot or cold) and makes decisions about data replication based on its classification. In addition, it compresses unused data to manage resources efficiently and reduce storage space usage.

Геопорталы как инструмент доступа к облачным хранилищам данных о метагеосистемах

The article is devoted to solving the scientific problem of spatial data storage management using geoportal systems on the basis of solving the scientific problem of analyzing the patterns of spatiotemporal organization of complex dynamic hierarchically organized metageosystems. The paper proves that the problem of integrating large arrays of spatial information in spatial data infrastructures (SDIs) can be solved by designing multi-model storages operating on the basis of database management systems of different classes. To improve the reliability of spatial data integration systems, it is advisable to focus on a microservice architecture that involves organizing the interaction of separate services, each of which performs its own specific task. The following microservices usage contexts can be distinguished in the IPD: geoportals, spatial data analysis, processing and management systems, as well as distributed cloud storages.

Thermal Management Effectiveness and Efficiency of a Fin Surrounded by a Phase Change Material (PCM)

Extended surfaces, also known as fins are used commonly for heat transfer enhancement in energy storage and thermal management problems. A fin may improve the rate of heat transfer by offering greater surface area. While the performance of a fin in a single-phase ambient such as air is well-understood, relatively lesser work is available on fin performance, including parameters such as fin effectiveness and efficiency, when embedded in a phase change material (PCM). A key theoretical challenge in such analysis is the transient nature of the phase change problem that must be combined with transient diffusion and phase change in the fin. This work presents theoretical analysis of phase change heat transfer between a base wall and a PCM in presence of a fin. The processes of thermal diffusion in the fin, thermal diffusion and phase change in the PCM are combined using perturbation analysis of a problem with time-dependent boundary condition. Results are found to be in good agreement with numerical simulations. Expressions for fin effectiveness and fin efficiency as functions of time are derived. The impact of various problem parameters such as fin geometry, material and the Stefan number on fin performance is analyzed. Expressions for efficiency and effectiveness of an array of equally-spaced fins are also derived. It is shown that while a fin provides additional surface area for enhanced melting of the PCM, transient diffusion in the fin may limit the benefit of the fin, especially at small times. On the other hand, fin performance at large times is limited by slow phase change in the PCM. Results presented here improve the fundamental understanding of PCM and fin based thermal management. Expressions for fin effectiveness and efficiency derived in this work offer useful tools for designing and improving the performance of practical fin and PCM based thermal management systems.

Stochastic Optimal Energy Storage Management for Energy Routers Via Compressive Sensing

The functionality of energy routing among microgrids is becoming increasingly important with the progress of deploying smart power systems all over the world. For higher energy routing performance and better renewable energy integration, a new type of electrical device, called energy router (ER), is being developed as a part of the infrastructure of the future energy Internet (EI). Generally, the long-term operation of ERs requires an effective energy management scheme for the energy storage inside these devices. In this article, considering the randomness of power generation by renewable energy sources and the stochastic power usage of loads in EI scenario, the compressive sensing is adopted for the solution to the nonlinear energy storage management problem which is essential for the design of ERs. The compressive sensing method used in this article is proven to be more efficient than the conventional Monte–Carlo methods and polynomial chaos expansion method, and the performance of the proposed method is evaluated with numerical examples.

Convergence and correctness of belief propagation for weighted min–max flow

In this paper, we investigate the performance of message-passing algorithms for the weighted min–max flow (WMMF) problem which was introduced by Ichimori et al. (1980). WMMF was well studied in combinational optimization, as it provides important applications in time transportation problem and the storage management problem. We develop a message-passing algorithm called min–max belief propagation (BP) for determining the optimal solution of WMMF. As the main result of this paper, we prove that for a digraph of size n, BP converges to the optimal solution within O(n3) time after O(n) iterations if the optimal solution of the underlying min–max flow problem instance is unique. To the best of our knowledge, the fastest polynomial time algorithm for WMMF runs in essentially O(n6) time among the known algorithms, where n is the number of vertices. On the other hand, it is one of a very few instances where BP is proved correct with fully-polynomial running time.

A MILP approach for detailed pipeline scheduling and storage management problem in the phosphate industry

Short-term detailed multiproduct pipeline scheduling is a complex problem with many industry-specific constraints. We are interested in solving an operational scheduling problem of a straight pipeline in the phosphate industry. In this article, we propose a mixed-integer linear programming model (MILP) with a discrete-time formulation to provide a detailed scheduling solution for a storage-sensitive problem. The objective is to determine the charging and discharging schedule of the slurry pipeline, to satisfy products demand while respecting all pipeline and storage constraints. The model was tested on different instances, proving to be able to minimize products sold-outs and the total water quantity scheduled.

Deep Reinforcement Learning Based Energy Storage Management Strategy Considering Prediction Intervals of Wind Power

Wind power generation development needs new methods to maintain the energy balance and stable operation by using energy storage management strategies. Most of the available storage control strategies do not take into account wind power uncertainty, which can be quantified by prediction intervals of wind power. This paper presents a data-driven energy storage management strategy considering prediction intervals of wind power for wind power microgrid operations. On the one hand, a power interval prediction model based on long short-term memory (LSTM) and lower upper bound estimation (LUBE) is established to quantify the uncertainty of wind power forecasting. On the other hand, the energy storage management problem is formulated as a Markov decision process (MDP), which is solved by deep reinforcement learning. According to the real-time state such as wind power, power prediction intervals, local load, dynamic electricity price and state of charge (SOC), the proposed strategy can make the charge/discharge schedule automatically. Effectiveness of the proposed energy storage management strategy is demonstrated on real wind farm dataset.

Privacy-preserving string search on encrypted genomic data using a generalized suffix tree

Abstract Background and objective Efficient sequencing technologies generate a plethora of genomic data and make it available to researchers. To compute a massive genomic dataset, outsourcing the data to the cloud is often required. Before outsourcing, data owners encrypt sensitive data to ensure data confidentiality. Outsourcing helps data owners to eliminate the local storage management problem. Since genome data is large in volume, executing researchers’ queries securely and efficiently is challenging. Methods In this paper, we propose a method to securely perform substring search and set-maximal search on a SNPs dataset using a generalized suffix tree. The proposed method guarantees the following: (1) data privacy, (2) query privacy, and (3) output privacy. It adopts the semi-honest adversary model, and the security of the data is guaranteed through encryption and garbled circuits. Results Our experimental results demonstrate that our proposed method can compute secure substring and set-maximal searches against a single-nucleotide polymorphism (SNPs) dataset of 2184 records (each record contains 10000 SNPs) in 2.3 and 2 s, respectively. Furthermore, we compared our results with existing techniques of secure substring and set-maximal search (Ishimaki et al., 2016; Shimizu et al., 2016) [1,2], and we achieved a 400 and 2 times improvement, respectively (Table 5).

A Narravite Cloud Storage Management using Multi-level Semantic Bonding Technique

The problem of cloud storage management has been well studied. The growing size and types of data increases the challenge in storage and retrieval. Number of approaches has been discussed for the problem of storage management. Text based clustering algorithms and Semantic based approaches are defined to improve the performance of storage management. However, the methods suffer to achieve higher performance in indexing and retrieval in terms of storage management. To solve this issue, an efficient semantic bonding measure based clustering and data management algorithm is presented. The method maintains ontology of various classes where each class has been mentioned in multiple levels. Each level of a class has specific properties and values. Using the semantic ontology, the method estimates MSB (Multi-level semantic Bonding) measure for different class of data. The same has been estimated for different level of semantic classes. Indexing of document class is performed based on MSB where the documents similarity has been measured using Topical Closure Measure (TCM). According to the value of TCM, the documents which are similar are identified and merge. The proposed algorithm improves the performance of document clustering and storage management in cloud environment

Roof-Top Stand-Alone PV Micro-Grid: A Joint Real-Time BES Management, Load Scheduling and Energy Procurement From a Peaker Generator

In this paper, we propose a user centric real-time (RT) technique for an optimal control of a distributed photovoltaic stand-alone micro-grid. Our objective is to minimize an average aggregated system cost over a finite time period considering a joint optimization problem of battery energy storage (BES) management, load scheduling, and energy procurement process from a peaker generator (PG). Due to online decision making, the dynamic control actions of the state of energy of the storage battery are correlated over time. Thus, we impose a constraint on the BES state of energy over the considered time period and eliminate the finite BES capacity constraint. In the load scheduling process, we account for the temporal variability and spatial uncertainty of each household load explicitly. We introduce the concept of block duration to optimize the energy procurement cost from a PG. We modify and then transform the problem to utilize the Lyapunov optimization technique. We show that the proposed solution to the joint optimization problem is asymptotically optimal with a bounded performance guarantee and is easy to implement. Simulations in different weather conditions show the effectiveness of our proposed user centric RT solution in terms of the selected performance metrics.

Secure Similar Patients Query on Encrypted Genomic Data

Both individuals and enterprises produce genomic data rapidly and continuously. There is a need to outsource such data to the cloud for better flexibility. Outsourcing also helps data owners by eliminating the local storage management problem. To protect data privacy and security, data owners must encrypt the sensitive data before outsourcing. Since genomic data are enormous in volume, executing researchers queries securely, and efficiently is a challenging task. In this paper, we introduce an indexing algorithm based on the prefix-tree to support similar patient queries. The proposed method guarantees the following: data privacy, query privacy, and output privacy. The privacy is guaranteed through encryption and garbled circuits considering the semi-honest adversary model. The overall computation is scalable and fast enough for real-life biomedical applications. Moreover, experimental results show that our method performs better than existing state-of-art techniques in this domain.

CTS2M: concurrent task scheduling and storage management for residential energy consumers under dynamic energy pricing

Dynamic energy pricing policy introduces real-time power-consumption-reflective pricing in the smart grid in order to incentivise energy consumers to schedule electricity-consuming applications (tasks) more prudently to minimise electric bills. This has become a particularly interesting problem with the availability of photovoltaic (PV) power generation facilities and controllable energy storage systems. This study addresses the problem of concurrent task scheduling and storage management for residential energy consumers with PV and storage systems, in order to minimise the electric bill. A general type of dynamic pricing scenario is assumed where the energy price is both time-of-use and power dependent. Tasks are allowed to support suspend-now and resume-later operations. A negotiation-based iterative approach has been proposed. In each iteration, all tasks are ripped-up and rescheduled under a fixed storage charging/discharging scheme, and then the storage control scheme is derived based on the latest task scheduling. The concept of congestion is introduced to gradually adjust the schedule of each task, whereas dynamic programming is used to find the optimal schedule. A near-optimal storage control algorithm is effectively implemented. Experimental results demonstrate that the proposed algorithm can achieve up to 60.95% in the total energy cost reduction compared with various baseline methods.

A new state-of-charge estimation method for electric vehicle lithium-ion batteries based on multiple input parameter fitting model

Summary The estimation of state-of-charge (SOC) is crucial to determine the remaining capacity of the Lithium-Ion battery, and thus plays an important role in many electric vehicle control and energy storage management problems. The accuracy of the estimated SOC depends mostly on the accuracy of the battery model, which is mainly affected by factors like temperature, State of Health (SOH), and chemical reactions. Also many characteristic parameters of the battery cell, such as the output voltage, the internal resistance and so on, have close relations with SOC. Battery models are often identified by a large amount of experiments under different SOCs and temperatures. To resolve this difficulty and also improve modeling accuracy, a multiple input parameter fitting model of the Lithium-Ion battery and the factors that would affect the accuracy of the battery model are derived from the Nernst equation in this paper. Statistics theory is applied to obtain a more accurate battery model while using less measurement data. The relevant parameters can be calculated by data fitting through measurement on factors like continuously changing temperatures. From the obtained battery model, Extended Kalman Filter algorithm is applied to estimate the SOC. Finally, simulation and experimental results are given to illustrate the advantage of the proposed SOC estimation method. It is found that the proposed SOC estimation method always satisfies the precision requirement in the relevant Standards under different environmental temperatures. Particularly, the SOC estimation accuracy can be improved by 14% under low temperatures below 0 °C compared with existing methods. Copyright © 2017 John Wiley & Sons, Ltd.

SILVERBACK+: scalable association mining via fast list intersection for columnar social data

We present Silverback+, a scalable probabilistic framework for accurate association rule and frequent item-set mining of large-scale social behavioral data. Silverback+ tackles the problem of efficient storage utilization and management via: (1) probabilistic columnar infrastructure and (2) using Bloom filters and sampling techniques. In addition, probabilistic pruning techniques based on Apriori method are developed, for accelerating the mining of frequent item-sets. The proposed target-driven techniques yield a significant reduction of the size of the frequent item-set candidates, as well as the required number of repetitive membership checks through a novel list intersection algorithm. Extensive experimental evaluations demonstrate the benefits of this context-aware consideration and incorporation of the infrastructure limitations when utilizing the corresponding research techniques. When compared to the traditional Hadoop-based approach for improving scalability by straightforwardly adding more hosts, Silverback+ exhibits a much better runtime performance, with negligible loss of accuracy.

Mean-Conditional Value-at-Risk Optimal Energy Storage Operation in the Presence of Transaction Costs

This paper addresses the formulation and solution of an optimal energy storage management problem under risk consideration and transaction costs of trading energy with the power grid. The price evolves as a stochastic process, capable of correctly explaining the seasonality effects as well as the tail fatness and spikiness in its distribution. Transaction costs capture the price impact of the storage operation on the electricity spot price. A risk analysis of an optimal risk neutral deterministic policy as well as the simple myopic policy indicates that the realized operational cost may notably differ from the expected cost by a considerable probability. This difference suggests that we need to consider risk. Using the downside risk measure of conditional value-at-risk, an optimal risk averse conversion and transmission strategy, among the grid, the renewable power generation source, and an energy storage is proposed to fully satisfy the electricity demand and minimize the expected operational cost as well as the risk. Our numerical study using data from NYISO demonstrates the impacts of risk consideration and the transaction cost parameters on the optimal strategy structure, its expected cost, and its risk.

Approximately nearly controllability of discrete-time bilinear control systems with control input characteristic

In this paper we study a class of discrete-time bilinear control systems that can be used to model the production storage management problems. A weaker but more applicable concept of controllability, approximately nearly controllability, is proposed under the framework of hypercyclicity from the theory of dynamical systems. The complete characteristic of the system structure for being approximately nearly controllable is provided. Necessary and sufficient conditions for the system to be approximately nearly controllable are presented. Moreover, the required number of different input values for the system to achieve the approximately nearly controllability during the evolutionary process is obtained.

Proposal of Storage Process Optimization in Manufacturing Enterprise

Article focuses on the issue of storage optimization and efficiency improvement through the control and overview of material flow in the chosen company. In the first chapter is processed introduction overview of case study. Second chapter is focused on defining the problem of current storage management. The third chapter exactly defined goals and application of solved issue which should by achieved using the proposed innovations in chapter five. In the forth chapter is current state of production storage process in the enterprise. Summary of the issue is last chapter called conclusion.

An adaptive per-application storage management scheme based on manifold learning in information centric networks

Information Centric Network (ICN) is an emerging network paradigm centered around the named contents rather than the host-to-host connectivity. The common characteristic of ICN leverages in-network caching to achieve an efficient and reliable content distribution but also brings challenges. The in-network caching technique equips all ICN routers with cache storage. However, no existing works focus on the cache storage sharing mechanism among different applications to satisfy the line speed requirements and diversity of applications in ICN. In this paper, we formulate the per-application storage management problem into an optimized resource allocation problem and introduce a manifold learning method to classify the priority of applications. Dynamic programming is adopted to solve the formulated problem and an adaptive per-application storage management scheme is proposed on the basis of the optimal solutions. Extensive experiments have been performed to evaluate the proposed scheme and show that our approach is superior to static partitioning and shared storage schemes.

Opportunistic data collection for disconnected wireless sensor networks by mobile mules

This paper considers a field with a number of isolated wireless sensor networks served by some mobile mules and base stations (BSs). Sensing data needs to be carried by mobile mules to BSs via opportunistic contact between them. Also, such contact may not be frequent. Thus there are four types of communications in this environment: (i) inter-node communications within a WSN, (ii) opportunistic WSN-to-mule communications, (iii) opportunistic mule-to-mule communications, and (iv) opportunistic mule-to-BS communications. In such disconnected WSNs, since sensors’ memory spaces are limited and data collection from isolated WSNs to mules and then to BSs relies on opportunistic communications in the sense that contact between these entities is occasional, storing and collecting higher-priority data is necessary. Therefore, there are two critical issues to be addressed: the data storage management in each isolated WSN and opportunistic data collection between these entities. We address the storage management problem by modeling the limited memory spaces of a WSN’s sensor nodes as a distributed storage system. Assuming that there is a sink in the WSN that will be visited by mobile mules occasionally, we address three issues: (i) how to buffer sensory data to reduce data loss due to a shortage of storage spaces, (ii) if dropping of data is inevitable, how to avoid higher-priority data from being dropped, and (iii) how to manage the data nearby the sink to facilitate the downloading jobs of mules when the downloading time is unpredictable. We propose a Distributed Storage Management (DSM) strategy based on a novel shuffling mechanism similar to heap sort. It allows nodes to exchange sensory data with neighbors efficiently in a distributed manner. For the opportunistic data collection problem, based on a utility model, we then develop an Opportunistic Data Exchange (ODE) strategy to guide two mules to exchange data that would lead to a higher reward. To the best of our knowledge, this is the first work addressing distributed storage strategy for isolated WSNs with opportunistic communications using mobile mules. We conduct extensive simulations to investigate the merit of DSM and ODE. The simulation results indicate that the level of data importance collected by our DSM is very close to a global optimization and our ODE could facilitate delivery of important data to BSs through mules. We also implement these strategies in a real sensor platform, which demonstrates that the simple and lightweight protocols can achieve our goals.

Secure and efficient public key management in next generation mobile networks

Employing public key-based security architecture is inevitable for the advanced security applications in the mobile networks. However, key storage management problems have arisen, because the public key computation is still the large overhead to USIM, and the mobile equipment has potential threats of the key leakage or loss. In order to solve such shortcomings, we improve the key-insulated models and propose "Trust Delegation" model that the overall security computations are operated in ME, while the initial private key still remains in the secure storage in USIM. Our model is resilient against not only key exposure but also key loss. Finally, we show that the overall transactions can be reduced to one-third than current 3GPP Generic Authentication Architecture.