Efficient File Research Articles

병렬 프로그램에서의 효율적인 대용량 파일 입출력 방식의 비교 연구

병렬 프로그램에서의 효율적인 대용량 파일 입출력 방식의 비교 연구

A forensic method for efficient file extraction in HDFS based on three-level mapping

The large scale and distribution of cloud computing storage have become the major challenges in cloud forensics for file extraction. Current disk forensic methods do not adapt to cloud computing well and the forensic research on distributed file system is inadequate. To address the forensic problems, this paper uses the Hadoop distributed file system (HDFS) as a case study and proposes a forensic method for efficient file extraction based on three-level (3L) mapping. First, HDFS is analyzed from overall architecture to local file system. Second, the 3L mapping of an HDFS file from HDFS namespace to data blocks on local file system is established and a recovery method for deleted files based on 3L mapping is presented. Third, a multi-node Hadoop framework via Xen virtualization platform is set up to test the performance of the method. The results indicate that the proposed method could succeed in efficient location of large files stored across data nodes, make selective image of disk data and get high recovery rate of deleted files.

Processing tracking in jMRUI software for magnetic resonance spectra quantitation reproducibility assurance

BackgroundProton magnetic resonance spectroscopy is a non-invasive measurement technique which provides information about concentrations of up to 20 metabolites participating in intracellular biochemical processes. In order to obtain any metabolic information from measured spectra a processing should be done in specialized software, like jMRUI. The processing is interactive and complex and often requires many trials before obtaining a correct result. This paper proposes a jMRUI enhancement for efficient and unambiguous history tracking and file identification.ResultsA database storing all processing steps, parameters and files used in processing was developed for jMRUI. The solution was developed in Java, authors used a SQL database for robust storage of parameters and SHA-256 hash code for unambiguous file identification. The developed system was integrated directly in jMRUI and it will be publically available. A graphical user interface was implemented in order to make the user experience more comfortable. The database operation is invisible from the point of view of the common user, all tracking operations are performed in the background.ConclusionsThe implemented jMRUI database is a tool that can significantly help the user to track the processing history performed on data in jMRUI. The created tool is oriented to be user-friendly, robust and easy to use. The database GUI allows the user to browse the whole processing history of a selected file and learn e.g. what processing lead to the results, where the original data are stored, to obtain the list of all processing actions performed on spectra.

CLDSafe: An Efficient File Backup System in Cloud Storage against Ransomware

CLDSafe: An Efficient File Backup System in Cloud Storage against Ransomware

File Transfer Between Charging Stations Goes Infrastructureless: A Code–Delay Tradeoff

In an energy harvesting vehicular network, efficient file transfer between two charging stations is challenging due to the lack of communication infrastructure. Naturally, a source station can allocate the packets to a set of vehicles, which serve as relays to opportunistically deliver these packets to the destination station while performing their own tasks. In this paper, we show that, with packet-level erasure coding (e.g., fountain codes), only a few redundant packets are required to achieve a significant reduction in file delay. As more redundant packets are added, however, the file delay improvement diminishes, and the storage cost rises. In both the single and multiple-destination scenarios, the fundamental tradeoffs between coding and file delay are characterized. For the multiple-destination scenario, there exists a unique and optimal coding rate that minimizes the average file delay. We propose a greedy algorithm that can achieve this minimum delay in practice. Finally, the proposed tradeoffs are verified through intensive numerical experiments.

E2FS: an elastic storage system for cloud computing

In cloud storage, replication technologies are essential to fault tolerance and high availability of data. While achieving the goal of high availability, replication brings extra number of active servers to the storage system. Extra active servers mean extra power consumption and capital expenditure. Furthermore, the lack of classification of data makes replication scheme fixed at the very beginning. This paper proposes an elastic and efficient file storage called E2FS for big data applications. E2FS can dynamically scale in/out the storage system based on real-time demands of big data applications. We adopt a novel replication scheme based on data blocks, which provides a fine-grained maintenance of the data in the storage system. E2FS analyzes features of data and makes dynamic replication decision to balance the cost and performance of cloud storage. To evaluate the performance of proposed work, we implement a prototype of E2FS and compare it with HDFS. Our experiments show E2FS can outperform HDFS in elasticity while achieving guaranteed performance for big data applications.

Design and analysis of parallel file downloading algorithms in peer-to-peer networks

It is well known that the method of parallel downloading can be used to reduce file download times in a peer-to-peer (P2P) network. There has been little investigation on parallel download and chunk allocation for source peers with random service capacities. The main contribution of this paper is to address the problem of efficient parallel file download in P2P networks with random service capacities. A precise analysis of the expected download time is given when the service capacity of a source peer is a random variable. A general framework is developed for analyzing the expected download time of a parallel download and chunk allocation algorithm, and is applied to the analysis of several algorithms. Two chunk allocation algorithms for parallel download are proposed. It is observed that the performance of parallel download can be significantly improved by using the method of probing high-capacity peers. One such algorithm is proposed and its expected parallel download time is analyzed. The performance of these parallel file download algorithms in P2P networks with random service capacities are compared. The above parallel download algorithms are extended to multiple file download by dividing source peers into clusters. It is noticed that there is an important issue of optimal parallelism which minimizes the combined effect of intracluster and intercluster overhead of parallel download and load imbalance.

Efficient File Search in Delay Tolerant Networks with Social Content and Contact Awareness

Distributed file searching in delay tolerant networks formed by mobile devices can potentially support various useful applications. In such networks, nodes often present certain social network properties of their holders in terms of contents (i.e., interests) and contacts. However, current methods in DTNs only consider either content or contact for file searching or dissemination, which limits the file sharing efficiency. In this paper, we first analyze real traces to confirm the importance and necessity of considering both content and contact in file search. We then propose Cont $^2$ , a social-aware file search method that exploits both node contents and contact patterns. First, considering people with common interests tend to share files and gather together, Cont $^2$ virtually groups common-interest nodes into a community to direct file search. Second, considering human mobility follows a certain pattern, Cont $^2$ exploits nodes’ contact frequencies with a community to expedite file searching. To further improve the searching efficiency, Cont $^2$ also integrates sub-communities and parallel forwarding as optional components for file searching. Trace-driven experiments on the GENI testbed and NS-2 simulator show that Cont $^2$ can effectively improve the search efficiency compared to current methods.

An Efficient File Hierarchy Attribute-Based Encryption Scheme in Cloud Computing

Ciphertext-policy attribute-based encryption (CP-ABE) has been a preferred encryption technology to solve the challenging problem of secure data sharing in cloud computing. The shared data files generally have the characteristic of multilevel hierarchy, particularly in the area of healthcare and the military. However, the hierarchy structure of shared files has not been explored in CP-ABE. In this paper, an efficient file hierarchy attribute-based encryption scheme is proposed in cloud computing. The layered access structures are integrated into a single access structure, and then, the hierarchical files are encrypted with the integrated access structure. The ciphertext components related to attributes could be shared by the files. Therefore, both ciphertext storage and time cost of encryption are saved. Moreover, the proposed scheme is proved to be secure under the standard assumption. Experimental simulation shows that the proposed scheme is highly efficient in terms of encryption and decryption. With the number of the files increasing, the advantages of our scheme become more and more conspicuous.

An Efficient Scheme to Establish Content and Audit Liveness in the Cloud

Background/Objectives: Data Storage in the cloud based services are evident everywhere in this day and age starting from Google doc based file management services, file storage services etc. Various schemes with respect to protecting this data range from ciphering services, auditing services, migration services etc. All these services help a cloud user to preserve their data based on a certain time threshold. Methods/Statistical Analysis: The current context of the paper concentrates on delivering an efficient data audit service. Previously these Audit services used to reside within the cloud domain but due to untrustworthiness of many cloud operators, Outside Auditor (OA) implementations were gearing up to validate cloud stored data. Findings: Earlier these OA’s used to download users data at their end to check their integrity leading to users privacy problems. Also it happens to be a time complex operation. Hence we propose an efficient file chunk sampling to detect anomalous changes in the cloud stored data. Index Hash structures play a vital role in this aspect. Applications/Improvements: This process is an iterative procedure instead of one time operation happening at regular intervals and importantly preserving user’s privacy.

Secure and efficient fine-grained multiple file sharing in cloud-assisted crowd sensing networks

Mobile crowd sensing significantly facilitates a broad range of emerging applications by treating mobile devices of ordinary users as basic sensing units to distribute sensing tasks and collect sensing data. Owing to the continuity of data sensing and richness of content, resource-constrained mobile devices always outsource their collected data into the cloud for file sharing. The existing work enables only batch decryption limited to RSA algorithm and merely allows one specific file to be shared among multiple authorized receivers by exploiting the technique of attribute-based encryption (ABE). In this paper, a generalized efficient batch cryptosystem GBC and its extension GBCS are firstly proposed to achieve both batch encryption and batch decryption from any public key encryption algorithm. Then, our proposed GBC is further extended to FMFS, TMFS and MMFS to address secure multiple file sharing in the cloud-assisted mobile crowd sensing network, respectively for fine-grained multi-receiver multi-file sharing, file authority transfer and multiple file owners’s settings. Finally, formal security proof is given to show our proposed generalized efficient batch cryptosystems GBC, GBCS and secure multiple file sharing schemes are respectively secure against adaptive chosen ciphertext attack (CCA2) in the random oracle model and under the assumption on the computational hardness of the subset decision problem in the standard model. Extensive evaluations illustrate our secure multiple file sharing in cloud-assisted crowd sensing network based on the newly-devised GBC dramatically outperforms the state-of-the-art in storage, computational and communication overhead.

파일 분산 저장 시스템의 에너지 효율성 증대를 위한 파일 블록 관리 기술

저장해야 하는 데이터양이 상당히 증가하여 필요 저장 장치의 수가 증가하게 되었다. 다수의 저장 장치 이용 시 일부 저장 장치가 사용 불가능하여도 파일의 가용성을 보장하는 파일 분산 저장 기술이 필수적이다. 최근 다수의 저장 장치로 구성된 파일 분산 저장 시스템의 에너지 소모가 문제가 되고 있다. 파일 분산 저장 시스템의 에너지 효율성을 향상시키기 위한 대표적인 기존 기술은 효율적인 파일 블록 배치를 통하여 사용량이 적은 시간에 일부 저장 장치를 절전 상태로 변경하는 것이다. 한 번 배치가 된 파일 블록의 재배치는 기존 연구에서 고려되지 않는데, 대량의 파일을 저장하고 있는 파일 분산 저장 시스템에서 파일 블록의 재배치에는 큰 비용이 필요할 수 있기 때문이다. 하지만 새 저장 장치나 파일이 추가되는 경우를 고려할 때 파일 블록의 재배치는 필수적이다. 본 논문은 파일 블록의 재배치 시 필요한 블록 이동수를 최소화하는 정수 최적화 문제를 제시하고 이를 분기한정법 알고리즘으로 해결하는 방법을 제안한다. 이를 통해 최소한의 파일 블록 이동으로 최대한의 많은 수의 저장 장치를 절전 상태로 변경할 수 있다. 하지만 정수 최적화 문제의 분기한정법 알고리즘을 통한 해결은 연산 속도가 문제 크기에 따라 지수 함수적으로 증가하는 문제가 있다. 따라서 본 논문에서는 모든 파일과 데이터 서버를 여러 집단으로 나누어 크기가 작은 문제 다수를 해결하는 방식을 제안한다. Because of rapid growth of data size, the number of data storage has been increased. When using multiple data storages, a distribute file system is essential to insure the availability of data files. The power consumption is a major problem when using a distributed file system with many data storages. Previous works have aimed at reducing the energy consumption with efficient file block layout by changing some data servers into stand-by mode. The file block migration has not been seriously considered because migration causes large cost. But when we consider addition of a new data server or file, file block migration is needed. This paper formulates the minimization of data block migration as an ILP optimization problem and solves it using branch-and-bound method. Using this technique, we can maximize the number of stand-by data servers with the minimum number of file block movement. However, computation time of branch-and-bound method of an ILP optimization problem increases exponentially as the problem size grows. Therefore this paper also proposes a data block and data server grouping method to solve many small ILP problems.

A secure and efficient file protecting system based on SHA3 and parallel AES

We propose a secure and efficient file protecting system (SEFPS) that uses currently excellent SHA3 and parallel AES.We prove the security of SEFPS in terms of confidentiality and integrity.We design and implement the three variants of SEFPS adopting GPU parallelism or/and CPU parallelism.We evaluate the SEFPS' performance on two representative platforms. There are many private or confidential files stored in computers or transferred on the Internet. People worry and even fear their security problems, such as stealing, breaking, forging, and so on, and urgently need a both secure and highly efficient (for better experience) file protecting system. Thus, we propose and implement a secure and efficient file protecting system (SEFPS) for this demand. SEFPS based on advanced SHA3 (Secure Hash Algorithm 3) and parallel AES (Advance Encryption Standard) can provide the protection of both confidentiality and integrity, and produce high performance by GPU (Graphics Processing Unit) parallelism or/and CPU (Central Processing Unit) parallelism. Correspondingly, SEFPS has three variants for GPU parallelism, CPU parallelism, and both of them. The first variant includes CPU Parallel Protecting (CPP) and CPU Parallel Unprotecting (CPUP). The second variant includes GPU Parallel Protecting (GPP) and GPU Parallel Unprotecting (GPUP). The third variant includes Hybrid Parallel Protecting (HPP) and Hybrid Parallel Unprotecting (HPUP). We design and implement them, and evaluate their performance on two representative platforms by some experiments. HPP and HPUP outperform GPP and GPUP, respectively, and outperform CPP and CPUP more. For those computers not equipped with Nvidia GPUs, CPP and CPUP can be employed, because they still outperform CSP and CSUP, respectively, where CSP and CSUP denote the serial implementation of SEFPS for protecting and for unprotecting, respectively. Moreover, we also prove the security of SEFPS in terms of integrity and confidentiality. Thus, SEFPS is a secure and efficient file protecting system, and can be used in computers no matter whether equipped with Nvidia GPUs or not.

Photon-HDF5: An Open File Format for Timestamp-Based Single-Molecule Fluorescence Experiments

We introduce Photon-HDF5, an open and efficient file format to simplify exchange and long-term accessibility of data from single-molecule fluorescence experiments based on photon-counting detectors such as single-photon avalanche diode, photomultiplier tube, or arrays of such detectors. The format is based on HDF5, a widely used platform- and language-independent hierarchical file format for which user-friendly viewers are available. Photon-HDF5 can store raw photon data (timestamp, channel number, etc.) from any acquisition hardware, but also setup and sample description, information on provenance, authorship and other metadata, and is flexible enough to include any kind of custom data. The format specifications are hosted on a public website, which is open to contributions by the biophysics community. As an initial resource, the website provides code examples to read Photon-HDF5 files in several programming languages and a reference Python library (phconvert), to create new Photon-HDF5 files and convert several existing file formats into Photon-HDF5. To encourage adoption by the academic and commercial communities, all software is released under the MIT open source license.

Content-based File Sharing in Peer-to-peer Networks Using Threshold

In content based file sharing peer-to-peer (P2P) [1] network model nodes share files directly with each other without a centralized server. In such a file sharing system, nodes meet and exchange requests and files in the format of text, short videos, and voice clips in different interest categories. Content is various and large file sharing such as the multimedia content is required with the rapid development of the wireless communication technology. File sharing can also mean having an allocated amount of personal file storage in a common file system. A P2P content based file sharing system, for efficient file searching, threshold takes advantage of node mobility by designating stable nodes, which have the most frequent contact with community members, as community coordinators for intra community searching, and highly mobile nodes that visit other communities frequently as community ambassadors for intercommunity searching. The large file sharing needs more stable end to end path and long transmission time. Last but not least, more relationship between nodes will be used to promote the file sharing process. Content based file sharing is helpful for taking certain decisions during file transmission. These decisions will benefit in proper utilization of network resources. In this paper content-based file sharing scheme using threshold is proposed. The user's interest is determined by the proposed scheme before searching and sharing the files in the peer-to-peer network. The resources in the network are utilized as per the contents of the files to be shared. The performance evaluation show that proposed system significantly lowers transmission cost and improves file sharing success rate compared to current methods.

Energy Efficient Metadata Management for Cloud Storage System

To effectively handle duplicate files, data deduplication schemes are widely used in many storage systems. Data deduplication algorithms reduce storage space by eliminating data to ensure that only single instance of data is stored in storage device. In this paper, we propose an energy efficient file synchronization scheme that provides hybrid data chunking using variable-length chunking (VLC) and fixed-length chunking (FLC). The main idea is to analyze similarities between old and new versions of data and decide which chunking method to apply in synchronizing the files. In particular, the proposed algorithm exploits the file similarity pattern for calculating the energy efficiency of chunking algorithms. We have developed an Android mobile application for file synchronization and measured energy consumption. The experiment results show that the proposed scheme helps save energy in synchronizing files, regardless of file types or amount of redundancies the files have.

Scalable and efficient file sharing in information-centric networking

Internet usage has been increasingly shifting toward massive content retrieval and distribution. To satisfy the demand, information-centric networking (ICN) approaches have emerged. These approaches have common features such as in-network caching and the decoupling of content and location to enable efficient content delivery. In this paper, we investigate the performance issue of file sharing in ICN. Compared to web and user generated content (UGC), file sharing entails relatively low request rates, but much larger content. These characteristics make it difficult for file sharing traffic to take advantage of in-network caching while competing with web and UGC traffic. For scalable and efficient file sharing, we design a peer-aware routing (PAR) protocol to utilize peers׳ storage in ICN. PAR uses a name-based shared tree without additional name resolution service. The shared tree is built at a network layer to connect peers for the same purpose, allowing a peer to discover the nearest peer holding the requested content in the tree. To evaluate the proposed PAR protocol, we conducted a large-scale simulation with a BitTorrent trace over the Internet-scale topology. We show that our protocol can substantially reduce inter-domain traffic, while keeping network overhead low.

A Proximity-Aware Interest-Clustered P2P File Sharing System

Efficient file query is important to the overall performance of peer-to-peer (P2P) file sharing systems. Clustering peers by their common interests can significantly enhance the efficiency of file query. Clustering peers by their physical proximity can also improve file query performance. However, few current works are able to cluster peers based on both peer interest and physical proximity. Although structured P2Ps provide higher file query efficiency than unstructured P2Ps, it is difficult to realize it due to their strictly defined topologies. In this work, we introduce a Proximity-Aware and Interest-clustered P2P file sharing System (PAIS) based on a structured P2P, which forms physically-close nodes into a cluster and further groups physically-close and common-interest nodes into a sub-cluster based on a hierarchical topology. PAIS uses an intelligent file replication algorithm to further enhance file query efficiency. It creates replicas of files that are frequently requested by a group of physically close nodes in their location. Moreover, PAIS enhances the intra-sub-cluster file searching through several approaches. First, it further classifies the interest of a sub-cluster to a number of sub-interests, and clusters common-sub-interest nodes into a group for file sharing. Second, PAIS builds an overlay for each group that connects lower capacity nodes to higher capacity nodes for distributed file querying while avoiding node overload. Third, to reduce file searching delay, PAIS uses proactive file information collection so that a file requester can know if its requested file is in its nearby nodes. Fourth, to reduce the overhead of the file information collection, PAIS uses bloom filter based file information collection and corresponding distributed file searching. Fifth, to improve the file sharing efficiency, PAIS ranks the bloom filter results in order. Sixth, considering that a recently visited file tends to be visited again, the bloom filter based approach is enhanced by only checking the newly added bloom filter information to reduce file searching delay. Trace-driven experimental results from the real-world PlanetLab testbed demonstrate that PAIS dramatically reduces overhead and enhances the efficiency of file sharing with and without churn. Further, the experimental results show the high effectiveness of the intra-sub-cluster file searching approaches in improving file searching efficiency.

A four-phase data replication algorithm for data grid

<p>Nowadays, scientific applications generate a huge amount of data in terabytes or petabytes. Data grids currently proposed solutions to large scale data management problems including efficient file transfer and replication. Data is typically replicated in a Data Grid to improve the job response time and data availability. A reasonable number and right locations for replicas has become a challenge in the Data Grid. In this paper, a four-phase dynamic data replication algorithm based on Temporal and Geographical locality is proposed. It includes: 1) evaluating and identifying the popular data and triggering a replication operation when the popularity data passes a dynamic threshold; 2) analyzing and modeling the relationship between system availability and the number of replicas, and calculating a suitable number of new replicas; 3) evaluating and identifying the popular data in each site, and placing replicas among them; 4) removing files with least cost of average access time when encountering insufficient space for replication. The algorithm was tested using a grid simulator, OptorSim developed by European Data Grid Projects. The simulation results show that the proposed algorithm has better performance in comparison with other algorithms in terms of job execution time, effective network usage and percentage of storage filled.</p>

Maximizing P2P File Access Availability in Mobile Ad Hoc Networks though Replication for Efficient File Sharing

File sharing applications in mobile ad hoc networks (MANETs) have attracted more and more attention in recent years. The efficiency of file querying suffers from the distinctive properties of such networks including node mobility and limited communication range and resource. An intuitive method to alleviate this problem is to create file replicas in the network. However, despite the efforts on file replication, no research has focused on the global optimal replica creation with minimum average querying delay. Specifically, current file replication protocols in mobile ad hoc networks have two shortcomings. First, they lack a rule to allocate limited resources to different files in order to minimize the average querying delay. Second, they simply consider storage as available resources for replicas, but neglect the fact that the file holders’ frequency of meeting other nodes also plays an important role in determining file availability. Actually, a node that has a higher meeting frequency with others provides higher availability to its files. This becomes even more evident in sparsely distributed MANETs, in which nodes meet disruptively. In this paper, we introduce a new concept of resource for file replication, which considers both node storage and meeting frequency. We theoretically study the influence of resource allocation on the average querying delay and derive a resource allocation rule to minimize the average querying delay. We further propose a distributed file replication protocol to realize the proposed rule. Extensive trace-driven experiments with synthesized traces and real traces show that our protocol can achieve shorter average querying delay at a lower cost than current replication protocols.