File Storage Research Articles

Streaming Large-Scale Microscopy Data to a Supercomputing Facility.

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s-1 (87,000 frames s-1), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera's data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

College Amenities Application (CampusIn)

Abstract—In the digital age, enhancing student experience and academic success through efficient services is essential for modern educational institutions. The College Amenities application serves as a comprehensive, multifunctional platform that integrates various aspects of academics into one seamless system. It offers a range of services, including food ordering, where students can conveniently browse menus, place orders, and track deliveries; event showcasing, which keeps students informed about campus events, workshops, social activities and placement information, providing access to job opportunities, internships, and career resources. Additionally, the application features like secure document storage, enabling students to organize and retrieve academic materials such as notes, assignments, and research papers with ease. By combining these features, the application improves campus connectivity, enhances student engagement, and supports academic and professional growth, making it a valuable tool for fostering a more efficient and interactive campus experience. Keywords: Service Automation, Campus dining, Event Registration, Placement Automation, File Storage.

Seamless, rapid, and accurate analyses of outbreak genomic data using split k-mer analysis.

Sequence variation observed in populations of pathogens can be used for important public health and evolutionary genomic analyses, especially outbreak analysis and transmission reconstruction. Identifying this variation is typically achieved by aligning sequence reads to a reference genome, but this approach is susceptible to reference biases and requires careful filtering of called genotypes. There is a need for tools that can process this growing volume of bacterial genome data, providing rapid results, but that remain simple so they can be used without highly trained bioinformaticians, expensive data analysis, and long-term storage and processing of large files. Here we describe split k-mer analysis (SKA2), a method that supports both reference-free and reference-based mapping to quickly and accurately genotype populations of bacteria using sequencing reads or genome assemblies. SKA2 is highly accurate for closely related samples, and in outbreak simulations, we show superior variant recall compared with reference-based methods, with no false positives. SKA2 can also accurately map variants to a reference and be used with recombination detection methods to rapidly reconstruct vertical evolutionary history. SKA2 is many times faster than comparable methods and can be used to add new genomes to an existing call set, allowing sequential use without the need to reanalyze entire collections. With an inherent absence of reference bias, high accuracy, and a robust implementation, SKA2 has the potential to become the tool of choice for genotyping bacteria. SKA2 is implemented in Rust and is freely available as open-source software.

Honey password vaults tolerating leakage of both personally identifiable information and passwords

Honey vaults are useful tools for password management. A vault usually contains usernames for each domain, and the corresponding passwords, encrypted with a master password chosen by the owner. By generating decoy vaults for incorrect master password attempts, honey vaults force attackers with the vault’s storage file to engage in online verification to distinguish the real vaults, thus thwarting offline guessing attacks. However, sophisticated attackers can acquire additional information, such as personally identifiable information (PII) and partial passwords contained within the vault from various data breaches. Since many users tend to incorporate PII in their passwords, attackers may utilize PII to distinguish the real vault. Furthermore, if attackers may learn partial passwords included in the real vault, it can exclude numerous decoy vaults without the need for online verification. Indeed, both leakages pose serious threats to the security of the existing honey vault schemes. In this paper, we explore two attack variants of the inspired attack scenario, where the attacker gains access to the vault’s storage file along with acquiring PII and partial passwords contained within the real vault, and design a new honey vault scheme. For security assurance, we prove that our scheme is secure against one of the aforementioned attack variants. Moreover, our experimental findings suggest enhancements in security against the other attack. In particular, to evaluate the security in multiple leakage cases where both the vault’s storage file and PII are leaked, we propose several new practical attacks (called PII-based attacks), building upon the existing practical attacks in the traditional single leakage case where only the vault’s storage file is compromised. Our experimental results demonstrate that certain PII-based attacks achieve a 63–70% accuracy in distinguishing the real vault from decoys in the best-performing honey vault scheme (Cheng et al. in Incrementally updateable honey password vaults, pp 857–874, 2021). Our scheme reduces these metrics to 41–50%, closely approaching the ideal value of 50%.

Efficient security interface for high-performance Ceph storage systems

Ceph portrays a resilient clustered storage solution with supporting object, block, and file storage capabilities with no single point of failure. Despite these qualifications, data confidentiality defines a concern in the system, as authentication and access control are the only data protection security services in Ceph. CephArmor was proposed as a third-party security interface to protect data confidentiality by adding an extra protection layer to data at rest. Despite the added layer, the initial design of the API needed to be more efficient in addressing security and performance simultaneously. In this study, we propose a new architectural design to address the associated issues with the preliminary prototype. Comprehensive performance and security analysis verify the improvement of the proposed method compared to the initial approach. The benchmark result has indicated a 37% improvement on average in IOPS, elapsed time, and bandwidth for the write benchmark compared to the initial model.

Spatial lung imaging in clinical and translational settings.

For many severe lung diseases, non-invasive biomarkers from imaging could improve early detection of lung injury or disease onset, establish a diagnosis, or help follow-up disease progression and treatment strategies. Imaging of the thorax and lung is challenging due to its size, respiration movement, transferred cardiac pulsation, vast density range and gravitation sensitivity. However, there is extensive ongoing research in this fast-evolving field. Recent improvements in spatial imaging have allowed us to study the three-dimensional structure of the lung, providing both spatial architecture and transcriptomic information at single-cell resolution. This fast progression, however, comes with several challenges, including significant image file storage and network capacity issues, increased costs, data processing and analysis, the role of artificial intelligence and machine learning, and mechanisms to combine several modalities. In this review, we provide an overview of advances and current issues in the field of spatial lung imaging.

Designing File Storage and Sharing System

Background: Files stored on a network are anticipated to be exchanged securely and efficiently. An internet-based file management platform that enables users to securely create, save, organize, collaborate, and share files. Sharing is a recently developed service idea that is frequently sought after by researchers and students. The suggested work offers many forms of data files, including documents and multimedia files. File storage has been a widely used storage method for many years. Materials and Methods: The proposed method used to build the website is Visual Studio 2015 with SQL server Database to provide file storage with different data type storage and it provides a service as an IT repository file storage system to help student and teachers to communicate with each other in academic way for publishing files in easy and in a flexible manner. Results: The proposed system results showed that the website can provide file upload with supported files are ".doc", ".docx", ".pptx", ".pdf", ".zip", ".rar", ".mp4", ".flv", ".avi", ".PDF", ".DOCX" and ".DOC". Results showed that the system uploaded file size 5M, 10 and 25 M smoothly. Conclusion: The work provides a flexible way to store and sharing files for different fields and we use it for education purposes. The results showed that the proposed system is very flexible in the process of uploading various files at a very high speed, due to the ease of uploading files to the server and accessing them by various students in the college.

A hybrid architecture for secure data sharing in multi-clouds system

Abstract Cloud computing is one of the most cutting-edge technologies around the world that plays a major role in IT industries and personal use. Several sectors are showing efforts in adopting cloud computing to their services, considering the provided cost reduction and process efficiency. However, outsourcing users’ sensitive data increases the concerns regarding the security, privacy and integrity of stored data. Therefore, there is a need to build a trusting relationship between users and cloud systems. Hence, in this paper, we propose a privacy-preserving framework, called Hybrid and Secure Data Sharing Architecture (HSDSA), for secure data storage in cloud systems. The basic idea of HSDSA is to improve data security in a multi-cloud environment using a combination of cryptography techniques. These techniques ensure that the user has total control over the data generation and management of the decryption without relying on a trusted authority. HSDSA provides removal of centralized file storage distribution and ensures data integrity in the recovery process. We evaluate our contribution under Cloudera, and the results demonstrate the efficiency of HSDSA compared with the existing system.

Using the Ethereum Blockchain in the Internet of Things Network for IT Diagnostics

The article discusses the use of Ethereum blockchain technology in the Internet of Things (IoT) network for IT diagnostics of patients, which increases data security and user privacy. This integration is proving effective for storing and managing sensitive data of patients with neurological diseases. An integrated system architecture has been developed that combines the IoT network, the IPFS (InterPlanetary File System) file structure with the Ethereum blockchain to create a reliable data storage model. This system ensures efficient, secure and transparent data processing, optimizing the processes of data registration, authorization and verification. Using IPFS for decentralized file storage, along with the Ethereum blockchain to create tamper-proof medical records, provides increased efficiency, scalability and privacy. During the experiments, the process of creating and testing the system was implemented, including setting up the environment, connecting an IPFS node, programming Ethereum smart contracts, sampling voice data and storing their hashes.

Challenges and Solutions for Sustainable ICT: The Role of File Storage

Digitalization has been increasingly recognized for its role in addressing numerous societal and environmental challenges. However, the rapid surge in data production and the widespread adoption of cloud computing has resulted in an explosion of redundant, obsolete, and trivial (ROT) data within organizations’ data estates. This issue adversely affects energy consumption and carbon footprint, leading to inefficiencies and a higher environmental impact. Thus, this opinion paper aims to discuss the challenges and potential solutions related to the environmental impact of file storage on the cloud, aiming to address the research gap in “digital sustainability” and the Green IT literature. The key findings reveal that technological issues dominate cloud computing and sustainability research. Key challenges in achieving sustainable practices include the widespread lack of awareness about the environmental impacts of digital activities, the complexity of implementing accurate carbon accounting systems compliant with existing regulatory frameworks, and the role of public–private partnerships in developing novel solutions in emerging areas such as 6G technology.

Integration of extended reality and CAE in the context of industry 4.0

ABSTRACT Simulation plays a major role in the development of new products and is considered one of the nine technological pillars of Industry 4.0. By encapsulating a plethora of digital tools, simulation has transitioned towards what is known as Computer Aided Engineering (CAE), which further enables engineers to validate the design and ensure product performance. The development of cutting-edge digital technologies, such as Extended Reality (XR), has also gained attention aiming at the facilitation of human-machine interface (HMI) and the creation of communication channels for improved remote collaboration. The contribution of this study lies in the design and development of a cloud-based framework for the integration of CAE tools with Mixed Reality (MR), specifically, Computational Fluid Dynamics (CFD), focusing on industrial refrigeration systems design and editing applications. Furthermore, the developed solution provides augmented 3D visualizations overlaid on the existing installations. The proposed framework supports the virtual co-existence of engineers during the design phase, thus enabling collaboration and mentoring between experienced and young engineers. Additionally, it can be realized as a Cloud platform, which supports the distribution of several services such as storage of CAD files, simulation results, and the XR application for the setup and visualization of simulation results.

Digital transformation in PAS file management: A case study of PT. AP I Bali

In the current era of globalization, digitization has become a significant advancement, often utilized by companies to streamline administrative processes and reduce reliance on paper-based systems. This research explores the effectiveness of using Google Drive for the monthly accumulation of Passenger Service Charge (PSC) files at PT. AP1 Bali. It aims to assess how digital technology can optimize administrative tasks, particularly the storage and management of PAS files. Additionally, the study outlines the procedural steps involved in the accumulation of these files, offering insights into improving service efficiency. The research is conducted through interviews, observations, and documentation at the airport, using qualitative methods to analyse the collected data. The process involves data reduction, summarization, and drawing conclusions to determine how effectively Google Drive enhances the company's file management system. The expected outcomes will provide PT. AP1 Bali with practical recommendations to optimize its PAS file accumulation process and improve overall service efficiency.

Digitization of logistics and supply chain management

The article presents theoretical and applied aspects of digitalization of logistics and supply chain management. It is emphasized that the processes of digitalization of supply chains and logistics are a modern requirement for increasing business efficiency, as well as solving environmental, social and management problems of successful companies. Digitization of supply chains and logistics is defined as the integration of digital technologies and the implementation of digital tools to optimize, automate and increase the efficiency of all stages of the logistics process. Key modern technologies in logistics and supply chains are examined, including artificial intelligence (AI), the Internet of Things (IoT) and Big Data. The role of AI in optimizing logistics processes, forecasting market behavior, inventory management and route planning is outlined. Emphasis is placed on the importance of IoT for tracking goods, monitoring storage conditions and vehicle maintenance. The role of Big Data in optimizing supply chains, improving decision-making and compliance with regulatory requirements is analyzed. The use of digital documentation and electronic document circulation are highlighted as important areas of digitization. The difference between digitization of documents and digitalization of document circulation is revealed: digitization, or digital preservation, refers to the transfer of information from paper documents to electronic format for convenient and safe storage of files and access to them in the future; digitalization involves the creation of documentation in computer applications, its intelligent processing and analysis, its accumulation in digital repositories, and the wide use of its data to solve various problems. It is noted that one of the main directions of digitization of documents is the automated reading of documents and data entry, which is implemented using specialized software for automatically extracting information from paper documents and entering it into digital systems. The list of the main logistics documents that are subject to digitization and the digitization of which leads to an increase in the efficiency of the supply chain (delivery confirmation, bill of lading, packing lists, invoices, cargo manifest, etc.) is provided. It was concluded that digitalization of logistics is a significant factor for modern companies seeking to increase their competitiveness in the market. It also enables the integration of all company divisions, improves product tracking, speeds up customs procedures, optimizes transport costs and increases the flexibility of return management. Despite the shortcomings that accompany this process, digitalization of logistics is necessary to achieve sustainable development and effective management of supply chains in today's world.

Integrating NoSQL, Hilbert Curve, and R*-Tree to Efficiently Manage Mobile LiDAR Point Cloud Data

The widespread use of Light Detection and Ranging (LiDAR) technology has led to a surge in three-dimensional point cloud data; although, it also poses challenges in terms of data storage and indexing. Efficient storage and management of LiDAR data are prerequisites for data processing and analysis for various LiDAR-based scientific applications. Traditional relational database management systems and centralized file storage struggle to meet the storage, scaling, and specific query requirements of massive point cloud data. However, NoSQL databases, known for their scalability, speed, and cost-effectiveness, provide a viable solution. In this study, a 3D point cloud indexing strategy for mobile LiDAR point cloud data that integrates Hilbert curves, R*-trees, and B+-trees was proposed to support MongoDB-based point cloud storage and querying from the following aspects: (1) partitioning the point cloud using an adaptive space partitioning strategy to improve the I/O efficiency and ensure data locality; (2) encoding partitions using Hilbert curves to construct global indices; (3) constructing local indexes (R*-trees) for each point cloud partition so that MongoDB can natively support indexing of point cloud data; and (4) a MongoDB-oriented storage structure design based on a hierarchical indexing structure. We evaluated the efficacy of chunked point cloud data storage with MongoDB for spatial querying and found that the proposed storage strategy provides higher data encoding, index construction and retrieval speeds, and more scalable storage structures to support efficient point cloud spatial query processing compared to many mainstream point cloud indexing strategies and database systems.

File chunking towards on-chain storage: a blockchain-based data preservation framework

The growing popularity of the most current wave of decentralized systems, powered by blockchain technology, which act as data vaults and preserve data, ensures that, once stored, it stays preserved, considered to be one of the most promising safe and immutable storage methods. The authors of this research suggest an on-chain storage framework that stores files inside blockchain transactions using file transforming, chunking, and encoding techniques. This study investigates the performance of on-chain file storage using a simulated network blockchain environment. Test files of varying sizes were deployed. Performance metrics, including consumed time in chunking, encoding, and distributing chunks among block transactions, were measured and analyzed. An analysis of the collected data was conducted to assess the framework’s performance. The result showed that selecting the appropriate chunk size significantly influences the overall performance of the system. We also explored the implications of our findings and offered suggestions for improving performance within the framework.

Fault diagnosis method for arrester in infrared images based on improved U-Net

Conducting target detection on arresters in practical settings is challenging due to complex backgrounds and occlusion by surrounding objects. This study proposes a method based on improved U-Net for segmenting arresters in infrared images, and achieves fault diagnosis by obtaining surface temperature of the arrester. Firstly, traditional standard convolution in the U-Net backbone is replaced with depthwise separable convolution, reducing computational complexity to about 11% of the original. Secondly, ECA is introduced in the skip connection section to enable cross-channel information exchange, improving segmentation accuracy by 3.21%. Thirdly, to reduce potential errors during the conversion of grayscale images into temperature maps, the data file storage format defined in DL/T 664 is adopted, ensuring the accuracy of the temperature matrix obtained for the arrester. Finally, three fault diagnosis methods based on the surface temperature distribution curve of the arrester were proposed, and the effectiveness of the methods was verified through experiments.

Implementation of Personal Cloud using Cryptography

Abstract: This Python-based personal cloud application leverages Flask and cryptography to establish a secure file storage system. Employing the Fernet encryption algorithm, the server encrypts uploaded files, ensuring data confidentiality. Users can securely upload, store, and download files, with each file automatically encrypted upon upload and decrypted for download. While this example prioritizes simplicity, a production-ready system should include robust user authentication, access controls, and further security enhancements. The abstract encapsulates the project's essence, highlighting its focus on file security within a personal cloud environment while acknowledging the need for additional features in a real-world deployment.

Analysis of Storage Spaces to Support the Health Service System at Santosa Hospital Bandung Central in 2021

One of the health services provided by the hospital is the medical record service. The implementation of good medical file storage through systematic procedures is one of the keys to the success of health service, it is also one of this study was to determine the medical record storage space, the obstacles that occurred, and the effort made in the storage room at the Santosa Hospital Bandung Central. The research method used is descriptive qualitative research, by describing the relevant data. Data collection techniques in the form of interviews and document studies. From the results of the study it can be said that the layout and environment for storing medical records at Santosa Hospital Bandung Central is appropriate, the number of visits that continues to increase is not comparable to the medical record storage space contained in the storage room, the use of medical records is an effort made by Santosa Hospital Bandung Central. The use of electronic medical records overcomes problems in the storage space so as to minimize disruption of health services provided by the Santosa Hospital Bandung Central to patients.

A Closer Look into IPFS: Accessibility, Content, and Performance

The InterPlanetary File System (IPFS) has recently gained considerable attention. While prior research has focused on understanding its performance characterization and application support, it remains unclear: (1) what kind of files/content are stored in IPFS, (2) who are providing these files, (3) are these files always accessible, and (4) what affects the file access performance. To answer these questions, in this paper, we perform measurement and analysis on over 4 million files associated with CIDs (content IDs) that appeared in publicly available IPFS datasets. Our results reveal the following key findings: (1) Mixed file accessibility: while IPFS is not designed for a permanent storage, accessing a non-trivial portion of files, such as those of NFTs and video streams, often requires multiple retrieval attempts, potentially blocking NFT transactions and negatively affecting the user experience. (2) Dominance of NFT (non-fungible token) and video files: about 50% of stored files are NFT-related, followed by a large portion of video files, among which about half are pirated movies and adult content. (3) Centralization of content providers: a small number of peers (top-50), mostly cloud nodes hosted by tech companies, serve a large portion (95%) of files, deviating from IPFS's intended design goal. (4) High variation of downloading throughput and lookup time: large file retrievals experience lower average throughput due to more overhead for resolving file chunk CIDs, and looking up files hosted by non-cloud nodes takes longer. We hope that our findings can offer valuable insights for (1) IPFS application developers to take into consideration these characteristics when building applications on top of IPFS, and (2) IPFS system developers to improve IPFS and similar systems to be developed for Web3.

Blockchain with optimal deep learning assisted secure data sharing and classification on future healthcare systems

In the future of healthcare, Blockchain (BC) technology holds immense potential for improving the security and privacy of data. By allowing the secure and immutable storage of medical files and healthcare-related transactions, BC ensured that sensitive medical data remains tamper-proof and open only to authorized parties. Patients have greater control over their data's development, revoking or granting access as required, but healthcare workers can streamline data sharing and ensure the integrity of important data. The decentralized nature of BC networks decreases the risk of centralized data breaches, eventually fostering trust and transparency in the healthcare ecosystems. Conversely, deep learning (DL) has great to revolutionize healthcare diagnostics in the future, offering quick and extremely accurate estimates of medical conditions. This technology has greatly enhanced patient solutions, decreased medical expenses, and improved the burden on medical staff by providing appreciated insights into an extensive range of conditions, from cancer to neurological disorders. With this stimulus, this study presents a novel BC with optimal DL-based secure data sharing and classification (BCODL-SDSC) technique in the future healthcare system. The goal of the BCODL-SDSC technique is to secure and thoroughly examine healthcare data using BC and DL techniques. Primarily, the BCODL-SDSC technique enables BC technology to store and maintain the patient’s data from the procedure of several transactions and enable access control to the various stakeholders. For the security of the medical images, the BCODL-SDSC technique applies the Fractional Order Lorenz system (FOLS) based encryption technique with tuna swarm optimization (TSO) algorithm based optimal key generation process. Finally, a multi-stage process performs the classification of the medical images: MobileNetv1 feature extractor, artificial rabbit’s optimization (ARO) based hyperparameter tuning, and stacked recurrent neural network (SRNN) based classification. The experimental outcome of the BCODL-SDSC technique was examined on a benchmark medical image database. An extensive comparative study reported that the BCODL-SDSC technique reaches an effective performance with other models with a maximum accuracy of 99.11%.