File System Structure Research Articles

ENCORE: a practical implementation to improve reproducibility and transparency of computational research

Reproducibility of computational research is often challenging despite established guidelines and best practices. Translating these guidelines into practical applications remains difficult. Here, we present ENCORE, an approach to enhance transparency and reproducibility by guiding researchers in how to structure and document a computational project. ENCORE builds on previous efforts in computational reproducibility and integrates all project components into a standardized file system structure. It utilizes pre-defined files as documentation templates, leverages GitHub for software versioning, and includes an HTML-based navigator. ENCORE is designed to be agnostic to the type of computational project, data, programming language, and ICT infrastructure, and does not rely on specific software tools. We also share our group’s experience using ENCORE, highlighting that the most significant challenge to the routine adoption of approaches like ours is the lack of incentives to motivate researchers to dedicate sufficient time and effort to ensure reproducibility.

Mapping Hierarchical File Structures to Semantic Data Models for Efficient Data Integration into Research Data Management Systems

Although other methods exist to store and manage data in modern information technology, the standard solution is file systems. Therefore, keeping well-organized file structures and file system layouts can be key to a sustainable research data management infrastructure. However, file structures alone lack several important capabilities for FAIR data management: the two most significant being insufficient visualization of data and inadequate possibilities for searching and obtaining an overview. Research data management systems (RDMSs) can fill this gap, but many do not support the simultaneous use of the file system and RDMS. This simultaneous use can have many benefits, but keeping data in RDMS in synchrony with the file structure is challenging. Here, we present concepts that allow for keeping file structures and semantic data models (in RDMS) synchronous. Furthermore, we propose a specification in yaml format that allows for a structured and extensible declaration and implementation of a mapping between the file system and data models used in semantic research data management. Implementing these concepts will facilitate the re-use of specifications for multiple use cases. Furthermore, the specification can serve as a machine-readable and, at the same time, human-readable documentation of specific file system structures. We demonstrate our work using the Open Source RDMS LinkAhead (previously named “CaosDB”).

IoT devices with data analytics capabilities Using machine learning algorithms for intelligent decision making: a review of patents and the cited academic literature

Innovation in IoT data analytics is essential for social and economic advancement as well as for opening up new business prospects. IoT data analytics applications have garnered a lot of attention lately for information mining, knowledge extraction, and prediction/inference making. Another field of study that has been effectively used to address a variety of networking issues, including resource allocation, routing, traffic engineering, and security, is machine learning. The use of machine learning (ML)-based techniques to enhance different Internet of Things applications has increased recently. Despite a wealth of research on machine learning and IoT data analytics, there is a dearth of studies that only address the development of ML-based methods for IoT data analysis. The present study examines the patenting practices of various international jurisdictions in the domain of IoT data analytics, in addition to the academically cited literature. Between 2000 and 2023, we gathered 24,931 patent applications, and 13,432 scholarly works were cited by the patents. Our research details the trends in the development of IoT data analytics technologies and the relationship between patenting activity and the referenced scientific literature. Additionally, it examines and displays the social networks found in scientific publications and patents. As a result, it discloses the patenting practices of international jurisdictions concerning IoT data analytics technologies, the degree to which different agents such as Lens Id and Applicants interact within social networks, and the connection between patents and science. According to this study, information retrieval, database structures, and file system structures make up the majority of the IPC categories for filed IoT data analytics patent applications. Chinese businesses and individuals make up the bulk of the patent Lens Id and Applicants. Computer science and medicine are scientific fields that are more strongly connected within the network of co-fields. The relationship between scientific works and inventions is not very strong.

Analysis of real-time operating systems’ file systems: Built-in cameras from vehicles

The vehicles we use daily record enormous information via the embedded system that is installed. This recorded information identifies traffic accidents and might reveal the truth of crimes committed using the vehicle. However, as the embedded systems mounted on vehicles must maintain real-time performance, they often have a real-time operating system (RTOS)-dedicated file system structure. Analysis of such systems is becoming a new challenge for digital forensics. This study analyzed the real-time operating system-dedicated file system of a vehicle's built-in camera. When the built-in camera of the vehicle operates normally, there are various ways to acquire video data. However, when the built-in cam circuit is damaged, the only available method is to extract the onboard memory. To analyze the dedicated file system, we analyzed the driver file in the system area using the reverse engineering technique. We could analyze various log files and user-setting files in multiple partitions stored in the analyzed memory. In addition, we proved that more video frames can be restored by extracting the unallocated area of the video storage partition. In the future, this method can be applied to analyze various RTOS and dedicated file systems installed in the vehicle.

Forensic Analysis of IoT File Systems for Linux-Compatible Platforms

Due to recent developments in IT technology, various IoT devices have been developed for use in various environments, such as card smart TVs, and smartphones Communication between IoT devices has become possible. Various IoT devices are found in homes and in daily life, and IoT technologies are being combined with vehicles, power, and wearables, amongst others. Although the usage of IoT devices has increased, the level of security technology applied to IoT devices is still insufficient. There is sensitive information stored inside IoT devices, such as personal information and usage history, so if security accidents happen, such as data leakage, it can be very damaging for users. Since research on data storage and acquisition in IoT devices is very important, in this paper we conducted a security analysis, from a forensic perspective, on IoT platform file systems used in various environments. The analysis was conducted on two mechanical platforms: Tizen (VDFS) and Linux (JFFS2 and UBIFS). Through file system metadata analysis, file system type, size, list of files and folders, deleted file information were obtained so that we could analyze file system structure with the obtained information. We also used the obtained information to check the recoverability of deleted data to investigate the recovery plan. In this study, we explain the characteristics of platforms used in various environments, and the characteristics of data stored in each platform. By analyzing the security issues of data stored during platform communications, we aimed to help in solving the problems affecting devices. In addition, we explain the analysis method for file system forensics so that it can be referred to in other platform forensics.

Forensic Recovery of File System Metadata for Digital Forensic Investigation

File system forensics is one of the most important elements in digital forensic investigations. To date, various file system forensic methods, such as analysis of tree structure and the recovery of deleted file data, have been studied. Among these file system forensic methods, the recovery of file system metadata is a key technique that makes digital forensic investigations possible by recovering metadata when it is not possible to obtain metadata in a regular manner because the file system structure is damaged due to an accident/disaster or cyber terrorism. Previous studies mainly focused on recovering record or entry data, which are the basic units of metadata, using carving techniques via a fixed values or values capable of range prediction at the beginning of the data. However, no studies have been conducted on metadata without such fixed values or values capable of range prediction. $LogFile, which is a metadata file of the New Technology File System (NTFS) that is one of the most used file systems at present, contains very important metadata that provide a history of all file system operations during a specific period. However, since there is no fixed value or a value capable of range prediction at the start position of the record, which is the basic unit of $LogFile, there have been no studies on recovery using record units, and only recovery by file and page have been possible. If the file header or page header of $LogFile is damaged, existing recovery methods cannot properly recover the metadata; in such cases, a record-level recovery method is required to recover the metadata. In this context, we investigated the mechanisms of record storage through a detailed analysis of the $LogFile structure and proposed a recovery method for records without fixed values. Our proposed method was implemented as a tool and verified through comparative experiments with existing forensic tools that recover $LogFile data. The experimental results showed that the proposed recovery method was able recover all the data that existing tools are unable to recover in situations where the $LogFile data were damaged. The implemented tools are released free of charge to contribute digital forensic community. Finally, we explained what important role $LogFile played in solving real-world cases and confirm the importance of recovering $LogFile data in situations where file systems may be damaged due to accidents and disasters.

Investigation of the computational complexity of methods for hiding information in cluster steganosystems

Several methods of technical steganography are currently known. Hiding information in a model in 3D printing, this industry of hiding information has certain advantages and disadvantages, namely: the relatively high cost of creating a hidden message and the difficulty in reading the information. The second area of technical steganography is related to network traffic. In this method, information can be hidden, for example, in the header fields of protocols, or, for example, the transmission of a hidden message by sending packets in a certain sequence. There are also methods of hiding information in the structure of the file system, but the known methods are either capable of hiding a small amount of information, or have an insufficient level of resistance to detection. Thus, an urgent task is to develop a method for hiding information, which is able to hide more information and has a higher level of resistance to detection, with a satisfactory level of computational complexity.
 This paper presents methods of technical steganography based on the structural features of file systems in storage media, in particular, hiding information in the FAT file system by mixing clusters of certain key files (cover files). Methods of hiding information in the structure of a clustered file system by mixing clusters of cover files require significant computational resources. In this paper, research has been carried out on methods to increase computational efficiency in terms of the amount of required RAM, and the number of the required number of basic operations to hide a message.

Comparative analysis and study of the properties of information carriers for steganographic data hiding in clustered file sys-tems

The paper studies and analyzes various modern information storage technologies, namely HDD, Flash-USB, SSD. We`ve analyzed different indicators such as the number of implemented products, price, speed of reading and writing. Besides, we`ve considered some indicators of the information carriers’ efficiency in terms of view of the possibility of using steganographic methods for hiding information in clustered file systems. It have been analyzed the speed of sequential reading / writing and the speed of access to a random cluster, corresponding to the speed of access to a fragmented file. For this task, we used the test results from the UserBenchmark resource. The testing has made using the Sequential and Random4k methods. In addition, we have provided an assessment of information carriers and have gave recommendations of using the particular information carrier and method for hiding data by mixing clusters in the structure of the file system. Besides, it was analyzed the dependence of the speed parameters of access to the cluster on the level of file fragmentation. Refinements are made of how an increase or decrease in the level of fragmentation (entanglement) affects the speed of access to the file, which is an important indicator when using the method of hiding data in the file system structure. The advantages and disadvantages of various types of storage devices have been discussed, and its comparative analysis was made. Moreover, we analyzed the features of the process of defragmentation of drives, and the influence of various factors on the overall level of fragmentation on the storage medium. We placed emphasis on the greater the level of fragmentation on the storage medium, the more information could be hidden. It was concluded that due to the widespread use of SSD / HDD drives, the method of hiding information in the structure of file systems, by mixing clusters of covering files, is relevant.

CernVM-FS Container Image Integration

Linux containers have gained widespread use in high energy physics, be it for services using container engines such as containerd/kubernetes, for production jobs using container engines such as Singularity or Shifter, or for development workflows using Docker as a local container engine. Thus the efficient distribution of the container images, whose size usually ranges from a few hundred megabytes to a few tens of gigabytes, is becoming a pressing concern. Because container images show similar characteristics than scientific application stacks, unpacking the images in CernVM-FS can remedy the distribution issues provided that the container engine at hand is able to use such unpacked images from CernVM-FS. In this contribution, we willl report on recent advances in the integration of Singularity, Docker, and containerd with CernVM-FS. We show improvements in the publishing of container images from a Docker registry that rely on new means of directly ingesting image tarballs. Well also show a repository file system structure for storing container images that are optimized for storing both container engines using flat root file systems (Singularity) as well as container engines using layers (containerd, Docker). To evaluate the benefits of our approach, we show concrete use cases and figures for production and development images from LHC experiments stored in the recently created unpacked.cern.ch repository.

Understanding the development trends of big data technologies: an analysis of patents and the cited scholarly works

Big data innovation is a key instrument for economic and social development and for the creation of new business opportunities. This study analyzes the patenting activities of global jurisdictions in the field of big data; as well as the scholarly cited works. We collected 13,112 patent applications between 1992 and 2019, and 642 cited scholarly works by the patents. Our findings report on the development trends in big data technologies, as well as on the link between patenting activities and the cited scientific works. It also analyzes and visualizes the social networks embedded in patents and the cited scientific works; thus, it reveals the patenting activities of global jurisdictions on big data technologies, the strength of the interaction of various agents, such as inventors and applicants within social networks, and the link between patents and the scientific world. This study shows that most of the big data patent applications filed fall within the IPC category of information retrieval, database structures, and file system structures. The majority of the applicants and inventors of the patents filed are Chinese companies and individuals. Scientific fields with stronger connections within the network of co-fields are computer science and medicine. There is a weak link between inventions and scientific works.

Reverse engineering of ReFS

File system forensics is an important part of Digital Forensics. Investigators of storage media have traditionally focused on the most commonly used file systems such as NTFS, FAT, ExFAT, Ext2-4, HFS+, APFS, etc. NTFS is the current file system used by Windows for the system volume, but this may change in the future. In this paper we will show the structure of the Resilient File System (ReFS), which has been available since Windows Server 2012 and Windows 8. The main purpose of ReFS is to be used on storage spaces in server systems, but it can also be used in Windows 8 or newer. Although ReFS is not the current standard file system in Windows, while users have the option to create ReFS file systems, digital forensic investigators need to investigate the file systems identified on a seized media. Further, we will focus on remnants of non-allocated metadata structures or attributes. This may allow metadata carving, which means searching for specific attributes that are not allocated. Attributes found can then be used for file recovery. ReFS uses superblocks and checkpoints in addition to a VBR, which is different from other Windows file systems. If the partition is reformatted with another file system, the backup superblocks can be used for partition recovery. Further, it is possible to search for checkpoints in order to recover both metadata and content.Another concept not seen for Windows file systems, is the sharing of blocks. When a file is copied, both the original and the new file will share the same content blocks. If the user changes the copy, new data runs will be created for the modified content, but unchanged blocks remain shared. This may impact file carving, because part of the blocks previously used by a deleted file might still be in use by another file. The large default cluster size, 64 KiB, in ReFS v1.2 is an advantage when carving for deleted files, since most deleted files are less than 64 KiB and therefore only use a single cluster. For ReFS v3.2 this advantage has decreased because the standard cluster size is 4 KiB.Preliminary support for ReFS v1.2 has been available in EnCase 7 and 8, but the implementation has not been documented or peer-reviewed. The same is true for Paragon Software, which recently added ReFS support to their forensic product. Our work documents how ReFS v1.2 and ReFS v3.2 are structured at an abstraction level that allows digital forensic investigation of this new file system. At the time of writing this paper, Paragon Software is the only digital forensic tool that supports ReFS v3.x.It is the most recent version of the ReFS file system that is most relevant for digital forensics, as Windows automatically updates the file system to the latest version on mount. This is why we have included information about ReFS v3.2. However, it is possible to change a registry value to avoid updating. The latest ReFS version observed is 3.4, but the information presented about 3.2 is still valid. In any criminal case, the investigator needs to investigate the file system version found.

Developing a File System Structure to Solve Healthy Big Data Storage and Archiving Problems Using a Distributed File System

Recently, the use of internet has become widespread, increasing the use of mobile phones, tablets, computers, Internet of Things (IoT) devices and other digital sources. In the health sector with the help of new generation digital medical equipment, this digital world also has tended to grow in an unpredictable way in that it has nearly 10% of the global wide data itself and continues to keep grow beyond what the other sectors have. This progress has greatly enlarged the amount of produced data which cannot be resolved with conventional methods. In this work, an efficient model for the storage of medical images using a distributed file system structure has been developed. With this work, a robust, available, scalable, and serverless solution structure has been produced, especially for storing large amounts of data in the medical field. Furthermore, the security level of the system is extreme by use of static Internet protocol (IP), user credentials, and synchronously encrypted file contents. One of the most important key features of the system is high performance and easy scalability. In this way, the system can work with fewer hardware elements and be more robust than others that use name node architecture. According to the test results, it is seen that the performance of the designed system is better than 97% from a Not Only Structured Query Language (NoSQL) system, 80% from a relational database management system (RDBMS), and 74% from an operating system (OS).

Data hiding in the FAT family file system structure

Methods of technical steganography, namely, those ones based on information messages hiding in the structure of the FAT family file system are studied. The basic method based on changing the numbering of cover files clusters is considered, its advantages and disadvantages are researched. An improved method is proposed that makes it possible to increase significantly the hidden data size. The results of the comparative analysis of hiding data in the structure of the FAT family file system are given.

The Scientific Filesystem.

BackgroundHere, we present the Scientific Filesystem (SCIF), an organizational format that supports exposure of executables and metadata for discoverability of scientific applications. The format includes a known filesystem structure, a definition for a set of environment variables describing it, and functions for generation of the variables and interaction with the libraries, metadata, and executables located within. SCIF makes it easy to expose metadata, multiple environments, installation steps, files, and entry points to render scientific applications consistent, modular, and discoverable. A SCIF can be installed on a traditional host or in a container technology such as Docker or Singularity. We start by reviewing the background and rationale for the SCIF, followed by an overview of the specification and the different levels of internal modules (“apps”) that the organizational format affords. Finally, we demonstrate that SCIF is useful by implementing and discussing several use cases that improve user interaction and understanding of scientific applications. SCIF is released along with a client and integration in the Singularity 2.4 software to quickly install and interact with SCIF. When used inside of a reproducible container, a SCIF is a recipe for reproducibility and introspection of the functions and users that it serves.ResultsWe use SCIF to evaluate container software, provide metrics, serve scientific workflows, and execute a primary function under different contexts. To encourage collaboration and sharing of applications, we developed tools along with an open source, version-controlled, tested, and programmatically accessible web infrastructure. SCIF and associated resources are available at https://sci-f.github.io. The ease of using SCIF, especially in the context of containers, offers promise for scientists’ work to be self-documenting and programatically parseable for maximum reproducibility. SCIF opens up an abstraction from underlying programming languages and packaging logic to work with scientific applications, opening up new opportunities for scientific software development.

Extending CMMN with entity life cycles

In the domain of business process modeling, a case denotes a situation that requires a customized treatment, and this may take place if the case workers are entitled to decide the tasks to perform as well as their ordering. In the recent CMMN (Case Management Model and Notation) standard, a case involves both an information structure and a process. The former is patterned on the file system structure and the latter is made up of stages, which are groupings of interrelated tasks. The standard leaves some open issues, such as the determination of the performers of the tasks, and the definition of the inputs and outputs of the tasks in terms of the information items affected. To address these issues, this paper proposes an extension to CMMN in which stages represent states of information items whose types along with their attributes and relationships are defined in an information model. The benefits of the extension are illustrated by means of an example that concerns the handling of papers submitted to conferences.

Extension of research data repository system to support direct compute access to biomedical datasets: enhancing Dataverse to support large datasets.

Access to experimental X-ray diffraction image data is important for validation and reproduction of macromolecular models and indispensable for the development of structural biology processing methods. In response to the evolving needs of the structural biology community, we recently established a diffraction data publication system, the Structural Biology Data Grid (SBDG, data.sbgrid.org), to preserve primary experimental datasets supporting scientific publications. All datasets published through the SBDG are freely available to the research community under a public domain dedication license, with metadata compliant with the DataCite Schema (schema.datacite.org). A proof-of-concept study demonstrated community interest and utility. Publication of large datasets is a challenge shared by several fields, and the SBDG has begun collaborating with the Institute for Quantitative Social Science at Harvard University to extend the Dataverse (dataverse.org) open-source data repository system to structural biology datasets. Several extensions are necessary to support the size and metadata requirements for structural biology datasets. In this paper, we describe one such extension-functionality supporting preservation of file system structure within Dataverse-which is essential for both in-place computation and supporting non-HTTP data transfers.

A New Design of In-Memory File System Based on File Virtual Address Framework

The emerging technologies of persistent memory, such as PCM, MRAM, provide opportunities for preserving files in memory. Traditional file system structures may need to be re-studied. Even though there are several file systems proposed for memory, most of them have limited performance without fully utilizing the hardware at the processor side. This paper presents a framework based on a new concept, “File Virtual Address Space”. A file system, Sustainable In-Memory File System (SIMFS), is designed and implemented, which fully utilizes the memory mapping hardware at the file access path. First, SIMFS embeds the address space of an open file into the process’ address space. Then, file accesses are handled by the memory mapping hardware. Several optimization approaches are also presented for the proposed SIMFS. Extensive experiments are conducted. The experimental results show that the throughput of SIMFS achieves significant performance improvement over the state-of-the-art in-memory file systems.

A unified framework for designing high performance in-memory and hybrid memory file systems

The emerging non-volatile memory technologies provide a new choice for storing persistent data in memory. Therefore, file system structure needs re-studying and re-designing. Our goal is to design a framework that gives high-performance in-memory file accesses and allows a file whose data can be stored across memory and block device. This paper presents a novel unified framework for in-memory and hybrid memory file systems based on the concept that each file has a contiguous Virtual Address Space. Within this framework, the file access for in-memory data can be efficiently handled by address translation hardware and the virtual address space of file. The file accesses for data in block devices are handled by a dedicated page fault handler for file system. A file system called Hybrid Memory File System (HMFS) is implemented based on this framework. Experimental results show that the throughput of HMFS approaches the memory bus bandwidth in best cases. Compared with in-memory file systems, HMFS reaches 5 times, 2.1 times, and 1.6 times faster than EXT4 on Ramdisk, RAMFS, and PMFS, respectively. Compared with EXT4 on SSD and EXT4 using page cache, HMFS also achieves 100 times and tens of times performance improvement, respectively.

Enabling Object Storage via shims for Grid Middleware

The Object Store model has quickly become the basis of most commercially successful mass storage infrastructure, backing so-called ”Cloud” storage such as Amazon S3, but also underlying the implementation of most parallel distributed storage systems. Many of the assumptions in Object Store design are similar, but not identical, to concepts in the design of Grid Storage Elements, although the requirement for ”POSIX-like” filesystem structures on top of SEs makes the disjunction seem larger. As modern Object Stores provide many features that most Grid SEs do not (block level striping, parallel access, automatic file repair, etc.), it is of interest to see how easily we can provide interfaces to typical Object Stores via plugins and shims for Grid tools, and how well experiments can adapt their data models to them. We present evaluation of, and first-deployment experiences with, (for example) Xrootd-Ceph interfaces for direct object-store access, as part of an initiative within GridPP[1] hosted at RAL. Additionally, we discuss the tradeoffs and experience of developing plugins for the currently-popular Ceph parallel distributed filesystem for the GFAL2 access layer, at Glasgow.

Design of Interlnk Protocol Packet at Embedded Linux

Aiming at the file transmission problem between electronic controller for an engine and the portable test equipment based on AT91SAM9263 and Linux, the Interlnk/Intersvr software and their communicating protocol are analyzed and studied which used by electronic controller. The message format and communicating process are made clear by putting up protocol analysis environment at laboratory and source code analysis, FAT file system structure and its access methods are studied, and the file transmission function of test equipment has been fulfilled. The results of test indicate that the desired goals have been reached.