Ext4 File System Research Articles

Apptainer Without Setuid

Apptainer (formerly known as Singularity) since its beginning implemented many of its container features with the assistance of a setuidroot program. It still supports that mode, but as of version 1.1.0 it no longer uses setuid by default. This is feasible because it now can mount squashfs filesystems, ext3 filesystems, and overlay filesystems using unprivileged user namespaces and FUSE. It also now enables unprivileged users to build containers, even without requiring system administrators to configure /etc/subuid and /etc/subgid unlike other “rootless” container systems. As a result, all the unprivileged functions can be used nested inside of another container, even if the container runtime prevents any elevated privileges. As of version 1.2.0 Apptainer also supports completely unprivileged encryption of Singularity Image Format (SIF) container files. Performance with a particularly challenging HEP benchmark using the FUSE-based mounts both with and without encryption is essentially identical to the previous methods that required elevated privileges to use the Linux kernel-based counterparts.

Separating the File System Journal to Reduce Write Amplification of Garbage Collection on ZNS SSDs

Solid-State Drives (SSDs), despite their outstanding I/O performance, suffer from the garbage collection overhead incurred by out-place update scheme that addresses the erase-before-write constraint of NAND flash memory. The Zoned Namespace SSD (ZNS SSD) is a state-of-the-art storage technology that divides NAND flash memory space within an SSD into zones and allows the host to directly assign certain zone for a write request. Therefore, the ZNS SSD can minimize the garbage collection overhead by allocating data with similar update patterns to the same zone. However, legacy journaling file systems still cannot actively exploit the advantages of ZNS SSDs, and journal data and file data with obviously different update patterns are written together in the same zone. In this paper, we propose new file system journal placement scheme that separates journal data into a dedicated zone to minimize write amplification due to garbage collection, noting the characteristics of journal data in the Ext4 file system. The proposed scheme is implemented in Linux kernel and carefully evaluated. The evaluation results show that our scheme removes the unnecessary copying of journal data during garbage collection of ZNS SSD and reduces the valid page copies caused by garbage collection by up to 26.8%.

Comparison of EXT4 and NTFS filesystem performance

The aim of this article is to compare the two most popular and competing file systems – EXT4 and NTFS in Ubutnu with the use of SSD disk.. At the beginning of this article, a critical description of the literature was made, explaining the purposefulness of the research undertaken. Additionally, the basics of the operation of both discussed file systems are explained. The research consisted of copying files between two partitions and measuring the time of this operation using a specially developed system shell script in the bash language. The conducted research has shown that the EXT4 system is more effective than the NTFS system.

Golden Eye

File systems are important sources of intelligence information and digital evidence. They have long attracted the interest of researchers in recovering files that are deleted from a hard disk. Existing file recovery studies rely heavily on an operating system (OS). However, it is often encountered that OS services are not available, making existing file recovery approaches unusable. To address this issue, the authors design and implement an OS-independent file recovery algorithm named Golden Eye (GE) by targeting the EXT4 file system. Fed the raw image obtained from a (sanitized) hard disk, GE can automatically recover any designated file or even the whole EXT4 file system. GE is based on the understanding of the file disk layout of EXT4 and does not need any support from additional hardware or software. Experimental results prove the feasibility and correctness of GE. This work not only solves the OS dependency problem that most existing file recovery work suffers from but also reveals the fact that even sanitized hard disks are still at risk of leaking sensitive data.

FUSE based file system for efficient storage and retrieval of fragmented multimedia files

Multimedia content providers (such as cyberlockers) are overwhelmed by huge volumes and large sizes of multimedia files. Cyberlockers provide Quality-of-Service (QoS) downloading to end users as the files are split into fragments. However, merging these fragments on the fly for streaming harms Quality-of-Experience (QoE) of users and overwhelms computational capacity. As a solution, this paper proposes a FUSE based file system, namely fumy, for efficient storage of large multimedia files as small sized fragments (for downloading) and virtually unifying them for their efficient retrieval (for streaming). We discuss virtual unification, namespace manipulation and operation redirection employed by fumy to achieve its goal. We also discuss the implementation details of fumy as a FUSE file system. We evaluated fumy for its performance on extended file systems in various operational and benchmarking configurations. Our results suggest that fumy adds minimal performance overhead to extended file systems when the fragment size is 512 MiB or higher, imposes least performance tax on ext4 file system, and even enhances the performance of extended file systems for specific workloads and fragments sizes. We recommend using ext4 file system with fragment size of 512 MiB for different workloads to achieve optimal performance of fumy.

Embedded Transaction Support Inside SSD With Small-Capacity Non-Volatile Disk Cache

Flash-based Solid State Drives (SSDs) have proved to be ideal devices that support embedded transaction protocols inside SSDs. Existing embedded transaction protocols effectively improve transaction throughput, but still incur high transaction overhead and long recovery time. In this paper, we propose a new embedded transaction protocol called Non-volatile Cache Transaction (NVCTX). NVCTX reduces transaction overhead and provides fast recovery by leveraging the small-capacity NVM-based disk cache from two aspects. First, we store transactional metadata, which is of small amount but is frequently accessed, in the NVM-based disk cache rather than in the flash memory. Second, we introduce two techniques to improve the performance when the capacity of the NVM-based disk cache is very limited. We have implemented NVCTX on a real hardware board, and modified ext4 file system and NVMe driver to be compatible with the transactional interfaces provided by NVCTX. For comparison, we implement other four protocols in the firmware. Evaluations using DBMS and file system workloads show that, compared to other protocols, NVCTX improves transaction throughput by up to 136.5%, 9.4%, 131.6% and 29.9%, reduces write traffic to flash memory by up to 42.8%, 4.1%, 62.4%, 31.2%, and shortens recovery time to 1/2574, 1/2559, 1/95 and 1/2 respectively.

Ext4 and XFS File System Forensic Framework Based on TSK

Recently, the number of Internet of Things (IoT) devices, such as artificial intelligence (AI) speakers and smartwatches, using a Linux-based file system has increased. Moreover, these devices are connected to the Internet and generate vast amounts of data. To efficiently manage these generated data and improve the processing speed, the function is improved by updating the file system version or using new file systems, such as an Extended File System (XFS), B-tree file system (Btrfs), or Flash-Friendly File System (F2FS). However, in the process of updating the existing file system, the metadata structure may be changed or the analysis of the newly released file system may be insufficient, making it impossible for existing commercial tools to extract and restore deleted files. In an actual forensic investigation, when deleted files become unrecoverable, important clues may be missed, making it difficult to identify the culprit. Accordingly, a framework for extracting and recovering files based on The Sleuth Kit (TSK) is proposed by deriving the metadata changed in Ext4 file system journal checksum v3 and XFS file system v5. Thereafter, by comparing the accuracy and recovery rate of the proposed framework with existing commercial tools using the experimental dataset, we conclude that sustained research on file systems should be conducted from the perspective of forensics.

IoT 기기 플래시 메모리의 Ext4 파일 시스템을 위한 디지털 포렌식 절차

최근 급격한 통신기술의 발달에 따라 IoT(Internet of Things)의 확산이 가속화되고 있으며 IoT 기기는 인간의 삶과 밀접한 분야에서 활용되어 범죄 및 사고 경황 정보 조사에 적극적으로 활용되고 있다. 이에 따라 디지털 포렌식의 중요성이 부각되며 그 수행 절차에 관한 연구들이 진행되고 있다. 하지만 IoT 기기는 상이한 규격으로 인해 실제 현장에서 사용될 수 없는 추상적인 방법론을 제시하는 연구가 대부분이다. 또한 펌웨어 분석 도구로 활발히 사용되는 binwalk 분석 도구는 ext4 파일 시스템에 대한 분석 및 추출을 제대로 수행하지 못하고 있다. 이에 본 논문에서는 이러한 문제점을 해결하기 위해 binwalk의 올바른 ext4 파일 시스템 추출 및 분석 방안을 제시하고 이를 이용해 IoT 기기의 플래시 메모리로부터 ext4 파일 시스템을 추출하는 절차를 제안한다. 또한, DJI Phantom 4 Pro V2.0 드론으로 PoC를 구현하여 제안하는 절차를 검증한다.

An Empirical Evaluation of NVM-Aware File Systems on Intel Optane DC Persistent Memory Modules

The emergence of non-volatile memories (NVM) brings new opportunities and challenges to data management system design. As an important part of the data management systems, several new file systems are developed to take advantage of the characteristics of NVM. However, these NVM-aware file systems are usually designed and evaluated based on simulations or emulations. In order to explore the performance and characteristics of these file systems on real hardware, in this article, we provide an empirical evaluation of NVM-aware file systems on the first commercially available byte-addressable NVM (i.e., the Intel Optane DC Persistent Memory Module (DCPMM)). First, to compare the performance difference between traditional file systems and NVM-aware file systems, we evaluate the performance of Ext4, XFS, F2FS, Ext4-DAX, XFS-DAX, and NOVA file systems on DCPMMs. To compare DCPMMs with other secondary storage devices, we also conduct the same evaluations on Optane SSDs and NAND-flash SSDs. Second, we observe how remote NUMA node access and device mapper striping affect the performance of DCPMMs. Finally, we evaluate the performance of the database (i.e., MySQL) on DCPMMs with Ext4 and Ext4-DAX file systems. We summarize several observations from the evaluation results and performance analysis. We anticipate that these observations will provide implications for various memory and storage systems.

Cyber Forensics on Internet of Things: Slicing and Dicing Raspberry Pi

Any device can now connect to the Internet, and Raspberry Pi is one of the more popular applications, enabling single-board computers to make robotics, devices, and appliances part of the Internet of Things (IoT). The low cost and customizability of Raspberry Pi makes it easily adopted and widespread. Unfortunately, the unprotected Raspberry Pi device—when connected to the Internet—also paves the way for cyber-attacks. Our ability to investigate, collect, and validate digital forensic evidence with confidence using Raspberry Pi has become important. This article discusses and presents techniques and methodologies for the investigation of timestamp variations between different Raspberry Pi ext4 filesystems (Raspbian vs. UbuntuMATE), comparing forensic evidence with that of other ext4 filesystems (i.e., Ubuntu), based on interactions within a private cloud, as well as a public cloud. Sixteen observational principles of file operations were documented to assist in our understanding of Raspberry Pi’s behavior in the cloud environments. This study contributes to IoT forensics for law enforcement in cybercrime investigations.

PerSeVerE: persistency semantics for verification under ext4

Although ubiquitous, modern filesystems have rather complex behaviours that are hardly understood by programmers and lead to severe software bugs such as data corruption. As a first step to ensure correctness of software performing file I/O, we formalize the semantics of the Linux ext4 filesystem, which we integrate with the weak memory consistency semantics of C/C++. We further develop an effective model checking approach for verifying programs that use the filesystem. In doing so, we discover and report bugs in commonly-used text editors such as vim, emacs and nano.

Spiffy

Many file-system applications such as defragmentation tools, file-system checkers, or data recovery tools, operate at the storage layer. Today, developers of these file-system aware storage applications require detailed knowledge of the file-system format, which requires significant time to learn, often by trial and error, due to insufficient documentation or specification of the format. Furthermore, these applications perform ad-hoc processing of the file-system metadata, leading to bugs and vulnerabilities. We propose Spiffy, an annotation language for specifying the on-disk format of a file system. File-system developers annotate the data structures of a file system, and we use these annotations to generate a library that allows identifying, parsing, and traversing file-system metadata, providing support for both offline and online storage applications. This approach simplifies the development of storage applications that work across different file systems because it reduces the amount of file-system--specific code that needs to be written. We have written annotations for the Linux Ext4, Btrfs, and F2FS file systems, and developed several applications for these file systems, including a type-specific metadata corruptor, a file-system converter, an online storage layer cache that preferentially caches files for certain users, and a runtime file-system checker. Our experiments show that applications built with the Spiffy library for accessing file-system metadata can achieve good performance and are robust against file-system corruption errors.

An Energy-Efficient and Fast Scheme for Hybrid Storage Class Memory in an AIoT Terminal System

Conventional main memory can no longer meet the requirements of low energy consumption and massive data storage in an artificial intelligence Internet of Things (AIoT) system. Moreover, the efficiency is decreased due to the swapping of data between the main memory and storage. This paper presents a hybrid storage class memory system to reduce the energy consumption and optimize IO performance. Phase change memory (PCM) brings the advantages of low static power and a large capacity to a hybrid memory system. In order to avoid the impact of poor write performance in PCM, a migration scheme implemented in the memory controller is proposed. By counting the write times and row buffer miss times in PCM simultaneously, the write-intensive data can be selected and migrated from PCM to dynamic random-access memory (DRAM) efficiently, which improves the performance of hybrid storage class memory. In addition, a fast mode with a tmpfs-based, in-memory file system is applied to hybrid storage class memory to reduce the number of data movements between memory and external storage. Experimental results show that the proposed system can reduce energy consumption by 46.2% on average compared with the traditional DRAM-only system. The fast mode increases the IO performance of the system by more than 30 times compared with the common ext3 file system.

Making Application-Level Crash Consistency Practical on Flash Storage

We present the design, implementation, and evaluation of a new file system, called ACCFS , supporting application-level crash consistency as its first-class citizen functionality. With ACCFS , application data can be correctly recovered in the event of system crashes without any complex update protocol at the application level. With the help of the SHARE interface supporting atomic address remapping at the flash storage layer, ACCFS can easily and efficiently achieve crash consistency as well as single-write journaling. We prototyped ACCFS by slightly modifying the full data journal mode in ext4, implemented the SHARE interface as firmware in a commercial SSD available in the market, and carried out various experiments by running ACCFS on top of the SSD. Our preliminary experimental results are very promising. For instance, the performance of an OLTP benchmark using MySQL/InnoDB engine can be boosted by more than 2–6x by offloading the responsibility of guaranteeing the atomic write of MySQL data pages from the InnoDB engine's own journaling mechanism to ACCFS . This impressive performance gain is in part due to the single-write journaling in ACCFS and in part comes from the fact that the frequent fsync() calls caused by the complex update protocol at the application level can be avoided. ACCFS is a practical solution for the crash consistency problem in that (1) the SHARE interface can be, like the TRIM command, easily supported by commercial SSDs, (2) it can be embodied with a minor modification on the existing ext4 file system, and (3) the existing applications can be made crash consistent simply by opening files in O_ATOMIC mode while the legacy applications can be run without any change.

UniBuffer: Optimizing Journaling Overhead With Unified DRAM and NVM Hybrid Buffer Cache

Journaling techniques play an important role in addressing the reliability issue of filesystems caused by the volatile dynamic random access memory (DRAM)-based buffer cache. However, journaling techniques introduce a large number of extra storage writes, which greatly degrades system performance, especially for mobile devices. Emerging nonvolatile memory (NVM) technologies bring a new perspective of solving the write amplification issue caused by journaling. By adopting NVM as the buffer cache, the committed data can be maintained in NVM before being written back to the storage, thus, eliminating the journaling overhead. However, simply utilizing NVM as the buffer cache suffers from the limited lifetime and the relatively longer write latency of NVM. In this paper, we present a hybrid buffer cache architecture, named UniBuffer, by combing NVM with dynamic random access memory (DRAM) to reduce the journaling overhead and overcome the inherent constraints of NVM. In UniBuffer, we first propose a journaling-aware page management (JAPM) policy to smartly allocate data to DRAM and NVM pages. JAPM puts infrequently updated data in NVM to reduce the journaling overhead and frequently updated data in DRAM to improve the write performance and lifetime of the hybrid buffer cache. In addition, since data in one journaling transaction may be dispersed in NVM and DRAM simultaneously, different committing policies are required for different storage media, respectively. In order to guarantee the atomicity of the transaction execution in the hybrid cache architecture, a partial in-place commit (PIPC) journaling scheme is proposed to coordinate different committing patterns for NVM and DRAM. We have implemented the proposed techniques on Linux 3.14.52 and measured the performance with representative I/O-intensive benchmarks. The experimental results show that our scheme effectively improves the I/O performance compared with the ext4 filesystem and prolongs the lifetime of the hybrid buffer cache compared with the union of buffer cache and journaling (UBJ) scheme.

ExtSFR: scalable file recovery framework based on an Ext file system

As the technologies based on the Internet of Things, the Cloud, Big Data, and mobile technology have recently become the engine of the next-generation fusion environment, the use of consumer electronics with Linux/Unix-based operating systems which include mobile and embedded operating systems has been gradually increasing. As these technologies are applied in the real world, digital forensics and post-processing techniques in the next-generation environment are required for security/privacy perspective. In this paper, an Ext2/3/4 file system’s file recovery framework which is suitable for the next-generation environment is proposed. Also, Ext4 used from small to large size file systems in recent consumer electronics such as home appliances, mobile devices, home-office devices, entertainment devices, etc. The proposed framework takes the various Ext4 file systems created in the IoT/Cloud/Big Data/mobile environment into account, and it is configured to accommodate not only Ext4 used mainly in the recent environment but also Ext2/3 legacy environment. Additionally, the proposed framework is implemented as a prototype and validated it by comparing it with the existing commercial technologies, showing that the accuracy and efficiency of the prototype of the proposed framework for large file system recovery rates are superior to those of the existing technologies.

WOJ: Enabling Write-Once Full-data Journaling in SSDs by using weak-hashing-based deduplication

Journaling is a commonly used technique in file systems to provide data reliability for applications. Full-data journaling, which stores all file system (data and metadata) updates in a journal before they are applied to their home locations, provides the strongest data reliability, reduces application developers’ efforts on application-level crash consistency, and helps to remove most crash-consistency vulnerabilities. However, file system users usually hesitate to use it as it doubles the write volume to the disk, leading to compromised performance. While fast SSDs have the potential to make full-data journaling affordable, its doubled writes threaten the devices’ durability, which is their Achilles heel.While data deduplication technique can be used to remove the second writes to the home locations, it can be too expensive to be a practical solution due to its high computation and space overheads as it has to compute and cache collision-resistant hashing values (fingerprints). The issue is especially serious for SSDs, which are becoming increasingly large and fast, but less tolerant of additional overhead in the I/O stack. Leveraging the fact that with data journal mode all writes to the home locations in a file system are preceded by corresponding writes to the journal, we propose Write-Once data Journaling (WOJ), which uses a weak-hashing-based deduplication dedicated for removing the second writes in data journaling. WOJ can reduce regular deduplication’s computation and space overheads significantly without compromising the correctness. To further reduce metadata persistency cost, WOJ is integrated with SSD’s FTL within the device. Experiment results show that the ext3 file system with data journaling on WOJ-enabled SSDs can deliver up to 2.7X higher throughput than that with regular deduplication, while both remove about half of the writes to the SSD.

Anti-forensics in ext4: On secrecy and usability of timestamp-based data hiding

Ext4 is a popular file system used by Android and many Linux distributions. With its rising pervasiveness, anti-forensic techniques like data hiding may be used to conceal data. This paper analyzes the feasibility of using timestamps of the ext4 file system to hide data. First, we examine the usage, the structure and the capacity of the available timestamps with a special focus on their sub-second granularity. The results reveal that the nanoseconds part of the ext4 timestamps can be used to build a system with steganographic strength. Second, we devise an ext4 anti-forensic technique that offers secrecy of the hidden data and easy usability in a wide range of scenarios. We provide a set of requirements (e.g., indistinguishability of regular and tampered timestamps) and a proof-of-concept implementation that is able to conceal arbitrary data within the file system timestamps. The evaluation shows that our implementation satisfies our requirements and actually works in practice.

Following the breadcrumbs: Timestamp pattern identification for cloud forensics

This study explores the challenges of digital forensics investigation in file access, transfer and operations, and identifies file operational and behavioral patterns based on timestamps—in both the standalone as well as interactions between Windows NTFS and Ubuntu Ext4 filesystems. File-based metadata is observed, and timestamps across different cloud access behavioral patterns are compared and validated. As critical metadata information cannot be easily observed, a rigorous iterative approach was implemented to extract hidden, critical file attributes and timestamps. Direct observation and cross-sectional analysis were adopted to analyze timestamps, and to differentiate between patterns based on different types of cloud access operations. Fundamental observation rules and characteristics of file interaction in the cloud environment are derived as behavioral patterns for cloud operations. This study contributes to cloud forensics investigation of data breach incidents where the crime clues, characteristics and evidence of the incidents are collected, identified and analyzed. The results demonstrate the effectiveness of pattern identification for digital forensics across various types of cloud access operations.

Block and Inode Based Analysis on EXT4 File System

Block and Inode Based Analysis on EXT4 File System