Unified Benchmark Research Articles

4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions

AbstractIn this paper, we present a novel visual SLAM and long-term localization benchmark for autonomous driving in challenging conditions based on the large-scale 4Seasons dataset. The proposed benchmark provides drastic appearance variations caused by seasonal changes and diverse weather and illumination conditions. While significant progress has been made in advancing visual SLAM on small-scale datasets with similar conditions, there is still a lack of unified benchmarks representative of real-world scenarios for autonomous driving. We introduce a new unified benchmark for jointly evaluating visual odometry, global place recognition, and map-based visual localization performance which is crucial to successfully enable autonomous driving in any condition. The data has been collected for more than one year, resulting in more than 300 km of recordings in nine different environments ranging from a multi-level parking garage to urban (including tunnels) to countryside and highway. We provide globally consistent reference poses with up to centimeter-level accuracy obtained from the fusion of direct stereo-inertial odometry with RTK GNSS. We evaluate the performance of several state-of-the-art visual odometry and visual localization baseline approaches on the benchmark and analyze their properties. The experimental results provide new insights into current approaches and show promising potential for future research. Our benchmark and evaluation protocols will be available at https://go.vision.in.tum.de/4seasons.

A novel configuration optimization approach for IES considering exergy-degradation and non-energy costs of equipment

Improving the exergy utilization and economic performance is the core goal for the configuration optimization of the integrated energy system (IES). Achieving such a goal requires accurate evaluation of the exergy efficiency and economy. However, IES is a multisource heterogeneous and strongly coupled system. The development of reasonable exergy efficiency-economy characteristic evaluation indicators is a complex issue that limits its application in IES configuration. This article proposes a novel IES configuration optimization approach based on the concept of the equipment exergyeconomic cost coefficient. The exergyeconomic cost coefficient consists of the exergy-degradation and non-energy costs, which can quantitatively characterize the exergy efficiency and economy of the IES under a unified benchmark. A mechanism model for the equipment exergyeconomic cost coefficient, output of various equipment and total exergyeconomic cost is established. Furthermore, exergy flow attribute classification method is proposed to obtain the equipment exergyeconomic cost coefficient in the IES. The proposed method is verified in a real-world study case located in Yancheng, China. Compared to the conventional configuration method, the capacity of the renewable energy and combined heat and power generation (CHP) are enhanced, while the capacity of purchased electricity and electric boiler (EB) are reduced. It can obtain the minimum total exergyeconomic cost (TEC) for the IES. This study offers a new direction for the optimal configuration of IES.

A Semi-Supervised Transformer-Based Deep Learning Framework for Automated Tooth Segmentation and Identification on Panoramic Radiographs.

Automated tooth segmentation and identification on dental radiographs are crucial steps in establishing digital dental workflows. While deep learning networks have been developed for these tasks, their performance has been inferior in partially edentulous individuals. This study proposes a novel semi-supervised Transformer-based framework (SemiTNet), specifically designed to improve tooth segmentation and identification performance on panoramic radiographs, particularly in partially edentulous cases, and establish an open-source dataset to serve as a unified benchmark. A total of 16,317 panoramic radiographs (1589 labeled and 14,728 unlabeled images) were collected from various datasets to create a large-scale dataset (TSI15k). The labeled images were divided into training and test sets at a 7:1 ratio, while the unlabeled images were used for semi-supervised learning. The SemiTNet was developed using a semi-supervised learning method with a label-guided teacher-student knowledge distillation strategy, incorporating a Transformer-based architecture. The performance of SemiTNet was evaluated on the test set using the intersection over union (IoU), Dice coefficient, precision, recall, and F1 score, and compared with five state-of-the-art networks. Paired t-tests were performed to compare the evaluation metrics between SemiTNet and the other networks. SemiTNet outperformed other networks, achieving the highest accuracy for tooth segmentation and identification, while requiring minimal model size. SemiTNet's performance was near-perfect for fully dentate individuals (all metrics over 99.69%) and excellent for partially edentulous individuals (all metrics over 93%). In edentulous cases, SemiTNet obtained statistically significantly higher tooth identification performance than all other networks. The proposed SemiTNet outperformed previous high-complexity, state-of-the-art networks, particularly in partially edentulous cases. The established open-source TSI15k dataset could serve as a unified benchmark for future studies.

GLFuse: A Global and Local Four-Branch Feature Extraction Network for Infrared and Visible Image Fusion

Infrared and visible image fusion integrates complementary information from different modalities into a single image, providing sufficient imaging information for scene interpretation and downstream target recognition tasks. However, existing fusion methods often focus only on highlighting salient targets or preserving scene details, failing to effectively combine entire features from different modalities during the fusion process, resulting in underutilized features and poor overall fusion effects. To address these challenges, a global and local four-branch feature extraction image fusion network (GLFuse) is proposed. On one hand, the Super Token Transformer (STT) block, which is capable of rapidly sampling and predicting super tokens, is utilized to capture global features in the scene. On the other hand, a Detail Extraction Block (DEB) is developed to extract local features in the scene. Additionally, two feature fusion modules, namely the Attention-based Feature Selection Fusion Module (ASFM) and the Dual Attention Fusion Module (DAFM), are designed to facilitate selective fusion of features from different modalities. Of more importance, the various perceptual information of feature maps learned from different modality images at the different layers of a network is investigated to design a perceptual loss function to better restore scene detail information and highlight salient targets by treating the perceptual information separately. Extensive experiments confirm that GLFuse exhibits excellent performance in both subjective and objective evaluations. It deserves note that GLFuse effectively improves downstream target detection performance on a unified benchmark.

A unified benchmark for deep reinforcement learning-based energy management: Novel training ideas with the unweighted reward

Deep reinforcement learning stands as a powerful force in the realm of intelligent control for hybrid power systems, yet some imperfections persist in the positive progression of learning-based strategies, necessitating the proposal of essential solutions to address these flaws. Firstly, a public and reliable benchmark model for hybrid powertrains and the optimization results of energy management strategies are essential. Hence, two Python-based standard deep reinforcement learning agents and four Simulink-based hybrid powertrains are employed, forming a co-simulation training approach as the reliable solution. Secondly, a detailed analysis from the perspectives of range, magnitude, and importance reveals that the optimization terms in traditional reward functions can mislead the agent during the training process and require cumbersome weight tuning. Accordingly, this paper proposes a novel training idea that combines the rule-based engine start-stop with an unweighted reward tailored for optimizing engine efficiency and facilitating training progress. Finally, a hardware-in-the-loop test is performed, treating the P2 hybrid electric vehicle as the target. The results show that two deep reinforcement learning-based energy management strategies achieved fuel economies of 6.537 L/100 km and 6.330 L/100 km, respectively, and more efficient and reasonable control sequences ensure the working state of the engine as well as the state of charge of batteries.

Manganite memristive devices: recent progress and emerging opportunities

Manganite-based memristive devices have emerged as promising candidates for next-generation non-volatile memory and neuromorphic computing applications, owing to their unique resistive switching behavior and tunable electronic properties. This review explores recent innovations in manganite-based memristive devices, with a focus on materials engineering, device architectures, and fabrication techniques. We delve into the underlying mechanisms governing resistive switching in manganite thin films, elucidating the intricate interplay of oxygen vacancies, charge carriers, and structural modifications. This review underscores breakthroughs in harnessing manganite memristors for a range of applications, from high-density memory storage to neuromorphic computing platforms that mimic synaptic and neuronal functionalities. Additionally, we discuss the role of characterization techniques and the need for a unified benchmark for these devices. We provide insights into the challenges and opportunities associated with the co-integration of manganite-based memristive devices with more mature technologies, offering a roadmap for future research directions.

ConVision Benchmark: A Contemporary Framework to Benchmark CNN and ViT Models

Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have shown remarkable performance in computer vision tasks, including object detection and image recognition. These models have evolved significantly in architecture, efficiency, and versatility. Concurrently, deep-learning frameworks have diversified, with versions that often complicate reproducibility and unified benchmarking. We propose ConVision Benchmark, a comprehensive framework in PyTorch, to standardize the implementation and evaluation of state-of-the-art CNN and ViT models. This framework addresses common challenges such as version mismatches and inconsistent validation metrics. As a proof of concept, we performed an extensive benchmark analysis on a COVID-19 dataset, encompassing nearly 200 CNN and ViT models in which DenseNet-161 and MaxViT-Tiny achieved exceptional accuracy with a peak performance of around 95%. Although we primarily used the COVID-19 dataset for image classification, the framework is adaptable to a variety of datasets, enhancing its applicability across different domains. Our methodology includes rigorous performance evaluations, highlighting metrics such as accuracy, precision, recall, F1 score, and computational efficiency (FLOPs, MACs, CPU, and GPU latency). The ConVision Benchmark facilitates a comprehensive understanding of model efficacy, aiding researchers in deploying high-performance models for diverse applications.

Text-to-Image Vehicle Re-Identification: Multi-Scale Multi-View Cross-Modal Alignment Network and a Unified Benchmark

Text-to-Image Vehicle Re-Identification: Multi-Scale Multi-View Cross-Modal Alignment Network and a Unified Benchmark

Evaluation of Ten Deep-Learning-Based Out-of-Distribution Detection Methods for Remote Sensing Image Scene Classification

Although deep neural networks have made significant progress in tasks related to remote sensing image scene classification, most of these tasks assume that the training and test data are independently and identically distributed. However, when remote sensing scene classification models are deployed in the real world, the model will inevitably encounter situations where the distribution of the test set differs from that of the training set, leading to unpredictable errors during the inference and testing phase. For instance, in the context of large-scale remote sensing scene classification applications, it is difficult to obtain all the feature classes in the training phase. Consequently, during the inference and testing phases, the model will categorize images of unidentified unknown classes into known classes. Therefore, the deployment of out-of-distribution (OOD) detection within the realm of remote sensing scene classification is crucial for ensuring the reliability and safety of model application in real-world scenarios. Despite significant advancements in OOD detection methods in recent years, there remains a lack of a unified benchmark for evaluating various OOD methods specifically in remote sensing scene classification tasks. We designed different benchmarks on three classical remote sensing datasets to simulate scenes with different distributional shift. Ten different types of OOD detection methods were employed, and their performance was evaluated and compared using quantitative metrics. Numerous experiments were conducted to evaluate the overall performance of these state-of-the-art OOD detection methods under different test benchmarks. The comparative results show that the virtual-logit matching methods without additional training outperform the other types of methods on our benchmarks, suggesting that additional training methods are unnecessary for remote sensing image scene classification applications. Furthermore, we provide insights into OOD detection models and performance enhancement in real world. To the best of our knowledge, this study is the first evaluation and analysis of methods for detecting out-of-distribution data in remote sensing. We hope that this research will serve as a fundamental resource for future studies on out-of-distribution detection in remote sensing.

H5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

SummaryParallel I/O is a critical technique for moving data between compute and storage subsystems of supercomputers. With massive amounts of data produced or consumed by compute nodes, high‐performant parallel I/O is essential. I/O benchmarks play an important role in this process; however, there is a scarcity of I/O benchmarks representative of current workloads on HPC systems. Toward creating representative I/O kernels from real‐world applications, we have created h5bench , a set of I/O kernels that exercise hierarchical data format version 5 (HDF5) I/O on parallel file systems in numerous dimensions. Our focus on HDF5 is due to the parallel I/O library's heavy usage in various scientific applications running on supercomputing systems. The various tests benchmarked in the h5bench suite include I/O operations (read and write), data locality (arrays of basic data types and arrays of structures), array dimensionality (one‐dimensional arrays, two‐dimensional meshes, three‐dimensional cubes), I/O modes (synchronous and asynchronous). In this paper, we present the observed performance of h5bench executed along several of these dimensions on existing supercomputers (Cori and Summit) and pre‐exascale platforms (Perlmutter, Theta, and Polaris). h5bench measurements can be used to identify performance bottlenecks and their root causes and evaluate I/O optimizations. As the I/O patterns of h5bench are diverse and capture the I/O behaviors of various HPC applications, this study will be helpful to the broader supercomputing and I/O community.

Collaborative License Plate Recognition via Association Enhancement Network With Auxiliary Learning and a Unified Benchmark

Collaborative License Plate Recognition via Association Enhancement Network With Auxiliary Learning and a Unified Benchmark

Alignment-Free RGBT Salient Object Detection: Semantics-Guided Asymmetric Correlation Network and a Unified Benchmark

Alignment-Free RGBT Salient Object Detection: Semantics-Guided Asymmetric Correlation Network and a Unified Benchmark

Dense Tiny Object Detection: A Scene Context Guided Approach and a Unified Benchmark

Dense Tiny Object Detection: A Scene Context Guided Approach and a Unified Benchmark

See, Perceive, and Answer: A Unified Benchmark for High-Resolution Postdisaster Evaluation in Remote Sensing Images

See, Perceive, and Answer: A Unified Benchmark for High-Resolution Postdisaster Evaluation in Remote Sensing Images

A unified benchmark for the unknown detection capability of deep neural networks

Deep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: Over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out these unknown samples, but they only considered narrow and specific tasks, referred to as misclassification detection, open-set recognition, or out-of-distribution detection. In this work, we argue that these tasks should be treated as fundamentally an identical problem because an ideal model should possess detection capability for all those tasks. Therefore, we introduce the unknown detection task, an integration of previous individual tasks, for a rigorous examination of the detection capability of deep neural networks on a wide spectrum of unknown samples. To this end, unified benchmark datasets on different scales were constructed and the unknown detection capabilities of existing popular methods were subject to comparison. We found that Deep Ensemble consistently outperforms the other approaches in detecting unknowns; however, all methods are only successful for a specific type of unknown. The reproducible code and benchmark datasets are available at https://github.com/daintlab/unknown-detection-benchmarks.

On the Performance Intricacies of Persistent Memory Aware Storage Engines

As key components of DBMSs, various storage engines and index structures have been proposed based on incorrect assumptions before PMem hardware is publicly available. Recent studies reveal that there is a significant performance gap in evaluating index structures on real PMem platforms as compared to DRAM-based emulators. However, a comprehensive evaluation for those PMem-aware database storage engines on real PMem hardware is still missing. Meanwhile, dynamic memory management is more important on PMem systems because PMem is slower than DRAM and unfriendly to random small-writes, and ensuring crash-consistency for the metadata of PMem allocators introduces extra overhead. Therefore, it is essential to understand the performance intricacies of PMem-aware database storage engines from the perspective of PMem allocators. This paper presents a systematic evaluation of three PMem-aware database storage engines using representative workloads and a unified benchmarking framework that is integrated with four PMem allocators. Besides the commonly used metrics, the impact of different hardware configurations (such as NUMA and eADR) on performance is also considered. Through in-depth analysis, we reveal caveats and pitfalls on using or designing PMem-aware storage engines and important insights that can serve as guidelines for future development of PMem allocators and other related components.

ECFuse: Edge-Consistent and Correlation-Driven Fusion Framework for Infrared and Visible Image Fusion.

Infrared and visible image fusion (IVIF) aims to render fused images that maintain the merits of both modalities. To tackle the challenge in fusing cross-modality information and avoiding texture loss in IVIF, we propose a novel edge-consistent and correlation-driven fusion framework (ECFuse). This framework leverages our proposed edge-consistency fusion module to maintain rich and coherent edges and textures, simultaneously introducing a correlation-driven deep learning network to fuse the cross-modality global features and modality-specific local features. Firstly, the framework employs a multi-scale transformation (MST) to decompose the source images into base and detail layers. Then, the edge-consistent fusion module fuses detail layers while maintaining the coherence of edges through consistency verification. A correlation-driven fusion network is proposed to fuse the base layers containing both modalities' main features in the transformation domain. Finally, the final fused spatial image is reconstructed by inverse MST. We conducted experiments to compare our ECFuse with both conventional and deep leaning approaches on TNO, LLVIP and M3FD datasets. The qualitative and quantitative evaluation results demonstrate the effectiveness of our framework. We also show that ECFuse can boost the performance in downstream infrared-visible object detection in a unified benchmark.

IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers

High scalability and low operating cost make black-box protocol fuzzing a vital tool for discovering vulnerabilities in the firmware of IoT smart devices. However, it is still challenging to compare black-box protocol fuzzers due to the lack of unified benchmark firmware images, complete fuzzing mutation seeds, comprehensive performance metrics, and a standardized evaluation framework. In this paper, we design and implement IoTFuzzBench, a scalable, modular, metric-driven automation framework for evaluating black-box protocol fuzzers for IoT smart devices comprehensively and quantitatively. Specifically, IoTFuzzBench has so far included 14 real-world benchmark firmware images, 30 verified real-world benchmark vulnerabilities, complete fuzzing seeds for each vulnerability, 7 popular fuzzers, and 5 categories of complementary performance metrics. We deployed IoTFuzzBench and evaluated 7 popular black-box protocol fuzzers on all benchmark firmware images and benchmark vulnerabilities. The experimental results show that IoTFuzzBench can not only provide fast, reliable, and reproducible experiments, but also effectively evaluate the ability of each fuzzer to find vulnerabilities and the differential performance on different performance metrics. The fuzzers found a total of 13 vulnerabilities out of 30. None of these fuzzers can outperform the others on all metrics. This result demonstrates the importance of comprehensive metrics. We hope our findings ease the burden of fuzzing evaluation in IoT scenarios, advancing more pragmatic and reproducible fuzzer benchmarking efforts.

FCEVAL: An effective and quantitative platform for evaluating fuzzer combinations fairly and easily

Multiple base fuzzers collaborate as a fuzzer combination. Fuzzer combinations have been proven to perform more robustly and efficiently when fuzzing complicated real-world programs. The efficiency of finding bugs with limited computational resources would greatly benefit from the fuzzer combinations chosen by an effective and quantitative performance evaluation. However, evaluating fuzzer combinations remains challenging due to the lack of infrastructure for collaborative fuzzing, high enough collaboration efficiency of base fuzzers, unified benchmarks, comprehensive metrics, and unified analysis methods of coverage and bugs. This prevents us from selecting efficient fuzzer combinations and thus impairs vulnerability mining on real-world targets. In this paper, we design and implement FCEVAL, the first open-source platform for evaluating fuzzer combinations. In detail, we propose a new test case-sharing policy for increasing fuzzing potential so that we can provide a more efficient running environment for fuzzer combinations and thus improve evaluation effectiveness. Also, we select a unified set of diverse benchmarks and comprehensive metrics while adopting unified independent methods of real-time coverage statistics and multiple-sanitizers-based bug analysis for evaluation fairness and quantification. In addition, we design tools and guidelines covering the whole evaluation process for usability. With the above methodologies, we first construct an infrastructure special for collaborative fuzzing as the base of FCEVAL. After comparing two test case-sharing policies on the infrastructure and choosing the promising one as a substantial part of FCEVAL, we leverage FCEVAL to evaluate fuzzer combinations for more than 40,000 CPU hours and come up with five important conclusions, including (a) an efficient test case-sharing policy improving fuzzing potential and thus evaluation effectiveness, (b) comprehensive metrics being essential, (c) 24-hour duration and 20 repetitions for evaluation being substantial, (d) independent analysis methods of code coverage and bugs deserving of extensive adoption, and (e) FCEVAL being able to evaluate fuzzer combinations effectively, fairly, comprehensively, and easily. Meanwhile, we suggest how to improve collaborative fuzzing. In addition, source codes and test data are publicly available.

On potential challenges of V2X sidelink relaying under interference: link-level and system-level simulation with neural network assisted

The ever-increasing demand for high data rates and high connection densities in the vehicle communication network, along with the widespread adoption of radio access over the Third Generation Partnership Project (3GPP) standard, has been a major driver for the research on cellular vehicle-to-everything (C-V2X) communication. Nevertheless, Wi-Fi and other wireless communication technology work on the 5.9 GHz unlicensed band has also undergone booming proliferation over the years. C-V2X users dedicated band on the 5.9 GHz spectrum may thus suffer from both co-channel and adjacent channel interference, which cannot be negligible, especially in urban scenarios. To this end, 3GPP has standardized relay technology in New Radio (NR) V2X sidelink to extend the transmission range under interference. In this paper, through a link-level and system-level simulation study, we evaluate the sidelink performance in relaying scenarios under different interference. Motivated by the recent success of deep learning, a novel neural network is further introduced as a unified benchmark for interference mitigation evaluation. Numerical results show that there exist challenges in the real-time optimization of transmission scheme selection and power allocation in relay-assisted cases. The simulation also reveals that the interference incurred by NR on unlicensed spectrum (NR-U) signals and other sidelink signals is intractable to be suppressed, which may bring potential challenges in future works.