Multi-stage Pipeline Research Articles

MixFuse: An iterative mix-attention transformer for multi-modal image fusion

Multi-modal image fusion plays a crucial role in various visual systems. However, existing methods typically involve a multi-stage pipeline, i.e., feature extraction, integration, and reconstruction, which limits the effectiveness and efficiency of feature interaction and aggregation. In this paper, we propose MixFuse, a compact multi-modal image fusion framework based on Transformers. It smoothly unifies the process of feature extraction and integration, As its core, the Mix Attention Transformer Block (MATB) integrates the Cross-Attention Transformer Module (CATM) and the Self-Attention Transformer Module (SATM). The CATM introduces a symmetrical cross-attention mechanism to identify modality-specific and general features, filtering out irrelevant and redundant information. Meanwhile, the SATM is designed to refine the combined features via a self-attention mechanism, enhancing the internal consistency and proper preservation of the features. This successive cross and self-attention modules work together to enhance the generation of more accurate and refined feature maps, which are essential for later reconstruction. Extensive evaluation of MixFuse on five public datasets shows its superior performance and adaptability over state-of-the-art methods. The code and model will be released at https://github.com/Bitlijinfu/MixFuse.

MixFormer: End-to-End Tracking With Iterative Mixed Attention.

Visual object tracking often employs a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, in this paper, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and we propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows us to extract target-specific discriminative features and perform extensive communication between target and search area. Based on MAM, we build our MixFormer trackers simply by stacking multiple MAMs and placing a localization head on top. Specifically, we instantiate two types of MixFormer trackers, a hierarchical tracker MixCvT, and a non-hierarchical simple tracker MixViT. For these two trackers, we investigate a series of pre-training methods and uncover the different behaviors between supervised pre-training and self-supervised pre-training in our MixFormer trackers. We also extend the masked autoencoder pre-training to our MixFormer trackers and design the new competitive TrackMAE pre-training technique. Finally, to handle multiple target templates during online tracking, we devise an asymmetric attention scheme in MAM to reduce computational cost, and propose an effective score prediction module to select high-quality templates. Our MixFormer trackers set a new state-of-the-art performance on seven tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10 k, OTB100, TOTB and UAV123. In particular, our MixViT-L achieves AUC scores of 73.3% on LaSOT, 86.1% on TrackingNet and 82.8% on TOTB.

A Semantic Spatial Structure-Based Loop Detection Algorithm for Visual Environmental Sensing

Loop closure detection is an important component of the Simultaneous Localization and Mapping (SLAM) algorithm, which is utilized in environmental sensing. It helps to reduce drift errors during long-term operation, improving the accuracy and robustness of localization. Such improvements are sorely needed, as conventional visual-based loop detection algorithms are greatly affected by significant changes in viewpoint and lighting conditions. In this paper, we present a semantic spatial structure-based loop detection algorithm. In place of feature points, robust semantic features are used to cope with the variation in the viewpoint. In consideration of the semantic features, which are region-based, we provide a corresponding matching algorithm. Constraints on semantic information and spatial structure are used to determine the existence of loop-back. A multi-stage pipeline framework is proposed to systematically leverage semantic information at different levels, enabling efficient filtering of potential loop closure candidates. To validate the effectiveness of our algorithm, we conducted experiments using the uHumans2 dataset. Our results demonstrate that, even when there are significant changes in viewpoint, the algorithm exhibits superior robustness compared to that of traditional loop detection methods.

High-performance subpixel edge location based on FPGA for horizon sensors

Horizon edge localization accuracy and speed are key factors in the performance of horizon sensors. This paper proposes a high-performance sub-pixel edge localization algorithm base on Field Programmable Gate Array (FPGA) for horizon sensors. The algorithm is carefully designed and simplified according to the computational capabilities and limitations of FPGAs. By making full use of the parallel computing capability of FPGA and carrying out multi-stage pipeline design, the algorithm can complete image acquisition, rough edge localization, sub-pixel edge localization, and projections from pixel points to unit vectors at the same time, which greatly reduces the delay caused by image processing. The experimental results show that FPGA-based image processing shows a significant superiority in terms of speed compared to traditional embedded processors.

High frequency attitude determination of micro satellites based on FPGA

The accuracy and speed of satellite attitude calculation algorithms are one of the key factors affecting the performance of satellite attitude control. This article proposes a high-performance attitude calculation algorithm for micro satellites based on Field Programmable Gate Array (FPGA). This algorithm is designed and simplified based on the parallel characteristics and abundant computing resources of FPGA. The algorithm fully utilizes the parallel computing power of FPGA and designs a multi-stage pipeline for complex matrix operations in Kalman filtering. It can simultaneously complete data collection and final attitude determination result output, greatly improving the output frequency of attitude information and reducing output delay. The experimental results show that compared to traditional embedded processors, FPGA based attitude calculation has significant advantages in terms of speed.

SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation

The Segment Anything Model (SAM) is a powerful foundation model that has revolutionised image segmentation. To apply SAM to surgical instrument segmentation, a common approach is to locate precise points or boxes of instruments and then use them as prompts for SAM in a zero-shot manner. However, we observe two problems with this naive pipeline: (1) the domain gap between natural objects and surgical instruments leads to inferior generalisation of SAM; and (2) SAM relies on precise point or box locations for accurate segmentation, requiring either extensive manual guidance or a well-performing specialist detector for prompt preparation, which leads to a complex multi-stage pipeline. To address these problems, we introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to effectively integrate surgical-specific information with SAM’s pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes and eliminates the use of explicit prompts for improved robustness and a simpler pipeline. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning, further enhancing the discrimination of the class prototypes for more accurate class prompting. The results of extensive experiments on both EndoVis2018 and EndoVis2017 datasets demonstrate that SurgicalSAM achieves state-of-the-art performance while only requiring a small number of tunable parameters. The source code is available at https://github.com/wenxi-yue/SurgicalSAM.

Multi-network approach for image segmentation in non-contrast enhanced cardiac 3D MRI of arrhythmic patients

Left atrial appendage (LAA) is the source of thrombi formation in more than 90% of strokes in patients with nonvalvular atrial fibrillation. Catheter-based LAA occlusion is being increasingly applied as a treatment strategy to prevent stroke. Anatomical complexity of LAA makes percutaneous occlusion commonly performed under transesophageal echocardiography (TEE) and X-ray (XR) guidance especially challenging. Image fusion techniques integrating 3D anatomical models derived from pre-procedural imaging into the live XR fluoroscopy can be applied to guide each step of the LAA closure. Cardiac magnetic resonance (CMR) imaging gains in importance for radiation-free evaluation of cardiac morphology as alternative to gold-standard TEE or computed tomography angiography (CTA). Manual delineation of cardiac structures from non-contrast enhanced CMR is, however, labor-intensive, tedious, and challenging due to the rather low contrast. Additionally, arrhythmia often impairs the image quality in ECG synchronized acquisitions causing blurring and motion artifacts. Thus, for cardiac segmentation in arrhythmic patients, there is a strong need for an automated image segmentation method. Deep learning-based methods have shown great promise in medical image analysis achieving superior performance in various imaging modalities and different clinical applications. Fully-convolutional neural networks (CNNs), especially U-Net, have become the method of choice for cardiac segmentation. In this paper, we propose an approach for automatic segmentation of cardiac structures from non-contrast enhanced CMR images of arrhythmic patients based on CNNs implemented in a multi-stage pipeline. Two-stage implementation allows subdividing the task into localization of the relevant cardiac structures and segmentation of these structures from the cropped sub-regions obtained from previous step leading to efficient and effective way of automated cardiac segmentation.

Sparse Action Tube Detection.

Action tube detection is a challenging task as it requires not only to locate action instances in each frame, but also link them in time. Existing action tube detection methods often employ multi-stage pipelines with complex designs and time-consuming linking procedure. In this paper, we present a simple end-to-end action tube detection method, termed as Sparse Tube Detector (STDet). Unlike those dense action detectors, our core idea is to use a set of learnable tube queries and directly decode them into action tubes (i.e., a set of tracked boxes with action label) from video content. This sparse detection paradigm shares several advantages. First, the large number of hand-crafted anchor candidates in dense action detectors is greatly reduced to a small number of learnable tubes, which results in a more efficient detection framework. Second, our learnable tube queries directly attend the whole video content, which endows our method with the capacity of capturing long-range information for action detection. Finally, our action detector is an end-to-end tube detection without requiring the linking procedure, which directly and explicitly predicts the action boundary instead of depending on the linking strategy. Extensive experiments shows that our STDet outperforms the previous state-of-the-art methods on two challenging untrimmed video action detection datasets of UCF101-24 and MultiSports. We hope our method will be an simple end-to-end tube detection baseline and can inspire new ideas in this direction.

6D Pose Estimation on Point Cloud Data through Prior Knowledge Integration: A Case Study in Autonomous Disassembly

The accurate estimation of 6D pose remains a challenging task within the computer vision domain, even when utilizing 3D point cloud data. Conversely, in the manufacturing domain, instances arise where leveraging prior knowledge can yield advancements in this endeavor. This study focuses on the disassembly of starter motors to augment the engineering of product life cycles. A pivotal objective in this context involves the identification and 6D pose estimation of bolts affixed to the motors, facilitating automated disassembly within the manufacturing workflow. Complicating matters, the presence of occlusions and the limitations of single-view data acquisition, notably when motors are placed in a clamping system, obscure certain portions and render some bolts imperceptible. Consequently, the development of a comprehensive pipeline capable of acquiring complete bolt information is imperative to avoid oversight in bolt detection. In this paper, employing the task of bolt detection within the scope of our project as a pertinent use case, we introduce a meticulously devised pipeline. This multi-stage pipeline effectively captures the 6D information with regard to all bolts on the motor, thereby showcasing the effective utilization of prior knowledge in handling this challenging task. The proposed methodology not only contributes to the field of 6D pose estimation but also underscores the viability of integrating domain-specific insights to tackle complex problems in manufacturing and automation.

Improving First-stage Retrieval of Point-of-interest Search by Pre-training Models

Point-of-interest (POI) search is important for location-based services, such as navigation and online ride-hailing service. The goal of POI search is to find the most relevant destinations from a large-scale POI database given a text query. To improve the effectiveness and efficiency of POI search, most existing approaches are based on a multi-stage pipeline that consists of an efficiency-oriented retrieval stage and one or more effectiveness-oriented re-rank stages. In this article, we focus on the first efficiency-oriented retrieval stage of the POI search. We first identify the limitations of existing first-stage POI retrieval models in capturing the semantic-geography relationship and modeling the fine-grained geographical context information. Then, we propose a Geo-Enhanced Dense Retrieval framework for POI search to alleviate the above problems. Specifically, the proposed framework leverages the capacity of pre-trained language models (e.g., BERT) and designs a pre-training approach to better model the semantic match between the query prefix and POIs. With the POI collection, we first perform a token-level pre-training task based on a geographical-sensitive masked language prediction and design two retrieval-oriented pre-training tasks that link the address of each POI to its name and geo-location. With the user behavior logs collected from an online POI search system, we design two additional pre-training tasks based on users’ query reformulation behavior and the transitions between POIs. We also utilize a late-interaction network structure to model the fine-grained interactions between the text and geographical context information within an acceptable query latency. Extensive experiments on the real-world datasets collected from the Didichuxing application demonstrate that the proposed framework can achieve superior retrieval performance over existing first-stage POI retrieval methods.

A marker-based neural network system for extracting social determinants of health.

The impact of social determinants of health (SDoH) on patients' healthcare quality and the disparity is well known. Many SDoH items are not coded in structured forms in electronic health records. These items are often captured in free-text clinical notes, but there are limited methods for automatically extracting them. We explore a multi-stage pipeline involving named entity recognition (NER), relation classification (RC), and text classification methods to automatically extract SDoH information from clinical notes. The study uses the N2C2 Shared Task data, which were collected from 2 sources of clinical notes: MIMIC-III and University of Washington Harborview Medical Centers. It contains 4480 social history sections with full annotation for 12 SDoHs. In order to handle the issue of overlapping entities, we developed a novel marker-based NER model. We used it in a multi-stage pipeline to extract SDoH information from clinical notes. Our marker-based system outperformed the state-of-the-art span-based models at handling overlapping entities based on the overall Micro-F1 score performance. It also achieved state-of-the-art performance compared with the shared task methods. Our approach achieved an F1 of 0.9101, 0.8053, and 0.9025 for Subtasks A, B, and C, respectively. The major finding of this study is that the multi-stage pipeline effectively extracts SDoH information from clinical notes. This approach can improve the understanding and tracking of SDoHs in clinical settings. However, error propagation may be an issue and further research is needed to improve the extraction of entities with complex semantic meanings and low-frequency entities. We have made the source code available at https://github.com/Zephyr1022/SDOH-N2C2-UTSA.

Design and Implementation of Convolutional Neural Network Accelerator Based on FPGA

Convolutional neural network, as a kind of feed-forward neural network, has been widely used in image recognition, speech processing and other fields in recent years. In this paper, an FPGA-based CNN gas pedal is designed to solve the problem of slow running and high-power consumption of CNN on resource-constrained hardware. The design invokes multi-stage pipeline parallel processing technology to accelerate convolutional operations; quantifies network parameters from 32-bit floating-point to 8-bit fixed-point while guaranteeing CNN accuracy, and uses data multiplexing to reduce resource consumption. Experimental results show that the design is 10 times faster than the intel i7-8700 and consumes only 1% of the power of the RTX 2060 at 50MHz.

Temporal Dropout for Weakly Supervised Action Localization

Weakly supervised action localization is a challenging problem in video understanding and action recognition. Existing models usually formulate the training process as direct classification using video-level supervision. They tend to only locate the most discriminative parts of action instances and produce temporally incomplete detection results. A natural solution for this problem, the adversarial erasing strategy, is to remove such parts from training so that models can attend to complementary parts. Previous works do it in an offline and heuristic way. They adopt a multi-stage pipeline, where discriminative regions are determined and erased under the guidance of detection results from last stage. Such a pipeline can be both ineffective and inefficient, possibly hindering the overall performance. On the contrary, we combine adversarial erasing with dropout mechanism and propose a Temporal Dropout Module that learns where to remove in a data-driven and online manner. This plug-and-play module is trained without iterative stages, which not only simplifies the pipeline but also makes the regularization during training easier and more adaptive. Experiments show that the proposed method outperforms previous erasing-based methods by a large margin. More importantly, it achieves universal improvement when plugged into various direct classification methods and obtains state-of-the-art performance.

A Real-Time Global Inference Network for One-Stage Referring Expression Comprehension.

Referring expression comprehension (REC) is an emerging research topic in computer vision, which refers to the detection of a target region in an image given a test description. Most existing REC methods follow a multistage pipeline, which is computationally expensive and greatly limits the applications of REC. In this article, we propose a one-stage model toward real-time REC, termed real-time global inference network (RealGIN). RealGIN addresses the issues of expression diversity and complexity of REC with two innovative designs: adaptive feature selection (AFS) and Global Attentive ReAsoNing (GARAN). Expression diversity concerns varying expression content, which includes information such as colors, attributes, locations, and fine-grained categories. To address this issue, AFS adaptively fuses features of different semantic levels to tackle the changes in expression content. In contrast, expression complexity concerns the complex relational conditions in expressions that are used to identify the referent. To this end, GARAN uses the textual feature as a pivot to collect expression-aware visual information from all regions and then diffuses this information back to each region, which provides sufficient context for modeling the relational conditions in expressions. On five benchmark datasets, i.e., RefCOCO, RefCOCO+, RefCOCOg, ReferIT, and Flickr30k, the proposed RealGIN outperforms most existing methods and achieves very competitive performances against the most advanced one, i.e., MAttNet. More importantly, under the same hardware, RealGIN can boost the processing speed by 10-20 times over the existing methods.

High-performance adaptive texture streaming for planetary-scale high-mobility information visualization

Planetary-scale terrain requires streaming of world-scale high-resolution data sets in real time. High-performance streaming of world-scale data sets to achieve high mobility in terms of speed and view such as look-around, side-ways, or backward for high fidelity display systems is challenging. Moreover, streaming data sets without any preprocessing overhead require high-performance decompression and re-projection in real time. This work proposes a high-performance adaptive texture streaming system to capture high-speed scenes with view switching capability over a real-world terrain for high-fidelity display system. The system proposes multi-stage pipeline to stream multi-resolution data sets with a stable high-performance rendering rate. The proposed pipeline stages are asynchronously connected using request queues with a cache hierarchy. Efficient algorithms for resource management, prefetching and caching, prioritization of load requests, and hardware-accelerated processing are proposed for texture streaming without frame-rate stuttering. The proposed algorithms adapt to viewer navigation, display resolution, and dynamic execution environment. The proposed method is evaluated for complex scenes, mobility in terms of speed and view, and high-fidelity display. Our method achieved a ground-speed of 2.4x105,1.2x105,and9.0x104 km/h for HD (1024x720), FHD (1920x1080), and UHD (3840x2160) display resolutions respectively with a stable frame rate of 850, 590, and 420 frames per second.

Fast point cloud registration algorithm using parallel computing strategy

At present, the traditional ICP (Iterative Closest Point) algorithm used by TOF (Time-Of-Flight) cameras has the problem of high time-consuming in nearest point search process, resulting in low data update rate. Therefore, based on the characteristics of the data structure of the ICP algorithm, a multi-stage pipeline acceleration system is proposed, and the point cloud registration algorithm used in this system is studied. Firstly, design multiple units to store point cloud data and serve as the main search module. Secondly, make each group of units perform parallel operations, input the stored point clouds into their respective calculation modules, select a model point, traverse all query points, and perform subtraction. Then replace the model point and repeat the process until all model points are calculated. Finally, the query point corresponding to the minimum value in each model point difference data is selected and the model point is output as a point pair to the CPU to complete the final algorithm iteration. The experimental results show that at 30000 points, a single closest point search takes 25.65ms; at 20000 points, a single closest point search takes 11.65ms; at 10000 points, a single closest point search takes 2.88ms. It basically meets the requirements of TOF camera on-orbit measurement for the stability and reliability of point cloud registration and high data update rate.

PyVHR: a Python framework for remote photoplethysmography.

Remote photoplethysmography (rPPG) aspires to automatically estimate heart rate (HR) variability from videos in realistic environments. A number of effective methods relying on data-driven, model-based and statistical approaches have emerged in the past two decades. They exhibit increasing ability to estimate the blood volume pulse (BVP) signal upon which BPMs (Beats per Minute) can be estimated. Furthermore, learning-based rPPG methods have been recently proposed. The present pyVHR framework represents a multi-stage pipeline covering the whole process for extracting and analyzing HR fluctuations. It is designed for both theoretical studies and practical applications in contexts where wearable sensors are inconvenient to use. Namely, pyVHR supports either the development, assessment and statistical analysis of novel rPPG methods, either traditional or learning-based, or simply the sound comparison of well-established methods on multiple datasets. It is built up on accelerated Python libraries for video and signal processing as well as equipped with parallel/accelerated ad-hoc procedures paving the way to online processing on a GPU. The whole accelerated process can be safely run in real-time for 30 fps HD videos with an average speedup of around 5. This paper is shaped in the form of a gentle tutorial presentation of the framework.

Semantic-aware Spatiotemporal Alignment of Natural Outdoor Surveys

This article presents a keyframe-based, innovative map registration scheme for applications that benefit from recurring data acquisition, such as long-term natural environment monitoring. The proposed method consists of a multistage pipeline, in which semantic knowledge of the scene is acquired using a pretrained neural network. The semantic knowledge is subsequently employed to constrain the Iterative Closest Point algorithm (ICP). In this article, semantic-aware ICP is used to build keyframes as well as to align them both spatially and temporally, with neighboring keyframes and those captured around the same area but at a different point in time, respectively. Hierarchical clustering of ICP-generated transformations is then used to both eliminate outliers and find alignment consensus, followed by an optimization scheme based on a factor graph that includes loop closure. To evaluate the proposed framework, data were captured using a portable robotic sensor suite consisting of three cameras, a three-dimensional lidar, and an inertial navigation system. The data were acquired monthly over 12 months by revisiting the same trajectory between August 2020 and July 2021.

Tuberculosis Bacteria Detection and Counting in Fluorescence Microscopy Images Using a Multi-Stage Deep Learning Pipeline

The manual observation of sputum smears by fluorescence microscopy for the diagnosis and treatment monitoring of patients with tuberculosis (TB) is a laborious and subjective task. In this work, we introduce an automatic pipeline which employs a novel deep learning-based approach to rapidly detect Mycobacterium tuberculosis (Mtb) organisms in sputum samples and thus quantify the burden of the disease. Fluorescence microscopy images are used as input in a series of networks, which ultimately produces a final count of present bacteria more quickly and consistently than manual analysis by healthcare workers. The pipeline consists of four stages: annotation by cycle-consistent generative adversarial networks (GANs), extraction of salient image patches, classification of the extracted patches, and finally, regression to yield the final bacteria count. We empirically evaluate the individual stages of the pipeline as well as perform a unified evaluation on previously unseen data that were given ground-truth labels by an experienced microscopist. We show that with no human intervention, the pipeline can provide the bacterial count for a sample of images with an error of less than 5%.

A Grid of Core-collapse Supernova Remnant Models. I. The Effect of Wind-driven Mass Loss

Abstract Massive stars can shed material via steady, line-driven winds, eruptive outflows, or mass transfer onto a binary companion. In the case of single stars, the mass is deposited by the stellar wind into the nearby environment. After the massive star explodes, the stellar ejecta interact with this circumstellar material (CSM), oftentimes resulting in bright X-ray line emission from both the shock-heated CSM and ejecta. The amount of material lost by the progenitor, the mass of ejecta, and its energetics all impact the bulk spectral characteristics of this X-ray emission. Here we present a grid of core-collapse supernova remnant models derived from models for massive stars with zero-age main-sequence masses of ∼10–30 M ☉ evolved from the pre-main-sequence stage with wind-driven mass loss. Evolution is handled by a multistage pipeline of software packages. First, we use mesa (Modules for Experiments in Stellar Astrophysics) to evolve the progenitors from pre-main-sequence to iron core collapse. We then use the Supernova Explosion Code (snec) to explode the mesa models, and we follow them for the first 100 days following core collapse. Finally, we couple the snec output, along with the CSM generated from mesa mass-loss rates, into the cosmic-ray hydrodynamics code to model the remnant phase to 7000 yr after core collapse. At the end of each stage, we compare our outputs with those found in the literature, and we examine any qualitative and quantitative differences in the bulk properties of the remnants and their spectra based on the initial progenitor mass, as well as mass-loss history.