Video Mosaicking Research Articles

Spatial Transformation Accelerator with Parallel Data Access Scheme for Sample Reconstruction

Spatial image transformation is a commonly used component in many image processing pipelines. It enables the correction of optical distortions, image registration onto a common reference plane, electronic image stabilisation, digital zoom, video mosaicking, etc. With the growing tendency to embed image processing in low-power devices, attaining an efficient transformation solution becomes increasingly decisive. Furthermore, interpolation is the key operation in achieving the high quality of the transformed data from the original data. Fortunately, different implementations have already seen several efficiency improvements in recent years. However, interpolation relies on sampling a set of neighbouring points from memory, which has yet to be addressed efficiently for smaller computational platforms with limited memory resources. In this work, we derive a generic mathematical model and circuit design principles for the spatial transformation accelerator design for N-dimensional data. Furthermore, we present an efficient simultaneous access scheme for high-quality signal reconstruction. Finally, the introduced ideas are verified in field programmable gate arrays using one-dimensional and two-dimensional data transformation use cases. The presented solution is able to transform images with sizes ranging from 256 × 256 to 8192 × 8192 and achieves a transfer rate of 275 frames per second with 512 × 512 images.

Coverage estimation of benthic habitat features by semantic segmentation of underwater imagery from South-eastern Baltic reefs using deep learning models

Underwater imagery (UI) is an important and sometimes the only tool for mapping hard-bottom habitats. With the development of new camera systems, from hand-held or simple “drop-down” cameras to ROV/AUV-mounted video systems, video data collection has increased considerably. However, the processing and analysing of vast amounts of imagery can become very labour-intensive, thus making it ineffective both time-wise and financially. This task could be simplified if the processes or their intermediate steps could be done automatically. Luckily, the rise of AI applications for automatic image analysis tasks in the last decade has empowered researchers with robust and effective tools. In this study, two ways to make UI analysis more efficient were tested with eight dominant visual features of the Southeastern Baltic reefs: 1) the simplification of video processing and expert annotation efforts by skipping the video mosaicking step and reducing the number of frames analysed; 2) the application of semantic segmentation of UI using deep learning models. The results showed that the annotation of individual frames provides similar results compared to 2D mosaics; moreover, the reduction of frames by 2–3 times resulted in only minor differences from the baseline. Semantic segmentation using the PSPNet model as the deep learning architecture was extensively evaluated, applying three variants of validation. The accuracy of segmentation, as measured by the intersection-over-union, was mediocre; however, estimates of visual coverage percentages were fair: the difference between the expert annotations and model-predicted segmentation was less than 6–8%, which could be considered an encouraging result.

Placental vessel segmentation and registration in fetoscopy: Literature review and MICCAI FetReg2021 challenge findings

Fetoscopy laser photocoagulation is a widely adopted procedure for treating Twin-to-Twin Transfusion Syndrome (TTTS). The procedure involves photocoagulation pathological anastomoses to restore a physiological blood exchange among twins. The procedure is particularly challenging, from the surgeon’s side, due to the limited field of view, poor manoeuvrability of the fetoscope, poor visibility due to amniotic fluid turbidity, and variability in illumination. These challenges may lead to increased surgery time and incomplete ablation of pathological anastomoses, resulting in persistent TTTS. Computer-assisted intervention (CAI) can provide TTTS surgeons with decision support and context awareness by identifying key structures in the scene and expanding the fetoscopic field of view through video mosaicking. Research in this domain has been hampered by the lack of high-quality data to design, develop and test CAI algorithms. Through the Fetoscopic Placental Vessel Segmentation and Registration (FetReg2021) challenge, which was organized as part of the MICCAI2021 Endoscopic Vision (EndoVis) challenge, we released the first large-scale multi-center TTTS dataset for the development of generalized and robust semantic segmentation and video mosaicking algorithms with a focus on creating drift-free mosaics from long duration fetoscopy videos. For this challenge, we released a dataset of 2060 images, pixel-annotated for vessels, tool, fetus and background classes, from 18 in-vivo TTTS fetoscopy procedures and 18 short video clips of an average length of 411 frames for developing placental scene segmentation and frame registration for mosaicking techniques. Seven teams participated in this challenge and their model performance was assessed on an unseen test dataset of 658 pixel-annotated images from 6 fetoscopic procedures and 6 short clips. For the segmentation task, overall baseline performed was the top performing (aggregated mIoU of 0.6763) and was the best on the vessel class (mIoU of 0.5817) while team RREB was the best on the tool (mIoU of 0.6335) and fetus (mIoU of 0.5178) classes. For the registration task, overall the baseline performed better than team SANO with an overall mean 5-frame SSIM of 0.9348. Qualitatively, it was observed that team SANO performed better in planar scenarios, while baseline was better in non-planner scenarios. The detailed analysis showed that no single team outperformed on all 6 test fetoscopic videos. The challenge provided an opportunity to create generalized solutions for fetoscopic scene understanding and mosaicking. In this paper, we present the findings of the FetReg2021 challenge, alongside reporting a detailed literature review for CAI in TTTS fetoscopy. Through this challenge, its analysis and the release of multi-center fetoscopic data, we provide a benchmark for future research in this field.

Placental vessel-guided hybrid framework for fetoscopic mosaicking

ABSTRACT Fetoscopic laser photocoagulation is used to treat twin-to-twin transfusion syndrome; however, this procedure is hindered because of difficulty in visualising the intraoperative surgical environment due to limited surgical field-of-view, unusual placenta position, limited manoeuvrability of the fetoscope and poor visibility due to fluid turbidity and occlusions. Fetoscopic video mosaicking can create an expanded field-of-view image of the fetoscopic intraoperative environment, which could support the surgeons in localising the vascular anastomoses during the fetoscopic procedure. However, classical handcrafted feature matching methods fail on in vivo fetoscopic videos. An existing state-of-the-art method on fetoscopic mosaicking relies on vessel presence and fails when vessels are not present in the view. We propose a vessel-guided hybrid fetoscopic mosaicking framework that mutually benefits from a placental vessel-based registration and a deep learning-based dense matching method to optimise the overall performance. A selection mechanism is implemented based on vessels’ appearance consistency and photometric error minimisation for choosing the best pairwise transformation. Using the extended fetoscopy placenta dataset, we experimentally show the robustness of the proposed framework, over the state-of-the-art methods, even in vessel-free, low-textured, or low illumination non-planar fetoscopic views.

Unlocking the Potential of Deep Learning for Migratory Waterbirds Monitoring Using Surveillance Video

Estimates of migratory waterbirds population provide the essential scientific basis to guide the conservation of coastal wetlands, which are heavily modified and threatened by economic development. New equipment and technology have been increasingly introduced in protected areas to expand the monitoring efforts, among which video surveillance and other unmanned devices are widely used in coastal wetlands. However, the massive amount of video records brings the dual challenge of storage and analysis. Manual analysis methods are time-consuming and error-prone, representing a significant bottleneck to rapid data processing and dissemination and application of results. Recently, video processing with deep learning has emerged as a solution, but its ability to accurately identify and count waterbirds across habitat types (e.g., mudflat, saltmarsh, and open water) is untested in coastal environments. In this study, we developed a two-step automatic waterbird monitoring framework. The first step involves automatic video segmentation, selection, processing, and mosaicking video footages into panorama images covering the entire monitoring area, which are subjected to the second step of counting and density estimation using a depth density estimation network (DDE). We tested the effectiveness and performance of the framework in Tiaozini, Jiangsu Province, China, which is a restored wetland, providing key high-tide roosting ground for migratory waterbirds in the East Asian–Australasian flyway. The results showed that our approach achieved an accuracy of 85.59%, outperforming many other popular deep learning algorithms. Furthermore, the standard error of our model was very small (se = 0.0004), suggesting the high stability of the method. The framework is computing effective—it takes about one minute to process a theme covering the entire site using a high-performance desktop computer. These results demonstrate that our framework can extract ecologically meaningful data and information from video surveillance footages accurately to assist biodiversity monitoring, fulfilling the gap in the efficient use of existing monitoring equipment deployed in protected areas.

High frame rate video mosaicking microendoscope to image large regions of intact tissue with subcellular resolution.

High-resolution microendoscopy (HRME) is a low-cost strategy to acquire images of intact tissue with subcellular resolution at frame rates ranging from 11 to 18 fps. Current HRME imaging strategies are limited by the small microendoscope field of view (∼0.5 mm2); multiple images must be acquired and reliably registered to assess large regions of clinical interest. Image mosaics have been assembled from co-registered frames of video acquired as a microendoscope is slowly moved across the tissue surface, but the slow frame rate of previous HRME systems made this approach impractical for acquiring quality mosaicked images from large regions of interest. Here, we present a novel video mosaicking microendoscope incorporating a high frame rate CMOS sensor and optical probe holder to enable high-speed, high quality interrogation of large tissue regions of interest. Microendoscopy videos acquired at >90 fps are assembled into an image mosaic. We assessed registration accuracy and image sharpness across the mosaic for images acquired with a handheld probe over a range of translational speeds. This high frame rate video mosaicking microendoscope enables in vivo probe translation at >15 millimeters per second while preserving high image quality and accurate mosaicking, increasing the size of the region of interest that can be interrogated at high resolution from 0.5 mm2 to >30 mm2. Real-time deployment of this high-frame rate system is demonstrated in vivo and source code made publicly available.

Robust mosaicking of maize fields from aerial imagery.

PremiseAerial imagery from small unmanned aerial vehicle systems is a promising approach for high‐throughput phenotyping and precision agriculture. A key requirement for both applications is to create a field‐scale mosaic of the aerial imagery sequence so that the same features are in registration, a very challenging problem for crop imagery.MethodsWe have developed an improved mosaicking pipeline, Video Mosaicking and summariZation (VMZ), which uses a novel two‐dimensional mosaicking algorithm that minimizes errors in estimating the transformations between successive frames during registration. The VMZ pipeline uses only the imagery, rather than relying on vehicle telemetry, ground control points, or global positioning system data, to estimate the frame‐to‐frame homographies. It exploits the spatiotemporal ordering of the image frames to reduce the computational complexity of finding corresponding features between frames using feature descriptors. We compared the performance of VMZ to a standard two‐dimensional mosaicking algorithm (AutoStitch) by mosaicking imagery of two maize (Zea mays) research nurseries freely flown with a variety of trajectories.ResultsThe VMZ pipeline produces superior mosaics faster. Using the speeded up robust features (SURF) descriptor, VMZ produces the highest‐quality mosaics.DiscussionOur results demonstrate the value of VMZ for the future automated extraction of plant phenotypes and dynamic scouting for crop management.

Real-time mosaicking for infrared videos from an oblique sweeping camera

Image mosaicking is widely used in Geographic Information Systems (GISs) for large-scale ground surface analysis. However, most existing mosaicking methods can only be used in offline processing due to the enormous amounts of computation. In this paper, we propose a novel and practical algorithm for real-time infrared video mosaicking. To achieve this, a novel fast template matching algorithm based on Sum of Cosine Differences (SCD) is proposed to coarsely match the sequential images. The high speed of the proposed template matching algorithm is obtained by computing correlation with Fast Fourier Transform (FFT). We also propose a novel fast Least Squares Matching (LSM) algorithm for inter-frame fine registration, which can significantly reduce the computation without degrading the matching accuracy. In addition, the proposed fast LSM can effectively adapt for noise degradation and geometric distortion. Based on the proposed fast template matching algorithm and fine registration algorithm, we develop a practical real-time mosaicking approach which can produce seamless mosaic image highly efficiently. Experiments on synthetic and real-world datasets demonstrate that the proposed algorithm is not just computationally efficient but also robust against various noise distortions.

Real-time video mosaicking to guide handheld in vivo microscopy.

Handheld and endoscopic optical-sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real-time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here, we modified a previous post-processing mosaicking method so that real-time mosaicking is possible at >30 frames/second when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real-time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real-time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual-axis confocal microscope.

Automatic video mosaicking algorithm via dynamic key-frame

Automatic video mosaicking is a challenging task in computer vision. Current researches consider either panoramic or mapping tasks on short videos. In this paper, an automatic mosaicking algorithm is proposed for both mapping and panoramic tasks based on the adapted key-frame on videos of any length. The speeded up robust features (SURF) and the grid motion statistic (GMS) algorithm are used for feature extraction and matching between consecutive frames, which are used to compute the transformation. In order to reduce the influence of the accumulated error during image stitching, an evaluation metric is put forward for the transformation matrix. Besides, a self-growth method is employed to stitch the global image for long videos. The algorithm is evaluated by using aerial-view and panoramic videos respectively on the graphic processing unit (GPU) device, which can satisfy the real-time requirement. The experimental results demonstrate that the proposed algorithm is able to achieve a better performance than the state-of-art.

Handheld line-scanned dual-axis confocal microscope with pistoned MEMS actuation for flat-field fluorescence imaging.

A handheld line-scanned dual-axis confocal (LS-DAC) microscope has been developed for high-speed (16frames/s) fluorescence imaging of tissues with sub-nuclear resolution. This is the first miniature fluorescence LS-DAC system that has been fully packaged for handheld clinical use on patients. A novel micro-electro-mechanical system scanning mechanism, with synchronized tilting and pistoning, is used to achieve flat-field en face imaging. We show that this facilitates video mosaicking to generate images that sample an extended lateral field of view.

Implementation of Histogram Based Image Fusion Technique for Underwater Image Enhancement in Reconfigurable Platform

Objectives: This paper proposes a novel histogram based image fusion method to refine the visibility of underwater images which are suffered from colour degradation and poor illumination due to ocean depth. Methods/Statistical Analysis: To improve the enhancement of underwater images, R channel is separately processed as red component attenuates heavily compared to blue and green component and is combined with histogram equalized original image. Then various weight maps, suitable for underwater images, were obtained from the single image and are processed before fusion is done. Findings: Automated Underwater Vehicles (AUV) is used for marine navigation, pipeline tracking, video mosaicking etc. Automation of video captured has to be used for various applications where image pre processing is necessary. Hardware implementation helps the automation process to be used in real time scenario. In this work Field Programmable Gate Array implementation (FPGA) of a novel algorithm for the underwater image enhancement is carried out. New weight maps suitable for underwater images were used for fusion as compared to existing methods and were compared with the existing methods. Altiumnano board was used for FPGA implementation. Application/Improvements: Degraded images need to be enhanced before processing it further applications. This work is mainly focussed on underwater images and can be used in AUVs and Remotely Operated Vehicles (ROV). Keywords: Histogram, Image Fusion, Underwater, AUV, FPGA

Seam-based variable-step Bresenham blending method for real-time video mosaicking

Image blending plays an important role in video mosaicking, which has a high demand for real-time performance and visual quality. This paper proposes a fast blending method based on Bresenham algorithm, which realizes blending by controlling the storing addresses of source pixels. The starting storing location is accurately computed based on the coordinates of the middle pixel of the seam instead of the first pixel’s, reducing the accumulated error along the seam significantly. The other storing addresses are acquired using a variable-step Bresenham method, which takes advantage of burst mode operation of a dynamic memory and can achieve a good trade-off between the operation convenience and memory requirement. By the proposed method, complicated calculations of storing addresses are simplified into integer additions and subtractions, which is more suitable for hardware implementation. A hardware architecture based on field programmable gate array is presented to evaluate the proposed method with clock frequency analysis and resource assessment. The experimental results show that the proposed method achieves good performance of high image quality, low computational complexity, and low memory requirement.

Retinal slit lamp video mosaicking.

PurposeTo this day, the slit lamp remains the first tool used by an ophthalmologist to examine patient eyes. Imaging of the retina poses, however, a variety of problems, namely a shallow depth of focus, reflections from the optical system, a small field of view and non-uniform illumination. For ophthalmologists, the use of slit lamp images for documentation and analysis purposes, however, remains extremely challenging due to large image artifacts. For this reason, we propose an automatic retinal slit lamp video mosaicking, which enlarges the field of view and reduces amount of noise and reflections, thus enhancing image quality.MethodsOur method is composed of three parts: (i) viable content segmentation, (ii) global registration and (iii) image blending. Frame content is segmented using gradient boosting with custom pixel-wise features. Speeded-up robust features are used for finding pair-wise translations between frames with robust random sample consensus estimation and graph-based simultaneous localization and mapping for global bundle adjustment. Foreground-aware blending based on feathering merges video frames into comprehensive mosaics.ResultsForeground is segmented successfully with an area under the curve of the receiver operating characteristic curve of 0.9557. Mosaicking results and state-of-the-art methods were compared and rated by ophthalmologists showing a strong preference for a large field of view provided by our method.ConclusionsThe proposed method for global registration of retinal slit lamp images of the retina into comprehensive mosaics improves over state-of-the-art methods and is preferred qualitatively.

Real‐time construction and visualisation of drift‐free video mosaics from unconstrained camera motion

This work proposes a novel approach for real-time video mosaicking facilitating drift-free mosaic construction and visualisation, with integrated frame blending and redundancy management, that is shown to be flexible to a range of varying mosaic scenarios. The approach supports unconstrained camera motion with in-sequence loop closing, variation in camera focal distance (zoom) and recovery from video sequence breaks. Real-time performance, over extended duration sequences, is realised via novel aspects of frame management within the mosaic representation and thus avoiding the high data redundancy associated with temporally dense, spatially overlapping video frame inputs. This managed set of image frames is visualised in real time using a dynamic mosaic representation of overlapping textured graphics primitives in place of the traditional globally constructed, and hence frequently reconstructed, mosaic image. Within this formulation, subsequent optimisation occurring during online construction can thus efficiency adjust relative frame positions via simple primitive position transforms. Effective visualisation is similarly facilitated by online inter-frame blending to overcome the illumination and colour variance associated with modern camera hardware. The evaluation illustrates overall robustness in video mosaic construction under a diverse range of conditions including indoor and outdoor environments, varying illumination and presence of in-scene motion on varying computational platforms.

Intraoperative imaging during Mohs surgery with reflectance confocal microscopy: initial clinical experience.

Mohs surgery for the removal of nonmelanoma skin cancers (NMSCs) is performed in stages, while being guided by the examination for residual tumor with frozen pathology. However, preparation of frozen pathology at each stage is time consuming and labor intensive. Real-time intraoperative reflectance confocal microscopy(RCM), combined with video mosaicking, may enable rapid detection of residual tumor directly in the surgical wounds on patients. We report our initial experience on 25 patients, using aluminum chloride for nuclear contrast. Imaging was performed in quadrants in the wound to simulate the Mohs surgeon’s examination of pathology. Images and videos of the epidermal and dermal margins were found to be of clinically acceptable quality. Bright nuclear morphology was identified at the epidermal margin and detectable in residual NMSC tumors. The presence of residual tumor and normal skin features could be detected in the peripheral and deep dermal margins. Intraoperative RCM imaging may enable detection of residual tumor directly on patients during Mohs surgery, and may serve as an adjunct for frozen pathology. Ultimately, for routine clinical utility, a stronger tumor-to-dermis contrast may be necessary, and also a smaller microscope with an automated approach for imaging in the entire wound in a rapid and controlled manner.

Motion Panoramas

AbstractIn this paper we describe a method for analysing video sequences and for representing them as mosaics or panoramas. Previous work on video mosaicking essentially concentrated on static scenes. We generalize these approaches to the case of a rotating camera observing both static and moving objects where the static portions of the scene are not necessarily dominant, as it has been often hypothesized in the past. We start by describing a robust technique for accurately aligning a large number of video frames under unknown camera rotations and camera settings. The alignment technique combines a feature‐based method (initialization and refinement) with rough motion segmentation followed by a colour‐based direct method (final adjustment). This precise frame‐to‐frame alignment allows the dynamic building of a background representation as well as an efficient segmentation of each image such that moving regions of arbitrary shape and size are aligned with the static background. Thus a motion panorama visualizes both dynamic and static scene elements in a geometrically consistent way. Extensive experiments applied to archived videos of track‐and‐field events validate the approach. Copyright © 2004 John Wiley & Sons, Ltd.

Video strip mosaicking: A two-dimensional approach by convergent image bridging

A principal advantage of video is to provide, in a cost-effective manner, information required for a very narrow and long strip target, utilizing narrow view angle and dynamic stereo coverage. A video mosaicking technique has been specifically developed with a focus on such a narrow and long strip target (e.g. 20-30m wide and several hundred kilometres long). This technique was developed by combining traditional analogue mosaicking and digital image processing techniques. An interactive approach, registering video frames in two convergent directions along a strip, produced an acceptable positional accuracy in terms of time and cost requirements. The results also suggest that video strip mosaicking could effectively be used for linear targets which require generalized thematic mapping of relatively simple classes. This development of a simplified two-dimensional image bridging approach has enabled airborne videography to progress beyond being used mainly as a 'snapshot' visual assessment tool. The low-cost operational use of convergent image bridging, requiring solely endlap, identifies video strip mosaicking as a potentially useful remote sensing tool.

Real-time video mosaicking of the ocean floor

An approach was developed for real-time creation of video mosaics of the ocean floor. The approach uses visual correspondence during both image acquisition and consolidation to insure that there are no gaps in the mosaic. The use of visual information for mosaic creation offers an accuracy previously unachievable in real time. A complete mosaicking approach was developed, including a technique for image registration and several strategies for image acquisition control. The approach was implemented and tested experimentally. Ocean-floor mosaics were created in real time using a remotely-operated vehicle. In addition, extensive mosaicking experiments were conducted in a test-tank environment using a semiautonomous underwater robot.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>