Multiple Image Frames Research Articles

Non-cooperative target recognition and relative motion estimation with inertial measurement unit assistance

This study investigated the problems of non-cooperative target recognition and relative motion estimation during spacecraft rendezvous maneuvers. A structure integrating an Inertial Measurement Unit (IMU) and a visual camera was presented. The angular velocity output of the IMU was used to calculate the motion trajectories of star points in multiple image frames, which can highlight the motion of non-cooperative targets with respect to the image background to improve the probability of target recognition. To solve the problem of target misidentification caused by new star points entering the field of view, a target-tracking link based on IMU prediction was introduced to track the position of the target in the image. Furthermore, a measurement model was constructed using the line-of-sight vector generated from target recognition, and the relative motion state was estimated using a Huber-based non-linear filter. Semi-physical and numerical simulations were performed to evaluate the effectiveness and efficiency of the proposed method.

AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

Effective data compression technology is essential for addressing data storage and transmission needs, especially given the escalating volume and complexity of data generated by contemporary astronomy. In this study, we propose utilizing deep learning-based lossless compression techniques to improve compression efficiency. We begin with a qualitative and quantitative analysis of the temporal and spatial redundancy in solar observation data. Based on this analysis, we introduce a novel deep learning-based framework called AstroDLLC for the lossless compression of astronomical solar images. AstroDLLC first segments high-resolution images into blocks to ensure that deep learning model training does not rely on high-computation power devices. It then addresses the non-normality of the partitioned data through simple reversible computational methods. Finally, it utilizes Bit-swap to train deep learning models that capture redundant features across multiple image frames, thereby enhancing compression efficiency. Comprehensive evaluations using data from the New Vacuum Solar Telescope reveal that AstroDLLC achieves a maximum compression ratio of 3.00 per image, surpassing Gzip, RICE, and other lossless technologies. The performance of AstroDLLC underscores its potential to address data compression challenges in astronomy.

Atmospheric and Radiometric Normalization of Satellite Images for Landscape-Level Environmental Monitoring: The Case of The Mediterranean Region

Ensuring atmospheric and radiometric consistency among the frameworks of satellite data used in regional studies is a critical requirement for change detection studies employed in regional planning monitoring. The purpose of this article is to provide a guide for the necessary atmospheric correction and radiometric normalization processes required in generating environmental data at the landscape level for physical planning. In this context, adjustments were made to remove atmospheric effects before merging multiple ASTER satellite image frames used in a project supported by TÜBİTAK, covering landscape-level environmental inventory and monitoring. The Dark Object Subtraction method with the Cos(t) model was utilized in the atmospheric correction process. Subsequently, separate regression relationships were computed for each band by considering overlapping areas on adjacent tracks of ASTER data, and radiometric normalization was performed based on these regression equations. Thus, differences between satellite images used in monitoring land changes and affecting multiple frames were minimized.

Towards Generating Authentic Human-Removed Pictures in Crowded Places Using a Few-Second Video.

If we visit famous and iconic landmarks, we may want to take a photo of them. However, such sites are usually crowded, and taking photos with only landmarks without people could be challenging. This paper aims to automatically remove people in a picture and produce a natural image of the landmark alone. To this end, it presents Thanos, a system to generate authentic human-removed images in crowded places. It is designed to produce high-quality images with reasonable computation cost using short video clips of a few seconds. For this purpose, a multi-frame-based recovery region minimization method is proposed. The key idea is to aggregate information partially available from multiple image frames to minimize the area to be restored. The evaluation result presents that the proposed method outperforms alternatives; it shows lower Fréchet Inception Distance (FID) scores with comparable processing latency. It is also shown that the images by Thanos achieve a lower FID score than those of existing applications; Thanos's score is 242.8, while those by Retouch-photos and Samsung object eraser are 249.4 and 271.2, respectively.

Workout Classification Using a Convolutional Neural Network in Ensemble Learning.

To meet the increased demand for home workouts owing to the COVID-19 pandemic, this study proposes a new approach to real-time exercise posture classification based on the convolutional neural network (CNN) in an ensemble learning system. By utilizing MediaPipe, the proposed system extracts the joint coordinates and angles of the human body, which the CNN uses to learn the complex patterns of various exercises. Additionally, this new approach enhances classification performance by combining predictions from multiple image frames using an ensemble learning method. Infinity AI's Fitness Basic Dataset is employed for validation, and the experiments demonstrate high accuracy in classifying exercises such as arm raises, squats, and overhead presses. The proposed model demonstrated its ability to effectively classify exercise postures in real time, achieving high rates in accuracy (92.12%), precision (91.62%), recall (91.64%), and F1 score (91.58%). This indicates its potential application in personalized fitness recommendations and physical therapy services, showcasing the possibility for beneficial use in these fields.

A feature map aggregation network for unconstrained video face recognition

Unconstrained video face recognition is an extension of face recognition technology, and it is an indispensable part of intelligent security and criminal investigation systems. However, general face recognition technology cannot be directly applied to unconstrained video face recognition, because the video contains fewer frontal face image frames and a single image contains less face feature information. To address the above problems, this work proposes a Feature Map Aggregation Network (FMAN) to achieve unconstrained video face recognition by aggregating multiple face image frames. Specifically, an image group is used as the input of the feature extraction network to replace a single image to obtain a multi-channel feature map group. Then a quality perception module is proposed to obtain quality scores for feature maps and adaptively aggregate image features from image groups at the feature map level. Finally, extensive experiments are conducted on the challenging face recognition benchmarks YTF, IJB-A and COX to evaluate the proposed method, showing a significant increase in accuracy compared to the state-of-the-art.

Maritime Tracking With Georeferenced Multi-Camera Fusion

Cameras form an essential part of any autonomous surface vehicle’s sensor package, both for COLREGs compliance to detect light signals and for identifying and tracking other vessels. Due to limited fields of view compared to more traditional autonomy sensors such as lidars and radars, an autonomous surface vessel will typically be equipped with multiple cameras which can induce biases when used in tracking if a target is present in multiple image frames. In this work, we propose a novel pipeline for camera-based maritime tracking that combines georeferencing with clustering-based multi-camera fusion for bias-free camera measurements with target range estimates. Using real-world datasets collected using the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">milliAmpere</i> research platform the performance of this pipeline exceeded a lidar benchmark across multiple performance measures, both in pure detection performance and as part of a JIPDA-based tracking system.

Laser Spot Centering Algorithm of Double-Area Shrinking Iteration Based on Baseline Method

High-precision laser spot center detection occupies an important position in optical measurement technology. In this paper, we propose a laser spot centering method to improve positioning accuracy. This method is an iterative double-area shrinkage approach based on the baseline method. The background noise baseline is calculated from the noise statistics of multiple background image frames acquired, and then the background noise is subtracted during the calculation while retaining the effective information of the spot region. The real spot area is located in the end by double-area shrinkage iteration to calculate the position of the spot center. Simulation and experimental results showed that our proposed method has strong anti-background noise interference ability, as well as higher positioning accuracy in locating the spot center than commonly used approaches; the maximum localization accuracy could reach 0.05 pixels, meeting the real-time requirements of the algorithm. The fluctuation range of measurement results was small when continuously detecting the center of the same laser spot, which could reach 0.04 and 0.03 pixels in the x- and y-directions, respectively. The result indicates that the method can meet the requirements of laser high-precision positioning.

A hybrid spatial–temporal deep learning architecture for lane detection

AbstractAccurate and reliable lane detection is vital for the safe performance of lane‐keeping assistance and lane departure warning systems. However, under certain challenging circumstances, it is difficult to get satisfactory performance in accurately detecting the lanes from one single image as mostly done in current literature. Since lane markings are continuous lines, the lanes that are difficult to be accurately detected in the current single image can potentially be better deduced if information from previous frames is incorporated. This study proposes a novel hybrid spatial–temporal (ST) sequence‐to‐one deep learning architecture. This architecture makes full use of the ST information in multiple continuous image frames to detect the lane markings in the very last frame. Specifically, the hybrid model integrates the following aspects: (a) the single image feature extraction module equipped with the spatial convolutional neural network; (b) the ST feature integration module constructed by ST recurrent neural network; (c) the encoder–decoder structure, which makes this image segmentation problem work in an end‐to‐end supervised learning format. Extensive experiments reveal that the proposed model architecture can effectively handle challenging driving scenes and outperforms available state‐of‐the‐art methods.

Three-dimensional reconstruction of wear particles by multi-view contour fitting and dense point-cloud interpolation

Three-dimensional (3D) surfaces of wear particles were reconstructed through multi-view contour fitting and dense point-cloud interpolation with an aim to obtain comprehensive features for wear debris analysis. Multiple image frames at different views were captured when the particles move with rotations through a micro-sized flow channel. The particle contours were extracted from multi-view images to build a 3D model based on particle contour fitting. The 3D modeling accuracy was then improved through an interpolated dense point-cloud. To validate the 3D model, an experiment was carried out to compare its performance to that of results obtained using laser scanning confocal microscopy. The results show that the errors of quantitative surface characterization using the arithmetical mean height (Sa) and the root-mean-square height (Sq) are less than 5.86%, and less than 17.12% for the kurtosis (Sku). The proposed method has great potential values in online condition monitoring of wear particles.

Deep Learning-Based Video System for Accurate and Real-Time Parking Measurement

Parking spaces are costly to build, parking payments are difficult to enforce, and drivers waste an excessive amount of time searching for empty lots. Accurate quantification would inform developers and municipalities in space allocation and design, while real-time measurements would provide drivers and parking enforcement with information that saves time and resources. In this paper, we propose an accurate and real-time video system for future Internet of Things (IoT) and smart cities applications. Using recent developments in deep convolutional neural networks (DCNNs) and a novel vehicle tracking filter, we combine information across multiple image frames in a video sequence to remove noise introduced by occlusions and detection failures. We demonstrate that our system achieves higher accuracy than pure image-based instance segmentation, and is comparable in performance to industry benchmark systems that utilize more expensive sensors such as radar. Furthermore, our system shows significant potential in its scalability to a city-wide scale and also in the richness of its output that goes beyond traditional binary occupancy statistics.

Cascaded Iterative Training Model and Parallel Multi-Classifiers for Visual Object Tracking

Because of the superiority in distinguishing the target from the background by learning mechanism, discriminative classifier attracts a great deal of attention in target detection and tracking systems. Generally, a discriminative classifier is learned from the initial image patch. However, the other algorithms, using only one image frame inhibit the performance of the classifier. A novel training strategy of cascade iteration, which is based on the multiple image frames, is used to initialize the discriminative classifier for the purpose of weakening the initialization limitation and improving the algorithms' adaptability and robustness. Furthermore, to cope with the changes in natural images, online learning is usually adopted to enhance the performance of classifiers, but it easily leads to tracking failures, especially in the case of full occlusion. In order to attenuate the influence of full occlusion, a multi-classifiers parallel tracking algorithm is proposed by designing the multi-classifiers and inducing extra constraints. The experiments are performed on the visual tracking benchmark, and the proposed algorithm is more robust while the target is fully occluded. The simulation results show that the proposed algorithm exhibits better performance compared to the state-of-the-art trackers, especially with KCF, TLD, and STRUCK.

Proximal robust factorization for piecewise planar reconstruction

In this paper, we aim to obtain a dense piecewise planar reconstruction of the scene from multiple image frames based on a factorization framework. Integrating all the relevant constraints in a global objective function, we are able to effectively leverage on the scene smoothness prior afforded by the dense formulation, as well as imposing the necessary algebraic constraints required by the shape matrix. These constraints also help to robustly decompose the measurement matrix into the underlying low-rank subspace and the sparse outlier part. Numerically, we achieve the constrained factorization and decomposition via modifying a recently proposed proximal alternating robust subspace minimization algorithm. The results show that our algorithm is effective in handling real life sequences, and outperforms other algorithms in recovering motions and dense scene estimate.

Diffeomorphic Multi-Frame Non-Rigid Registration of Cell Nuclei in 2D and 3D Live Cell Images.

To gain a better understanding of cellular and molecular processes, it is important to quantitatively analyze the motion of subcellular particles in live cell microscopy image sequences. Since, generally, the subcellular particles move and cell nuclei move as well as deform, it is important to decouple the movement of particles from that of the cell nuclei using non-rigid registration methods. We have developed a diffeomorphic multi-frame approach for non-rigid registration of cell nuclei in 2D and 3D live cell fluorescence microscopy images. Our non-rigid registration approach is based on local optic flow estimation, exploits information from multiple consecutive image frames, and determines diffeomorphic transformations in the log-domain, which allows efficient computation of the inverse transformations. To register single images of an image sequence to a reference image, we use a temporally weighted mean image, which is constructed based on inverse transformations and multiple consecutive frames. Using multiple consecutive frames improves the registration accuracy compared to pairwise registration, and using a temporally weighted mean image significantly reduces the computation time compared with previous work. In addition, we use a flow boundary preserving method for regularization of computed deformation vector fields, which prevents from over-smoothing compared to standard Gaussian filtering. Our approach has been successfully applied to 2D and 3D synthetic as well as real live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise, multi-frame, and temporal groupwise registration has been carried out.

The performance analysis of three-dimensional track-before-detect algorithm based on Fisher-Tippett-Gnedenko theorem

The dim moving target tracking from the infrared image sequence in the presence of high clutter and noise has been recently under intensive investigation. The track-before-detect (TBD) algorithm processing the image sequence over a number of frames before decisions on the target track and existence is known to be especially attractive in very low SNR environments (⩽3dB). In this paper, we shortly present a three-dimensional (3-D) TBD with dynamic programming (TBD-DP) algorithm using multiple IR image sensors. Since traditional two-dimensional TBD algorithm cannot track and detect the along the viewing direction, we use 3-D TBD with multiple sensors and also strictly analyze the detection performance (false alarm and detection probabilities) based on Fisher-Tippett-Gnedenko theorem. The 3-D TBD-DP algorithm which does not require a separate image registration step uses the pixel intensity values jointly read off from multiple image frames to compute the merit function required in the DP process. Therefore, we also establish the relationship between the pixel coordinates of image frame and the reference coordinates.

Chemical exchange saturation transfer (CEST) imaging with fast variably-accelerated sensitivity encoding (vSENSE).

The widespread clinical use of chemical exchange saturation transfer (CEST) imaging is hampered by relatively long scan times due to its requirement that multiple saturation-offset image frames be acquired. Here, a novel variably-accelerated sensitivity encoding (vSENSE) method is proposed that provides faster CEST acquisition than conventional SENSE. The vSENSE method fully samples one CEST saturation frame, then undersamples the other frames variably. The fully-sampled frame, in conjunction with newly proposed incoherence absorption and artifact suppression strategies, improves the accuracy of sensitivity maps and permits higher acceleration factors for the other undersampled frames than regular SENSE. vSENSE is validated in a phantom, a normal volunteer and eight brain tumor patients at 3 Tesla. vSENSE with an acceleration factor of four generated a 3-6 times smaller error on average than conventional SENSE (P ≤ 0.02), with acceleration factors of 2-4, as compared with full k-space reconstruction. vSENSE permitted four-fold acceleration for amide proton transfer-weighted images, while regular SENSE could only provide a factor of two. When conventional SENSE is used with vSENSE's variable undersampling pattern, erroneous (∼9%) z-spectra result. The vSENSE method enabled twice the acceleration and generated more accurate images than conventional SENSE. Magn Reson Med 77:2225-2238, 2017. © 2016 International Society for Magnetic Resonance in Medicine.

Compressive Parameter Estimation for Correlated Frames in MIMO Visible Light Communications

Estimation of the image location of a mobile LED array transmitter on the camera sensor plane of a digital camera is considered. A Bayesian framework is used to design an initial large measurement matrix over multiple image frames to exploit inter-frame temporal correlations. This large matrix is then factorized into two component matrices; one is used as the front-end measurement matrix with the potential for hardware implementation while the other is designed for software implementation. The factorization can be performed offline through fixed point iterations. The proposed measurement matrix design directly exploits possible transmitter mobility through a parametric framework. The findings are validated through analysis, computer simulation and hardware experiments. Sub-pixel localization accuracy with significant compression is achieved, and the proposed compressive parameter estimation outperforms results obtained using a Gaussian measurement matrix.

VIDEO-BASED POINT CLOUD GENERATION USING MULTIPLE ACTION CAMERAS

Abstract. Due to the development of action cameras, the use of video technology for collecting geo-spatial data becomes an important trend. The objective of this study is to compare the image-mode and video-mode of multiple action cameras for 3D point clouds generation. Frame images are acquired from discrete camera stations while videos are taken from continuous trajectories. The proposed method includes five major parts: (1) camera calibration, (2) video conversion and alignment, (3) orientation modelling, (4) dense matching, and (5) evaluation. As the action cameras usually have large FOV in wide viewing mode, camera calibration plays an important role to calibrate the effect of lens distortion before image matching. Once the camera has been calibrated, the author use these action cameras to take video in an indoor environment. The videos are further converted into multiple frame images based on the frame rates. In order to overcome the time synchronous issues in between videos from different viewpoints, an additional timer APP is used to determine the time shift factor between cameras in time alignment. A structure form motion (SfM) technique is utilized to obtain the image orientations. Then, semi-global matching (SGM) algorithm is adopted to obtain dense 3D point clouds. The preliminary results indicated that the 3D points from 4K video are similar to 12MP images, but the data acquisition performance of 4K video is more efficient than 12MP digital images.

Gaze behavior data profiling and analysis system platform based on visual content representation

This paper presents a real-time system platform for profiling and analyzing the gaze behavior based on visual contents. The proposed system captures the gaze information from multiple users and provides the ability to measure the degree of visual content perception by the users through statistical analysis. Visual content representation scheme is presented for capturing and annotating the gaze behavior effectively. Information correlation property among multiple image frames is defined for providing the ability to analyze the pattern of perception of user based on complex visual contents. In order to monitor the real time gaze behavior of the users, the monitoring rules are incorporated in to the representation template. The number of users and the duration of observation time may significantly increase the profile data size. To alleviate the problem, an adaptive data compression techniques is incorporated. The capabilities and functionalities of the proposed system are verified for multiple users with different gaze behavior on a sequence of commercial image data.

SU-E-J-151: An Off-Line QA Tool for Evaluating Reproducibility of Deep Inhalation Breath-Hold Treatment for Breast Radiotherapy

Purpose: To develop an off‐line QA tool for evaluating reproducibility of deep inhalation breath hold (DIBH) breast treatment using a spirometer‐based breathing control system with video guidance. Methods: Tangential beam plans (open medial field; lateral IMRT field) used for whole breast radiation with DIBH‐guided by a spirometer‐based breathing control system have been studied. An imaging sequence was set‐up at the treatment machine to acquire multiple (>15) EPID image frames during beam‐on with source‐to‐imager distance 150cm. Images were exported from the EPID and imported into an off‐line QA tool developed in‐house for evaluation of intra‐and inter‐fractional uncertainties for DIBH treatments. The user defines a region of interest (ROI) (e.g., expander, clip(s), chestwall) on the first (reference) image. A rectangular imaging template is created from the ROI with a defined expansion (e.g., 1cm). The template is then matched to each of the following image frames using cross‐correlation. Template movement is used to estimate breast motion during one or multiple breath‐hold sessions. The breath‐hold lung volume information is extracted from the spirometer and compared to the corresponding motion curve estimated from the EPID images. Results: Motions of different regions of the breast can be visualized and shown to be constant within 1mm at the imager (∼.7mm in the breast) for some patients. Both drifting and random motions can be identified on individual patients on specific days. Off‐line analysis of these data may provide a useful tool to assure that individual patients are adequately performing the DIBH technique, and to identify patients who require new margins or planning strategies. Conclusion: An off‐line QA motion evaluation tool has been designed for DIBH breast treatment, and is an effective method for evaluation of DIBH reproducibility both intra‐and inter‐fraction.