Pair Of Frames Research Articles

Based on improved joint detection and tracking of UAV for multi-target detection of livestock

In agriculture, specifically livestock monitoring, drones' ability to track multiple targets is essential for advancing the field. However, limited computing resources and unpredictable drone movements often cause issues like ambiguous video frames, object obstructions, and size deviations. These inconsistencies reduce tracking accuracy, making traditional algorithms inadequate for handling drone footage. This study introduces an enhanced deep learning-based multi-target drone tracker framework that enables real-time processing. The proposed method combines object detection and tracking by leveraging consecutive frame pairs to extract and share features, enhancing computational efficiency. It employs diverse loss functions to address class and sample distribution imbalances and includes a composite deblurring module to enhance detection accuracy. Object association utilizes a dual regress bounding box technique, aiding in object identification verification and predictive motion. Live tracking is achieved by predicting object locations in subsequent frames, enabling real-time tracking. Evaluation against leading benchmarks shows that the system improves precision and speed, achieving a 4.3 % increase in Multi-Object Tracking Accuracy (MOTA) and a 7.7 % boost in F1 score.

Investigation of the impact of local properties of 3D data on the accuracy of block matching-based speckle tracking echocardiography

Abstract Introduction: Due to its simplicity, block-matching is a popular motion-tracking method used in speckle-tracking echocardiography. Improvement of its robustness and accuracy is thus of prime interest. Although it seems plausible that the quality of block matching-based tracking depends on the local properties of image data, and thus, it should be possible to assess in advance how well certain portions of the image data are suited for displacement estimation, the potential relationship has not been studied extensively. Material and methods: This study aimed to search for a relationship between selected features of echocardiographic data and the quality of local displacement estimation. The study used a 3D synthetic B-mode imaging sequence data with known ground truth. Frame-to-frame displacements were estimated for 9856 points in five different frame pairs with mean inter-frame displacements of 0.15, 0.87, 2, 3.02, and 3.84 mm. In each case, tracking errors were evaluated against thirteen grayscale image features, the displacement’s magnitudes, and the normalized cross-correlation (NCX) values. Additionally, a multi-variable regression model was applied to test the combined ability of the proposed features to predict tracking quality. Results: Median tracking error magnitudes were 0.06, 0.13, 0.28, 0.74, and 1.5 mm for each image pair. Weak correlation between errors and individual data features was found only in the case of 3 features: NCX (Pearson’s correlation coefficients in the range of −0.366 to −0.223), number of speckles within the kernel (−0.283, −0.282, and −0.214 for three lowest deformations) and mean of the 3D gradient (−0.252, −0.237 and −0.25). The regression model, however, provided significant prediction improvement with R2 exceeding 0.5. Conclusions: In conclusion, only a weak relationship between the individual investigated kernel features and tracking accuracy has been established, but their combined strength can be assessed as at least moderate.

Self-supervised multi-object tracking based on metric learning

The current paradigm of joint detection and tracking still requires a large amount of instance-level trajectory annotation, which incurs high annotation costs. Moreover, treating embedding training as a classification problem would lead to difficulties in model fitting. In this paper, we propose a new self-supervised multi-object tracking based on the real-time joint detection and embedding (JDE) framework, which we termed as self-supervised multi-object tracking (SS-MOT). In SS-MOT, the short-term temporal correlations between objects within and across adjacent video frames are both considered as self-supervised constraints, where the distances between different objects are enlarged while the distances between same object of adjacent frames are brought closer. In addition, short trajectories are formed by matching pairs of adjacent frames using a matching algorithm, and these matched pairs are treated as positive samples. The distances between positive samples are then minimized for futher the feature representation of the same object. Therefore, our method can be trained on videos without instance-level annotations. We apply our approach to state-of-the-art JDE models, such as FairMOT, Cstrack, and SiamMOT, and achieve comparable results to these supevised methods on the widely used MOT17 and MOT20 challenges.

Self-supervised rigid object 3-D motion estimation from monocular video

Independent object 3D motion estimation is a fundamental problem in 3D computer vision. Directly segmenting and estimating rigid object 3D motion from a consistent video frame is an ill-posed problem. We present a self-supervised framework for segmenting moving independent rigid objects and estimating their motion information including location, driving direction, and speed from a monocular video. Specifically, we first estimate depth, optical flow, and camera pose between a pair of video frames, and then synthesize a new 3D viewpoint from this pair. Subsequently, the Motion Recurrent All-Pairs Field Transforms (MRAFT) module is introduced to extract 3D scene flow and a motion area binary mask from a pair of images and depth. After that, a Rigid Object Motion Estimation Module (ROMEM) with a slot attention mechanism is proposed to extract rigid object motion masks from a multi-layer motion field, including optical flow, depth changes, refined scene flow, and motion masks. Finally, 2D images and 3D scene reconstruction errors are used to facilitate self-supervised training for rigid object motion. Experiments on the FlyingThings3D and KITTI datasets demonstrate that our method outperforms other advanced algorithms in estimating depth, optical flow, scene flow, and rigid moving object masks, demonstrating the benefits of our approach.

Perceiving Actions via Temporal Video Frame Pairs

Video action recognition aims at classifying the action category in given videos. In general, semantic-relevant video frame pairs reflect significant action patterns such as object appearance variation and abstract temporal concepts like speed, rhythm, and so on. However, existing action recognition approaches tend to holistically extract spatiotemporal features. Though effective, there is still a risk of neglecting the crucial action features occurring across frames with a long-term temporal span. Motivated by this, in this article, we propose to perceive actions via frame pairs directly and devise a novel Nest Structure with frame pairs as basic units. Specifically, we decompose a video sequence into all possible frame pairs and hierarchically organize them according to temporal frequency and order, thus transforming the original video sequence into a Nest Structure. Through naturally decomposing actions, the proposed structure can flexibly adapt to diverse action variations such as speed or rhythm changes. Next, we devise a Temporal Pair Analysis module (TPA) to extract discriminative action patterns based on the proposed Nest Structure. The designed TPA module consists of a pair calculation part to calculate the pair features and a pair fusion part to hierarchically fuse the pair features for recognizing actions. The proposed TPA can be flexibly integrated into existing backbones, serving as a side branch to capture various action patterns from multi-level features. Extensive experiments show that the proposed TPA module can achieve consistent improvements over several typical backbones, reaching or updating CNN-based SOTA results on several challenging action recognition benchmarks.

TQU-SLAM Benchmark Dataset for Comparative Study to Build Visual Odometry Based on Extracted Features from Feature Descriptors and Deep Learning

The problem of data enrichment to train visual SLAM and VO construction models using deep learning (DL) is an urgent problem today in computer vision. DL requires a large amount of data to train a model, and more data with many different contextual and conditional conditions will create a more accurate visual SLAM and VO construction model. In this paper, we introduce the TQU-SLAM benchmark dataset, which includes 160,631 RGB-D frame pairs. It was collected from the corridors of three interconnected buildings comprising a length of about 230 m. The ground-truth data of the TQU-SLAM benchmark dataset were prepared manually, including 6-DOF camera poses, 3D point cloud data, intrinsic parameters, and the transformation matrix between the camera coordinate system and the real world. We also tested the TQU-SLAM benchmark dataset using the PySLAM framework with traditional features such as SHI_TOMASI, SIFT, SURF, ORB, ORB2, AKAZE, KAZE, and BRISK and features extracted from DL such as VGG, DPVO, and TartanVO. The camera pose estimation results are evaluated, and we show that the ORB2 features have the best results (Errd = 5.74 mm), while the ratio of the number of frames with detected keypoints of the SHI_TOMASI feature is the best (rd=98.97%). At the same time, we also present and analyze the challenges of the TQU-SLAM benchmark dataset for building visual SLAM and VO systems.

Frames and system of translates of Hilbert C⁎-module valued functions

In this paper, we investigate three types of problems dealing with frames in L2(Rn,H), where H denotes a Hilbert C⁎-module. First, we obtain a characterization for the system of translates {Tkϕ:k∈Z} to be a frame sequence. Then we investigate the system of generalized translates GJ associated with a non-singular matrix D. Here we obtain a characterization for GJ to be a frame for L2(Rn,H). Given two systems of such translates, we obtain a necessary and sufficient condition for such systems to form a pair of dual frames. Finally, we study semi-discrete frames for L2(Rn,H), where H is a separable Hilbert space.

A Deep Learning Framework for Start-End Frame Pair-Driven Motion Synthesis.

A start-end frame pair and a motion pattern-based motion synthesis scheme can provide more control to the synthesis process and produce content-various motion sequences. However, the data preparation for the motion training is intractable, and concatenating feature spaces of the start-end frame pair and the motion pattern lacks theoretical rationality in previous works. In this article, we propose a deep learning framework that completes automatic data preparation and learns the nonlinear mapping from start-end frame pairs to motion patterns. The proposed model consists of three modules: action detection, motion extraction, and motion synthesis networks. The action detection network extends the deep subspace learning framework to a supervised version, i.e., uses the local self-expression (LSE) of the motion data to supervise feature learning and complement the classification error. A long short-term memory (LSTM)-based network is used to efficiently extract the motion patterns to address the speed deficiency reflected in the previous optimization-based method. A motion synthesis network consists of a group of LSTM-based blocks, where each of them is to learn the nonlinear relation between the start-end frame pairs and the motion patterns of a certain joint. The superior performances in action detection accuracy, motion pattern extraction efficiency, and motion synthesis quality show the effectiveness of each module in the proposed framework.

Joint Stroke Tracing and Correspondence for 2D Animation

To alleviate human labor in redrawing keyframes with ordered vector strokes for automatic inbetweening, we for the first time propose a joint stroke tracing and correspondence approach. Given consecutive raster keyframes along with a single vector image of the starting frame as a guidance, the approach generates vector drawings for the remaining keyframes while ensuring one-to-one stroke correspondence. Our framework trained on clean line drawings generalizes to rough sketches, and the generated results can be imported into inbetweening systems to produce inbetween sequences. Hence, the method is compatible with standard 2D animation workflow. An adaptive spatial transformation module (ASTM) is introduced to handle non-rigid motions and stroke distortion. We collect a dataset for training with 10k+ pairs of raster frames and their vector drawings with stroke correspondence. Comprehensive validations on real clean and rough animated frames manifest the effectiveness of our method and superiority to existing methods.

Efficient Search and Detection of Faint Moving Objects in Image Data

The search and detection of faint moving objects in image data can enable discoveries of small solar system bodies. To detect objects fainter than the single-frame sensitivity limit, track-before-detect methods can improve the signal-to-noise ratio of the object of interest by incoherently adding the object’s signal across multiple frames. However, traditional track-before-detect techniques can become computationally intensive over large search volumes. In this work, we present a computational approach to significantly speed up the search process by applying dynamic-programming techniques to implement the discrete X-ray transform. In this approach, image frames are processed in stages, in each of which pairs of frames are shifted and added to generate short-track segments, which are combined in later stages to form longer tracks. The algorithm speedup comes from the fact that a single short track segment can be reused multiple times for different longer tracks without the need for recomputing. Benchmark testing with simulated data shows that the method presented in this paper results in a significant reduction in runtime in comparison to a traditional track-before-detect approach. As a proof of concept, we demonstrated the applicability of the technique in performing a blind search for faint asteroids in image data collected from the Transiting Exoplanet Survey Satellite, leading to the detection of more than a thousand asteroids below the single-frame detection limit with moderate computational resources. The approach presented in this work has the potential to enable efficient discovery of previously undetected faint solar system objects across multiple orbit classes.

Video Frame Interpolation With Many-to-Many Splatting and Spatial Selective Refinement.

In this work, we first propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. Given a frame pair, we estimate multiple bidirectional flows to directly forward warp the pixels to the desired time step before fusing any overlapping pixels. In doing so, each source pixel renders multiple target pixels and each target pixel can be synthesized from a larger area of visual context, establishing a many-to-many splatting scheme with robustness to undesirable artifacts. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames, hence achieving fast multi-frame interpolation. However, directly warping and fusing pixels in the intensity domain is sensitive to the quality of motion estimation and may suffer from less effective representation capacity. To improve interpolation accuracy, we further extend an M2M++ framework by introducing a flexible Spatial Selective Refinement (SSR) component, which allows for trading computational efficiency for interpolation quality and vice versa. Instead of refining the entire interpolated frame, SSR only processes difficult regions selected under the guidance of an estimated error map, thereby avoiding redundant computation. Evaluation on multiple benchmark datasets shows that our method is able to improve the efficiency while maintaining competitive video interpolation quality, and it can be adjusted to use more or less compute as needed.

UAS-based methodology to create digital models of bridges and viaducts

The use of Unmanned Aerial Systems allows to exploit the aerial photogrammetry technique to obtain high-resolution orthophotos with large images, owing to latest generation payloads. This approach allows to acquire metric data of a structure through the analysis of pairs of stereometric frames. Aerial photogrammetry forms the basis for the creation of point cloud data models used to develop real-time digital replication of objects (or Digital Twins systems)to monitor, analyze and simulate their behavior in the real world, along with Finite Element models for structural analysis, material simulation and design of complex components. In addition, BIM models are developed that integrate detailed information on infrastructure geometry, materials, and performance into a single collaborative 3D model, for improved management and planning during the bridge lifespan. The results obtained by applying these techniques to an Italian bridge will be examined, providing crucial support to experts in infrastructure assessment and monitoring.

Review Paper on IoT Thread Detection using Deep CNN Classifier

Abnormal activity will lead to uncommon changes in the crowd behavior. In other words, the crowd motion changes conform to certain rules for valid behaviors, while for abnormal events the motion changes are uncontrolled. The motion-changed rules to detect and localize abnormal behavior in crowd videos. Specifically, we first generate the motion patterns based on the descriptor of collectiveness. Then each frame pair is represented as a transfer matrix whose elements are the difference of a set of motion patterns. Thereafter, the motion-changed rules are constructed in the transformation space using a bag-of-words approach. Finally, the proposed approach measures the similarity between motion-changed rules and the incoming video data to examine whether the actions are anomalous. The approach is tested on the UMN dataset and a challenging dataset of crowd videos taken from the railway station. The experimental results demonstrate the effectiveness of the proposed method for detection abnormal behavior

A Fast Method for Robust Video Watermarking Based on Zernike Moments

Watermarking by Zernike moments has been proven to be effective in providing high rotational resistance. However, due to the high computational complexity, the conventional video watermarking methods using Zernike moments are developed for videos with low resolution. Moreover, according to the properties of Zernike moments, only the matrices of equal height and width can be calculated since the inscribed circle of the original image matrix is selected as the area to be processed, but most of the available videos on the Internet do not meet such requirement. To solve the above problem, this paper proposes a fast watermarking method based on Zernike moments for high resolution videos to resist various attacks. In the proposed method, the frames of a video sequence are firstly grouped, from which a certain number of frame pairs are then selected for watermark embedding. For each frame pair to be embedded, we partition one frame into a set of disjoint blocks and apply singular value decomposition to each block to obtain a square feature matrix. Thereafter, by calculating all the Zernike moments, secret information is embedded into the selected Zernike moments to achieve superior robustness while keeping imperceptibility. Finally, according to the video encoding framework, we overwrite the frame difference of the frame pair by the watermark to resist compression and transcoding attacks. Furthermore, we propose two optional methods for compensating the special cases of rotation and scaling attacks during watermark detection. Experiments demonstrate the advantage of our method over the existing robust watermarking methods.

Quantitative bounds for unconditional pairs of frames

We formulate a quantitative finite-dimensional conjecture about frame multipliers and prove that it is equivalent to Conjecture 1 in [22]. We then present solutions to the conjecture for certain classes of frame multipliers. In particular, we prove that for all C>0 and N∈N the following is true: Let (xj)j=1N and (fj)j=1N be sequences in a finite dimensional Hilbert space which satisfy ‖xj‖=‖fj‖ for all 1≤j≤N and‖∑j=1Nεj〈x,fj〉xj‖≤C‖x‖,for all x∈ℓ2M and |εj|=1. If the frame operator for (fj)j=1N has eigenvalues λ1≥...≥λM and β>0 is such that λ1≤βM−1∑j=1Mλj then (fj)j=1N has Bessel bound 27β2C. The same holds for (xj)j=1N.

Characterization of rationally sampled quaternionic dual Gabor frames

This paper addresses quaternionic dual Gabor frames under the condition that the products of time‐frequency shift parameters are rational numbers. For a general overcomplete quaternionic Gabor frame with the product of time‐frequency shift parameters not equal to , we show that its corresponding frame and translation operators do not commute, which leads to its canonical dual frame not having the Gabor structure, but it may have other dual frames with Gabor structure. We characterize when two quaternionic Gabor Bessel sequences form a pair of dual frames, and present a class of quaternionic dual Gabor frames. We also characterize quaternionic Gabor Riesz bases and prove that their canonical dual frames have Gabor structure.

Gabor's “complex signal” revisited: Complexifying frames and bases

AbstractIn 1946, Dennis Gabor introduced the analytic signal for real‐valued signals f. Here, H is the Hilbert transform. This complexification of functions allows for an analysis of their amplitude and phase information and has ever since given well‐interpretable insight into the properties of the signals over time. The idea of complexification has been reconsidered with regard to many aspects: examples are the dual tree complex wavelet transform, or via the Riesz transform and the monogenic signal, that is, a multi‐dimensional version of the Hilbert transform, which in combination with multi‐resolution approaches leads to Riesz wavelets, and others. In this context, we ask two questions: Which pairs of real orthonormal bases (ONBs), Riesz bases, frames and Parseval frames and can be “rebricked” to complex‐valued ones ? And which real operators A allow for rebricking via the ansatz ? In this short note, we give answers to these questions with regard to a characterization which linear operators A are suitable for rebricking while maintaining the structure of the original real valued family. Surprisingly, the Hilbert transform is not among them.

Video Global Motion Compensation Based on Affine Inverse Transform Model.

Global motion greatly increases the number of false alarms for object detection in video sequences against dynamic backgrounds. Therefore, before detecting the target in the dynamic background, it is necessary to estimate and compensate the global motion to eliminate the influence of the global motion. In this paper, we use the SURF (speeded up robust features) algorithm combined with the MSAC (M-Estimate Sample Consensus) algorithm to process the video. The global motion of a video sequence is estimated according to the feature point matching pairs of adjacent frames of the video sequence and the global motion parameters of the video sequence under the dynamic background. On this basis, we propose an inverse transformation model of affine transformation, which acts on each adjacent frame of the video sequence in turn. The model compensates the global motion, and outputs a video sequence after global motion compensation from a specific view for object detection. Experimental results show that the algorithm proposed in this paper can accurately perform motion compensation on video sequences containing complex global motion, and the compensated video sequences achieve higher peak signal-to-noise ratio and better visual effects.

Multi-object tracking via deep feature fusion and association analysis

We describe a tracking-by-detection framework for multi-object tracking (MOT). It first detects the objects of interest in each frame of the video, followed by identifying association with the object detected in the previous frame. A deep association network is described to perform object feature matching in the arbitrary two frames to infer association degree of objects, and then similarity matrix loss is used to calculate association between each object in different frames to achieve an accurate tracking. The novelty of the work lies in the design of a multi-scale fusion strategy by gradually concatenating sub-networks of low-resolution feature maps in parallel to the main network of high-resolution feature maps, in the construction of a deeper backbone network which can enhance the semantic information of object features, and in the use of a siamese network for training a pair of discontinuous frames. The main advantage of our framework is that it avoids missing detection and partial detection. It is particularly suitable for solving the problem of object ID switch caused by occlusion, entering and leaving of objects. Our method is evaluated and demonstrated on the publicly available MOT15, MOT16, MOT17 and MOT20 benchmark datasets. Compared with the state-of-the-art methods, our method achieves better tracking performance, and is therefore, more suited for MOT tasks.

Semi-supervised learning improves the performance of cardiac event detection in echocardiography

Detection of end-diastole (ED) and end-systole (ES) frames in echocardiography video is a critical step for assessment of cardiac function. A recently released large public dataset, i.e., EchoNet-Dynamic, could be used as a benchmark for cardiac event detection. However, only a pair of ED and ES frames are annotated in each echocardiography video and the annotated ED comes before ES in most cases. This means that only a few frames during systole in each video are utilizable for training, which makes it challenging to train a cardiac event detection model using the dataset. Semi-supervised learning (SSL) could alleviate the problems. An architecture combining convolutional neural network (CNN), recurrent neural network (RNN) and fully-connected layers (FC) is adopted. Experimental results indicate that SSL brings at least three benefits: faster convergence rate, performance improvement and more reasonable volume curves. The best mean absolute errors (MAEs) for ED and ES detection are 40.2 ms (2.1 frames) and 32.6 ms (1.7 frames), respectively. In addition, the results show that models trained on apical four-chamber (A4C) view could work well on other standard views, such as other apical views and parasternal short axis (PSAX) views.