Motion Vectors Research Articles

MNSS: Neural Supersampling Framework for Real-Time Rendering on Mobile Devices.

Although neural supersampling has achieved great success in various applications for improving image quality, it is still difficult to apply it to a wide range of real-time rendering applications due to the high computational power demand. Most existing methods are computationally expensive and require high-performance hardware, preventing their use on platforms with limited hardware, such as smartphones. To this end, we propose a new supersampling framework for real-time rendering applications to reconstruct a high-quality image out of a low-resolution one, which is sufficiently lightweight to run on smartphones within a real-time budget. Our model takes as input the renderer-generated low resolution content and produces high resolution and anti-aliased results. To maximize sampling efficiency, we propose using an alternate sub-pixel sample pattern during the rasterization process. This allows us to create a relatively small reconstruction model while maintaining high image quality. By accumulating new samples into a high-resolution history buffer, an efficient history check and re-usage scheme is introduced to improve temporal stability. To our knowledge, this is the first research in pushing real-time neural supersampling on mobile devices. Due to the absence of training data, we present a new dataset containing 57 training and test sequences from three game scenes. Furthermore, based on the rendered motion vectors and a visual perception study, we introduce a new metric called inter-frame structural similarity (IF-SSIM) to quantitatively measure the temporal stability of rendered videos. Extensive evaluations demonstrate that our supersampling model outperforms existing or alternative solutions in both performance and temporal stability.

Cost-Effective Ground-Moving Object Detection Method in Aerial Video by Change Detection of Delaunay Triangulation

This paper is dedicated to developing a cost-effective ground-moving object detection method in aerial videos. Without limitations on types, quantity, and distribution of moving objects, which were required by the previous methods, the proposed approach can detect various ground-moving objects in aerial videos captured through diverse flying states in various ground appearances. The proposed method is mainly composed of generation of the appropriate feature points, detection of moving objects, and target tracking. In the originality of our work, a novel detection strategy designed for ground-moving objects is based on change detection of Delaunay triangulation (CDDT) and a three-step motion vector search-based tracking algorithm is further exploited for enhancing the detection rate. Experimental results show that our method can achieve the detection rate of at least 95% (roughly similar to the famous existing state-of-the-art methods) and 0.03 second/frame (far less than the famous existing methods) using test videos (containing only several moving objects distributed in a sparse space) in the previous methods compared. Besides, the average detection rate of 86.81%, average false detection rate of 9.44%, and a frame rate of about 33[Formula: see text]fps can be obtained using our test videos captured in the complicated ground appearances. This result makes the proposed method more attractive for detecting various ground-moving objects in aerial videos, when compared to other approaches, and can also achieve cost-effective performance.

Block based motion estimation model using CNN with representative point matching algorithm for object tracking in videos

Motion estimation is considered significant for tracking the movement of an object in video sequences, and it is widely used in various video processing applications. Traditionally, many researchers focus on pixel-based motion estimation for object tracking, but it experienced increased computation time and cost. To reduce computation time, the utilization of a block-based motion estimation approach for object tracking is a recent trend. The existing block-based approach faces difficulty in finding representative points within the intensity domain. Therefore, this current research merged the deep learning approach with a block-matching algorithm for achieving efficient object tracking. In this proposed work, initially, video sequences are collected from a benchmark video dataset. Then, the acquired video sequences are segmented into frames. From the segmented frames, current and previous frames are considered for motion estimation. Frames are sent for the data augmentation process in which the process of flipping, cropping, and rotation is carried out. Then, the augmented frames are sent into Convolutional Neural Network (CNN) for feature extraction. Representative Point Matching (RPM) is used to estimate the motion vector based on the extracted features. After estimating the motion vector, the similarity between two consecutive frames is found using Structural Similarity Index (SSIM) technique. Finally, based on the similarity score, the movement of an object in the video is tracked effectively. Simulation analysis of the proposed block-based motion estimation model is done by evaluating some performance metrics. RMSE, PSNR, Execution Time, SSIM, and accuracy obtained for the proposed model are 27.5, 26.5 db, 31 sec, 0.91, and 94 %. This analysis suggested that the proposed CNN-RPM motion estimation model performs better in tracking the movement of the object.

Evolving Particles in the 2022 Hunga Tonga—Hunga Ha'apai Volcano Eruption Plume

AbstractThe Multi‐angle Imaging SpectroRadiometer (MISR) aboard NASA's Terra satellite observed the Hunga Tonga—Hunga Ha'apai (HTHH) 15 January eruption plume on eight occasions between 15 and 23 January 2022. From the MISR multi‐angle, multi‐spectral imagery we retrieve aerosol plume height geometrically, along with plume‐level motion vectors, and derive radiometrically constraints on particle effective size, shape, and light‐absorption properties. Parts of two downwind aerosol layers were observed in different places and times, one concentrated in the upper troposphere (11–18 km ASL), and a mid‐stratosphere layer ∼23–30+ km ASL. After the initial day (1/15), the retrievals identified only spherical, non‐light‐absorbing particles, typical of volcanic sulfate/water particles. The near‐tropopause plume particles show constant, medium‐small (several tenths of a micron) effective size over 4 days. The mid‐stratosphere particles were consistently smaller, but retrieved effective particle size increased between 1/17 and 1/23, though they might have decreased slightly on 1/22. As a vast amount of water was also injected into the stratosphere by this eruption, models predicted relatively rapid sulfate particle growth from the modest amounts of SO2 gas injected by the eruption to high altitudes along with the water (Zhu et al., 2022, https://doi.org/10.5194/acp‐22‐10267‐2022). MISR observations up to 10 days after the eruption are consistent with these model predictions. The possible decrease in stratospheric particle size after initial growth was likely caused by evaporation, as the plume mixed with drier, ambient air. Particles in the lower‐elevation plume observed on 1/15 were larger than all the downwind aerosols and contained significant non‐spherical (likely ash) particles.

Convective Phenomenes in the Context of Meso-β-scale Convective Structures

This review article presents the results of radar studies of convective phenomena in Moldavia and the North Caucasus using Eulerian (ECS) and Lagrangian (LCS) coordinate systems. Application of the Lagrangian approach allowed us to exclude the influence of the tropospheric displacement and to obtain integral grid patterns of thunderstorm-hail processes. These structures are especially well manifested at small wind shears (up to 1 m/sec/km). At large shears, the mesh structures are transformed predominantly into linear structures. The methodology for obtaining integral pictures of radio echoes of thunderstorm processes is described. Intersections of linear elements, which we call faces, occur at nodes. The latter plays a particularly important role in the dynamics and kinematics of convective storms. The latter play a particularly important role in the dynamics and kinematics of convective storms. The development of storms occurs along the facets and at the nodes of meso-β-scale convective structures (MMCS), which explains the mechanisms of splitting and merging of storms: in the first case the facets diverge, in the second case they converge. The relations of motion vectors for different types of storms are obtained. It is shown that the direction of the radio echo canopy coincides with the storm motion trajectory; the evolution vector (propagation) for the most powerful storms deviates from the storm displacement direction by 80°–135°. The structure of the updated band within which the Flanking Line is formed for supercells and multicells is studied. Mnemonic rules have been derived that allow one to infer from instantaneous patterns of anvil orientation and the mutual location of storms whether they are converging or diverging, and to identify left- or right-moving storms. A hypothesis on the internal structure of the cold front of the 2nd kind is stated. The main conclusion of the work is that the evolution of storms is determined by the configuration of meso-β-scale convective structures. This explains various convective phenomena from unified positions. The results are applicable in works on modification of convective cloudiness, for ultra-short-term forecasts of dangerous phenomena, storm warnings of the population, rescue services, etc.

Reconstruction flow recurrent network for compressed video quality enhancement

We present a reconstruction flow for the task of compressed video quality enhancement (VQE). Compressed videos often suffer from various coding artifacts, such as blocking and blurring, especially under low bit-rate. VQE aims to suppress these artifacts to improve the visual quality. Frame similarity can be utilized to enhance low-quality frames given their neighboring high-quality frames, for which motion estimation becomes important. Previous approaches often calculate optical flow for the motion compensation. On the other hand, video coding contains a rich set of block motion vectors, forming a coding flow, which may or may not correspond to the scene motion, but to places that deliver the minimum compression error. In contrast, such a valuable coding flow has always been ignored in VQE previously. In this work, we combine these two motion sources into a new flow, namely reconstruction flow, for the purpose of high-quality VQE. Specifically, we estimate optical flows from RGB frames and extract coding flows from coding streams, which are then merged by a fusion module to generate reconstruction flow. Besides, our network is built upon a recurrent network to utilize global temporal information. The deep features are warped according to the reconstruction flow and fed into the subsequent reconstruction module with spatial-variant kernel attention. Our method is evaluated on the leading MFQE2.0 dataset, which demonstrates superior performances when compared to the existing state-of-the-art methods.

Real-Time Motion Blur Using Multi-Layer Motion Vectors

Traditional methods for motion blur, often relying on a single layer, deviate from the correct colors. We propose a multilayer rendering method that closely approximates the motion blur effect. Our approach stores motion vectors for each pixel, divides these vectors into multiple sample points, and performs a backward search from the current pixel. The color at a sample point is sampled if it shares the same motion vector as its origin. This procedure repeats across layers, with only the nearest color values sampled for depth testing. The average color sampled at each point becomes that of the motion blur. Our experimental results indicate that our method significantly reduces the color deviation commonly found in traditional approaches, achieving structural similarity index measures (SSIM) of 0.8 and 0.92, which represent substantial improvements over the accumulation method.

Compositionality in perception: A framework.

Perception involves the processing of content or information about the world. In what form is this content represented? I argue that perception is widely compositional. The perceptual system represents many stimulus features (including shape, orientation, and motion) in terms of combinations of other features (such as shape parts, slant and tilt, common and residual motion vectors). But compositionality can take a variety of forms. The ways in which perceptual representations compose are markedly different from the ways in which sentences or thoughts are thought to be composed. I suggest that the thesis that perception is compositional is not itself a concrete hypothesis with specific predictions; rather it affords a productive framework for developing and evaluating specific empirical hypotheses about the form and content of perceptual representations. The question is not just whether perception is compositional, but how. Answering this latter question can provide fundamental insights into perception. This article is categorized under: Philosophy > Representation Philosophy > Foundations of Cognitive Science Psychology > Perception and Psychophysics.

Atmospheric motion vector (AMV) error characterization and bias correction by leveraging independent lidar data: a simulation using an observing system simulation experiment (OSSE) and optical flow AMVs

Abstract. Accurate estimation of global winds is crucial for various scientific and practical applications, such as global chemical transport modeling and numerical weather prediction. One valuable source of wind measurements is atmospheric motion vectors (AMVs), which play a vital role in the global observing system and numerical weather prediction models. However, errors in AMV retrievals need to be addressed before their assimilation into data assimilation systems, as they can affect the accuracy of outputs. An assessment of the bias and uncertainty in passive-sensor AMVs can be done by comparing them with information from independent sources such as active-sensor winds. In this paper, we examine the benefit and performance of a colocation scheme using independent and sparse lidar wind observations as a dependent variable in a supervised machine learning model. We demonstrate the feasibility and performance of this approach in an observing system simulation experiment (OSSE) framework, with reference geophysical state data obtained from high-resolution Weather Research and Forecasting (WRF) model simulations of three different weather events. Lidar wind data are typically available in only one direction, and our study demonstrates that this single component of wind in high-precision active-sensor data can be leveraged (via a machine learning algorithm to model the conditional mean) to reduce the bias in the passive-sensor winds. Further, this active-sensor wind information can be leveraged through an algorithm that models the conditional quantiles to produce stable estimates of the prediction intervals, which are helpful in the design and application of error analysis, such as quality filters.

Medical image registration with object deviation estimation through motion vectors using octave and level sampling

Medical image analysis presents a significant problem in the field of image registration. Recently, medical image registration has been recognized as a helpful tool for medical professionals. Current state-of-the-art approaches solely focus on source image registration and lack quantitative measurement for object deviation in terms of loosening, subsidence and anteversion related to surgery. In this article, we have provided motion vectors for recognizing the object deviation, in addition to detecting and selecting the feature points. Firstly, the feature points will be detected using Hessian matrix determinants and octave and level sampling. Then the strongest feature points are selected which will be utilized for identifying the object deviation with respect to the reference image through motion vectors. The objective of this work is to combine image registration and temporal differencing to achieve independent motion detection. In comparison to state-of-the-art approaches, the proposed methodology achieves higher Information Ratio (IR), Mutual Information Ratio (MIR) and their lower bounds for image registration.

ECMWF Ensemble Forecasts of Six Tropical Cyclones That Formed during a Long-Lasting Rossby Wave Breaking Event in the Western North Pacific

The ECMWF‘s ensemble (ECEPS) predictions are documented for the lifecycles of six tropical cyclones (TCs) that formed during a long-lasting Rossby wave breaking event in the western North Pacific. All six TC tracks started between 20° N and 25° N, and between 136° E and 160° E. All five typhoons recurved north of 30° N, and the three typhoons that did not make landfall had long tracks to 50° N and beyond. The ECEPS weighted mean vector motion track forecasts from pre-formation onward are quite accurate, with track forecast spreads that are primarily related to initial position uncertainties. The ECEPS intensity forecasts have been validated relative to the Joint Typhoon Warning Center (JTWC) Working Best Track (WBT) intensities (when available). The key results for Tokage (11 W) were the ECEPS forecasts of the intensification to a peak intensity of 100 kt, and then a rapid decay as a cold-core cyclone. For Hinnamnor (12 W), the key result was the ECEPS intensity forecasts during the post-extratropical transition period when Hinnamnor was rapidly translating poleward through the Japan Sea. For Muifa (14 W), the key advantage of the ECEPS was that intensity guidance was provided for longer periods than the JTWC 5-day forecast. The most intriguing aspect of the ECEPS forecasts for post-Merbok (15 W) was its prediction of a transition to an intense, warm-core vortex after Merbok had moved beyond 50° N and was headed toward the Aleutian Islands. The most disappointing result was that the ECEPS over-predicted the slow intensification rate of Nanmadol (16 W) until the time-to-typhoon (T2TY), but then failed to predict the large rapid intensification (RI) following the T2TY. The tentative conclusion is that the ECEPS model‘s physics are not capable of predicting the inner-core spin-up rates when a small inner-core vortex is undergoing large RI.

Arctic Winds Retrieved from FY-3D Microwave Humidity Sounder-II 183.31 GHz Brightness Temperature Using Atmospheric Motion Vector Method

In this study, we develop an Atmospheric Motion Vector (AMV)-based method for retrieving wind vectors using 183.31 GHz water-vapor absorption channels. The method involves tracking water-vapor features from image triplets and subsequently deriving wind fields from motion vectors. The height of the derived wind for each channel is determined by calculating the weighing function peak using monthly averaged ERA5 reanalysis data. By utilizing Microwave Humidity Sounder-II (MWHS-II) brightness temperatures from the five channels centered around 183.31 GHz, wind vectors are retrieved within the Arctic region for the entire year of 2022. The retrieval quality is evaluated through comparative analysis with ERA5 reanalysis data and the Visible Infrared Imaging Radiometer Suite (VIIRS) wind product. The resultant vector root mean square errors (RMSEs) are approximately 4.5 m/s for the three lower-height channels and 5.5 m/s for the two upper-height channels. These findings demonstrate a wind retrieval performance comparable to the existing methods, highlighting its potential for augmenting wind availability at lower height levels.

Efficient Task-driven Video Data Privacy Protection for Smart Camera Surveillance System

As one of the most commonly used AIoT sensors, smart cameras and their supporting services, namely cloud video surveillance (CVS) systems, have brought great convenience to people’s lives. Recent CVS providers use different machine learning techniques to improve their services (regarded as tasks) based on the uploaded video. However, uploading data to the CVS providers may cause severe privacy issues. Existing works that remove privacy information could not achieve a high tradeoff between data usability and privacy, because the importance of information varies with the task. In addition, it is challenging to design a real-time privacy protection mechanism, especially in resource-constrained smart cameras. In this work, we design a task-driven and efficient video privacy protection mechanism for a better tradeoff between privacy and data usability. We use Class Activation Mapping to protect privacy while preserving data usability. To improve the efficiency, we utilize the motion vector and residual matrix produced during video codec. Our work outperforms the region of interest–based methods in data protection while preserving data usability. The attack accuracy drops 70%, while the task accuracy is comparable to those without protection (within ± 4%). The average protection frame rate of the High Definition video can exceed 16 fps+ even on a CPU.

Anomaly detection in structural dynamic systems via nonlinearity occurrence analysis using video data

Dynamic loads in disaster events, such as earthquakes, often cause structural damages that severely affect serviceability and even reduce the safety of civil structures or render them unsafe. Most of these damage scenarios can cause the transition from a linear to nonlinear behavior in structural dynamics. Therefore, extracting such nonlinear behaviors is an effective strategy for structural anomaly detection. In recent years, the rapid development of media devices has led to the widespread use of video data, primarily because of the ease of acquisition and lack of contact measurement. This study developed a novel method for detecting anomalies due to structural nonlinearity from video data that could capture target dynamic behaviors via non-contact measurement. The method is based on optical flow, extended repulsive force networks, and morphological operations for extracting singularities due to nonlinear events in the estimated motion vector field in video data. Subsequently, shaking table tests on a three-story shear building model containing a novel controllable hinge bearing verified the effectiveness of the developed method. The results of the feature enhancement process demonstrate the accurate localization of anomalous regions and effective mitigation of unwanted interference using the proposed method. The proposed method facilitates the use of video-based technology to evaluate rapid damage conditions post-earthquake.

3D-AMM: a 3D artificial moment method for path planning of manipulator in multiple obstacles scenario

PurposeIn response to the issue of traditional algorithms often falling into local minima or failing to find feasible solutions in manipulator path planning. The purpose of this paper is to propose a 3D artificial moment method (3D-AMM) for obstacle avoidance for the robotic arm's end-effector.Design/methodology/approachA new method for constructing temporary attractive points in 3D has been introduced using the vector triple product approach, which generates the attractive moments that attract the end-effector to move toward it. Second, distance weight factorization and spatial projection methods are introduced to improve the solution of repulsive moments in multiobstacle scenarios. Third, a novel motion vector-solving mechanism is proposed to provide nonzero velocity for the end-effector to solve the problem of limiting the solution of the motion vector to a fixed coordinate plane due to dimensionality constraints.FindingsA comparative analysis was conducted between the proposed algorithm and the existing methods, the improved artificial potential field method and the rapidly-random tree method under identical simulation conditions. The results indicate that the 3D-AMM method successfully plans paths with smoother trajectories and reduces the path length by 20.03% to 36.9%. Additionally, the experimental comparison outcomes affirm the feasibility and effectiveness of this method for obstacle avoidance in industrial scenarios.Originality/valueThis paper proposes a 3D-AMM algorithm for manipulator path planning in Cartesian space with multiple obstacles. This method effectively solves the problem of the artificial potential field method easily falling into local minimum points and the low path planning success rate of the rapidly-exploring random tree method.

Transverse Spin Hall Effect and Twisted Polarization Ribbons at the Sharp Focus

Transverse Spin Hall Effect and Twisted Polarization Ribbons at the Sharp Focus

Wind Distribution in the Eye of Tropical Cyclone Revealed by a Novel Atmospheric Motion Vector Derivation

AbstractObservations of wind distribution in the eye of tropical cyclones (TCs) are still limited. In this study, a method to derive atmospheric motion vectors (AMVs) for TCs is developed, where selection from multiple local rotation speeds is made by considering continuity among neighboring grid points. The method is applied to 2.5‐min interval image sequences of three TCs, Lan (2017), Haishen (2020), and Nanmadol (2022), observed by the Himawari‐8 satellite. The results are compared with AMVs derived from research‐based 30‐s Himawari‐8 special observations conducted for Haishen and Nanmadol, as well as with in‐situ dropsonde observations conducted for Lan and Nanmadol. In these storms, the AMVs obtained from the 2.5‐min interval images in the eye are found to be in good agreement with the dropsonde observations. Examinations of AMVs in the eye reveal transient azimuthal wavenumber‐1 features in all three TCs. These features are consistent with algebraically growing wavenumber‐1 disturbances, which transport angular momentum inward and accelerate the eye rotation. In the case of Lan, the angular velocity in the eye increased by approximately 1.5 times within 1 hr. This short‐term increase is further examined. Visualization of low‐level vorticity in the eye and angular momentum budget analysis suggest that angular momentum transport associated with mesovortices played an important role in the increase of tangential wind and the homogenization of angular velocity in the eye of Lan.

Advancements in Motion Detection Within Video Streams Through the Integration of Optical Flow Estimation and 3D-Convolutional Neural Network Architectures

Motion detection in video streams is important for many applications, such as autonomous navigation, human-computer interaction, and spying. Conventional techniques, which depend on backdrop removal or frame differencing, frequently break down when dealing with intricate motion patterns and occlusions. Current techniques struggle to handle occlusions and correctly capture complex motion patterns. Furthermore, these techniques could be computationally expensive, especially when dealing with big amounts of video data. An integrated strategy combining optical flow estimation with 3D convolutional neural network (3D-CNN) architectures is presented to address these limitations and boost motion detection systems' accuracy and efficiency. The suggested technique is unusual as it combines 3D CNNs with optical flow estimates. Motion vectors representing dynamic scene changes are produced by optical flow estimates, and spatiotemporal characteristics are extracted by 3D CNNs processing together with optical flow information. By using the complementing capabilities of both approaches, the method performs better in motion detection applications such as object tracking, action recognition, and anomaly detection in video streams. The efficiency of the strategy is assessed by experiments carried out on benchmark datasets. The results show that it is more accurate, resistant against occlusions, and computationally efficient than existing approaches. The suggested approach offers a viable way to improve motion detection capabilities for a range of practical uses. The results show a 98% improvement in processing efficiency and motion detection accuracy when compared to baseline methods. More research and advancement in this area are made possible by the attainable way that MATLAB's suggested approach offers to enhance motion detection skills in a variety of real-world applications.

Motion Vector Estimation Method of Dynamic Image Sequence Using Neural Network in the Context of Internet of Things

Abstract To improve the low resolution of dynamic images caused by motion vector, a neural network-based motion vector estimation method for dynamic image sequences is proposed in this study. First, a sum of absolute differences (SAD) method is used to determine the search range of motion vector estimation for dynamic image sequences, and a sped up robust features (SURF) algorithm is used to extract the motion vector features from the determined search range. Then, the self-organizing neural network is used to obtain the motion vector estimation results according to the motion vector features of the extracted dynamic images. Finally, a weighted median vector filter is used to correct the pseudo-nonlinear motion vector in the motion vector estimation results to improve the motion vector estimation performance of dynamic image sequences. The experimental results show that the algorithm can compensate for the image according to the motion vector estimation results of dynamic image sequences, and the image quality is obviously improved with high peak signal-to-noise ratio.

ConvMatch: Rethinking Network Design for Two-View Correspondence Learning.

Multilayer perceptron (MLP) has become the de facto backbone in two-view correspondence learning, for it can extract effective deep features from unordered correspondences individually. However, the problem of natively lacking context information limits its performance although many context-capturing modules are appended in the follow-up studies. In this paper, from a novel perspective, we design a correspondence learning network called ConvMatch that for the first time can leverage a convolutional neural network (CNN) as the backbone, inherently capable of context aggregation. Specifically, with the observation that sparse motion vectors and a dense motion field can be converted into each other with interpolating and sampling, we regularize the putative motion vectors by estimating the dense motion field implicitly, then rectify the errors caused by outliers in local areas with CNN, and finally obtain correct motion vectors from the rectified motion field. Moreover, we propose global information injection and bilateral convolution, to fit the overall spatial transformation better and accommodate the discontinuities of the motion field in case of large scene disparity. Extensive experiments reveal that ConvMatch consistently outperforms state-of-the-arts for relative pose estimation, homography estimation, and visual localization.