Scene Motion Research Articles

Multimodal Locomotion Control of Needle-Like Microrobots Assembled by Ferromagnetic Nanoparticles

Microrobots have great potential for multiple applications, such as targeted drug delivery and micromanipulation. Several kinds of microrobots assembled by nanoparticles have been proposed by researchers. However, they are difficult to adapt for complex environments, for example, traversing porous materials or climbing over obstacles. Many such environments require multimodal motion control. In this article, we proposed a needle-like microrobot assembled by ferromagnetic nanoparticles, which enables three types of locomotion, defined as axial motion, lateral motion, and rolling motion. The influence of velocities by input frequencies and the lengths of the microrobots are investigated theoretically and experimentally. Moreover, visual feedback path-following control methods are designed for each motion type, and experimentally verified. The needle-like microrobots enable traversing the nonwoven layer of a clinical mask and a seaweed silk barrier by axial motion, which showed a separation and a reaggregation during the traversing process. The needle-like microrobots with rolling motion are capable of climbing over obstacles, which expands its motion scene from 2-D to 3-D. They can overcome even higher obstacles by assembling themselves to form the longer needle-like microrobots. We further expect that with the proposed multimodal motion control of nanoparticle microrobots, they may achieve complex tasks at the microscale.

Visual motion perception as online hierarchical inference

Identifying the structure of motion relations in the environment is critical for navigation, tracking, prediction, and pursuit. Yet, little is known about the mental and neural computations that allow the visual system to infer this structure online from a volatile stream of visual information. We propose online hierarchical Bayesian inference as a principled solution for how the brain might solve this complex perceptual task. We derive an online Expectation-Maximization algorithm that explains human percepts qualitatively and quantitatively for a diverse set of stimuli, covering classical psychophysics experiments, ambiguous motion scenes, and illusory motion displays. We thereby identify normative explanations for the origin of human motion structure perception and make testable predictions for future psychophysics experiments. The proposed online hierarchical inference model furthermore affords a neural network implementation which shares properties with motion-sensitive cortical areas and motivates targeted experiments to reveal the neural representations of latent structure.

Multiple Video Frame Interpolation via Enhanced Deformable Separable Convolution.

Generating non-existing frames from a consecutive video sequence has been an interesting and challenging problem in the video processing field. Typical kernel-based interpolation methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels, which circumvents the time-consuming, explicit motion estimation in the form of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods are prone to yield less plausible results. In addition, they cannot directly generate a frame at an arbitrary temporal position because the learned kernels are tied to the midpoint in time between the input frames. In this paper, we try to solve these problems and propose a novel non-flow kernel-based approach that we refer to as enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases to make the network obtain information from non-local neighborhood. During the learning process, different intermediate time step can be involved as a control variable by means of an extension of coord-conv trick, allowing the estimated components to vary with different input temporal information. This makes our method capable to produce multiple in-between frames. Furthermore, we investigate the relationships between our method and other typical kernel- and flow-based methods. Experimental results show that our method performs favorably against the state-of-the-art methods across a broad range of datasets. Code will be publicly available on URL: https://github.com/Xianhang/EDSC-pytorch.

Instance-Aware Multi-Object Self-Supervision for Monocular Depth Prediction

This paper proposes a self-supervised monocular image-to-depth prediction framework that is trained with an end-to-end photometric loss that handles not only 6-DOF camera motion but also 6-DOF moving object instances. Self-supervision is performed by warping the images across a video sequence using depth and scene motion including object instances. One novelty of the proposed method is the use of the multi-head attention of the transformer network that matches moving objects across time and models their interaction and dynamics. This enables accurate and robust pose estimation for each object instance. Most image-to-depth predication frameworks make the assumption of rigid scenes, which largely degrades their performance with respect to dynamic objects. Only a few SOTA papers have accounted for dynamic objects. The proposed method is shown to outperform these methods on standard benchmarks and the impact of the dynamic motion on these benchmarks is exposed. Furthermore, the proposed image-to-depth prediction framework is also shown to be competitive with SOTA video-to-depth prediction frameworks.

Spatial–Temporal Convolutional Model for Urban Crowd Density Prediction Based on Mobile-Phone Signaling Data

Urban crowd density prediction is essential for transport demand management and public safety monitoring. Existing studies for crowd density prediction only focus on a few transport modes which do not cover all urban crowds. Moreover, most existing models rely on grid-based divisions, which cannot accommodate unevenly distributed populations. This study proposes a novel model for predicting crowd density in urban areas with mobile-phone signaling data (MPSD). MPSD contain diversified travel and activity information that is not limited to specific transport modes and are practical for activity-travel research. We introduce methods for preprocessing MPSD and extracting information on crowd density and activity choice behaviors (such as activity location, activity type, and activity duration). A novel spatial-temporal convolutional model is proposed for urban crowd density prediction. We design an attention-based feature fusion block, enabling the model to comprehensively consider historical crowd density, activity choices, and external factors (such as weather conditions and holidays). With tailored spatial-temporal convolution blocks, our proposed model successfully captures spatial-temporal information related to crowd movement in different scenes. Unlike existing grid-based models, the proposed model can predict crowd density in irregular-shaped regions. Experimental results show that our model outperforms baseline models that do not apply activity information when selecting different prediction offsets, especially for long-term predictions. The correlation between prediction error and the regularity of crowd activities is also analyzed. The subsequent parameter analysis shows that the influence of external factors on urban crowd movement varies according to activity choices and other travel-related factors like commuting distance.

Visual oscillation effects on dynamic balance control in people with multiple sclerosis

BackgroundPeople with multiple sclerosis (PwMS) have balance deficits while ambulating through environments that contain moving objects or visual manipulations to perceived self-motion. However, their ability to parse object from self-movement has not been explored. The purpose of this research was to examine the effect of medial–lateral oscillations of the visual field and of objects within the scene on gait in PwMS and healthy age-matched controls using virtual reality (VR).MethodsFourteen PwMS (mean age 49 ± 11 years, functional gait assessment score of 27.8 ± 1.8, and Berg Balance scale score 54.7 ± 1.5) and eleven healthy controls (mean age: 53 ± 12 years) participated in this study. Dynamic balance control was assessed while participants walked on a treadmill at a self-selected speed while wearing a VR headset that projected an immersive forest scene. Visual conditions consisted of (1) no visual manipulations (speed-matched anterior/posterior optical flow), (2) 0.175 m mediolateral translational oscillations of the scene that consisted of low pairing (0.1 and 0.31 Hz) or (3) high pairing (0.15 and 0.465 Hz) frequencies, (4) 5 degree medial–lateral rotational oscillations of virtual trees at a low frequency pairing (0.1 and 0.31 Hz), and (5) a combination of the tree and scene movements in (3) and (4).ResultsWe found that both PwMS and controls exhibited greater instability and visuomotor entrainment to simulated mediolateral translation of the visual field (scene) during treadmill walking. This was demonstrated by significant (p < 0.05) increases in mean step width and variability and center of mass sway. Visuomotor entrainment was demonstrated by high coherence between center of mass sway and visual motion (magnitude square coherence = ~ 0.5 to 0.8). Only PwMS exhibited significantly greater instability (higher step width variability and center of mass sway) when objects moved within the scene (i.e., swaying trees).ConclusionResults suggest the presence of visual motion processing errors in PwMS that reduced dynamic stability. Specifically, object motion (via tree sway) was not effectively parsed from the observer’s self-motion. Identifying this distinction between visual object motion and self-motion detection in MS provides insight regarding stability control in environments with excessive external movement, such as those encountered in daily life.

Learning Laparoscope Actions via Video Features for Proactive Robotic Field-of-View Control

Smart laparoscope motion control for adjusting surgical field-of-view is an increasingly hot topic in robot-assisted surgery. Previous off-the-shelf methods have been conducted in reactive ways which heavily rely on human input signals, e.g., gaze or voice, thus cannot avoid cognitive burdens to surgeons. In this paper, we introduce a novel proactive framework that learns the motion strategy from clinical surgical videos to achieve autonomous laparoscope control in surgery. In specific, we first propose a robust estimation method to acquire laparoscope motion in dynamic surgical scenes, which provides the basis for later supervised learning process. To imitate operating manners in different surgical domains, we design consistent dynamic image-level motion features that consist of semantic segmentation and dense optical flow. A sequence-to-sequence recurrent network is proposed to learn the connection between previous and future motion features at multiple temporal scales within a time-dependent sliding window. Then a laparoscope action head is built to convert the motion features into three-dimensional actions, hence establishing an end-to-end laparoscope motion imitation network. We have extensively validated the proposed proactive field-of-view control framework on both our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ex-vivo</i> phantom-based robotic platform, as well as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-vivo</i> videos collected from real clinical videos. Our promising results demonstrate the feasibility to directly learn laparoscope behavioral actions from collected video data of experienced surgical assistants.

Comparison of super-resolution and noise reduction for passive single-photon imaging

Single-photon sensitive image sensors have recently gained popularity in passive imaging applications where the goal is to capture photon flux (brightness) values of different scene points in the presence of challenging lighting conditions and scene motion. Recent work has shown that high-speed bursts of single-photon timestamp information captured using a single-photon avalanche diode camera can be used to estimate and correct for scene motion thereby improving signal-to-noise ratio and reducing motion blur artifacts. We perform a comparison of various design choices in the processing pipeline used for noise reduction, motion compensation, and upsampling of single-photon timestamp frames. We consider various pixelwise noise reduction techniques in combination with state-of-the-art deep neural network upscaling algorithms to super-resolve intensity images formed with single-photon timestamp data. We explore the trade space of motion blur and signal noise in various scenes with different motion content. Using real data captured with a hardware prototype, we achieved super-resolution reconstruction at frame rates up to 65.8 kHz (native sampling rate of the sensor) and captured videos of fast-moving objects. The best reconstruction is obtained with the motion compensation approach, which achieves a structural similarity (SSIM) of about 0.67 for fast-moving rigid objects. We are able to reconstruct subpixel resolution. These results show the relative superiority of our motion compensation compared to other approaches that do not exceed an SSIM of 0.5.

Design of Sports Training Simulation System for Children Based on Improved Deep Neural Network.

With the development of AI technology, human-computer interaction technology is no longer the traditional mouse and keyboard interaction. AI and VR have been widely used in early childhood education. In the process of the slow development and application of voice interaction, visual interaction, action interaction, and other technologies, multimodal interaction technology system has become a research hotspot. In this paper, dynamic image capture and recognition technology is integrated into early childhood physical education for intelligent interaction. According to the basic movement process and final node matching in children's sports training to judge children's physical behavior ability, attention is paid to identify the accuracy and safety of movement. The input images and questions are from the abstract clipart dataset of dynamic image recognition and the self-made 3D dataset of Web3D dynamic motion scene with the same style, which is similar to the action content in the actual preschool training teaching. Therefore, according to the idea of process capture and target recognition, on the basis of the original conditions of the recognition model, a new recognition model is developed through Zheng's target detector. The modified model is characterized by higher accuracy. Weapons need to combine process recognition and result recognition. The experimental results show that the improved model has the obvious advantages of high precision and fast speed, which provides a new research idea for the development of children's physical training simulation.

Computer Digital Technology Combined with Dynamic Visual Communication Sensors in Target Tracking with Big Data

The detection algorithm is explored to improve the dynamic visual sensors (DVS) combined with computer digital technology, build a DVS network, and complete the monitoring and tracking of the target. Ultimately, the problem that needs to be solved is the poor quality of traditional communication sensor data transmission, which needs to be improved by DVS. Firstly, the structure and function of the network are described through dynamic visual perception requirements analysis. Secondly, by introducing a target tracking algorithm that combines event flow and grayscale images, two methods are proposed, namely, the event flow noise reduction method based on event density and the optical flow detection feature tracking algorithm. Finally, through experiments, the tracking and detection effect of the optical flow detection algorithm on the target object in the dark environment is verified in the high-speed motion scene and the reflection environment. The results show that the average error of target object detection and tracking is 3.2 pixels in a dark environment. The average error of target tracking in high-speed motion scenes and reflective environments is 4.86 pixels and 2.88 pixels, respectively. This research has practical reference value for the digital and intelligent development of digital video surveillance systems.

Learning temporal regularized and weighted surrounding-aware correlation filter for unmanned aerial vehicle object tracking

Correlation filter (CF)-based trackers have demonstrated promising performance in the field of unmanned aerial vehicle (UAV) tracking, benefiting from their high computational efficiency. However, model degradation and the boundary effect remain two major obstacles for CF-based trackers. We present an innovative temporal regularized and weighted surrounding-aware CF for UAV object tracking. First, we considered the consistency of response between the current frame and the previous frame of an object in a time series, which effectively enhances the discriminability and generalization capability of CF-based trackers to abrupt appearance variations. Second, we developed an innovative weighted context learning strategy that is able to enlarge the visual field of trackers to cope with the boundary effect. Specifically, we extract weighted surrounding patches with the same size and shape of the object to empower the discriminating ability of the CF. Finally, to validate the effectiveness of our proposed method, substantial comparison experiments with other state-of-the-art algorithms were conducted on three UAV standard data sets. The experimental results demonstrate that the algorithm proposed has good robustness for occlusion, deformation, fast motion, and other challenging disturbances in complex scenes and effectively tracks the target on the platform of a UAV.

Insulator Hydrophobic Image Edge Detection Algorithm considering Deconvolution and Deblurring Algorithm

In this paper, the Gram matrix is used to calculate the correlation of the filter response sets under different scale kernels learned by each layer of the network in the deconvolution, and the loss between the corresponding feature response correlations in the multilayer network is calculated. Linear summation is used to obtain a stable, multiscale image model representation. This paper extracts the contours of the salient areas of the image and adjusts the parameters of the deconvolution network to learn the salient area patterns of the image. At the same time, for the image to be generated, a shape template is used to limit the range of the area to be generated in order to obtain a shape image with similar patterns. When the spatial relative relationship characteristics of the image constituent objects are obvious, we appropriately add high-level semantic feature activation values for reinforcement. This paper solves the estimation of the unknown blur kernel by using image prior knowledge, filtering and gradient domain algorithms and other different technologies to obtain image jitter or scene movement information and estimate the size, location, and density of the blur kernel. This paper studies a relatively robust deconvolution model, which is insensitive to random noise, has stable effects, and can overcome the water ripple effect caused by the usual convolution process. This paper attempts to study the fuzzy model with variable space. The usual blur is a spatial invariant model; that is, a single kernel is used to describe the motion of all pixels on the image. By selecting different characteristic parameters, this paper conducts experimental research on some existing hydrophobic indicator function methods and calculates the relationship between characteristic parameters and hydrophobicity when different hydrophobic indicator functions are adopted. One characteristic of the hydrophobic image of composite insulators is low contrast. The traditional method of converting color images to grayscale images cannot improve the image contrast. This paper analyzes the hydrophobic image of the composite insulator, and the extracted B channel component image of the hydrophobic image improves the contrast of the image and facilitates the subsequent segmentation of water traces and background. In this paper, the water repellent image's watermark area is counted, and connected-domain wave processing is used to limit the area of water droplets retained, thereby improving the efficiency of filtering water droplets without having a big impact on the image as a whole. The problem of uneven illumination is an unavoidable problem in the field of image processing, and the resulting reflection problem brings difficulties to image processing. This article regards the reflective area of the watermark as a “hole” and uses the idea of “hole filling” to eliminate the reflective point, which weakens the reflection problem to a certain extent.

Elements of exogenous attentional cueing preserved during optokinetic motion of the visual scene.

Navigating through our environment raises challenges for perception by generating salient background visual motion and eliciting prominent eye movements to stabilise the retinal image. It remains unclear if exogenous spatial attentional orienting is possible during background motion and the eye movements it causes and whether this compromises the underlying neural processing. To test this, we combined exogenous orienting, visual scene motion, and electroencephalography (EEG). A total of 26 participants viewed a background of moving black and grey bars (optokinetic stimulation). We tested for effects of non-spatially predictive peripheral cueing on visual motion discrimination of a target dot, presented either at the same (valid) or opposite (invalid) location as the preceding cue. Valid cueing decreased reaction times not only when participants kept their gaze fixed on a central point (fixation blocks) but also even when there was no fixation point, so that participants performed intensive, repetitive tracking eye movements (eye movement blocks). Overall, manual response reaction times were slower during eye movements. Cueing also produced reliable effects on neural activity on either block, including within the first 120 ms of neural processing of the target. The key pattern with larger event-related potential (ERP) amplitudes on invalid versus valid trials showed that the neural substrate of exogenous cueing was highly similar during eye movements or fixation. Exogenous peripheral cueing and its neural correlates are robust against distraction from the moving visual scene, important for perceptual cognition during navigation.

Learning Camera Control in Dynamic Scenes from Limited Demonstrations

AbstractIn this work, we present our strategy for camera control in dynamic scenes with multiple people (sports teams). We learn a generic model of the player dynamics offline in simulation. We use only a few sparse demonstrations of a user's camera control policy to learn a reward function to drive camera motion in an ongoing dynamic scene. Key to our approach is the creation of a low‐dimensional representation of the scene dynamics which is independent of the environment action and rewards, which enables learning the reward function using only a small number of examples. We cast the user‐specific control objective as an inverse reinforcement learning problem, aiming to learn an expert's intention from a small number of demonstrations. The learned reward function is used in combination with a visual model predictive controller (MPC). We learn a generic scene dynamics model that is agnostic to the user‐specific reward, enabling reusing the same dynamics model for different camera control policies. We show the effectiveness of our method on simulated and real soccer matches.

Visual search in naturalistic scenes from foveal to peripheral vision: A comparison between dynamic and static displays.

How important foveal, parafoveal, and peripheral vision are depends on the task. For object search and letter search in static images of real-world scenes, peripheral vision is crucial for efficient search guidance, whereas foveal vision is relatively unimportant. Extending this research, we used gaze-contingent Blindspots and Spotlights to investigate visual search in complex dynamic and static naturalistic scenes. In Experiment 1, we used dynamic scenes only, whereas in Experiments 2 and 3, we directly compared dynamic and static scenes. Each scene contained a static, contextually irrelevant target (i.e., a gray annulus). Scene motion was not predictive of target location. For dynamic scenes, the search-time results from all three experiments converge on the novel finding that neither foveal nor central vision was necessary to attain normal search proficiency. Since motion is known to attract attention and gaze, we explored whether guidance to the target was equally efficient in dynamic as compared to static scenes. We found that the very first saccade was guided by motion in the scene. This was not the case for subsequent saccades made during the scanning epoch, representing the actual search process. Thus, effects of task-irrelevant motion were fast-acting and short-lived. Furthermore, when motion was potentially present (Spotlights) or absent (Blindspots) in foveal or central vision only, we observed differences in verification times for dynamic and static scenes (Experiment 2). When using scenes with greater visual complexity and more motion (Experiment 3), however, the differences between dynamic and static scenes were much reduced.

Design of ECG Signal Generator Based on Motion Scene

With the normalization of cardiovascular diseases(CVD) in many countries, portable ECG monitoring devices have gradually become popular. However, since the current portable ECG monitoring equipment lacks a unified testing standard specification, a suitable calibration device is required for restraint. This research simulates a multi-state ECG signal generation system built using Arduino uno board for calibration of portable ECG monitoring devices. The system can generate human ECG signals of different intensities under different heart rates based on the operation of the host computer. The equipment has the advantages of simple, easy-to-use and low cost, and can be used as an ECG signal source for a portable ECG monitoring device in a simple laboratory environment.

Learning Dense and Continuous Optical Flow From an Event Camera.

Event cameras such as DAVIS can simultaneously output high temporal resolution events and low frame-rate intensity images, which own great potential in capturing scene motion, such as optical flow estimation. Most of the existing optical flow estimation methods are based on two consecutive image frames and can only estimate discrete flow at a fixed time interval. Previous work has shown that continuous flow estimation can be achieved by changing the quantities or time intervals of events. However, they are difficult to estimate reliable dense flow, especially in the regions without any triggered events. In this paper, we propose a novel deep learning-based dense and continuous optical flow estimation framework from a single image with event streams, which facilitates the accurate perception of high-speed motion. Specifically, we first propose an event-image fusion and correlation module to effectively exploit the internal motion from two different modalities of data. Then we propose an iterative update network structure with bidirectional training for optical flow prediction. Therefore, our model can estimate reliable dense flow as two-frame-based methods, as well as estimate temporal continuous flow as event-based methods. Extensive experimental results on both synthetic and real captured datasets demonstrate that our model outperforms existing event-based state-of-the-art methods and our designed baselines for accurate dense and continuous optical flow estimation.

Reconstructing Clear Image for High-Speed Motion Scene With a Retina-Inspired Spike Camera

Conventional digital cameras typically accumulate all the photons within an exposure period to form a snapshot image. It requires the scene to be quite still during the imaging time, otherwise it would result in blurry image for the moving objects. Recently, a retina-inspired spike camera has been proposed and shown great potential for recording high-speed motion scenes. Instead of capturing the visual scene by a single snapshot, the spike camera records the dynamic light intensity variation continuously. Each pixel on spike camera sensor accumulates the incoming photons independently and persistently, which fires a spike and restarts the photon accumulation immediately once the dispatch threshold is reached, producing a continuous stream of spikes recorded at very high temporal resolution. To recover the dynamic scene from captured spike stream, this paper presents an image reconstruction approach for spike camera. In order to generate high-quality reconstruction, we investigate the temporal correlation along motion trajectories and exploit it via adaptive temporal filtering. In particular, we present a hierarchical motion-aligned temporal filtering scheme, combining short-term filtering with long-term filtering to take advantage of long-term temporal correlation with low model complexity. Experimental results demonstrate that the proposed scheme outperforms the existing schemes significantly, producing much better objective and subjective qualities for spike camera image reconstruction.

Digital Twin Analysis for Driving Risks Based on Virtual Physical Simulation Technology

A digital twin is a mapping of real world objects in virtual space. For the study of traffic safety issues, digital twins have great potential to facilitate more accurate and detailed risk analysis. In this study, a digital twin method for highway driving safety analysis is proposed, which consists of three parts: Extracting vehicle motion information in the real world, constructing vehicle motion scenes in the virtual world, and analyzing vehicle driving risks. Firstly, aerial video of vehicle motion is captured by drone, while the microscopic vehicle trajectories are extracted from the video using machine vision algorithms. Secondly, the digital twin of vehicles and roads is constructed, while the motion behavior of vehicles is mapped in a digital space based on virtual physical simulation technology. Finally, according to the stability and trajectory deviation, the driving risks of the vehicle are evaluated, including sideslip, rollover, and guardrail collision. Through a case study, the effectiveness of the proposed digital twin method in driving risk assessment is verified, and one vehicle is found to have a higher driving risk.

Single Image Joint Motion Deblurring and Super-Resolution Using the Multi-Scale Channel Attention Modules

During the last decade, deep convolutional neural networks have significantly advanced the single image super-resolution techniques reconstructing realistic textural and spatial details. In classical image super-resolution problems, it is assumed that the low-resolution image has a certain downsampling degradation. However, complicated image degradations are inevitable in real-world scenarios, and motion blur is a common type of image degradation due to camera or scene motion during the image capturing process. This work proposes a fully convolutional neural network to reconstruct high-resolution sharp images from the given motion blurry low-resolution images. The deblurring subnetwork is based on multi-stage progressive architecture, while the super-resolution subnetwork is designed using the multi-scale channel attention modules. A simple and effective training strategy is employed where a pre-trained frozen deblurring module is used to train the super-resolution module. The deblurring module is unfrozen in the last training phase. Experiments show that, unlike the other methods, the proposed method reconstructs relatively small structures and textural details while successfully removing the complex motion blur. The implementation code and the pre-trained model are publicly available at https://github.com/misakshoyan/joint-motion-deblur-and-sr.