Motion Smear Research Articles

TKO‐SLAM: Visual SLAM algorithm based on time‐delay feature regression and keyframe pose optimization

AbstractThis paper addresses the challenge of generating clear image frames and minimizing the loss of keyframes by a robot engaging in rapid large viewing angle motion. These issues often lead to detrimental consequences such as trajectory drifting and loss during the construction of curved motion trajectories. To tackle this, we proposed a novel visual simultaneous localization and mapping (SLAM) algorithm, TKO‐SLAM, which is based on time‐delay feature regression and keyframe position optimization. TKO‐SLAM uses a multiscale recurrent neural network to rectify object deformation and image motion smear. This network effectively repairs the time‐delay image features caused by the rapid movement of the robot, thereby enhancing visual clarity. Simultaneously, inspired by the keyframe selection strategy of the ORB‐SLAM3 algorithm, we introduced a grayscale motion‐based image processing method to supplement keyframes that may be omitted due to the robot's rapid large viewing angle motion. To further refine the algorithm, the time‐delay feature regression image keyframes and adjacent secondary keyframes were used as dual measurement constraints to optimize camera poses and restore robot trajectories. The results of experiments on the benchmark RGB‐D data set TUM and real‐world scenarios show that TKO‐SLAM algorithm achieves more than 10% better localization accuracy than the PKS‐SLAM algorithm in the rapid large viewing angle motion scenario, and has advantages over the SOTA algorithms.

SURVEY ACCURACY AND SPATIAL RESOLUTION BENCHMARK OF A CAMERA SYSTEM MOUNTED ON A FAST FLYING DRONE

Abstract. Many drones are used to obtain high resolution imagery. Subsequent 3D object point derivation from images of these systems is an established technique. While rotor-craft drones are often used to capture fine, detailed structures and objects in small-scale areas fixed-wing versions are commonly used to cover larger areas even far beyond line of sight. Usually, these drones fly at much higher velocities during data acquisition and therefore the according sensor requirements are much higher.This paper presents the evaluation of a prototype camera system for fast flying fixed-wing drones. Focus of investigation is to find out if higher operating velocities, up to 100 km/h during image acquisition, has any influence on photogrammetric survey and image quality itself. It will be shown that images, obtained by the presented camera system and carrier, do not suffer from motion blur and that the overall survey accuracy is approximately 1/4 of ground sample distance.Survey accuracy analysis is carried out using standard photgrammetric procedures using signaled control- and checkpoints and verifying their conformity in image space and object space.Fundamentals of image quality will be introduced, as well asan approach to determine and evaluate motion smear of remote sensing senors (in theory and practical use case). Furthermore, it will be shown that the designed camera system mounted on a fixed-wing carrier does not suffer from motion smear.

Paint it black: Efficacy of increased wind turbine rotor blade visibility to reduce avian fatalities.

As wind energy deployment increases and larger wind‐power plants are considered, bird fatalities through collision with moving turbine rotor blades are expected to increase. However, few (cost‐) effective deterrent or mitigation measures have so far been developed to reduce the risk of collision. Provision of “passive” visual cues may enhance the visibility of the rotor blades enabling birds to take evasive action in due time. Laboratory experiments have indicated that painting one of three rotor blades black minimizes motion smear (Hodos 2003, Minimization of motion smear: Reducing avian collisions with wind turbines). We tested the hypothesis that painting would increase the visibility of the blades, and that this would reduce fatality rates in situ, at the Smøla wind‐power plant in Norway, using a Before–After–Control–Impact approach employing fatality searches. The annual fatality rate was significantly reduced at the turbines with a painted blade by over 70%, relative to the neighboring control (i.e., unpainted) turbines. The treatment had the largest effect on reduction of raptor fatalities; no white‐tailed eagle carcasses were recorded after painting. Applying contrast painting to the rotor blades significantly reduced the collision risk for a range of birds. Painting the rotor blades at operational turbines was, however, resource demanding given that they had to be painted while in‐place. However, if implemented before construction, this cost will be minimized. It is recommended to repeat this experiment at other sites to ensure that the outcomes are generic at various settings.

Intelligent Visual Tracking in Unstabilized Videos

Visual tracking requirement is increasing day by day due to the availability of high-performance digital cameras at low prices. Visual tracking becomes a complex problem when cameras suffer with unwanted and unintentional motion, resulting in motion-blurred unstabilized video. The problem in hand becomes more challenging when the target of interest is to be detected automatically in this unstabilized video. This paper presents a comprehensive single intelligent solution for these problems. The proposed algorithm auto-detects the camera motion, filters out the unintentional motion while stabilizing the video keeping intentional motion only using speeded-up robust features (SURF) technique. Motion smear due to unstabilization is also removed, providing sharp stabilized video output with video quality enhancement of up to 20dB. Gabor filter is used innovatively for auto-detection of target of interest in each stabilized frame. Then the target is tracked using SURF method.

The impact of retinal motion on stereoacuity for physical targets

In a series of studies using physical targets, we examined the effect of lateral retinal motion on stereoscopic depth discrimination thresholds. We briefly presented thin vertical lines, along with a fixation marker, at speeds ranging from 0 to 16 deg·s−1. Previous investigations of the effect of retinal motion on stereoacuity consistently show that there is little impact of retinal motion up to 2 deg·s−1, however, thresholds appear to rise steeply at higher velocities (greater than 3 deg·s−1). These prior experiments used computerized displays to generate their stimuli. In contrast, with our physical targets we find that stereoacuity is stable up to 16 deg·s−1, even in the presence of appreciable smearing due to visual persistence. We show that this discrepancy cannot be explained by differences in viewing time, prevalence of motion smear or by high frequency flicker due to display updates. We conclude that under natural viewing conditions observers are able to make depth discrimination judgements using binocular disparity signals that are rapidly acquired at stimulus onset.

Development of method for active alignment of multiple time delay and integration detectors in the optically butted focal plane assembly of high-resolution spaceborne imaging systems

Time delay and integration (TDI) detectors are preferred image sensors for high-resolution spaceborne imaging systems for cartographic applications as they provide higher sensitivity by integrating charges over multiple TDI stages in push broom mode. Large swath requirements are met using multiple TDI detectors in optical butting configuration in the focal plane assembly (FPA) to form a single image line on the ground. These detectors need to be aligned with subpixel accuracy to avoid motion smear and errors in the image mosaicking process. Alignment of the multiple TDI detectors in the FPA is a challenging task and is accomplished using custom methods and setups. Passive alignment techniques rely on very high-resolution microscopes for achieving desired accuracies. Active method requires TDI detectors to be operated in staring mode, which is architecturally and functionally not supported by TDI detectors. We developed a scheme for operating the TDI detectors in staring mode without any smear effects to aid in the active alignment process. Based on this, we propose an active alignment approach. The proposed approach is highly advantageous as it can be used throughout the imaging system development phase to demonstrate stability of the alignment and also helps in cross validation with the passive alignment. The developed method was test validated on qualification model FPA, and alignment accuracy of the order of 0.4 μm in tilt and 1.2 μm in line shift has been demonstrated, against the requirements of ±1.5 μm. This method has been extensively used in Cartosat-2S series imaging systems and will be very useful for upcoming high-resolution missions of the Indian Space Research Organisation.

Spatial-bisection acuity in infantile nystagmus

This study measured spatial bisection acuity for horizontally and vertically separated line targets in five observers with infantile nystagmus syndrome (INS) and no obvious associated sensory abnormalities, and in two normal observers during comparable horizontal retinal image motion. For small spatial separations between the line targets, bisection acuity for both horizontally and vertically separated lines is worse in the observers with IN than normal observers. In four of the five observers with IN, bisection acuity for small target separations is poorer for horizontally compared to vertically separated lines. Because the motion smear generated by the retinal image motion during IN would be expected to influence horizontally separated targets, the degradation of bisection acuity for both vertical and horizontally separated lines indicates that a sensory neural deficit contributes to impaired visual functioning in observers with idiopathic IN.

Hořava gravity versus thermodynamics: The black hole case

Under broad assumptions breaking of Lorentz invariance in gravitational theories leads to tension with unitarity because it allows for processes that apparently violate the second law of thermodynamics. The crucial ingredient of this argument is the existence of black hole solutions with the interior shielded from infinity by a causal horizon. We study how the paradox can be resolved in the healthy extension of Horava gravity. To this aim we analyze classical solutions describing large black holes in this theory with the emphasis on their causal structure. The notion of causality is subtle in this theory due to the presence of instantaneous interactions. Despite this fact, we find that within exact spherical symmetry a black hole solution contains a space-time region causally disconnected from infinity by a surface of finite area -- the `universal horizon'. We then consider small perturbations of arbitrary angular dependence in the black hole background. We argue that aspherical perturbations destabilize the universal horizon and, at non-linear level, turn it into a finite-area singularity. The causal structure of the region outside the singularity is trivial. If the higher-derivative terms in the equations of motion smear the singularity while preserving the trivial causal structure of the solutions, the thermodynamics paradox would be obviated. As a byproduct of our analysis we prove that the black holes do not have any non-standard long-range hair. We also comment on the relation with Einstein-aether theory, where the solutions with universal horizon appear to be stable.

Influence of Motion Smear on Visual Acuity in Simulated Infantile Nystagmus

In persons with infantile nystagmus (IN), visual acuity correlates with the duration of the foveation period of the nystagmus waveform, i.e., when the retinal image is on or near the fovea and moves with low velocity. In this study, we asked how acuity is affected by the non-foveating phases of the nystagmus waveform, when the velocity of retinal image motion is substantially higher. Visual acuity was measured in three normal observers for high contrast, four-orientation single T-stimuli, presented during image motion that simulated either the whole jerk-IN waveform (whole-waveform) or only the foveation periods of the IN waveform (foveation-only). Simulated foveation durations ranged from 20 to 120 ms. For both motion waveforms, we displayed the acuity target for different number of cycles to examine whether acuity benefits from multiple presentations of the stimulus. As expected, visual acuity improves with longer simulated foveation durations in both the whole-waveform and foveation-only conditions. Acuity is consistently better (by ∼0.1 logarithm of the minimum angle of resolution) in the foveation-only than the whole-waveform condition, indicating that the high-velocity image motion during the simulated IN waveform has a detrimental effect. This difference in acuity between the two waveform conditions increases with the number of cycles, apparently because summation occurs across cycles in the foveation-only condition but not in the whole-waveform condition. In normal observers, visual acuity in the presence of a simulated nystagmus waveform is limited not only by the duration of the foveation periods, but also by the non-foveating phases of the waveform. However, because persons with IN report little or no motion smear in association with their nystagmus, it remains unclear whether the rapid retinal image motion during the non-foveating phases of the nystagmus waveform generates a similar degradation of visual acuity in IN.

Visual contrast processing is largely unaltered during saccades

Saccadic suppression refers to a reduction in visual sensitivity during saccadic eye movements. This reduction is conventionally regarded as mediated by either of two sources. One is a simple passive process of motion smear during saccades also accompanied by visual masking exerted by high-contrast pre- and post-saccadic images. The other is an active process exerted by a neural mechanism that significantly reduces visual processing so that the perception of a stable visual environment is not disrupted during saccades. Some studies have actually shown that contrast sensitivity is significantly lower during saccades than under fixation, but these experiments were not designed in a way that could weigh the differential contribution of active and passive sources of saccadic suppression. We report the results of measurements of psychometric functions for contrast detection using stimuli that are only visible during saccades, thus effectively isolating any visual processing that actually takes place during the saccades and also preventing any pre- and post-saccadic visual masking. We also report measurements of psychometric functions for detection under fixation for stimuli that are comparable in duration and spatio-temporal characteristics to the intrasaccadic retinal stimulus. Whether during saccades or under fixation, the psychometric functions for detection turned out to be very similar, leaving room only for a small amount of sensitivity reduction during saccades. This suggests that contrast processing is largely unaltered during saccades and, thus, that no neural mechanism seems to be actively involved in saccadic suppression.

Perception of motion smear during visually induced illusory self motion

Perception of motion smear during visually induced illusory self motion

Effect of motion smear on perceived speed in low luminance

Effect of motion smear on perceived speed in low luminance

Direction-of-motion discrimination is facilitated by visible motion smear

Recent evidence indicates that motion smear can provide useful information for the detection and discrimination of motion. Further, it has been shown that the perception of motion smear depends critically on the density of dots in a random-dot (RD) stimulus. Therefore, in the present experiments, the contribution of perceived motion smear to direction-of-motion discrimination was evaluated using RD targets of different densities. Thresholds for direction-of-motion discrimination and the extent of perceived motion smear were determined for RD stimuli with densities of 1, 2, and 10 dots/deg(2), presented for 200 msec at a velocity of 4, 8, or 12 deg/sec. To evaluate the contribution of information about orientation from motion smear, thresholds for orientation discrimination were measured using parallel lines with the same length as the extent of perceived smear. Despite the opportunity for increased summation as RD density increases, our results indicate that direction-of-motion discrimination worsens. Because perception of motion smear is reduced with an increase in RD density, our results are consistent with a facilitation of direction-of-motion discrimination by visible motion smear.

The temporal impulse response function during smooth pursuit

Recent studies indicate that the extent of perceived motion smear is attenuated asymmetrically during smooth pursuit eye movements, based on the relative directions of the target and eye motion. We conducted two experiments to determine if the reduction of perceived smear during pursuit might be associated with an acceleration of the temporal impulse response function (TIRF). In Experiment 1, two-pulse increment sensitivity was determined during fixation and rightward pursuit for sequential flashes of a long horizontal line, presented with stimulus-onset asynchronies between 5.9 and 234 ms. In Experiment 2, temporal contrast sensitivity was measured during fixation and rightward pursuit for a vertical 1 cpd grating with retinal image velocities between 4 and 30 Hz. During pursuit, grating motion was either in the same or the opposite direction as the eye movement. TIRFs were modeled as the impulse responses of a second-order, low-pass linear system, fit to the two-pulse increment sensitivity data by an optimization procedure and to the temporal contrast sensitivity results by iterative Fourier synthesis. The results indicate that the natural temporal frequency of the fitted TIRFs was approximately 10% higher during pursuit than fixation. In Experiment 2, the increased natural frequency of the TIRF was restricted to the condition in which the grating moved spatially in the opposite direction of the pursuit eye movement. The results are consistent with the hypothesis that extra-retinal signals reduce the extent of perceived motion smear during pursuit, in part by increasing the speed of visual processing preferentially for one direction of image motion.

Invertible motion blur in video

We show that motion blur in successive video frames is invertible even if the point-spread function (PSF) due to motion smear in a single photo is non-invertible. Blurred photos exhibit nulls (zeros) in the frequency transform of the PSF, leading to an ill-posed deconvolution. Hardware solutions to avoid this require specialized devices such as the coded exposure camera or accelerating sensor motion. We employ ordinary video cameras and introduce the notion of null-filling along with joint-invertibility of multiple blur-functions. The key idea is to record the same object with varying PSFs, so that the nulls in the frequency component of one frame can be filled by other frames. The combined frequency transform becomes null-free, making deblurring well-posed. We achieve jointly-invertible blur simply by changing the exposure time of successive frames. We address the problem of automatic deblurring of objects moving with constant velocity by solving the four critical components: preservation of all spatial frequencies, segmentation of moving parts, motion estimation of moving parts, and non-degradation of the static parts of the scene. We demonstrate several challenging cases of object motion blur including textured backgrounds and partial occluders.

Asymmetrical perception of motion smear in infantile nystagmus

Normal observers perceive less motion smear if a target moves in the opposite direction of a smooth eye movement than if the target moves to produce the same retinal image speed in the same direction as the eye movement. This study investigated whether a similar asymmetrical attenuation of perceived motion smear occurs in observers with infantile nystagmus (IN). Observers ( N = 3) viewed a laser spot that moved for 100 or 125 ms to the right or left at a speed between 5 and 60°/s during the slow phase of jerk IN. After each trial, the observer adjusted the length of a bright line to match the extent of the perceived smear. Across observers, the average duration of perceived smear was 39 and 106 ms, respectively, for relative motion of the laser spot in the opposite vs. the same direction as the IN slow phase. In one observer with periodic alternating nystagmus, the direction of spot motion that produced less perceived smear reversed with an alternation in the direction of the IN slow phase. The reduction of perceived motion smear for relative target motion in the opposite direction of IN slow phases is attributed to extra-retinal signals that accompany IN. As during normal eye movements, the reduction of perceived smear for this direction of relative motion should foster the perception of clarity in the stationary visual world.

Apparent speed increases at low luminance

To investigate the effect of luminance on apparent speed, subjects adjusted the speed of a low-luminance rotating grating (0.31 cd/m(2)) to match that of a high-luminance one (1260 cd/m(2)). Above 4 Hz, subjects overestimated the speed of the low-luminance grating. This overestimation increased as a function of temporal rate and reached 30% around 10 Hz temporal rates. The speed overestimation became significant once the lower luminance was 2.4 log units lower than the high luminance comparison. Next the role of motion smear in speed overestimation was examined. First it was shown that the length of the perceived motion smear increased at low luminances. Second, the length of the visible smear was manipulated by changing the presentation time of the stimuli. Speed overestimation was reduced at shorter presentation times. Third the speed of a blurred stimulus was compared to a stimulus with sharp edges and the blurred stimulus was judged to move faster. These results indicate that the length of motion smear following a target contributes to its perceived speed and that this leads to speed overestimation at low luminance where motion traces lengthen because of increased persistence.

Signals of eye-muscle proprioception modulate perceived motion smear

Achieving clear perception during eye movements is one of the major challenges that the human visual system has to face every day. Like most light sensitive mechanisms, the human visual system has a finite integration time that may cause moving images to appear smeared. By comparing the perceived motion smear during ongoing eye movements and fixation, previous studies indicated that smear is reduced by a neural compensation mechanism that uses "extra-retinal information" about eye movements. However, it is not clear whether eye-muscle proprioception (afferent input), internal copies of efferent oculomotor commands (efference copy), or both contribute to the smear reduction. The present study found that similar reductions of perceived motion smear occur during passive eye movement (which is signaled only by eye-muscle proprioception) and during active pursuit tracking (for which efference copy signals exist as well). These results reveal a novel neural contribution for maintaining visual clarity and stand in contrast to previous reports that eye-muscle proprioception makes only a minor contribution to visual perception.

The temporal impulse response function in infantile nystagmus

Despite rapid to-and-fro motion of the retinal image that results from their incessant involuntary eye movements, persons with infantile nystagmus (IN) rarely report the perception of motion smear. We performed two experiments to determine if the reduction of perceived motion smear in persons with IN is associated with an increase in the speed of the temporal impulse response. In Experiment 1, increment thresholds were determined for pairs of successively presented flashes of a long horizontal line, presented on a 65-cd/m 2 background field. The stimulus-onset asynchrony (SOA) between the first and second flash varied from 5.9 to 234 ms. In experiment 2, temporal contrast sensitivity functions were determined for a 3-cpd horizontal square-wave grating that underwent counterphase flicker at temporal frequencies between 1 and 40 Hz. Data were obtained for 2 subjects with predominantly pendular IN and 8 normal observers in Experiment 1 and for 3 subjects with IN and 4 normal observers in Experiment 2. Temporal impulse response functions (TIRFs) were estimated as the impulse response of a linear second-order system that provided the best fit to the increment threshold data in Experiment 1 and to the temporal contrast sensitivity functions in Experiment 2. Estimated TIRFs of the subjects with pendular IN have natural temporal frequencies that are significantly faster than those of normal observers ( ca. 13 vs. 9 Hz), indicating an accelerated temporal response to visual stimuli. This increase in response speed is too small to account by itself for the virtual absence of perceived motion smear in subjects with IN, and additional neural mechanisms are considered.

Direction and extent of perceived motion smear during pursuit eye movement

Smooth pursuit eye movements superimpose additional motion on the retinal image of untracked visual targets, potentially leading to the perception of motion smear and a distortion of the perceived direction of motion. Previously, we demonstrated an attenuation of perceived motion smear when the untracked target moves in the opposite direction of an ongoing pursuit eye movement. In this study, the extent of perceived motion smear and the direction of perceived smear were compared for a single bright dot that moved in a wide range of directions with respective to horizontal pursuit at 8 deg/s. Comparable data were obtained during fixation as a control. The results indicate that a significant attenuation of perceived motion smear occurs when the dot’s motion includes a horizontal component in the opposite direction of eye movement. In contrast, the direction of perceived smear approximates the trajectory of the retinal image motion, during both fixation and pursuit. These results suggest a selective application of extra-retinal signals to compensate specific aspects of visual perception that results from the retinal image motion during smooth pursuit eye movements.