Aperture Problem Research Articles

The kinetic Orbison illusion.

In the typical Orbison illusion, the sides of a square placed on concentric circles appear to be bending toward the center of the circles. We report a motion version of the Orbison illusion (namely, the kinetic Orbison illusion). When a dot moves along a square trajectory against a background of concentric circles, the sides of the trajectory appear to bend toward the center and the corners appear to be sharpened. In the present study, observers adjusted the shape of a comparison stimulus to the shape of the perceived trajectory by bending the sides. The amount of illusion was operationally defined as the largest discrepancy between the square and adjusted shape in the comparison stimulus. It was found that the illusory bending was more than twice the static Orbison illusion and reached a maximum of 7.3% of the length of one side. Experiments including a comparison between fixation and pursuit conditions revealed that the main cause of the kinetic illusion was not motion streaks of the dot crossing background circles. We propose an alternative hypothesis based on induced motion generated by background circle motion, the direction of which is misperceived owing to the aperture problem.

Neural network‐based optical flow versus traditional optical flow techniques with thermal aerial imaging in real‐world settings

AbstractThe study explores the feasibility of optical flow‐based neural network from real‐world thermal aerial imagery. While traditional optical flow techniques have shown adequate performance, sparse techniques do not work well during cold‐soaked low‐contrast conditions, and dense algorithms are more accurate in low‐contrast conditions but suffer from the aperture problem in some scenes. On the other hand, optical flow from convolutional neural networks has demonstrated good performance with strong generalization from several synthetic public data set benchmarks. Ground truth was generated from real‐world thermal data estimated with traditional dense optical flow techniques. The state‐of‐the‐art Recurrent All‐Pairs Field Transform for the Optical Flow model was trained with both color synthetic data and the captured real‐world thermal data across various thermal contrast conditions. The results showed strong performance of the deep‐learning network against established sparse and dense optical flow techniques in various environments and weather conditions, at the cost of higher computational demand.

Reformulating Optical Flow to Solve Image-Based Inverse Problems and Quantify Uncertainty.

From meteorology to medical imaging and cell mechanics, many scientific domains use inverse problems (IPs) to extract physical measurements from image movement. To this end, motion estimation methods such as optical flow (OF) pre-process images into motion data to feed the IP, which then inverts for the measurements through a physical model. However, this combined OFIP pipeline exacerbates the ill-posedness inherent to each technique, propagating errors and preventing uncertainty quantification. We introduce a Bayesian PDE-constrained framework that transforms visual information directly into physical measurements in the context of probability distributions. The posterior mean is a constrained IP that tracks brightness while satisfying the physical model, thereby translating the aperture problem from the motion to the underlying physics; whereas the posterior covariance derives measurement error out of image noise. As we illustrate with traction force microscopy, our approach offers several advantages: more accurate reconstructions; unprecedented flexibility in experiment design (e.g., arbitrary boundary conditions); and the exclusivity of measurement error, central to empirical science, yet still unavailable under the OFIP strategy.

Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compression.

Block based motion estimation is integral to inter prediction processes performed in hybrid video codecs. Prevalent block matching based methods that are used to compute block motion vectors (MVs) rely on computationally intensive search procedures. They also suffer from the aperture problem, which tends to worsen as the block size is reduced. Moreover, the block matching criteria used in typical codecs do not account for the resulting levels of perceptual quality of the motion compensated pictures that are created upon decoding. Towards achieving the elusive goal of perceptually optimized motion estimation, we propose a search-free block motion estimation framework using a multi-stage convolutional neural network, which is able to conduct motion estimation on multiple block sizes simultaneously, using a triplet of frames as input. This composite block translation network (CBT-Net) is trained in a self-supervised manner on a large database that we created from publicly available uncompressed video content. We deploy the multi-scale structural similarity (MS-SSIM) loss function to optimize the perceptual quality of the motion compensated predicted frames. Our experimental results highlight the computational efficiency of our proposed model relative to conventional block matching based motion estimation algorithms, for comparable prediction errors. Further, when used to perform inter prediction in AV1, the MV predictions of the perceptually optimized model result in average Bjontegaard-delta rate (BD-rate) improvements of -1.73% and -1.31% with respect to the MS-SSIM and Video Multi-Method Assessment Fusion (VMAF) quality metrics, respectively, as compared to the block matching based motion estimation system employed in the SVT-AV1 encoder.

Pattern Motion Direction Is Encoded in the Population Activity of Macaque Area MT.

Direction selective neurons in macaque primary visual cortex are narrowly tuned for orientation, and are thus afflicted by the aperture problem. At the next stage of motion processing, in the middle temporal (MT) area, some cells appear to solve this problem, responding to the pattern motion direction of plaids. Models have been proposed to account for this computation, but they do not replicate the diversity of responses observed in MT. We recorded from 386 cells in area MT of two male macaques, while presenting a wide range of random-line stimuli and their compositions into noise plaids. As we broadened the range of stimuli used to probe the cells, yielding ever more challenging conditions for extracting pattern motion, the diversity of the responses observed increased, and the fraction of cells that faithfully encoded pattern motion direction shrank. However, we show here that a pattern motion signal is present at the population level. We identified four mechanisms, one never proposed before, that together might account for the observed diversity in single-cell responses. Pattern motion is thus extracted in area MT, but it is encoded across the population, and not in a small subset of pattern neurons.SIGNIFICANCE STATEMENT Some neurons in the middle temporal area of macaques solve the aperture problem, signaling the direction of motion of complex patterns. As the number of pattern types used to probe this mechanism is increased, fewer and fewer cells retain this capability. We show here that different cells fail in different ways, and that simply summing their responses averages away their failures, yielding a clear pattern motion signal. Similar encodings, which unequivocally violate the "neuron as a feature detector" hypothesis that has dominated sensory processing theories for the past 50 years, might apply throughout the brain.

The false aperture problem: Global motion perception without integration of local motion signals.

Early direction-selective neurons in the primary visual cortex are widely considered to be the main neural basis underlying motion perception even though motion perception can also rely on attentively tracking the position of objects. Because of their small receptive fields, early direction-selective neurons suffer from the aperture problem, which is assumed to be overcome by integrating inputs from many early direction-selective neurons. Because the perceived motion of objects sometimes depends on static form information and does not always match the mean direction of local motion signals, the general consensus is that motion integration is form dependent and complex. Based on the fact that early direction-selective neurons respond to motion only within a short temporal window, the present study used stroboscopic motion to test their contribution to motion perception of objects. For conditions under which the perceived motion was impaired by stroboscopic motion, the perceived motion matched the mean direction of local motion signals and was form independent. For classic conditions under which the perceived motion could not be explained by a simple form independent averaging of local motion signals, neutralizing the contribution of early direction-selective neurons using stroboscopic motion had little impact on the perceived motion, which demonstrates that the perceived motion relied on position tracking, not on early direction-selective neurons. When the perceived motion relies on position tracking, assuming that motion perception relies on early direction-selective neurons can lead to erroneously postulate the existence of complex or form-dependent integration of inputs from early direction-selective neurons. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A Comparison of Dense and Sparse Optical Flow Techniques for Low-Resolution Aerial Thermal Imagery.

It is necessary to establish the relative performance of established optical flow approaches in airborne scenarios with thermal cameras. This study investigated the performance of a dense optical flow algorithm on 14 bit radiometric images of the ground. While sparse techniques that rely on feature matching techniques perform very well with airborne thermal data in high-contrast thermal conditions, these techniques suffer in low-contrast scenes, where there are fewer detectable and distinct features in the image. On the other hand, some dense optical flow algorithms are highly amenable to parallel processing approaches compared to those that rely on tracking and feature detection. A Long-Wave Infrared (LWIR) micro-sensor and a PX4Flow optical sensor were mounted looking downwards on a drone. We compared the optical flow signals of a representative dense optical flow technique, the Image Interpolation Algorithm (), to the Lucas–Kanade (LK) algorithm in OpenCV and the visible light optical flow results from the PX4Flow in both X and Y displacements. The to LK was found to be generally comparable in performance and better in cold-soaked environments while suffering from the aperture problem in some scenes.

Ambiguous lateralization of complex binaural stimuli emerge from neurons constrained to physiologically relevant ranges of ITD and frequency—The binaural aperture

Although we perceive a spatially contiguous, integrated world, sensory systems sample information discretely via small spatial and narrow frequency receptive fields (RFs) in vision and audition, respectively. Undersampling of stimuli by narrow RFs produces ambiguous representations (e.g., the classic “aperture problem” that visual motion cannot be uniquely determined by one RF). Here we posit a similar problem for encoding interaural time differences (ITDs). Current models of ITD processing propose large ensembles of ITD-sensitive neurons spanning wide ranges of preferred ITD and characteristic frequency (CF). To account for anomalous lateralization of narrowband but robust lateralization of broadband stimuli, such models require two paradoxical assumptions that have recently come into question: (1) the existence of neurons preferring larger than physiologically possible ITDs; and (2) population biases overemphasizing small ITDs. Here, we consider an alternative account based on a highly constrained ITDxCF aperture through which the stimuli are perceived. ITDs bounded by ±π interaural phase difference and CFs by dominance weighting, where ∼600-750 Hz contributes most to lateralization. A more detailed consideration of this “binaural aperture” sheds light on how ambiguous lateralization data emerge from limited ensembles of neurons constrained to physiologically relevant ranges of ITD and frequency. [R01-DC011555 (DJT), R01-DC014948 (MJG), and R01-DC016643 (GCS).]

Design of a fast steering mirror driven by piezoelectric ceramics

To solve the small-optical aperture and stroke problem of existing fast steering mirror (FSM), we focus on the design of FSM driven by piezoelectric ceramics in space laser communication and lidar systems. The structure design of the FSM and the theoretical analysis of the piezoelectric actuators are carried out. The special structure and installation process of the mirror assembly are designed to ensure the accuracy of the mirror surface. Theoretical calculation and simulation analysis are conducted to evaluate the mirror’s output angle. The dynamic model of the FSM is established to analyze the stiffness. The operating principle and characteristic analysis results of FSM are verified by experimental tests. The results show that the FSM can provide a mechanical excursion angle of ±2.35 mrad, the closed-loop linearity of X axis and Y axis are 0.71% and 0.67%, respectively, and the closed-loop bandwidth of the FSM is 28 Hz. The surface shape accuracy of the mirror after installation can reach 1 / 50λ.

Spatio-Temporal Dual-Branch Network With Predictive Feature Learning for Satellite Video Object Segmentation

Satellite video is an important new earth observation data source that can be used to acquire large-scale dynamic information. Satellite video object segmentation (SVOS) is aimed at separating the foreground and background of satellite video and is a fundamental processing task for satellite video. To date, the state-of-the-art research into SVOS has mainly focused on unsupervised target extraction methods through hand-crafted features and post-processing operations, which are prone to obtaining incomplete contours of the targets and result in foreground aperture problem. Furthermore, the small size of targets and appearance deformation make the SVOS more difficult. Therefore, in this article, a spatio-temporal dual-branch network is proposed with predictive feature learning for the SVOS task. The proposed model consists of a temporal coherence branch and a spatial segmentation branch. In the temporal coherence branch, the Wasserstein generative adversarial network (WGAN) architecture is utilized for the future frame prediction to exploit temporal information, which captures the dynamic appearance and motion cues from the unlabeled satellite video data through predictive feature learning module in an adversarial manner. As a result, the proposed method can obtain segmentation results with temporal consistency, while avoiding the generation of optical flow images. In the spatial segmentation branch, a fully convolutional network (FCN) is used to extract the high-level spatial information of the satellite video and achieve end-to-end SVOS, without any post-processing operations. In the network implementation, boundary loss is used to solve the highly unbalanced segmentation problem caused by small size of the targets. The two branches of the network are also mutually constrained, to improve the final object segmentation results. The visual and quantitative results of three experiments all demonstrate that the proposed method outperforms the other current SVOS models.

HARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow

Event-based vision sensors produce asynchronous event streams with high temporal resolution based on changes in the visual scene. The properties of these sensors allow for accurate and fast calculation of optical flow as events are generated. Existing solutions for calculating optical flow from event data either fail to capture the true direction of motion due to the aperture problem, do not use the high temporal resolution of the sensor, or are too computationally expensive to be run in real time on embedded platforms. In this research, we first present a faster version of our previous algorithm, ARMS (Aperture Robust Multi-Scale flow). The new optimized software version (fARMS) significantly improves throughput on a traditional CPU. Further, we present hARMS, a hardware realization of the fARMS algorithm allowing for real-time computation of true flow on low-power, embedded platforms. The proposed hARMS architecture targets hybrid system-on-chip devices and was designed to maximize configurability and throughput. The hardware architecture and fARMS algorithm were developed with asynchronous neuromorphic processing in mind, abandoning the common use of an event frame and instead operating using only a small history of relevant events, allowing latency to scale independently of the sensor resolution. This change in processing paradigm improved the estimation of flow directions by up to 73% compared to the existing method and yielded a demonstrated hARMS throughput of up to 1.21 Mevent/s on the benchmark configuration selected. This throughput enables real-time performance and makes it the fastest known realization of aperture-robust, event-based optical flow to date.

Supplemental Material for The False Aperture Problem: Global Motion Perception Without Integration of Local Motion Signals

Supplemental Material for The False Aperture Problem: Global Motion Perception Without Integration of Local Motion Signals

The aperture problem for binaural hearing can explain ambiguous lateralization of complex binaural stimuli

Although we perceive a spatially contiguous, integrated world, sensory systems sample information discretely via small spatial and narrow frequency receptive fields (RFs) in vision and audition, respectively. Undersampling of stimuli by narrow RFs produces ambiguous representations (e.g., the classic “aperture problem” that visual motion cannot be uniquely determined by one RF). Here, we posit a similar problem for encoding low-frequency (<1500 Hz) interaural time differences (ITDs). Current models of ITD processing propose large ensembles of ITD-sensitive neurons spanning wide ranges of preferred ITD and characteristic frequency (CF). To account for anomalous lateralization of narrowband but robust lateralization of broadband stimuli, such models require two paradoxical assumptions that have recently come into question; (1) the existence of neurons preferring larger than physiologically possible ITDs, and (2) population biases overemphasizing small ITDs. Here, we consider an alternative account based on a narrow and weighted ITDxCF aperture through which the stimuli are perceived. ITDs are bounded by ±π interaural phase difference and CFs by dominance weighting, where ∼600–750 Hz contributes most to lateralization. A more detailed consideration of this “binaural aperture” sheds light on how lateralization data emerge from limited ensembles of neurons constrained to physiologically relevant ranges of ITD and frequency.

Adaptation to one perceived motion direction can generate multiple velocity aftereffects.

Sensory adaptation is a useful tool to identify the links between perceptual effects and neural mechanisms. Even though motion adaptation is one of the earliest and most documented aftereffects, few studies have investigated the perception of direction and speed of the aftereffect at the same time, that is the perceived velocity. Using a novel experimental paradigm, we simultaneously recorded the perceived direction and speed of leftward or rightward moving random dots before and after adaptation. For the adapting stimulus, we chose a horizontally-oriented broadband grating moving upward behind a circular aperture. Because of the aperture problem, the interpretation of this stimulus is ambiguous, being consistent with multiple velocities, and yet it is systematically perceived as moving at a single direction and speed. Here we ask whether the visual system adapts to the multiple velocities of the adaptor or to just the single perceived velocity. Our results show a strong repulsion aftereffect, away from the adapting velocity (downward and slower), that increases gradually for faster test stimuli as long as these stimuli include some velocities that match some of the ambiguous ones of the adaptor. In summary, the visual system seems to adapt to the multiple velocities of an ambiguous stimulus even though a single velocity is perceived. Our findings can be well described by a computational model that assumes a joint encoding of direction and speed and that includes an extended adaptation component that can represent all the possible velocities of the ambiguous stimulus.

Detection of Partial Discharge in GIS and Transformer Under Impulse Voltage by Fluorescent Optical Fiber Sensor

Partial discharge (PD) is a common phenomenon in high voltage (HV) equipment, which will radiate light signals outward in the process of discharge, by detecting this optical signal, we can judge whether the HV equipment have PDs or not, the conventional detection methods under impulse voltage are very complex due to the interference of electromagnetic (EM) wave, but optical method does not have this problem. Fluorescent optical fiber is not affected by numerical aperture problem, so the sensitivity of fluorescent optical fiber sensor is much higher than that of traditional quartz optical fiber sensor. In this work, the fluorescent optical fiber sensor was manufactured and applied to the PD detection in gas insulated switch (GIS) and transformer, fluorescent optical fiber was used as a probe, which was connected with the transmission optical fiber by coupler, and the photomultiplier tube was used to convert the optical signal into electrical signal for displaying by oscilloscope. The discharge signal can be detected by the sensor when the impulse voltage was applied to the GIS and transformer, and it was less affected by the interference of EM wave signals. As the PD source at different positions in GIS, we found that the optical signal attenuates greatly when the optical was blocked by some internal structures, and the attenuation was increase with the increase of distance between PD source and the sensor. The optical signals of PDs can also be detected by the sensor in the transformer, but the attenuation of the optical signal in the transformer was greater than that in GIS, it was greatly affected by the blocking of windings. The two experiments were verified in the TracePro simulation software, and the average deviations between results of experiments and simulations were less than 10%.

Multiple Aperture Shear-Interferometry (MArS): a solution to the aperture problem for the form measurement of aspheric surfaces.

Multiple Aperture Shear-Interferometry (MArS) is a shape measurement technique that uses multi-spot illumination to overcome the problem of a limited observation aperture of conventional interferometric techniques and thus considerably simplifies the measurement of optical aspheres and freeform surfaces. Using a shear interferometry setup, MArS measures the coherence function in order to obtain wave vector distributions created from multi-spot LED illumination reflected by the specimen. Based on the wave vectors we reconstruct the surface topography of aspheric lenses using an inverse ray tracing approach and prior knowledge about the individual source locations. We present the topographic measurement of two aspheric lenses with different global curvature radii measured with the same identical reflection setup. In addition, we examine the achievable accuracy of the wave vector measurement using a single light source to find physical limits of MArS.

Considerations on Bearing Capacity Factors of Rough Strip Footing Using the Stress Characteristics Method

In this study, to evaluate the bearing capacity of rough strip footing located on horizontal ground, a numerical algorithm has been proposed based on the stress characteristic lines method. To solve the aperture problem created beneath the footing, a solution procedure is suggested. The triple bearing capacity factors, the stress characteristics fields as well as geometric properties of plastic and non-plastic regions under the footing have been extracted. The obtained results are in an extremely good agreement with those presented in the literature. It was found that as the angle of internal friction increased, the bearing capacity factors and plastic span along the ground surface continuously increased. The maximum depth of the plastic domain affected by the stress characteristics fields was 1.63 times the footing's width and gradually moves toward footing edges. The depth of the non-plastic region is also increased and for $$\varphi = 45^{^\circ }$$ reaches 75% of the footing's width. Moreover, the width of the non-plastic region coincides with the footing base gradually moving toward the footing edges so that the ratio of the maximum width of the non-plastic curved wedge to footing width tends to one.

Fast hybrid numerical-asymptotic boundary element methods for high frequency screen and aperture problems based on least-squares collocation

We present a hybrid numerical-asymptotic (HNA) boundary element method (BEM) for high frequency scattering by two-dimensional screens and apertures, whose computational cost to achieve any prescribed accuracy remains bounded with increasing frequency. Our method is a collocation implementation of the high order hp HNA approximation space of Hewett et al. (IMA J Numer Anal 35:1698–1728, 2015), where a Galerkin implementation was studied. An advantage of the current collocation scheme is that the one-dimensional highly oscillatory singular integrals appearing in the BEM matrix entries are significantly easier to evaluate than the two-dimensional integrals appearing in the Galerkin case, which leads to much faster computation times. Here we compute the required integrals at frequency-independent cost using the numerical method of steepest descent, which involves complex contour deformation. The change from Galerkin to collocation is nontrivial because naive collocation implementations based on square linear systems suffer from severe numerical instabilities associated with the numerical redundancy of the HNA basis, which produces highly ill-conditioned BEM matrices. In this paper we show how these instabilities can be removed by oversampling, and solving the resulting overdetermined collocation system in a weighted least-squares sense using a truncated singular value decomposition. On the basis of our numerical experiments, the amount of oversampling required to stabilise the method is modest (around 25% typically suffices), and independent of frequency. As an application of our method we present numerical results for high frequency scattering by prefractal approximations to the middle-third Cantor set.

A Method to Detect and Track Moving Airplanes from a Satellite Video

In recent years, satellites capable of capturing videos have been developed and launched to provide high definition satellite videos that enable applications far beyond the capabilities of remotely sensed imagery. Moving object detection and moving object tracking are among the most essential and challenging tasks, but existing studies have mainly focused on vehicles. To accurately detect and then track more complex moving objects, specifically airplanes, we need to address the challenges posed by the new data. First, slow-moving airplanes may cause foreground aperture problem during detection. Second, various disturbances, especially parallax motion, may cause false detection. Third, airplanes may perform complex motions, which requires a rotation-invariant and scale-invariant tracking algorithm. To tackle these difficulties, we first develop an Improved Gaussian-based Background Subtractor (IPGBBS) algorithm for moving airplane detection. This algorithm adopts a novel strategy for background and foreground adaptation, which can effectively deal with the foreground aperture problem. Then, the detected moving airplanes are tracked by a Primary Scale Invariant Feature Transform (P-SIFT) keypoint matching algorithm. The P-SIFT keypoint of an airplane exhibits high distinctiveness and repeatability. More importantly, it provides a highly rotation-invariant and scale-invariant feature vector that can be used in the matching process to determine the new locations of the airplane in the frame sequence. The method was tested on a satellite video with eight moving airplanes. Compared with state-of-the-art algorithms, our IPGBBS algorithm achieved the best detection accuracy with the highest F1 score of 0.94 and also demonstrated its superiority on parallax motion suppression. The P-SIFT keypoint matching algorithm could successfully track seven out of the eight airplanes. Based on the tracking results, movement trajectories of the airplanes and their dynamic properties were also estimated.

Real-Time High Speed Motion Prediction Using Fast Aperture-Robust Event-Driven Visual Flow.

Optical flow is a crucial component of the feature space for early visual processing of dynamic scenes especially in new applications such as self-driving vehicles, drones and autonomous robots. The dynamic vision sensors are well suited for such applications because of their asynchronous, sparse and temporally precise representation of the visual dynamics. Many algorithms proposed for computing visual flow for these sensors suffer from the aperture problem as the direction of the estimated flow is governed by the curvature of the object rather than the true motion direction. Some methods that do overcome this problem by temporal windowing under-utilize the true precise temporal nature of the dynamic sensors. In this paper, we propose a novel multi-scale plane fitting based visual flow algorithm that is robust to the aperture problem and also computationally fast and efficient. Our algorithm performs well in many scenarios ranging from fixed camera recording simple geometric shapes to real world scenarios such as camera mounted on a moving car and can successfully perform event-by-event motion estimation of objects in the scene to allow for predictions of upto 500 ms i.e., equivalent to 10 to 25 frames with traditional cameras.