Virtual Image Plane Research Articles

Wavefront sensing with a gradient phase filter

Wavefront sensors have now become core components in the fields of metrology of optical systems, biomedical optics, and adaptive optics systems for astronomy. However, none of the designs used or proposed so far achieve simultaneously a high spatial resolution at the pupil of the tested optics and absolute measurement accuracy comparable to those of modern laser-interferometers. This paper presents an improved wavefront sensor concept that reaches both previous goals. This device named Crossed-sine phase sensor (CSPS) is based on a fully transparent gradient phase filter (GPF) placed at an intermediate location between the virtual pupil and image planes of the tested optics. The theoretical principle of the sensor is described in Fourier optics formalism. Numerical simulations confirm that a measurement accuracy of λ/100 RMS is achievable. The CSPS also offers the advantages of being quasi-achromatic and working on spatially or spectrally extended, natural or artificial light sources.

Effects of Focal Distance on Near-Field Depth Perception and Accommodative Response in a VariFocal Optical See-Through Augmented Reality Display.

Through a human-subject experiment, we investigated the effects of focal distance on depth perception and accommodative response in an optical see-through augmented reality (AR) display. The display was able to provide focus cues and was rigorously calibrated. The near-field distances ranging between 3 diopters and 1 diopter were considered as target distance. In the experiment, it was found that the perceived depth of a virtual object was significantly biased along with the focal distance of virtual image plane of the display. In addition, the experimental results implied that the perceived depth of a virtual object would be potentially more accurate in the condition where the focal distance of virtual image plane was consistent with the target distance than in the conditions where it could deviate from the target distance. Regarding accommodative response, it was found that the response to a virtual object changed along with the focal distance of virtual image plane as well as the target distance. However, the changing rate depending on target distance was less steep in the conditions where the focal distance could be mismatched with the target distance than in the condition where it was consistent with the target distance. In the consistent condition, the changing rate of accommodative responses to virtual objects were similar to that for their physical counterparts.

Initial Structure Design and Optimization of an Automotive Remote Head-Up Display

Aiming at the problems of difficult construction and occlusion of the initial structure of an automotive augmented reality head-up display (AR-HUD), a method to quickly build the initial structure of an automotive AR-HUD is proposed. Firstly, the position of the mirrors in the initial structure is calculated based on the Rodrigues rotation formula. Secondly, the position of the mirrors is restricted by constraints during optimization to prevent the problem of structural occlusion. Finally, a virtual image display system with a visual distance of 7.5 m and a field of view angle of 10° × 5° is designed. The image quality analysis of the optimized system shows that the light spots in each field of view are all within Airy spots. At the spatial cutoff frequency of the virtual image plane of the optical system, the modulation transfer function (MTF) value of the full field of view is basically greater than 0.5, and the distortion is less than 1%. Finally, using the image for simulation, the results of the simulation image are satisfying, which proves the validity and feasibility of the structural design. It provides a useful reference for the structural design of the remote head-up display system.

Image-based prescribed performance visual servoing control of a QUAV with hysteresis quantised input

ABSTRACT In this paper, a novel image-based prescribed performance visual servoing control scheme with hysteresis quantised input is proposed for quadrotor unmanned aerial vehicles (QUAVs, for short). First, based on the perspective projection principle, effective image features are defined on a image plane called virtual image plane, and the decoupled image feature dynamics are achieved with respect to target points. Then, the image feature dynamics and QUAV dynamics are combined to derive a nonlinear dynamic system, and the nonlinear system is decoupled into two subsystems based on the commonly used inner–outer loop control framework. Furthermore, the prescribed performance control-based visual servoing control scheme with hysteresis quantizer is designed, and the necessary stability proof is presented. Finally, the numerical simulations and experiments are conducted to validate the effectiveness of the proposed control scheme.

Robust Image-Based Landing Control of a Quadrotor on an Unpredictable Moving Vehicle Using Circle Features

This paper addresses the landing problem of a quadrotor on an unpredictable moving vehicle, using a robust image-based visual servoing (IBVS) method. The circle-based image moments are defined to construct image dynamics, and the passivity-like property of the circle features is preserved by reprojecting to the virtual image plane. The landing control system is decoupled into translation and rotation modules due to the rotation invariance of the proposed circle features. First, by exploiting the error transformation in the image space, a robust IBVS controller can overcome the lack of both the desired depth information of target features and the velocity feedback of the target. Next, an adaptive geometric attitude controller is developed directly using rotation matrices to avoid the singularities of Euler-angles and the ambiguity of quaternions. One benefit of the proposed scheme is that it can potentially improve the camera visibility, guarantee the transient and steady-state behaviors in image space, and be efficiently implemented on the low-cost quadrotor. Finally, The stability analysis is presented using Lyapunov stability theory on cascaded systems, and the effectiveness of the proposed control strategy is demonstrated through simulations and experiments. Note to Practitioners—The motivation of this paper is to investigate a practical control strategy for the image-based landing control of underactuated quadrotors on an unpredictable moving vehicle. In most of the existing image-based landing control schemes for underactuated quadrotors, having the prior predictive model of the moving landing vehicle to provide a feed-forward compensation during the landing maneuver. However, due to the fact that the landing environment and vehicle are primarily stochastic, resulting in no predictive models are valid in practice. Therefore, this paper suggests a robust image-based landing control strategy without the model or state of the moving landing vehicle. In particular, a novel virtual circle feature, possessing the characteristic of rotation invariance, is designed for the landing of underactuated quadrotors, which decouples the landing system and simplifies the control design. Moreover, the image feature errors are directly retained within prescribed performance funnels in the image space. As a result, the transient and steady-state landing behaviors can be implicitly guaranteed in Cartesian space. The stability and convergence of the system are analyzed mathematically and the experiment using quadrotors provides promising results. In ongoing research, we are addressing the issues of collision avoidances and unknown disturbances to provide a more realistic setup for the autonomous deployment and recovery of underactuated quadrotors in GPS-denied environments.

Effect Analysis of Nursing Intervention on Lower Extremity Deep Venous Thrombosis in Patients

In the modern era, nursing intervention is an increased commitment to patient quality and protection that allows nurses to make evidence-based healthcare decisions. The challenging characteristic of patients such as high deep venous thrombosis (DVT) and respiratory embolisms (RE) are significant health conditions that lead to post-operative severe injury and death. In this article, hybrid machine learning (HML) is used for senile patients with lower extremity fractures during the perioperative time and the clinical effectiveness of early stages nursing protocol for deep venous thrombosis of patients and nurses. A three-dimensional shape model of the user interface is shown the examined vessels, which have compression measurements mapped to the surface as colors and virtual image plane representation of DVT. The measures of comprehension have been validated using HML model segmentation experts and contrasted with paired f-tests to reduce the incidence of lower extremity deep venous thrombosis in patients and nurses.

Visual servoing of quadrotor UAVs for slant targets with autonomous object search

In this paper, an enhanced visual servoing method is designed for a quadrotor unmanned aerial vehicle (UAV) based on virtual plane image moments, under underactuation and tight coupling constraints of UAV kinematics. Moreover, in order to make the UAV search visual targets autonomously in target vicinity during flight, a flexible flight system is developed with stages of take-off, target searching, and image-based visual servoing (IBVS). With dual-camera sensor configuration, the UAV system searches targets from given directions while making localization. A virtual image plane is constructed and image moments are adopted to decouple UAV lateral movement. For a non-horizonal target, homography is utilized to construct the target plane and transform it into a horizonal plane. Backstepping techniques are used to derive the nonlinear controller to realize the IBVS strategy. Stability analysis proves global asymptotic performance of the closed-loop system. Experimental verification shows feasibility of the overall flight system and effectiveness of the visual servoing controller.

Image Dynamics-Based Visual Servo Control for Unmanned Aerial Manipulatorl With a Virtual Camera

This article proposes a visual servo control method based on image dynamics for an unmanned aerial manipulator (UAM) combining an unmanned aerial vehicle (UAV) with a multidegree-of-freedom onboard manipulator. First, taking into account the dynamic characteristics of the coupled UAM, the dynamic model of the coupled system is derived. Then, based on the perspective projection model, the appropriate image features are defined on the virtual image plane to obtain decoupled image feature dynamics. Combining image features dynamics and UAM dynamics, the position and attitude controller of the UAM flying platform with the center of gravity compensation is derived. Furthermore, the image feature motion is further applied to the positioning control of the end-effector of the onboard manipulator. In addition, the UAM motion distribution strategy is designed to coordinate the movement between the UAM flying platform and the onboard manipulator. The simulation results illustrate the performance of the proposed visual servo control based on image dynamics. As a result, the trajectories of the feature points are smooth in the real image plane, which means the UAM movement is stable. Finally, the experimental results in an outdoor environment verify the practicability and effectiveness of the proposed visual servo control method for autonomous operations. The proposed control method manages the UAM from an unknown initial position to the desired position, which can be applied to various types of aerial operations missions such as object transmission.

Image-based finite-time visual servoing of a quadrotor for tracking a moving target

This paper proposes an image-based visual servoing control method for a moving target of a quadrotor UAV (QUAV). Firstly, the dynamic image model with moving target parameters is established based on the image moment features in the virtual camera plane. For the unpredictability of the moving target in space, we use a high-order differentiator to estimate the state parameters of the moving target. In order to solve the problem of image depth information caused by a monocular camera, we derive a nonlinear finite-time linear velocity observer from the virtual image plane, which can not only estimate the linear velocity information of QUAV but also avoid the measurement of image depth. Based on the above information, we design the global finite-time controller and use Lyapunov theory to prove the finite-time stability of the system. Finally, the numerical simulations verify the convergence of the proposed control scheme, and the ROS gazebo simulations demonstrate the improved performance of the proposed control scheme in tracking error.

Research on a Visual Servoing Control Method Based on Perspective Transformation under Spatial Constraint

Visual servoing has been widely employed in robotic control to increase the flexibility and precision of a robotic arm. When the end-effector of the robotic arm needs to be moved to a spatial point without a coordinate, the conventional visual servoing control method has difficulty performing the task. The present work describes space constraint challenges in a visual servoing system by introducing an assembly node and then presents a two-stage visual servoing control approach based on perspective transformation. A virtual image plane is constructed using a calibration-derived homography matrix. The assembly node, as well as other objects, are projected into the plane after that. Second, the controller drives the robotic arm by tracking the projections in the virtual image plane and adjusting the position and attitude of the workpiece accordingly. Three simple image features are combined into a composite image feature, and an active disturbance rejection controller (ADRC) is established to improve the robotic arm’s motion sensitivity. Real-time simulations and experiments employing a robotic vision system with an eye-to-hand configuration are used to validate the effectiveness of the presented method. The results show that the robotic arm can move the workpiece to the desired position without using coordinates.

Design and Implementation of Print Advertisement Based on Computer Image Interactive Virtualization

This study aimed to further enhance and enrich users’ visual experience, provide users with a more infectious image sense, and further promote the development of multidimensional, virtualized, and interactive graphic advertising design applications. This study presents a design idea of print advertisement based on computer image interactive virtual technology. Based on the in-depth analysis of the multidimensional expression of plane image interactive virtualization, this idea carries out interactive virtual design with the help of interactive virtual image segmentation algorithm and plane advertising image element optimization technology. Through the test of image interactive edge extraction function on MATLAB platform, the simulation results show that the edge width of single side extraction results with different gray levels of the same width and different widths of the same gray level are 9 pixels, the gray values are 5a0, 110, and 200, respectively, and the edge width corresponds to 3, 9, and 15 pixels, respectively. The results show that the system can effectively enhance the image definition and obtain better print advertising design effect.

Virtual camera-based visual servoing for rotorcraft using monocular camera and gyroscopic feedback

In this paper, an image-based visual servoing (IBVS) control strategy with a virtual camera frame is proposed for multirotor vehicles. Compared to previous works, the proposed IBVS controller requires minimal sensors in the position loop: a monocular camera and gyroscope. To achieve this, the visual feature projected onto the virtual image plane is associated with the plane’s normal instead of relying on an additional attitude estimator and prior knowledge of the plane’s inclination. Furthermore, we show that the ratio velocity, when scaled by the image moment, exhibits a similar dynamics to the linear velocity. The finding allows the quantity to be recovered and used for control with a monocular camera without other metric cues. To provide feedback for the controller, an Extended Kalman filter for estimating the ratio velocity, target plane’s inclination, and relative rotation between the current and reference camera frame is developed using only monocular vision and gyroscopic measurements. To validate the proposed controller and estimation strategy, both simulation and real-world flight experiments were carried out. The quadrotor smoothly and robustly tracked both dynamic horizontal and static inclined targets, without prior knowledge of the target’s inclination. Overall, the proposed regime offers a lightweight and robust alternative IBVS solution for rotorcraft.

B-spline surface based 3D reconstruction method for deflectometry.

In the field of optical three-dimension (3-D) measurement, reconstruction usually is completed by the integration of a two-dimensional (2-D) gradient data set. Position and posture of camera and shape of the surface under test determine the location of gradient data which usually is on quadrilateral grids. This paper proposes a B-spline surface-based 3D reconstruction method for deflectometry, which reconstructs the surface under test with its 2-D gradient data set. The 2-D gradient data set consists of gradient data and the 2-D location of the gradient data in the camera coordinate system. The 2-D gradient data set is first transferred to the cameras' virtual image plane, so it locates on rectangular grids. Then, based on the properties of the B-spline basis function and characteristics of the camera, linear equations are derived to solve control points along the virtual image plane. The solved control points reconstruct the surface under test in the camera coordinate system. The property of the B-spline basis function determines the relationship between the depth of the surface and its derivative. The characteristic of the camera determines the relationship between the depth of the surface and the 2-D location of the gradient data. Meanwhile, the accuracy of the 2-D location can also be improved by the linear equations. Finally, simulated and actual experiments show that the proposed method is accurate and efficient at reconstructing surfaces in deflectometry.

Field-of-view correction in a VR HMD display with varifocal optics

Nowadays, we observe a progressive increase amount of virtual technologies application. VR devices, i.e. HMDs, are still struggling with the VAC. It is possible to “implement” the accommodation mechanism in HMDs using a varifocal system. By using focus tuneable lenses, we achieve high speed and smoothness of displacement of the virtual image plane. This solution involves also unfavourable changes of geometrical parameters of the projected image i.e. change in magnification. This can be solved by software modification parameters of the virtual camera. In this paper, the developed construction of varifocal HMD is presented, including the method of maintaining the constancy magnification. Full Text: PDF ReferencesY-J. Wang , Y-H. Lin, "Liquid crystal technology for vergenceaccommodation conflicts in augmented reality and virtual reality systems: a review", Liq. Cryst. Rev. 9. 35 (2021). CrossRef P.P. Monticone, M. Menozzi, M. Schlup, "Instrument for high-speed recording of accommodation of the human eye", Biomed. Tech. 55: 83 (2010). CrossRef G. Kramida, "Resolving the vergence-accommodation confict in headmounted displays", IEEE Trans. Vis. Comput. Graph. 22, 1912 (2016). CrossRef S. Shiwa, K. Omura, and F. Kishino, "Proposal for a 3-D display with accommodative compensation: 3DDAC," J. Soc. Inf. Disp. 4, 255 (1996). CrossRef K. Rathinavel, G. Wetzstein, H. Fuchs, "Varifocal Occlusion-Capable Optical See-through Augmented Reality Display based on Focus-tunable Optics," IEEE Trans. Vis. Comput. Graph. 25, 3125 (2019). CrossRef

Microsphere-assisted, nanospot, non-destructive metrology for semiconductor devices

As smaller structures are being increasingly adopted in the semiconductor industry, the performance of memory and logic devices is being continuously improved with innovative 3D integration schemes as well as shrinking and stacking strategies. Owing to the increasing complexity of the design architectures, optical metrology techniques including spectroscopic ellipsometry (SE) and reflectometry have been widely used for efficient process development and yield ramp-up due to the capability of 3D structure measurements. However, there has been an increasing demand for a significant reduction in the physical spot diameter used in the SE technique; the spot diameter should be at least 10 times smaller than the cell dimension (~30 × 40 μm2) of typical dynamic random-access memory to be able to measure in-cell critical dimension (CD) variations. To this end, this study demonstrates a novel spectrum measurement system that utilizes the microsphere-assisted super-resolution effect, achieving extremely small spot spectral metrology by reducing the spot diameter to ~210 nm, while maintaining a sufficiently high signal-to-noise ratio. In addition, a geometric model is introduced for the microsphere-based spectral metrology system that can calculate the virtual image plane magnification and depth of focus, providing the optimal distance between the objective lens, microsphere, and sample to achieve the best possible imaging quality. The proof of concept was fully verified through both simulations and experiments for various samples. Thus, owing to its ultra-small spot metrology capability, this technique has great potential for solving the current metrology challenge of monitoring in-cell CD variations in advanced logic and memory devices.

On-Site Global Calibration of Mobile Vision Measurement System Based on Virtual Omnidirectional Camera Model

The mobile vision measurement system (MVMS) is widely used for location and attitude measurement in aircraft takeoff and landing, and its on-site global calibration is crucial to obtaining high-accuracy measurement aimed at obtaining the transformation relationship between the MVMS coordinate system and the local-tangent-plane coordinate system. In this paper, several new ideas are proposed to realize the global calibration of the MVMS effectively. First, the MVMS is regarded as azimuth and pitch measurement equipment with a virtual single image plane at focal length 1. Second, a new virtual omnidirectional camera model constructed by three mutual orthogonal image planes is put forward, which effectively resolves the problem of global calibration error magnification when the angle between the virtual single image plane and view axis of the system becomes small. Meanwhile, an expanded factorial linear method is proposed to solve the global calibration equations, which effectively restrains the influence of calibration data error. Experimental results with synthetic data verify the validity of the proposed method.

P‐23: Image Adaptation to Human Vision (Eyeglasses Free): Full Visual‐Corrected Function in Light‐Field Near‐to‐Eye Displays

Many design factors are highly demanded in developing optimum near‐to‐eye display, such as higher resolution and wider FOV for vivid image quality, embedded vision correction for visual comfort, and light, compact and large eye box size for wearing comfort and stylish. Several optical approaches could achieve the visual correction, such as time‐multiplexed light field, space‐multiplexed light field, holographic display, multi‐focal planes, etc. In this study, based on the space‐multiplexed light field, the authors propose a special method that can adapt the variable pupil distance and deliver several virtual image planes at the same time by an algorithm of image processing. LFNEDs utilize this method which can adapt to the diversified facial shape and avoid wearing prescription glasses in using AR/VR glasses, no matter users have myopia, hyperopia, and astigmatism.

Linear Velocity-Free Visual Servoing Control for Unmanned Helicopter Landing on a Ship With Visibility Constraint

In this article, a constrained image-based visual servoing control method for the shipboard landing problem of unmanned helicopters is proposed. First, the pitch and roll motion of ship are predicted by an autoregressive (AR) model to determine an appropriate period for landing. Subsequently, a novel robust sliding mode controller without linear velocity measurements is developed on the basis of the perspective image feature in a virtual image plane. Meanwhile, a modified Chebyshev neural network (CNN) is proposed to estimate the uncertainties, including the linear acceleration of ship and translational perturbation, while an adaptive law is employed to compensate the influence of rotational disturbances. The whole controller only requires the measurements feedback of a vision sensor and an inertial measurement unit (IMU). Ulteriorly, to prevent the visual target on the ship from going beyond the field of view of camera, the constrained controller is developed by a control barrier function and a quadratic programming, where the unknown relative velocity is estimated by a velocity observer. Finally, simulations are implemented to substantiate the capability of the presented shipboard landing control method.

X-ray coherent diffraction imaging with an objective lens: Towards three-dimensional mapping of thick polycrystals

We report on a new x-ray imaging method, which combines the high spatial resolution of coherent diffraction imaging with the ability of dark field microscopy to map grains within thick polycrystalline specimens. An x-ray objective serves to isolate a grain and avoid overlap of diffraction spots. Iterative oversampling routines are used to reconstruct the shape and strain field within the grain from the far field intensity pattern. The limitation on resolution caused by the finite numerical aperture of the objective is overcome by the Fourier synthesis of several diffraction patterns. We demonstrate the method by an experimental study of a ~500 nm Pt grain for the two cases of a real and a virtual image plane. In the latter case the spatial resolution is 13 nm rms. Our results confirm that no information on the pupil function of the lens is required and that lens aberrations are not critical.

Robust Vision-Based Control of a Rotorcraft UAV for Uncooperative Target Tracking.

This paper investigates the problem of using an unmanned aerial vehicle (UAV) to track and hover above an uncooperative target, such as an unvisited area or an object that is newly discovered. A vision-based strategy integrating the metrology and the control is employed to achieve target tracking and hovering observation. First, by introducing a virtual camera frame, the reprojected image features can change independently of the rotational motion of the vehicle. The image centroid and an optimal observation area on the virtual image plane are exploited to regulate the relative horizontal and vertical distance. Then, the optic flow and gyro measurements are utilized to estimate the relative UAV-to-target velocity. Further, a gain-switching proportional-derivative (PD) control scheme is proposed to compensate for the external interference and model uncertainties. The closed-loop system is proven to be exponentially stable, based on the Lyapunov method. Finally, simulation results are presented to demonstrate the effectiveness of the proposed vision-based strategy in both hovering and tracking scenarios.