Camera Path Research Articles

Improving the Manipulability of a Redundant Arm Using Decoupled Hybrid Visual Servoing

This study proposes a hybrid visual servoing technique that is optimised to tackle the shortcomings of classical 2D, 3D and hybrid visual servoing approaches. These shortcomings are mostly the convergence issues, image and robot singularities, and unreachable trajectories for the robot. To address these deficiencies, 3D estimation of the visual features was used to control the translations in Z-axis as well as all rotations. To speed up the visual servoing (VS) operation, adaptive gains were used. Damped Least Square (DLS) approach was used to reduce the robot singularities and smooth out the discontinuities. Finally, manipulability was established as a secondary task, and the redundancy of the robot was resolved using the classical projection operator. The proposed approach is compared with the classical 2D, 3D and hybrid visual servoing methods in both simulation and real-world. The approach offers more efficient trajectories for the robot, with shorter camera paths than 2D image-based and classical hybrid VS methods. In comparison with the traditional position-based approach, the proposed method is less likely to lose the object from the camera scene, and it is more robust to the camera calibrations. Moreover, the proposed approach offers greater robot controllability (higher manipulability) than other approaches.

Sleep Deprivation Adversely Impacts Resident Performance for Simulated Arthroscopy

PurposeThe purpose of the study was to assess the performance of residents in orthopaedics before and after a 24-hour shift on a shoulder arthroscopy simulator. The primary study endpoint was an overall performance score (OPS) generated by the simulator.MethodsA prospective, comparative study of 120 simulator trials by 10 resident junior surgeons was performed in our university hospital’s simulation center between May and November 2018. To avoid memorization bias, all participants performed the same exercise 10 times on a VirtaMed ArthroS simulator prior to the study. Each resident’s performance (the OPS, the operating time, the proportion of procedures with iatrogenic lesions, the camera path length and the hook path length) in two different simulated arthroscopy exercise tasks was assessed once before and once after a 24-hour shift. This sequence was performed three times during the semester, and the change over time in performance was also evaluated.ResultsThe OPS was significantly lower after the night shift (P = 0.035 for the first exercise, and P = 0.025 for the second).ConclusionIn a group of previously trained resident junior surgeons, overall performance with an arthroscopy simulator was significantly worse after a 24-hour shift. The study of secondary parameters of the OPS and the subgroup analysis based on the sleep time and Epworth score vary depending on the type of exercise performed arthroscopically. However, the use of a simulator after a night shift did not prevent the trainee from improving his/her level of performance over time.Level of EvidenceII, a prospective, comparative study

Image-based 3D reconstruction precision using a camera mounted on a robot arm

Abstract In this paper, we study 3D object reconstruction based on a set of 2D images. In order to get the best camera path that increases accuracy we focus on this strategy to be used. Euclidean 3D image-based reconstruction is developed in three steps, which are primitive extraction, correspondence of these primitives and then triangulation. The extraction and triangulation are purely geometrical, whereas the matching step can have precision issues especially in the case of noisy images. An experimental study is carried out where a camera is attached to a robot arm and moved precisely relative to a scene containing a checkerboard calibration pattern. The reconstruction results are compared with values of motion given to the robot. A geometric and analytical study of the impact of the motion of the camera with respect to the scene on the error of a 3D image-based reconstructed point was also carried out. It has been demonstrated that the impact of a correspondence error on the reconstruction accuracy point varies drastically depending on the image capture strategy.

Optimized hybrid decoupled visual servoing with supervised learning

This study proposes an optimized hybrid visual servoing approach to overcome the imperfections of classical two-dimensional, three-dimensional and hybrid visual servoing methods. These imperfections are mostly convergence issues, non-optimized trajectories, expensive calculations and singularities. The proposed method provides more efficient optimized trajectories with shorter camera path for the robot than image-based and classical hybrid visual servoing methods. Moreover, it is less likely to lose the object from the camera field of view, and it is more robust to camera calibration than the classical position-based and hybrid visual servoing methods. The drawbacks in two-dimensional visual servoing are mostly related to the camera retreat and rotational motions. To tackle these drawbacks, rotations and translations in Z-axis have been separately controlled from three-dimensional estimation of the visual features. The pseudo-inverse of the proposed interaction matrix is approximated by a neuro-fuzzy neural network called local linear model tree. Using local linear model tree, the controller avoids the singularities and ill-conditioning of the proposed interaction matrix and makes it robust to image noises and camera parameters. The proposed method has been compared with classical image-based, position-based and hybrid visual servoing methods, both in simulation and in the real world using a 7-degree-of-freedom arm robot.

Ten hours of simulator training in arthroscopy are insufficient to reach the target level based on the Diagnostic Arthroscopic Skill Score

Simulator arthroscopy training has gained popularity in recent years. However, it remains unclear what level of competency surgeons may achieve in what time frame using virtual training. It was hypothesized that 10h of training would be sufficient to reach the target level defined by experts based on the Diagnostic Arthroscopic Skill Score (DASS). The training concept was developed by ten instructors affiliated with the German-speaking Society of Arthroscopy and Joint Surgery (AGA). The programme teaches the basics of performing arthroscopy; the main focus is on learning and practicing manual skills using a simulator. The training was based on a structured programme of exercises designed to help users reach defined learning goals. Initially, camera posture, horizon adjustment and control of the direction of view were taught in a virtual room. Based on these skills, further training was performed with a knee model. The learning progress was assessed by quantifying the exercise time, camera path length and instrument path length for selected tasks. At the end of the course, the learners' performance in diagnostic arthroscopy was evaluated using DASS. Participants were classified as novice or competent based on the number of arthroscopies performed prior to the assessment. Except for one surgeon, 131 orthopaedic residents and surgeons (29 women, 102 men) who participated in the seven courses agreed to anonymous data analysis. Fifty-eight of them were competents with more than ten independently performed arthroscopies, and 73 were novices, with fewer than ten independently performed arthroscopies. There were significant reductions in exercise time, camera path length and instrument path length for all participants after the training, indicating a rapid increase in performance. No difference in camera handling between the dominant and non-dominant sides was found in either group. The competents performed better than the novices in various tasks and achieved significantly better DASS values on the final performance test. Our data have demonstrated that arthroscopic skills can be taught effectively on a simulator, but a 10-h course is not sufficient to reach the target level set by experienced arthroscopists. However, learning progress can be monitored more objectively during simulator training than in the operating room, and simulation may partially replace the current practice of arthroscopic training. III.

Module-Based Arthroscopic Knee Simulator Training Improves Technical Skills in Naive Learners: A Randomized Trial

PurposeTo compare the effectiveness, in comparison to a control group (C), of module-based training (MBT) and traditional learning (TL) as a means of acquiring arthroscopic skills on an arthroscopic surgery simulator.MethodsThirty health sciences students with no previous arthroscopy experience were recruited and randomized into 1 of 3 groups: MBT, TL, or C (1:1:1 ratio). Participants in MBT were required to independently practice on a VirtaMed ArthroS simulator (VirtaMed AG, Zurich, Switzerland) for a minimum of 2 hours per week, whereas TL received one-on-one coaching by a senior orthopaedic resident for 15 minutes per week. The control group received no training. All groups were assessed at baseline and at 4 weeks based on objective measures generated by the surgical simulator (procedure time, camera path length, meniscus cutting score, detailed visualization, safety score and total score), and subjective ratings scales (Objective Assessment of Arthroscopic Skill [OAAS] global assessment form, and Competency-Based Assessment form).ResultsParticipants in the MBT group trained on average 113 min/week whereas the TL group trained on average 24 min/week. Three-way repeated-measures analysis of variance showed significant group by time interactions for procedure time (P = .006), camera path length (P = .008), safety score (P = .013), total score (P = .003), OAAS form (P < .001), and Competency-Based Assessment form (P < .001). MBT group was superior to C group for procedure time (P = .02), camera path length (P = .003), total score (P = .004), and OAAS form (P = .021), but there were no significant post-hoc differences between MBT and TL groups, or TL and C groups after Bonferroni correction. Total practice time explained 37.5% of the final simulator total score variance.ConclusionsKnee arthroscopy simulation training with self-learning modules can improve skills in areas such as procedure time, camera path length, and total score in untrained participants compared with a control group.Clinical RelevanceModule-based simulation training provides additional training time and improves technical skills in naive health science students. It is hoped that this effect can be preserved and applied to junior resident developing in a busy residency program.

Origami-inspired New Material Feeding Mechanism for Soft Growing Robots to Keep the Camera Stay at the Tip by Securing its Path

Soft growing robots that extend through tip eversion have attracted considerable interest owing to their unique locomotion. Although the availability of visual feedback from the tip of such robots would greatly enhance their usefulness in field explorations, given that the material at the tip continually moves as the robot grows, mounting a camera at the tip of the robot and securing it in place during robot growth has been a major challenge. Previous designs have severely impeded these robots’ intrinsic advantages, and it remains challenging to ensure that the camera remains at the tip without encumbering the robots’ natural ability to morph its shape and grow along or over an obstacle. In this letter, for the first time, we propose a method to maintain the camera at the tip during robot growth, while retaining the compliant nature of the robot. An origami-inspired new material feeding mechanism is designed to secure the camera's path from the base to the tip, which allows us to control the camera's growing speed independently of the robot. We experimentally investigate the required control parameters and present a vision-based control method for camera position adjustment. Finally, we demonstrate the feasibility of the proposed design in simulated experimental scenarios, including a narrow passage and a cluttered environment

A virtual environment for evaluation of computer vision algorithms under general airborne camera imperfections

AbstractThis paper describes a camera simulation framework for validating machine vision algorithms under general airborne camera imperfections. Lens distortion, image delay, rolling shutter, motion blur, interlacing, vignetting, image noise, and light level are modelled. This is the first simulation that considers all temporal distortions jointly, along with static lens distortions in an online manner. Several innovations are proposed including a motion tracking system allowing the camera to follow the flight log with eligible derivatives. A reverse pipeline, relating each pixel in the output image to pixels in the ideal input image, is developed. It is shown that the inverse lens distortion model and the inverse temporal distortion models are decoupled in this way. A short-time pixel displacement model is proposed to solve for temporal distortions (i.e. delay, rolling shutter, motion blur, and interlacing). Evaluation is done by several means including regenerating an airborne dataset, regenerating the camera path on a calibration pattern, and evaluating the ability of the time displacement model to predict other frames. Qualitative evaluations are also made.

The Arthroscopic Surgery Skill Evaluation Tool Global Rating Scale is a Valid and Reliable Adjunct Measure of Performance on a Virtual Reality Simulator for Hip Arthroscopy

The purpose of this study is to further evaluate the construct validity and interobserver reliability of a hip arthroscopy virtual simulator using the Arthroscopic Surgery Skill Evaluation Tool (ASSET) global rating scale. Thirty participants (23 male/7 female) completed a diagnostic arthroscopy and a loose body retrieval simulation on the VirtaMed Arthros Hip Simulator (Zurich, Switzerland) twice at a minimum of 1 week apart. Subjects consisted of 12 novices (medical students, postgraduate year [PGY] 1-2), 5 intermediate trainees (PGY3-4), 9 senior trainees (PGY5 and fellows), and 4 attending faculty. Simulator metrics were recorded and then compiled to generate a total simulator score (TSS). The loose body retrieval was graded using the ASSET scoring tool. Inter-rater and intrarater reliability for the ASSET for 2 blinded raters and construct validity of the ASSET and the TSS were calculated. Correlation between the TSS, ASSET and individual simulator metrics was determined. Prior simulation experience (P ≤ 0.01) correlated with higher TSS and higher ASSET, while video game experience correlated with higher TSS on the diagnostic module only (P= 0.004). There was a significant difference in ASSET score among all experience groups (P < 0.04). Novices had the lowest mean ASSET whereas experts had the highest mean ASSET with a difference of 17.4 points. Overall performance on the surgical module significantly correlated with the ASSET score (r= 0.444, P= 0.016). There was a significant positive correlation among higher ASSET and number of loose bodies retrieved, operation time, camera path and grasper path length, and percentage of cartilage injury. ASSET demonstrated excellent intrarater reliability and showed substantial or better inter-reliability in 8 of 9 domains. The VirtaMed hip arthroscopy simulator demonstrated good construct validity and excellent reliability for simulator-based metrics and ASSET score. Use of both simulator metrics and ASSET offers a more comprehensive performance assessment on hip arthroscopy simulation than either measure alone. As virtual reality simulation for arthroscopy becomes more commonplace in orthopaedic training, evaluation of the most effective objective and subjective measures of performance is necessary to optimize simulation training.

A new evolving mechanism of genetic algorithm for multi-constraint intelligent camera path planning

A new evolving mechanism of genetic algorithm for multi-constraint intelligent camera path planning

A Hierarchical Superpixel-Based Approach for DIBR View Synthesis.

View synthesis allows observers to explore static scenes using aligned color images and depth maps captured in a preset camera path. Among the options, depth-image-based rendering (DIBR) approaches have been effective and efficient since only one pair of color and depth map is required, saving storage and bandwidth. The present work proposes a novel DIBR pipeline for view synthesis that properly tackles the different artifacts that arise from 3D warping, such as cracks, disocclusions, ghosts, and out-of-field areas. A key aspect of our contributions relies on the adaptation and usage of a hierarchical image superpixel algorithm that helps to maintain structural characteristics of the scene during image reconstruction. We compare our approach with state-of-the-art methods and show that it attains the best average results in two common assessment metrics under public still-image and video-sequence datasets. Visual results are also provided, illustrating the potential of our technique in real-world applications.

Infrared video image stabilization algorithm based on joint camera path

To solve the problem of infrared video image stabilization caused by parallax variation, the infrared video image stabilization algorithm based on joint camera path was proposed, which included four steps: preprocessing, joint camera path solving, multi-path optimization and motion compensation. Firstly, histogram equalization, feature point extraction, matching processing and pre-mapping were needed for infrared image. Then, each frame of image was divided into m×n grid cells, and local homography matrixs obtained from the corresponding grid in each frame were multiplied by the mapping motion representation based on the grid to obtain the joint camera path. Then, the path of the joint camera was smoothed by the strategy of ‘multi-path optimization’. Finally, the smooth path was used to stabilize the video. The experimental results show that this method can effectively deal with the nonlinear motion caused by parallax, which is better than the traditional image stabilization algorithm, and can achieve good image stabilization even when the feature points have partial occlusion.

Calculation of Noise Components for Bidirectional Path Tracing with Photon Maps

The classic Monte-Carlo ray tracing is a powerful method which allows to simulate virtually all effects in ray optics, but it may be inadmissibly slow for many cases, such as calculation of images seen by a lens or pin-hole camera. In this cases another stochastic method is more efficient such as the bi-directional ray Monte-Carlo tracing with photon maps (BDPM). The level of noise i.e. the r.m.s. (root mean square) of pixel luminance calculated in one iteration of the method, depends on various parameters of the method, such as the number of light and camera paths, radius of integration sphere etc. so it is desirable to be able to predict this dependence to choose optimal parameters of the method. It was shown that this r.m.s is a sum of 3 functions scaled by reverse number of camera and light rays. These functions themselves are independent of the number of rays, so knowing them one can predict the noise for any number of rays and thus find the optimal one. These functions are a sort of correlations and their calculation from ray tracing is not a trivial problem. In this paper we describe a practical method of calculation and demonstrate the usage of its results for the choice of ray number.

Capturing Subjective First-Person View Shots with Drones for Automated Cinematography

We propose an approach to capture subjective first-person view (FPV) videos by drones for automated cinematography. FPV shots are intentionally not smooth to increase the level of immersion for the audience, and are usually captured by a walking camera operator holding traditional camera equipment. Our goal is to automatically control a drone in such a way that it imitates the motion dynamics of a walking camera operator, and, in turn, capture FPV videos. For this, given a user-defined camera path, orientation, and velocity, we first present a method to automatically generate the operator’s motion pattern and the associated motion of the camera, considering the damping mechanism of the camera equipment. Second, we propose a general computational approach that generates the drone commands to imitate the desired motion pattern. We express this task as a constrained optimization problem, where we aim to fulfill high-level user-defined goals, while imitating the dynamics of the walking camera operator and taking the drone’s physical constraints into account. Our approach is fully automatic, runs in real time, and is interactive, which provides artistic freedom in designing shots. It does not require a motion capture system, and works both indoors and outdoors. The validity of our approach has been confirmed via quantitative and qualitative evaluations.

Constructing big panorama from video sequence based on deep local feature

Constructing high-quality panorama is a fundamental task in both computer vision and computer graphics communities, thus, leading to many image stitching approaches for panorama construction. However, panorama constructed by traditional image stitching has a limited angle of view and has an expensively computational cost. To defend the issues, in this paper, we proposed to use video sequence as input for constructing big panorama, resulting in a high-quality panoramic image, the presented method is built on the stabilized video and robust feature tracking method. Specifically, the input video sequences are captured by moving hand cameras which can be any type of consumer-level camera. To mitigate the affections from rolling shutters in videos, a novel video stabilization method is introduced to filter the unstable camera's path, then resulting in a stabilized video for panorama construction. Additionally, a deep local feature-based feature tracking method is proposed to produce feature correspondences between consecutive video frames for camera motion estimation used in both video stabilization and image stitching. Finally, a comprehensive experiment conducted on the benchmarking datasets is presented to demonstrate the effectiveness of the proposed method.

Simulation-Based Arthroscopic Skills Using a Spaced Retraining Schedule Reduces Short-Term Task Completion Time and Camera Path Length

To investigate whether acquiring basic knee arthroscopic skills via a spaced retraining schedule could prevent skills deterioration and achieve further skills improvement. In the learning phase, 16 residents with no previous hands-on experience in practicing arthroscopic skills were asked to perform basic arthroscopic tasks on a simulator until they attained perfect scores in each task. Immediately after completing the learning phase, a pretest was performed to assess their performance. Next, they were randomly assigned into 2 groups. The spaced retraining group, which undertook a spaced repetitive training phase with a fixed-time interval, returned on days 2, 4 and 6 to repeat the same tasks for 20 minutes per day, whereas the control group did nothing. On day 7, all participants performed a posttest. A 2× 2 mixed analysis of variance model was used for statistical analysis. Significant differences between the 2 groups were found in task completion time (P= .003) and camera path length (P= .043) but not cartilage injury (P= .186). Residents in the spaced retraining group decreased their task completion time (163.2 ± 23.9 seconds) whereas the task time in the control group increased (351.3 ± 25.5 seconds). The same pattern was found with the camera path length. Implementing a spaced retraining schedule in 1 week resulted in a reduced task completion time and camera path length but no significant reduction in cartilage injury. It appears that introducing a spaced retraining schedule to retain arthroscopic skills acquired through massed learning may be advantageous. In consideration of the training time available to residents and the trend toward massed learning, this spaced retraining schedule may offer a cost-effective and convenient way for residents to maintain and improve their basic arthroscopic skills with no significant increase in time invested.

Real-time video stabilization via camera path correction and its applications to augmented reality on edge devices

With the rapid development of edge devices such as mobile phones, in the past decade, many videos have appeared on the professional video website and they are easy to retrieve. However, most of the videos captured by mobile cameras are jittery, and even motion blurred. These low-quality videos may affect the experience of users. Thus, the problem that how to eliminate the jittery issues, and make unstable video became stable one is very urgent. To enhance the stability of low-quality video, in this paper, we propose a novel method for video stabilization. The proposed method is called SimpleStab, and consists of motion estimation, trajectory smoothing, and compositing image. The SimpleStab is not only able to process offline videos but also can deal with live video streaming due to the novel architecture. We conduct a comprehensive experiment on the benchmarking dataset and make comparison with the state-of-the-art approaches. Experimental results show that the performance of SimpleStab is superior to the state-of-the-art methods.

Virtual reality simulator improves the acquisition of basic arthroscopy skills in first-year orthopedic surgery residents

IntroductionArthroscopy training using a virtual reality (VR) simulator is said to improve the training of orthopedic surgery residents, although it has never been evaluated in a large representative population of first-year residents. HypothesisWe hypothesized that first-year residents who train on a VR simulator would improve their basic arthroscopy skills more than residents who use other training methods. The primary aim was to compare various arthroscopy-learning techniques after 6 months of training. Population and methodsThe study population consisted of 107 first-year residents who were tested twice on a VR arthroscopy simulator (December 2017 and June 2018). The residents were divided into three groups: no specific arthroscopy training (A), non-specific and one-off arthroscopy training (B), 6 months of VR arthroscopy simulator training (C). During the testing, they had to perform the Periscoping exercise (orientation of angled scope) and the Catch the Stars Glenohumeral exercise (extraction of loose bodies). The parameters analyzed were time (s), camera alignment relative to horizontal (%), camera path length (cm) and grasper path length (cm). ResultsAfter 6 months, there was a significant difference between groups during the Periscoping exercise in the time (A: 137.8 s; B: 126.7 s; C: 92.2 s) (p<0.0001), camera alignment (A: 93%; B: 98%; C: 97%) (p=0.0028), camera path length (A: 116.9cm; B: 112.5cm; C: 67.3cm) (p<0.0001) and during the Catch the Stars Glenohumeral exercise in the time (A: 112.2 s; B: 103 s; C: 61.4 s) (p<0.0001), camera path length (A: 46.3cm; B: 40.9cm; C: 32.9cm) (p<0.0153) and grasper path length (A: 146.4cm; B: 142.2cm; C: 95.8cm) (p<0.0001). DiscussionThe residents who participated in the VR arthroscopy simulator training program for 6 months had better results when performing practical exercises and standard arthroscopy tasks than those who did not receive any training or only received only one-off training. Their final performance indicated technical mastery that the other residents had not achieved. Level of EvidenceII, Prospective, comparative, non-randomized study.

Evaluation of arthroscopic skills with a virtual reality simulator in first-year orthopaedic residents.

This study of residents' initial performance was performed to determine which factors predisposed residents for success in demonstrating the best arthroscopic skills. Each orthopaedic first-year resident was officially invited to take part in a one hour evaluation on a VirtaMed™ ArthroS™ simulator. On the FAST module, the Periscoping exercise was chosen to test for use of angled optics. The Shoulder Module was chosen to test their ability to extract intra-articular foreign bodies using the Catch the Stars exercise. The variables such as time, camera alignment, camera path length, and grasper path length were analysed. Residents completed a questionnaire prior to the evaluation. Their results were analysed according to gender, orientation assessment, and surgical history. A total of 34 women and 82 men were included in the study. In the Periscoping exercise, a significant difference between women and men in the time variables was noticed (275 ± 82 and 195 ± 71; p < 0.00001) and camera path length (207 ± 60 and 170 ± 66; p = 0.00094). For the Catch the Stars exercise, there was a significant difference between women and men for the time values (249 ± 114 and 201 ± 99; p = 0.01246) and grasper path length (290 ± 130 and 229 ± 108; p = 0.00493). After multivariate analysis, no influence of self-assessed spatial perception (p=0.1), number of arthroscopic procedures (p=0.39), or laparoscopic procedures (p=0.43) to which they had already assisted was found. This study shows a significant difference in skills regarding spatial recognition and triangulation related to gender at the beginning of specialization training. It also demonstrates that male medical students are more attracted by surgical departments during their medical training.

3D Viewpoint Estimation Based on Aesthetics

Research on viewpoint in 3D scenes has been a cutting-edge subject in computer vision and 3D scene understanding. Calculating a high quality 3D viewpoint in virtual scenes is conducive to a more comprehensive analysis of 3D graphic structure, which can contribute to an overall understanding of virtual environments. 3D viewpoint research can also be applied to the analysis of structural relationships in virtual scenes and to exploit hidden hierarchical structures, thus it plays a role in fields such as scientific visualization computing or image-based scene modeling. Viewpoint research combined with relevant research results of human visual perception or visual psychology can be used to analyze the visual interest and focus areas of 3D objects in human vision, thus effectively improving the quality of mesh simplification and rendering efficiency, as well as providing an optimization basis for highly complex scenes. In fields such as 3D computer games, virtual reality, and landscape animation, research on high-quality viewpoints can be adopted to optimize global lighting and illumination. Additionally, practical applications can be realized through user attention analysis and aesthetic improvement strategies based on high-quality viewpoints, such as intelligent modeling and scene optimization calculations. In common fields such as 3D model retrieval, practical applications can be realized for saliency research, scientific visualization computing, and medical 3D imaging. In-depth research has been performed and extensive applications have been developed that make use of viewpoint analysis. In this paper, we introduce an optimization strategy to calculate high-quality virtual viewpoints for aesthetic images. A novel framework is proposed for modifying and optimizing the viewpoint calculation model by combining a multibranch CNN and a viewpoint correction method, thus realizing rendered images with a higher aesthetic quality. By attempting to fully integrate the visual perception with the geometric information calculation, we expect this method to achieve more comprehensive applications in many practice areas in the future, such as the realization of aesthetically based virtual camera path planning and analysis of the aesthetic characteristics of virtual environments.