Robotic Camera Research Articles

A Review of Camera Viewpoint Automation in Robotic and Laparoscopic Surgery

Complex teleoperative tasks, such as surgery, generally require human control. However, teleoperating a robot using indirect visual information poses many technical challenges because the user is expected to control the movement(s) of the camera(s) in addition to the robot’s arms and other elements. For humans, camera positioning is difficult, error-prone, and a drain on the user’s available resources and attention. This paper reviews the state of the art of autonomous camera control with a focus on surgical applications. We also propose potential avenues of research in this field that will support the transition from direct slaved control to truly autonomous robotic camera systems.

Greenhouse microclimatic environment controlled by a mobile measuring station

This paper investigates a greenhouse microclimatic environment controlled by a mobile measuring station with the aim of improving performance by using wireless sensor networks (WSN) technology. The algorithms for the mobile measuring station that perform navigation tasks are called Bug algorithms. The existing potential field method based algorithms are improved with an RSSI signal propagation model and implemented on a two-wheel driven robot developing system and their performances are measured and analyzed. The implementation part is done on Boe-Bot equipped with SunSPOT enabling the wireless control ability. The control surface is generally made with LabVIEW and a relational database. Control strategy selection system is supported by Fuzzy Analytic Hierarchy Process (FAHP). Navigation of the mobile measuring robot can be done manually, relying on the visual data from the robot's camera, or it can be switched to automatic mode where the developed algorithm does the navigation job. Mobile robot navigation is based on the potential field method considering a wide range of energy-aware parameters.

Utilization of the Oculus Rift HMD in Mobile Robot Teleoperation

This paper mentions some problems related to utilization of a head-mounted display (HMD) for remote control of mobile robots by a human operator and also presents a possible solution. Considered is specifically the new HMD device called Oculus Rift, which is a very interesting device because of its great parameters and low price. The device is described in the beginning, together with some of the specific principles of the Oculus 3D display. Then follows the design of a new graphical user interface for teleoperation, with main focus on visualization of stereoscopic images from robot cameras. Demonstrated is also a way how to display additional data and information to the operator. The overall aim is to create a comfortable and highly effective interface suitable both for exploration and manipulation tasks in mobile robotics.

Toward Reactive Vision-Guided Walking on Rough Terrain: An Inverse-Dynamics Based Approach

This work presents a method to handle walking on rough terrain using inverse dynamics control and information from a stereo vision system. The ideal trajectories for the center of mass (CoM) and the next position of the feet are given by a pattern generator. The pattern generator is able to automatically find the footsteps for a given direction. Then, an inverse dynamics control scheme relying on a quadratic programming optimization solver is used to let each foot go from its initial to final position, controlling also the CoM and the waist. A 3D model reconstruction of the ground is obtained through the robot cameras located on its head as a stereo vision pair. The model allows the system to know the ground structure where the swinging foot is going to step on. Thus, contact points can be handled to adapt the foot position to the ground conditions.

Chain of refined perception in self-optimizing assembly of micro-optical systems

Abstract. Today, the assembly of laser systems requires a large share of manual operations due to its complexity regarding the optimal alignment of optics. Although the feasibility of automated alignment of laser optics has been shown in research labs, the development effort for the automation of assembly does not meet economic requirements – especially for low-volume laser production. This paper presents a model-based and sensor-integrated assembly execution approach for flexible assembly cells consisting of a macro-positioner covering a large workspace and a compact micromanipulator with camera attached to the positioner. In order to make full use of available models from computer-aided design (CAD) and optical simulation, sensor systems at different levels of accuracy are used for matching perceived information with model data. This approach is named "chain of refined perception", and it allows for automated planning of complex assembly tasks along all major phases of assembly such as collision-free path planning, part feeding, and active and passive alignment. The focus of the paper is put on the in-process image-based metrology and information extraction used for identifying and calibrating local coordinate systems as well as the exploitation of that information for a part feeding process for micro-optics. Results will be presented regarding the processes of automated calibration of the robot camera as well as the local coordinate systems of part feeding area and robot base.

Dynamic liquefaction of Jurassic sand dunes: processes, Origins, and implications

ABSTRACTSoft‐sediment deformation occurs in ancient eolian dune deposits, but understanding its extent and the conditions and forces behind the deformation often remains elusive. Here we gain insight into the aerial extent and environmental conditions related to intense soft‐sediment deformation, based on an exceptional three dimensional (3D) exposure of eolian dune deposits and preserved geomorphic landscape expressions in the Jurassic Navajo Sandstone at White Pocket, Vermilion Cliffs National Monument, Arizona. Deformation features include elongate northeast trending decameter‐scale mounds and raised ridges cored by deformed and upturned eolian dune sets, overlain by a massive blanket sandstone with breccia blocks. The geomorphic mounds display ~40–60 m spacing, roughly perpendicular to the southerly paleoflow dune foreset directions. The geometry of the deformation is imaged by oblique aerial photography using cameras mounted on a remote control airplane and high resolution panoramas with a robotic camera mount.We interpret the exquisitely preserved deformation features as liquefaction‐induced ground failure, consistent with theoretical and laboratory studies of deformation in saturated sand. A shallow water table affected by differential dune loading facilitated lateral spreading and failure. The transition to steady‐state flow liquefaction near the top of the shallow water table destroyed original sedimentary structure, creating a massive sand blanket that entrained brecciated blocks ripped up during flow. The water‐pressurized, upwelled sediment created the mounds (a relief inversion). In underlying deformed dune sets, deformation was intense but sedimentary structure was not obliterated as deformation progressed from initial cyclic mobility to strain‐softening, but ceased prior to steady‐state flow liquefaction. The spatial extent, topographic relief, and intensity of dynamic deformation suggest an origin of strong ground motion driven by long‐duration surface waves of a large earthquake (> Mw 7–8), possibly related to back arc thrusting from Jurassic subduction of the Pacific plate under North America. Copyright © 2014 John Wiley & Sons, Ltd.

Solo surgery – Early results of robot-assisted three-dimensional laparoscopic hysterectomy

Objective: Report of our initial experience in laparoscopic hysterectomy by a solo surgeon using a robotic camera system with three-dimensional visualisation. Material and methods: This novel device (Einstein Vision®, B. Braun, Aesculap AG, Tuttlingen, Germany) (EV) was used for laparoscopic supracervical hysterectomy (LASH) performed by one surgeon. Demographic data, clinical and surgical parameters were evaluated. Our first 22 cases, performed between June and November 2012, were compared with a cohort of 22 age-matched controls who underwent two-dimensional LASH performed by the same surgeon with a second surgeon assisting. Results: Compared to standard two-dimensional laparoscopic hysterectomy, there were no significant differences regarding duration of surgery, hospital stay, blood loss or incidence of complications. The number of trocars used was significantly higher in the control group (p <.0001). All hysterectomies in the treatment group were performed without assistance of a second physician. Conclusion: Robot-assisted solo surgery laparoscopic hysterectomy is a feasible and safe procedure. Duration of surgery, hospital stay, blood loss, and complication rates are comparable to a conventional laparoscopic hysterectomy.

Optimal Cable Tension Distribution of the High-Speed Redundant Driven Camera Robots Considering Cable Sag and Inertia Effects

Camera robots are high-speed redundantly cable-driven parallel manipulators that realize the aerial panoramic photographing. When long-span cables and high maneuverability are involved, the effects of cable sags and inertias on the dynamics must be carefully dealt with. This paper is devoted to the optimal cable tension distribution (OCTD for short) of the camera robots. Firstly, each fast varying-length cable is discretized into some nodes for computing the cable inertias. Secondly, the dynamic equation integrated with the cable inertias is set up regarding the large-span cables as catenaries. Thirdly, an iterative optimization algorithm is introduced for the cable tension distribution by using the dynamic equation and sag-to-span ratios as constraint conditions. Finally, numerical examples are presented to demonstrate the effects of cable sags and inertias on determining tensions. The results justify the convergence and effectiveness of the algorithm. In addition, the results show that it is necessary to take the cable sags and inertias into consideration for the large-span manipulators.

3P2-L06 RGB-Dカメラを用いた円柱状物体の検出及び把持手法(作業をするロボット(2))

This paper presents a method for grasping a cylindrical object by estimating its position and diameter using an RGB-D camera. The method takes into account both estimation errors and computational cost. Based on the results of a preliminary study, we demonstrate a grasping strategy using our prototypical manipulator.

Mathematical model of the aerial robotic camera base on its geometric relationship

This paper introduces a novel mathematical model of the Aerial Robotic Camera (Cable-suspended Parallel Robot-CPR system). The novelty of the CPR model is the geometric relationship between the camera motion in the Cartesian coordinates (external coordinates) and motors angular positions in the joint coordinates (internal coordinates). This relationship is defined by the Jacobian matrix which is used for solving the kinematics and dynamics of the Aerial Robotic Camera. For the computation of the dynamic model, the Lagrange principle of virtual work is used. The Jacobian matrix used in the formulation of the Lagrange principle of virtual work has been adopted according to the construction of the mechanism of the Aerial Robotic Camera. The specific structures of the Aerial Robotic Camera play an important role in defining the kinematic and dynamic models of the CPR system. The general form of the CPR mathematical model is defined and presented. Several numerical examples are used for the CPR model validation.

On the Minimum Cable Tensions for the Cable-Based Parallel Robots

This paper investigates the minimum cable tension distributions in the workspace for cable-based parallel robots to find out more information on the stability. First, the kinematic model of a cable-based parallel robot is derived based on the wrench matrix. Then, a noniterative polynomial-based optimization algorithm with the proper optimal objective function is presented based on the convex optimization theory, in which the minimum cable tension at any pose is determined. Additionally, three performance indices are proposed to show the distributions of the minimum cable tensions in a specified region of the workspace. An important thing is that the three performance indices can be used to evaluate the stability of the cable-based parallel robots. Furthermore, a new workspace, the Specified Minimum Cable Tension Workspace (SMCTW), is introduced, within which all the minimum tensions exceed a specified value, therefore meeting the specified stability requirement. Finally, a camera robot parallel driven by four cables for aerial panoramic photographing is selected to illustrate the distributions of the minimum cable tensions in the workspace and the relationship between the three performance indices and the stability.

Oriented Planetary Exploration Robotic Vision Binocular Camera Calibration

One of the goals of planetary exploration is to cache rock samples for subsequent return to Earth in the future Mars Sample Return (MSR) mission. This paper presents a method of binocular camera calibration. Robotic vision has become a very popular field in recent years due to the numerous promising applications it may enhance. However, errors within the cameras and in their perception of their environment can cause applications in robotics to fail. To help correct these intrinsic and extrinsic imperfections, stereo camera calibrations are performed. There are currently many accurate methods of camera calibration available; however, most or all of them are barely theoretical and difficultly practical. Because the authors' robot need to accurately approach and placement scientific objects, in this paper the authors present an image rectification method that is an extension to two-step method. There is an additional step for correcting the distorted image coordinates. The image rectification is performed and accurately compensates for radial and tangential distortions. Finally, through Matlab tool developed the results of experiment are accurate and available.

Task Shedding and Control Performance as a Function of Perceived Automation Reliability and Time Pressure

Research has demonstrated that workload and past machine performance influences operator allocation of task responsibilities to machines. We extended past investigations by offering task operators the opportunity to relinquish task control to a robotic entity. Forty-three participants navigated a remotely controlled vehicle around a prescribed course under conditions of low or high time pressure. While navigating, they could allocate camera monitoring to a low-or high-reliability automated agent. Results showed most participants retained control of the camera; others relinquished control immediately. Time pressure and reliability did not interact to influence task performance. Course navigation time was faster under high time pressure but errors were unaffected. Bivariate correlations revealed a positive relation between self-ratings of robotic expertise and pressure to perform, and between pressure to perform and errors committed during navigation. These results demonstrate low levels of trust in the robotic camera and comparative sensitivity of navigation time to time pressure.

Very-high-resolution time-lapse photography for plant and ecosystems research.

• Premise of the study: Traditional photography is a compromise between image detail and area covered. We report a new method for creating time-lapse sequences of very-high-resolution photographs to produce zoomable images that facilitate observation across a range of spatial and temporal scales.• Methods and Results: A robotic camera mount and software were used to capture images of the growth and movement in Brassica rapa every 15 s in the laboratory. The resultant time-lapse sequence (http://timemachine.gigapan.org/wiki/Plant_Growth) captures growth detail such as circumnutation. A modified, solar-powered system was deployed at a remote field site in southern Arizona. Images were collected every 2 h over a 3-mo period to capture the response of vegetation to monsoon season rainfall (http://timemachine.gigapan.org/wiki/Arizona_Grasslands).• Conclusions: A technique for observing time sequences of both individual plant and ecosystem response at a range of spatial scales is available for use in the laboratory and in the field.

Calibrating distance mapping of non‐SVP catadioptric camera of the soccer robot

PurposeThe purpose of the present paper is to propose a full model‐based method for distance‐mapping calibration for the non‐SVP (non‐single viewpoint) catadioptric camera of the soccer robot. The method should be easy to operate, efficient, accurate, and scalable to fit larger field sizes.Design/methodology/approachThe distance‐mapping model was first constructed based on the imaging principle. The authors then calibrated the internal parameters using the mirror boundary and used the mirror center to choose the correct pose from two possible solutions. The authors then proposed a three‐point method based on a unique solution case of the non‐SVP P3P (perspective‐three‐point) problem to solve the external parameters. Lastly, they built the distance mapping by back‐projection.FindingsThe simulation experimental results have shown that the authors' method is very accurate even when there is severe misalignment between the mirror and the camera and that all calibration operations, except the calibration of a standard camera, can be completed in 1 min. The result of the comparison with the traditional calibration method shows that the authors' method is superior to the traditional method in terms of accuracy and efficiency.Originality/valueThe proposed calibration method is scalable to larger fields because it only uses the boundary of the mirror and three feature points on the field, and does not need additional calibration objects. Additionally, an automatic calibration method that can be used during the game can be easily developed based on this method. Moreover, the proposed mirror‐pose‐selection method and a unique solution to the non‐SVP P3P problem are especially useful for a non‐SVP catadioptric camera.

Design and Implementation of the Control Device for a Cable-Driven Camera Robot

Due to the advantages of strong bearing capacity, compact structure and large working space, cable-driven camera robots are being used increasingly. This paper focuses on the design and implementation method of the control device for a four-cable-driven camera robot. Firstly, a kinematics model is built and the variations of the cable lengths are analyzed along with the camera robot motion; then, the console and actuator of the camera robot are designed; again, PC software transferring command codes from the camera robot console to the actuator and displaying camera robot motion state is developed; finally, the stability and rationality of the control device are verified by the experiment.

Feasibility of an endoscopic approach to the axillary nerve and the nerve to the long head of the triceps brachii with the help of the Da Vinci Robot

Surgery to transfer the axillary nerve and the nerve of the long head of the triceps presents two obstacles: 1) the access portals are not standardized and 2) the nerves are for their larger part approached through large incisions. The goal of this study was to explore the feasibility of an endoscopic microsurgical approach. The posterior aspect of a cadaver shoulder was approached through three communicating mini-incisions. The Da Vinci robot camera was installed on a central trocart, and the instrument arms on the adjacent trocarts. A gas insufflation distended the soft tissues up to the lateral axillary space. The branches of the axillary nerve and the nerve to the long head of the triceps brachii muscle were identified. The dissection of the axillary nerve trunk and its branches was easy. The posterior humeral circumflex veins and artery were dissected as well without any difficulty. Finding the nerve to the long head of the triceps brachii was found to be more challenging because of its deeper location. Robots properties allow performing conventional microsurgery: elimination of the physiologic tremor and multiplication of the movements. They also facilitate the endoscopic approach of the peripheral nerves, as seen in our results on the terminal branches of the axillary nerve and the nerve to the long head of the triceps brachii.

Robotic Partial Nephrectomy with Excision of Renal Vein Thrombus

Introduction: Renal cell carcinoma (RCC) has a predilection for extension into veins that can prevent the ability to perform partial nephrectomy when otherwise possible. Objective: Partial nephrectomy with venous tumor thrombus is feasible when venous drainage of the remaining kidney is preserved and negative surgical margins can be achieved, but this has not previously been described with robotic surgery. We present two representative cases from our experience of robotic partial nephrectomy (RPN) in the setting of venous extension of RCC. Techniques are illustrated for addressing tumor extension into a vein branch within the parenchyma in one patient and a second case with a tumor extending into the main renal vein. Methods: Standard four-port RPN was performed in both cases using two 8-mm ports for robotic instruments and two 12-mm ports, one each for the robotic camera and assistant. One 4.3-cm tumor was found to have an extension into a branch of the renal vein on intraoperative ultrasound. The laparoscopic ultrasound is a powerful tool that allows identification of the thrombus extent readily in experienced hands. The mass was excised with the vein and tumor thrombus as one unit, following the vein into the parenchyma until the end of the tumor thrombus was found. Another patient undergoing heminephrectomy for a 9.3-cm RCC had an extension from one of the two major vein branches into the main renal vein. Laparoscopic ultrasound was again used to determine the length of the thrombus and where the lumen of the vein became patent for placement of a bulldog clamp. The tumor and associated branch were excised as one unit, extracting the tumor thrombus from the main renal vein by making a linear incision along its length until reaching the end of the thrombus, allowing a traumatic removal. Suture reconstruction for drainage of the remaining half of the kidney was then performed, flushing the lumen of the renal vein with saline before unclamping. The remainder of the renorrhaphy can be performed in a standard fashion in these cases. Results: Both were performed with less than 30 minutes of warm ischemia time. Both patients were discharged on the first postoperative day without complications and with negative surgical margins. Conclusion: RPN can be successfully performed for selected tumors with venous tumor thrombus. Whereas local control can likely be achieved, systemic recurrence remains a high risk in such T3a patients. No competing financial interests exist. Runtime of video: 6 mins 50 secs

English

English

The head butler with design modification: a useful mountable shelf for the operating table in robotic surgery.

The 'head butler' or 'iron maiden', as it is sometimes referred to, is a mountable shelf that is attached to the main operating room table using stirrup clamps or brackets. It provides an elevated flat surface, which protects the patient's face and chest from inadvertent trauma of robotic arms, instruments and camera while also serving as a platform for placement of instrumentation that minimizes handoffs to improve workflow. Its canted design also prevents slippage of instruments off the operating field during steep Trendelenburg positioning. This operating room equipment was designed to improve efficiency in an operating room set-up while ensuring safety during patient positioning when performing robotic procedures. A modification of the original Head Butler(®) (Tri-Medical Corporation, Portland, OR, USA) is presented here which improves upon the design by avoiding brachial plexus nerve injury to the patient due to pressure from the uprights pressing on the patient's shoulders during use.