Remote Operation Research Articles

Quantifying marine growth on pipelines with archival remotely operated vehicle footage – a case study

Abstract. Marine growth can substantially affect the hydrodynamic properties of subsea infrastructure. The ability to accurately quantify this growth is essential for designing and maintaining subsea structures. Traditionally, quantifying marine growth has been done by applying simple ratio equations putting in relationship the known distance of an object and its length observed in the image to the length of an unknown object only observed in an image. This method is very sensitive if measurements are not taken in the same plane. Hence, this study aims to establish a method for creating a 3D model of a subsea pipeline segment based on Structure from Motion (SfM). The proof of concept is done using a case study. For this study, Remotely Operated Vehicle (ROV) footage from a 2019 inspection survey is utilised. The scenario in which the procedure is tested represents a difficult, but realistic test case. Despite the lack of metadata, video quality and obstructions (e.g. watermarks), meaningful results were produced. The SfM workflow included masking obstructions (e.g. watermarks), and an iterative camera alignment procedure to generate 3D models, enabling the measurement of marine growth using either the models or ortho images created. While the resulting models were sufficiently accurate to provide meaningful measurements for harder growth types, the method faced challenges in accurately modelling softer growth types, suggesting the need for further refinement. Nevertheless, SfM was proven to be a more reliable method of measuring biofouling than the ratio approach that been traditionally used.

Improving Underwater Photogrammetric 3D Reconstruction Processing of Shipwreck Sites

Abstract. Applying image processing algorithms to enhance the clarity of underwater images can significantly assist in the visual interpretation of subsea environments. Taking advantage of techniques to maximise the information extracted from the captured input data to produce the highest quality output ensures a higher utilisation of the valuable recorded content which may have been collected at the high cost of a laborious diver survey or an expensive ROV (Remotely Operated Vehicle) expedition. Adopting photogrammetric 3D reconstruction processing in underwater imaging has enabled the creation of high-fidelity digital 3D models which have been used to map subsea infrastructure for maintenance and repair, coral reefs for ecological monitoring, and in the primary context of this article, shipwrecks sites for maritime archaeology and accident investigation. Alongside exploring algorithms to reduce the characteristic haze of underwater photography, this article illustrates the advantages of optimising the processes utilised at distinct stages of the photogrammetric 3D reconstruction workflow to improve the photorealism of the digital 3D models.

In situ monitoring of defects in steel structures using a robot‐assisted ultrasonic inspection technique

AbstractIncidents in extreme environments can bring unprecedented consequences that would endanger the lives of many humans. Real‐time and early detection of cracks and defects in the steel used for the construction of industrial sites can bring many benefits to remote operators and site managers. Although different methods and systems have been proposed in the past, they are not suitable for real‐time fault detection or long‐term operation in harsh environments. Therefore, this paper proposes a low‐power robotic platform with a magnetic actuator mechanism which is capable of wireless inspection of steel structures using the ultrasonic method. The proposed magnetic motion uses the inertia force generated by the piston vibration and the difference in frictional force during movement. The paper evaluates the performance of the steel structure defect detection system from four different perspectives: sensing principle, wireless communication network, driving part, and power supply.

Combined Method Comprising Low Burden Physiological Measurements with Dry Electrodes and Machine Learning for Classification of Visually Induced Motion Sickness in Remote-Controlled Excavator

The construction industry is actively developing remote-controlled excavators to address labor shortages and improve work safety. However, visually induced motion sickness (VIMS) remains a concern in the remote operation of construction machinery. To predict the occurrence and severity of VIMS, we developed a prototype system that acquires multiple physiological signals with different mechanisms under a low burden and detects VIMS from the collected data. Signals during VIMS were recorded from nine healthy adult males operating excavator simulators equipped with multiple displays and a head-mounted display. Light gradient-boosting machine-based VIMS detection binary classification models were constructed using approximately 30,000 s of time-series data, comprising 23 features derived from the physiological signals. These models were validated using leave-one-out cross-validation on seven participants who experienced severe VIMS and evaluated through area under the curve (AUC) scores. The mean receiver operating characteristic curve AUC score was 0.84, and the mean precision–recall curve AUC score was 0.71. All features were incorporated into the models, with saccade frequency and skin conductance response identified as particularly important. These trends aligned with subjective assessments of VIMS severity. This study contributes to advancing the use of remote-controlled machinery by addressing a critical challenge to operator performance and safety.

Synthesis-in-place of V2O5 nanobelts for wide range humidity detection

Resistive type humidity sensors offer advantages in simplicity, low cost, compact size, and remote operation. Due to the benefits, they have great potential for use in a variety of environments and future applications, including the Internet of Everything (IoE). However, resistive humidity sensors have limitations in detecting extreme humidity levels below 10 % or above 90 % relative humidity (RH) and are also vulnerable to high temperatures, which hinders their use in harsh environments applications. For broader applications, humidity sensors with reliable performance across the full range of humidity levels and high stability at elevated temperatures need to be developed. Here, we report high performance resistive humidity sensors based on V2O5 nanobelts operating at high temperatures. The V2O5 nanobelts are directly deposited between Pt electrodes by O2-assisted physical vapor transport (PVT) method. The V2O5 nanobelt humidity sensor exhibits reliable humidity sensing properties over a wide range from 1 % to 100 % RH. Furthermore, long-term stability and reproducibility of V2O5 nanobelts are demonstrated through 25 consecutive humid air pulses and measurements obtained after 4 months of storage. The high performance of the V2O5 nanobelt humidity sensor is advantageous for practical use in applications ranging from IoE environments to harsh industrial conditions.

Boredom, vigilance, and repetition during remote control room work for autonomous ships

Abstract Introduction: Remote Operating Centers (ROCs) enable human oversight of remotely operated and autonomous ships. However, the “human-in-the-loop” model of automated control oversight implies prolonged periods of inactivity punctuated by safety-critical action. This paper investigates the extent to which this role is aligned with humans’ innate attentional resources in terms of boredom, vigilance, and repetition. Method: A simulator study was completed with ROC operators (volunteer participants with relevant expertise). One group completed a short scenario (5 minutes; N = 16); the other completed a long scenario (30 minutes; N = 16). The duration marked the onset of a stimulus (an alarm), prompting manual takeover. In a mixed method approach, four sources of data were collected, consisting of pupillometry, questionnaire, interview, and video data. Results: Pupillometry data showed larger pupillary constrictions in the 30-minute group, indicative of decreased cognitive activity (effect size 0.25 mm). The questionnaire showed that operators in the 30-minute scenario were more likely to experience their tasks as unchallenging, understimulating, and lacking in meaning. Finally, a relationship was found between repetitive content and varying perceptions of time duration. Moreover, operators perceived scenarios as longer when they detected an “oddball” in the repetitive series (a gorilla passenger). Conclusion: ROC work as currently imagined is prone to boredom, vigilance decrement, and suppression of repetition. The study adds to the empirical research about boredom and its associated cognitive states in control room work. The study also contributes to design considerations impacting safety and organizational aspects like operator training and retention.

Wavelet-based spatiotemporal sparse quaternion dictionary learning for reconstruction of multi-channel vibration data

Accurate reconstruction of multi-source data information plays an essential role in remote intelligent operation, and prognostic and health maintenance (RIO-PHM) of industrial systems. However, traditional signal reconstruction methods such as compressed sensing fail to capture the spatio-temporal characteristics and coupling nature of the multi-source or multi-channel data. To alleviate this bottleneck, in this paper, a new wavelet-based spatiotemporal sparse quaternion dictionary learning (WSTS-QDL) algorithm is formulated for the reconstruction of multi-channel vibration data for the first time. Specifically, the wavelet-based spatiotemporal sparse low-rank matrix (SLRM) algorithm is elaborated and the sparse coefficient matrix of the multi-channel signals can be estimated using the split Bregman iteration (SBI) technique. Then, quaternion-based dictionary learning is introduced for multi-channel data reconstruction according to the updated sparse coefficient matrix during quaternion dictionary learning. Eventually, two equipmental scenarios including run-to-failure data of rolling bearing in bearing test bench and vibration signal of the failured bearing in corn thresher, are utilized for verification. The experimental results demonstrate that the highest recovery accuracy performance with scoring function and the lowest errors are achieved by the proposed method in contrast with the state-of-the-art benchmarks such as orthogonal matching pursuit (OMP) method and spatiotemporal sparse Bayesian learning (SSBL), and the waveform of phase space trajectory and points cloud distribution of the Poincare section in the original signal are similar/same to that obtained by the proposed method.

Enhancing Situational Awareness for Remote Control of Ship Cranes

Abstract Cargo handling in ship operations poses risks due to heavy loads, necessitating innovation to mitigate accidents. Automation and remote control offers safer and more efficient operations by moving people from hazardous situations. Smaller and remote ports often rely on geared ships, which are typically manually operated. A first step towards automation is to remotely control these cranes. In remote control, the operator needs to gain situational awareness through sensor inputs. Previous research has shown that there is a need to investigate the sensor configuration and placement to provide situational awareness, particularly considering the placement of cargo onboard. This study seeks to identify needed sensor configurations for situational awareness in remote control of ship cranes using a simulator framework. A modular crane simulator framework has been developed to evaluate situational awareness for different sensor configurations. The sensor configuration has been evaluated with feedback from a crane expert. This research has identified potential sensor configurations for enhancing situational awareness in remote crane operations. The developed simulator framework has demonstrated its flexibility and efficiency as a platform for testing and optimizing cargo handling operations.

Quantum yield and thermal sensitivity of SrGd2O4:Yb, Tm up-conversion nanoparticles

Luminescence thermometry is an effective method for temperature measurements using the remote operating mode. In this work, a series of Tm3+ (1 at%) and Yb3+ (2, 4, and 6 at%) ions doped SrGd2O4 up-conversion (UC) nanoparticles were synthesized by the modified sol-gel method, and fully characterized toward optimization of their composition and optical characteristic. Crystallization of single-phase nanoparticles with homogeneous doping was confirmed by XRPD, TEM/HRTEM, and STEM/EDS analysis in all samples, independently of their stoichiometry. The UC emissions recorded under 976 nm excitation revealed typical Tm3+ transitions, showing that the optimal sensitizer doping concentration is 4 at%. For this sample, a lifetime of 342 μs, and a total quantum yield of 1.12 % were determined at room temperature for the emission observed in the Vis part of up-conversion spectra. The temperature-dependent emission spectra, measured from 263 to 363 K, implied its excellent temperature-sensing capabilities. The luminescent intensity ratio (LIR) was employed for blue I479/485 and IR I795/807 Tm3+ emissions. The absolute and relative temperature sensitivity continuously decreases with the increase of the temperature, showing the values of 4.05x10−3 K-1 and 0.41 % K−1 (for blue) and 3.8x10−3 K-1 and 0.31 % K−1 (for IR) at the temperature of 300 K. The results indicate that SrGd2O4 doped with 4 % Yb3+ and 1 % Tm3+ may be employed for reliable temperature measurement over an investigated temperature range.

Convolutional Long Short-Term Memory Predictor for Collaborative Remotely Operated Vehicle Trajectory Tracking in a Leader–Follower Formation Subject to Communication and Sensor Latency in the Presence of External Disturbances

Nowadays, collaborative operations between Remotely Operated Vehicles (ROVs) face considerable challenges, particularly in leader–follower schemes. The underwater environment imposes limitations on acoustic modems, leading to reduced transmission speeds and increased latency in ROV position and speed transmission. This complicates effective communication between the ROVs. Traditional methods, such as Recursive Least Squares (RLS) predictors and the Kalman filter, have been employed to address these issues. However, these approaches have limitations in handling non-linear patterns and disturbances in underwater environments. This paper introduces a Convolutional Long Short-Term Memory (ConvLSTM) predictor designed to enhance communication and trajectory tracking between ROVs in a leader–follower scheme. The proposed ConvLSTM aims to address the shortcomings of previous methods by adapting effectively to varying conditions, including ocean currents, communication delays, and signal interruptions. Simulations were conducted to evaluate ConvLSTM’s performance and compare it with other advanced predictors, such as Multilayer Perceptron (MLP) and Long Short-Term Memory (LSTM), under different conditions. The results demonstrated that ConvLSTM achieved a 13.9% improvement in trajectory tracking, surpassing other predictors in scenarios that replicate real underwater conditions and multi-vehicle communication. These results highlight ConvLSTM’s potential to significantly enhance the performance and stability of collaborative ROV operations in dynamic underwater environments.

Touchpoints and systems for unmanned autonomous urban ferry operation

Abstract Currently, both experimental and commercial autonomous ferry services have a safety operator onboard who monitors the ferry systems, takes care of the passengers’ safety, and gives them the necessary information. Service touchpoints are all the interaction points between the autonomous ferry and the passengers throughout the passenger journey, from planning the trip to reaching their destination. To reach a state where the ferry service can be unmanned, by moving the safety operator into a remote operation centre, several additional systems and service touchpoints are needed to support the passengers’ safety and provide them with necessary information. This article discusses (1) the service touchpoints and systems needed for an autonomous urban ferry service, (2) how these touchpoints provide the passengers with the necessary information to foster trust and perceived safety, and (3) shows some examples and design suggestions for the experimental autonomous urban ferry milliAmpere2. The service touchpoints presented are (1) a ticket system, (2) an automatic ferry ordering system for on-demand operation, (3) an automatic passenger handling system for safe boarding of the ferry, (4) a passenger information system to provide visual and audio feedback about the ferry service, and (5) a passenger-operator communication system in case contact between the ferry and the remote operation centre need to be established.

Design, Fabrication and Aerodynamic Analysis of a RC Airplane

Abstract An internal receiver and a hand-held transmitter enable the remote operation of a (model) airplane, often known as an RC plane or airplane, to be operated remotely. The objective is to create a high-performance, stable, and maneuverable RC airplane maintaining an optimum AOA suitable for recreational and hobbyist use. Since the used electronic components are freely available in markets and do not require user programming, this paper did not focus on them. In this study, computational fluid dynamic (CFD) analysis was used to discuss wing shape, and control surface configuration to optimize lift, drag, and stability also calculated the optimum angle of attack which was found to be below 15 degrees. The N10 airfoil was taken into consideration for the wing design based on its asymmetrical shape for better lift and aerodynamic performance. Overall, this article provides valuable insights and guidelines for the design and development of RC airplanes, fostering advancements in this exciting field of aviation.

Grasping Control for Kinematically Redundant Parallel Robots With a Remotely Operated Gripper

Grasping Control for Kinematically Redundant Parallel Robots With a Remotely Operated Gripper

The Development of Remotely Operated Underwater Vehicle for Plastic Waste Detection with Raspberry Pi

Abstract Underwater Plastic waste has had a serious impact on the environment. Unfortunately, Detecting and collecting underwater plastic waste is still a challenge, especially in Indonesia because of its position in the water, and the water turbidity. In previous research, a computer algorithm for detecting underwater plastic waste has been developed with Yolov3. To improve the research, we developed a Remotely Operated Underwater Vehicle (ROUV) with Raspberry Pi through the Research and Development (R&D) method. This ROUV is like a submarine that operates in the water and is integrated with a camera and computer algorithm to detect plastic waste. To get close to the real condition, we created an artificial water environment in a pond with various turbidity (from 20 to 200 Nephelometric Turbidity Unit). We use 30 pieces of plastic waste in several colors: white, red, black, and transparent. The plastic waste dipped in the water and was then detected by the ROUV camera. The result shows that the effective threshold for object detection is around 100 Nephelometric Turbidity Unit (NTU). Below 100 NTU, although the number of detected plastic images decreases as the turbidity increases, the visibility and contrast still allow for reasonably good object detection. The average confidence value for detection at turbidity levels below 100 NTU is 77%. At high turbidity conditions, the detection model may require adjustment or retraining with data that reflects those turbidity conditions.

Human-robot collaborative decision method of hexapod robot based on prior knowledge and negotiation strategy

Against the unstructured terrain environment, human drivers often rely on remote operating patterns to accomplish tasks with robots. With the gradual improvement in robot intelligence, a key problem that urgently requires attention is how to bridge cognitive and perceptual differences between human and robot in collaborative decision-making processes. Particularly effective resolution of differences in decision outcomes is crucial. Therefore, this paper utilizes the decision logic process and prior knowledge model of human drivers to form a robot's decision framework, narrowing the cognitive differences between human and robot. Besides, embedding the human-robot-environment features that affect the decision-making process as prior knowledge into the robot's decision-making system can effectively reduce perceptual gaps. Under the same decision framework, a negotiation strategy is proposed for forming a unified decision outcome between human and robot by finding an optimal concession rate. This study utilizes a hexapod robot simulated driving platform to gather experimental data and develop prior knowledge for robots. It also leverages virtual reality equipment and visual augmentation technologies to establish a remote human-robot collaborative decision-making experimental system. The experimental results conducted in complex obstacle terrain demonstrate the effectiveness of adopting the negotiation strategy for human-robot collaborative decision-making. Compared with the individual human or robot decision, the human-robot collaborative decision method significantly enhances robot stability, reduces energy consumption, and minimizes collisions.

Full-Body Pose Estimation of Humanoid Robots Using Head-Worn Cameras for Digital Human-Augmented Robotic Telepresence

We envision a telepresence system that enhances remote work by facilitating both physical and immersive visual interactions between individuals. However, during robot teleoperation, communication often lacks realism, as users see the robot’s body rather than the remote individual. To address this, we propose a method for overlaying a digital human model onto a humanoid robot using XR visualization, enabling an immersive 3D telepresence experience. Our approach employs a learning-based method to estimate the 2D poses of the humanoid robot from head-worn stereo views, leveraging a newly collected dataset of full-body poses for humanoid robots. The stereo 2D poses and sparse inertial measurements from the remote operator are optimized to compute 3D poses over time. The digital human is localized from the perspective of a continuously moving observer, utilizing the estimated 3D pose of the humanoid robot. Our moving camera-based pose estimation method does not rely on any markers or external knowledge of the robot’s status, effectively overcoming challenges such as marker occlusion, calibration issues, and dependencies on headset tracking errors. We demonstrate the system in a remote physical training scenario, achieving real-time performance at 40 fps, which enables simultaneous immersive and physical interactions. Experimental results show that our learning-based 3D pose estimation method, which operates without prior knowledge of the robot, significantly outperforms alternative approaches requiring the robot’s global pose, particularly during rapid headset movements, achieving markerless digital human augmentation from head-worn views.

New carnivorous sponges (Porifera: Cladorhizidae) from Western Australia, collected by a Remotely Operated Vehicle (ROV).

The last two decades have reinvigorated systematic research on predatory sponges, mainly fuelled by advances in technology that have facilitated collection in deep-water habitats. This research presents six new species of carnivorous sponges from the family Cladorhizidae Dendy, 1922 from the western continental margin of Australia. The new species are Abyssocladia johnhooperi nov. sp., Abyssocladia aurora nov. sp., Abyssocladia janusi nov. sp., Axoniderma challengeri nov. sp., Cladorhiza vanessaekins nov. sp. and Nullarbora ningalooa nov. sp.. This material was collected by ROV during expeditions FK200308 to the Ningaloo Canyons expedition off the mid-west coast near Ningaloo, and FK200126 to the Southwest Australian canyons expedition, in Western Australia. These and other expeditions by the Schmidt Ocean Institute in 2020-21 formed a campaign around Australia's deep sea and mesophotic environments, which has vastly increased our understanding of biodiversity in these habitats.

Biomimetic Transparent Slippery Surface for the Locomotion of Photocontrol Droplets and Bubbles.

Directed transportation and collection of liquids and bubbles play a vital role in the survival of ecosystems. Among them, the optical response control is widely used in the fields of microfluidic chips and chemical synthesis because of its high remote operation and fast response speed. However, due to poor light transmission, the development direction of traditional near-infrared (NIR) absorbing materials in the field of visualization is limited, and there are few reports of manufacturing an operating platform that can realize the directional movement of droplets/bubbles on a single platform. Here, a transparent photo-responsive PBFS platform is prepared for droplet and bubble manipulation by coating the etched glass substrate with Prussian blue (PB) nanocubes. When near-infrared (NIR) irradiation on the PBFS platform, PB nanocubes trigger heat production by photothermal means, due to the action of Marangoni force, the surface tension on the left and right sides of the droplets and bubbles is not uniform, forming a surface tension gradient, thereby driving the movement of the droplets and bubbles. The control platform has good application potential in the field of microchemical reaction and biomedical engineering and brings new solutions to the field of transparent photothermal materials.

Investigation of human interaction with an augmented remote operating system for scaling in mining applications

Thanks to the advent of telepresence applications, we can remotely take control and operate industrial machinery. Teleoperation removes operators from hazardous workplaces such as mining and plays an essential role in the safety of workers. In addition, augmented telepresence can introduce information that helps the user understand the remote scene. However, remote operation presents challenges since the information received is more limited than what could be perceived by being physically present, such as accurately judging depth. This study investigates how well operators interact with an augmented remote operation scaling system (AROSS) in a mining context when different computer-generated visual interfaces are provided. The system can achieve five visual interfaces: Disocclusion Augmentation view using selective content removal, Novel Perspective view generation, Lidar view, Right (Original) view, and Left (Original) view. We performed two experiments in a mine-like laboratory to analyze human interaction with the designed prototype by applying a mixed research methodology that used questionnaires, interviews, and observations. This mixed methodology consisted of quality of experience methods to discover the users’ requirements from a technological standpoint and user experience methods (i.e., user-centric approaches). We investigated 10 and 11 users’ interactions in two experimental studies. The first experiment focused on identifying small patterns (e.g., cracks in the mine wall), and the second focused on depth and three-dimensional understanding. We considered the first experiment a feasibility test to understand how to conduct the second experiment. Therefore, we designed the second test to assess the technical readiness of AROSS from the users’ perspective. The overall conclusion yields a comprehensive understanding of users’ perceptions and experiences. The quality of experience results favored Left and Right (Original) views for remote control, indicating a preference among remote operators using natural (Original) views due to their facilitation of environmental comprehension. User experience analysis revealed the reason why other views were less favored and what their potential benefits are. Specifically, Novel Perspective and Lidar views were found helpful for depth perception, and Disocclusion Augmentation view functionality could be enhanced if robot arm position tracking is enabled. These insights inform design recommendations, emphasizing the value of incorporating Disocclusion Augmentation and Novel Perspective views and suggesting improvements to enhance system usability.

Optogenetic Tools for Regulating RNA Metabolism and Functions.

RNA molecules play a vital role in linking genetic information with various cellular processes. In recent years, a variety of optogenetic tools have been engineered for regulating cellular RNA metabolism and functions. These highly desirable tools can offer non-intrusive control with spatial precision, remote operation, and biocompatibility. Here, we would like to review these currently available approaches that can regulate RNAs with light: from non-genetically encodable chemically modified oligonucleotides to genetically encoded RNA aptamers that recognize photosensitive small-molecule or protein ligands. Some key applications of these optogenetic tools will also be highlighted to illustrate how they have been used for regulating all aspects of the RNA life cycle: from RNA synthesis, maturation, modification, and translation to their degradation, localization, and phase separation control. Some current challenges and potential practical utilizations of these RNA optogenetic tools will also be discussed.