Semi-autonomous Control Research Articles

Activities of Daily Living Object Dataset: Advancing Assistive Robotic Manipulation with a Tailored Dataset

The increasing number of individuals with disabilities—over 61 million adults in the United States alone—underscores the urgent need for technologies that enhance autonomy and independence. Among these individuals, millions rely on wheelchairs and often require assistance from another person with activities of daily living (ADLs), such as eating, grooming, and dressing. Wheelchair-mounted assistive robotic arms offer a promising solution to enhance independence, but their complex control interfaces can be challenging for users. Automating control through deep learning-based object detection models presents a viable pathway to simplify operation, yet progress is impeded by the absence of specialized datasets tailored for ADL objects suitable for robotic manipulation in home environments. To bridge this gap, we present a novel ADL object dataset explicitly designed for training deep learning models in assistive robotic applications. We curated over 112,000 high-quality images from four major open-source datasets—COCO, Open Images, LVIS, and Roboflow Universe—focusing on objects pertinent to daily living tasks. Annotations were standardized to the YOLO Darknet format, and data quality was enhanced through a rigorous filtering process involving a pre-trained YOLOv5x model and manual validation. Our dataset provides a valuable resource that facilitates the development of more effective and user-friendly semi-autonomous control systems for assistive robots. By offering a focused collection of ADL-related objects, we aim to advance assistive technologies that empower individuals with mobility impairments, addressing a pressing societal need and laying the foundation for future innovations in human–robot interaction within home settings.

A Semi-Autonomous Control Mode for Flexible Steerable Intraluminal Platforms

A Semi-Autonomous Control Mode for Flexible Steerable Intraluminal Platforms

Ground Adaptability of Crawler Mobile Robots with Sub-Crawler Rotary Joint Compliance

Remote-controlled mobile robots are expected to be used in difficult- or impossible-to-access environments for inspection workers and responders, such as in investigations and search activities at accident/disaster sites and inspection/investigation work at plants/infrastructure. Among ground mobile robots, crawler mobile robots with sub-crawlers (also known as ground-adaptive crawler robots) excel at in-ground adaptability and stack escape; however, their operators require advanced remote-control technology and experience. Therefore, the introduction of semi-autonomous control to assist the operator is required. In this study, the principle of the pushing-up sequence and the possibility of mobiligence emerging from interaction with obstacles caused by the robot movement were described. In addition, the sub-crawler rotary joint’s compliance, which significantly contributes to ground adaptability, was hypothesized, and a compliance control system design method that uses the sub-crawler constraint angle as a design condition was proposed. It was confirmed that the model robot for the evaluation, which used the proposed method, could adapt to unknown obstacles without measuring their height and shape and traverse them based on experimental results. In addition, based on the numerical calculation results, it was determined that the optimum solution for the restriction angle of the sub-crawler was approximately 35°–50° from the perspective of propulsive force and tumble stability.

Designing an Uncrewed Aircraft Systems Control Model for an Air-to-Ground Collaborative System

<div>In autonomous technology, uncrewed aircraft systems have already become the preferred platform for the research and development of flight control systems. Although they are subjected to following and satisfying complicated scenarios of control stations, this high dependency on a specific control framework limits them in their application process and reduces the flight self-organizing network. In this article, we present a developed multilayer control system protocol with the additional supportive manned aircraft layer (<i>Tender</i>). The novelty of the introduced model is that uncrewed aircraft systems are monitored and navigated by the tender, and then based on the suggested scheme, data flows are controlled and transferred across the network by the developed cloud–robotics approach in the ground station layer. Therefore, it has been tried to design a semi-autonomous control network to gather data that combines human observation and the automotive nature of uncrewed aircraft systems. To ensure the accuracy and correctness of the model, we simulate our approach in the software-in-the-loop using its web-based interface with new configurations in the hardware and software architecture of the network. Results will be examined by the in-order per message delay, which has recorded a considerably low latency in both the uplink and downlink data transmission processes. This optimization is achieved along with maintaining the quality of data.</div>

Assessing wind impact on semi-autonomous drone landings for in-contact power-line inspection

In recent years, the use of inspection drones has become increasingly popular for high-voltage electric cable inspections due to their efficiency, cost-effectiveness, and ability to access hard-to-reach areas. However, safely landing drones on power lines, especially under windy conditions, remains a significant challenge. This study introduces a semi-autonomous control scheme for landing on an electrical line with the NADILE drone (an experimental drone based on original LineDrone key features for inspection of power lines) and assesses the operating envelope under various wind conditions. A Monte Carlo method is employed to analyze the success probability of landing given initial drone states. The system’s performance is assessed by testing two landing strategies, adjusting controller parameters, and considering four different levels of wind intensity. The results show that a two-stage landing strategy offers higher probabilities of landing success and gives insight regarding the best controller parameters and the maximum wind level for which the system is robust. Finally, an experimental demonstration of the system landing autonomously on a power line is presented.

A Semiautonomous Control Strategy Based on Computer Vision for a Hand–Wrist Prosthesis

Alleviating the burden on amputees in terms of high-level control of their prosthetic devices is an open research challenge. EMG-based intention detection presents some limitations due to movement artifacts, fatigue, and stability. The integration of exteroceptive sensing can provide a valuable solution to overcome such limitations. In this paper, a novel semiautonomous control system (SCS) for wrist–hand prostheses using a computer vision system (CVS) is proposed and validated. The SCS integrates object detection, grasp selection, and wrist orientation estimation algorithms. By combining CVS with a simulated EMG-based intention detection module, the SCS guarantees reliable prosthesis control. Results show high accuracy in grasping and object classification (≥97%) at a fast frame analysis frequency (2.07 FPS). The SCS achieves an average angular estimation error ≤18° and stability ≤0.8° for the proposed application. Operative tests demonstrate the capabilities of the proposed approach to handle complex real-world scenarios and pave the way for future implementation on a real prosthetic device.

The Classification and New Trends of Shared Control Strategies in Telerobotic Systems: A Survey.

Shared control, which permits a human operator and an autonomous controller to share the control of a telerobotic system, can reduce the operator's workload and/or improve performances during the execution of tasks. Due to the great benefits of combining the human intelligence with the higher power/precision abilities of robots, the shared control architecture occupies a wide spectrum among telerobotic systems. Although various shared control strategies have been proposed, a systematic overview to tease out the relation among different strategies is still absent. This survey, therefore, aims to provide a big picture for existing shared control strategies. To achieve this, we propose a categorization method and classify the shared control strategies into 3 categories: Semi-Autonomous control (SAC), State-Guidance Shared Control (SGSC), and State-Fusion Shared Control (SFSC), according to the different sharing ways between human operators and autonomous controllers. The typical scenarios in using each category are listed and the advantages/disadvantages and open issues of each category are discussed. Then, based on the overview of the existing strategies, new trends in shared control strategies, including the "autonomy from learning" and the "autonomy-levels adaptation," are summarized and discussed.

A semi-autonomous robot control based on bone layer transition detection for a safe pedicle tapping.

Automatic robotic platforms for robot-aided spinal surgery are mostly employed for drilling the pedicle screw path and do not adapt the tool rotational speed depending on the variation of the bone density. This feature is highly desirable in control strategies for robot-aided pedicle tapping, which may result in a poor quality thread if the surgical tool speed is not adequately tuned depending on the bone density to be threaded. Therefore, the objective of this paper is to propose a novel semi-autonomous control for robot-aided pedicle tapping that is able to (i) identify the bone layer transition, (ii) adapt the tool velocity depending on the detected bone layer density and (iii) stop the tool tip before propulsion of the bone boundaries. The proposed semi-autonomous control for pedicle tapping consists of: (i) a hybrid position/force control loop that allows the surgeon to move the surgical tool along a pre-planned axis and (ii) a velocity control loop that allows him/her to finely tune the tool rotational speed by modulating the tool-bone interaction force along the same axis. The velocity control loop integrates also a bone layer transition detection algorithm that dynamically limits the tool velocity depending on the bone layer density. The approach was tested on the Kuka LWR4+ provided with an actuated surgical tapper which was used to tap a wood specimen simulating the bone layer density characteristics and bovine bones. A normalized maximum time delay in the bone layer transition detection of 0.25 was achieved by the experiments. A success rate of [Formula: see text] was achieved for all the tested tool velocities. The proposed control achieved a maximum steady-state error of 0.4 rpm. The study demonstrated high capability of the proposed approach to i) promptly detect transition among the specimen layers and ii) adapt the tool velocities depending on the detected layers.

Electromagnetic Manipulation System for Semi-Autonomous Control of Small-Scale Magnetic Objects With Sequential Programming

In this study, an electromagnetic coil system was developed for semi-autonomous manipulation of microrobots in three-dimensional (3D) space with sequential programming. Cylindrical-shaped cores were used for the electromagnetic coils; the coils were arranged in a hemispherical configuration to allow open access to the workspace for imaging tools. The system consisted of four main parts: the electromagnetic coils, an electronic control unit, optical cameras, and a direct-manipulation graphical user interface (GUI). The system was capable of producing a magnetic field of up to 18 mT with a maximum current of 8 A current. Using this system, magnetic force and torque can be applied for semi-autonomous manipulation of magnetic objects, with different time intervals, by instantly inputting the required fields and gradients via the GUI. Sequential programming of the magnetic field allowed for better controllability and enhanced the repeatability of the system. The versatility of the system was demonstrated by moving a magnetic guidewire, micromagnet, and nanoparticles to their intended targets to assess the potential application of the system in biomedical fields.

Comparison of Semi-autonomous Mobile Robot Control Strategies in Presence of Large Delay Fluctuation

We propose semi-autonomous control strategies to assist in the teleoperation of mobile robots under unstable communication conditions. A short-term autonomous control system is the assistance in the semi-autonomous control strategies, when the teleoperation is compromised. The short-term autonomous control comprises of lateral and longitudinal functions. The lateral control is based on an artificial potential field method where obstacles are repulsive, and a route is attractive. LiDAR-based artificial potential field methods are well studied. We present a novel artificial potential field method based on color and depth images. Benefit of a camera system compared to a LiDAR is that a camera detects color, is cheaper, and does not have moving parts. Moreover, utilization of active sensors is not desired in the particle accelerator environment. A set of experiments with a robot prototype are carried out to validate this system. The experiments are carried out in an environment which mimics the accelerator tunnel environment. The difficulty of the teleoperation is altered with obstacles. Fully manual and autonomous control are compared with the proposed semi-autonomous control strategies. The results show that the teleoperation is improved with autonomous, delay-dependent, and control-dependent assist compared to the fully manual control. Based on the operation time, control-dependent assist performed the best, reducing the time by 12% on the tunnel section with most obstacles. The presented system can be easily applied to common industrial robots operating e.g. in warehouses or factories due to hardware simplicity and light computational demand.

Semi-autonomous control of prosthetic hands based on multimodal sensing, human grasp demonstration and user intention

Semi-autonomous control strategies for prosthetic hands provide a promising way to simplify and improve the grasping process for the user by adopting techniques usually applied in robotic grasping. Such strategies endow prosthetic hands with the ability to autonomously select and execute grasps while keeping the user in the loop to intervene at any time for triggering, accepting or rejecting decisions taken by the controller in an intuitive and easy way. In this paper, we present a semi-autonomous control strategy that allows the user to perform fluent grasping of everyday objects based on a single EMG channel and a multi-modal sensor system embedded in the hand for object perception and autonomous grasp execution. We conduct a user study with 20 subjects to assess the effectiveness and intuitiveness of our semi-autonomous control strategy and compare it to a conventional electromyography-based control strategy. The results show that the workload is reduced by 25.9% compared to conventional electromyographic control, the physical demand is reduced by 60% and the grasping process is accelerated by 19.4%.

Computer Vision-Based Adaptive Semi-Autonomous Control of an Upper Limb Exoskeleton for Individuals with Tetraplegia

We propose the use of computer vision for adaptive semi-autonomous control of an upper limb exoskeleton for assisting users with severe tetraplegia to increase independence and quality of life. A tongue-based interface was used together with the semi-autonomous control such that individuals with complete tetraplegia were able to use it despite being paralyzed from the neck down. The semi-autonomous control uses computer vision to detect nearby objects and estimate how to grasp them to assist the user in controlling the exoskeleton. Three control schemes were tested: non-autonomous (i.e., manual control using the tongue) control, semi-autonomous control with a fixed level of autonomy, and a semi-autonomous control with a confidence-based adaptive level of autonomy. Studies on experimental participants with and without tetraplegia were carried out. The control schemes were evaluated both in terms of their performance, such as the time and number of commands needed to complete a given task, as well as ratings from the users. The studies showed a clear and significant improvement in both performance and user ratings when using either of the semi-autonomous control schemes. The adaptive semi-autonomous control outperformed the fixed version in some scenarios, namely, in the more complex tasks and with users with more training in using the system.

Continuous Semi-autonomous Prosthesis Control Using a Depth Sensor on the Hand.

Modern myoelectric prostheses can perform multiple functions (e.g., several grasp types and wrist rotation) but their intuitive control by the user is still an open challenge. It has been recently demonstrated that semi-autonomous control can allow the subjects to operate complex prostheses effectively; however, this approach often requires placing sensors on the user. The present study proposes a system for semi-autonomous control of a myoelectric prosthesis that requires a single depth sensor placed on the dorsal side of the hand. The system automatically pre-shapes the hand (grasp type, size, and wrist rotation) and allows the user to grasp objects of different shapes, sizes and orientations, placed individually or within cluttered scenes. The system “reacts” to the side from which the object is approached, and enables the user to target not only the whole object but also an object part. Another unique aspect of the system is that it relies on online interaction between the user and the prosthesis; the system reacts continuously on the targets that are in its focus, while the user interprets the movement of the prosthesis to adjust aiming. Experimental assessment was conducted in ten able-bodied participants to evaluate the feasibility and the impact of training on prosthesis-user interaction. The subjects used the system to grasp a set of objects individually (Phase I) and in cluttered scenarios (Phase II), while the time to accomplish the task (TAT) was used as the performance metric. In both phases, the TAT improved significantly across blocks. Some targets (objects and/or their parts) were more challenging, requiring thus significantly more time to handle, but all objects and scenes were successfully accomplished by all subjects. The assessment therefore demonstrated that the system is indeed robust and effective, and that the subjects could successfully learn how to aim with the system after a brief training. This is an important step toward the development of a self-contained semi-autonomous system convenient for clinical applications.

Dimensional Extension of Nonlinear Dynamics for Similar Task Differentiation in Semi-autonomous Control

Dimensional Extension of Nonlinear Dynamics for Similar Task Differentiation in Semi-autonomous Control

Impact of Shared Control Modalities on Performance and Usability of Semi-autonomous Prostheses.

Semi-autonomous (SA) control of upper-limb prostheses can improve the performance and decrease the cognitive burden of a user. In this approach, a prosthesis is equipped with additional sensors (e.g., computer vision) that provide contextual information and enable the system to accomplish some tasks automatically. Autonomous control is fused with a volitional input of a user to compute the commands that are sent to the prosthesis. Although several promising prototypes demonstrating the potential of this approach have been presented, methods to integrate the two control streams (i.e., autonomous and volitional) have not been systematically investigated. In the present study, we implemented three shared control modalities (i.e., sequential, simultaneous, and continuous) and compared their performance, as well as the cognitive and physical burdens imposed on the user. In the sequential approach, the volitional input disabled the autonomous control. In the simultaneous approach, the volitional input to a specific degree of freedom (DoF) activated autonomous control of other DoFs, whereas in the continuous approach, autonomous control was always active except for the DoFs controlled by the user. The experiment was conducted in ten able-bodied subjects, and these subjects used an SA prosthesis to perform reach-and-grasp tasks while reacting to audio cues (dual tasking). The results demonstrated that, compared to the manual baseline (volitional control only), all three SA modalities accomplished the task in a shorter time and resulted in less volitional control input. The simultaneous SA modality performed worse than the sequential and continuous SA approaches. When systematic errors were introduced in the autonomous controller to generate a mismatch between the goals of the user and controller, the performance of SA modalities substantially decreased, even below the manual baseline. The sequential SA scheme was the least impacted one in terms of errors. The present study demonstrates that a specific approach for integrating volitional and autonomous control is indeed an important factor that significantly affects the performance and physical and cognitive load, and therefore these should be considered when designing SA prostheses.

Artificial Perception and Semiautonomous Control in Myoelectric Hand Prostheses Increases Performance and Decreases Effort

Dexterous control of upper limb prostheses with multiarticulated wrists/hands is still a challenge due to the limitations of myoelectric man-machine interfaces. Multiple factors limit the overall performance and usability of these interfaces, such as the need to control degrees of freedom sequentially and not concurrently, and the inaccuracies in decoding the user intent from weak or fatigued muscles. In this article, we developed a novel man-machine interface that endows a myoelectric prosthesis (MYO) with artificial perception, estimation of user intention, and intelligent control (MYO-PACE) to continuously support the user with automation while preparing the prosthesis for grasping. We compared the MYO-PACE against state-of-the-art myoelectric control (pattern recognition) in laboratory and clinical tests. For this purpose, eight able-bodied and two amputee individuals performed a standard clinical test consisting of a series of manipulation tasks (portion of the SHAP test), as well as a more complex sequence of transfer tasks in a cluttered scene. In all tests, the subjects not only completed the trials faster using the MYO-PACE but also achieved more efficient myoelectric control. These results demonstrate that the implementation of advanced perception, context interpretation, and autonomous decision-making into active prostheses improves control dexterity. Moreover, it also effectively supports the user by speeding up the preshaping phase of the movement and decreasing muscle use.

Research and Implementation of a Humanoid Robot Control Method Based on Action Mapping

In order to reduce programming difficulty and let humanoid robot act more humanlike, we designed a robotics control method for humanoid robots. Firstly, based on human body movements collected by Kinect camera, we calculated 3D data of joint point with pinhole model, and established space model of robot and skeleton model of human. Then, we transformed 3D coordinates of human joints into joint values by space vector method, mapping to the joints of the robot and guides the robot to move. Finally, with semi-autonomous control experiment, it shows that humanoid robot action mapping strategy is feasible and effective. Compared with the existing similar research, this paper implements the action guidance control of humanoid robot, enhances the intelligent degree of robot and expands the application scene and scope of humanoid robot. Depending on the human-like appearance, humanoid robot with action mapping could provide a much more intelligent interaction method.

Human interface design for semi-autonomous control of leader-follower excavator based on variable admittance and stagnation/trajectory bifurcation of nonlinear dynamics

Conventional remote operation of an excavator has low work efficiency comparing with on-site operations. This is because it is difficult for an operator to recognize the excavator status and surrounding environments. Moreover, there are restrictions such as a limitation on the amount of information that can be transmitted, delays in communication, and harsh environments that cause sensor failure. Therefore, we have developed a semi-autonomous control system that consists of autonomy and human manipulation. This paper proposes (1) a bilateral control system based on variable admittance and (2) an autonomous control using nonlinear dynamics that has attractors with stagnation and trajectory-bifurcation. (1) In the proposed method, the admittance parameters are changed based on the difference between the leader and follower position. We implement the proposed method into a prototype of excavator and verify the operation with two types of external environmental forces: free movement and contact with a rock. (2) The attractor’s stagnation is designed as a sink of vector field converging to an arbitrary point on the trajectory. A stagnation is placed at a trajectory’s bifurcation point, and the operator selects the next operation by adding a force to the leader system, which is measured through a force sensor. We design two types of bifurcating attractors and conduct an experimental verification of the proposed semi-autonomous control system.

A Development of a Capacitive Voltage Divider for High Voltage Measurement as Part of a Combined Current and Voltage Sensor

This article deals with the development of capacitive voltage divider for high voltage measurements and presents a method of analysis and optimization of its parameters. This divider is a part of a combined voltage and current sensor for measurements in high voltage power networks. The sensor allows continuous monitoring of the network distribution status and performs a quick diagnosis and location of possible network failures. Deployment of these devices will support semi-autonomous control of power networks and it can be considered as a step from traditional power grids toward smart grids. This is a worldwide trend connected with increasing number of renewable energy sources and plug-in electric vehicles as described in. In this way, it contributes to the reliability of the distribution network. Together with automated control techniques and fault location methods, it enables its self-healing capability. The following characteristics required for the sensor include: current measurement error up to 2 %, voltage measurement error up to 0.5 %, and power measurement error up to 5 %. At the same time, it is necessary that the sensor is cost-effective - relatively cheap. There were selected capacitors made in series production for the capacitive divider designing. The capacitive voltage divider was tested in terms of time and temperature stability; the results are described in the paper. Then, the method of mathematical correction of a temperature dependence of the capacitive voltage divider was suggested and tested.

Four-Way Classification of EEG Responses To Virtual Robot Navigation.

Studies have shown the possibility of using brain signals that are automatically generated while observing a navigation task as feedback for semi-autonomous control of a robot. This allows the robot to learn quasi-optimal routes to intended targets. We have combined the subclassification of two different types of navigational errors, with the subclassification of two different types of correct navigational actions, to create a 4-way classification strategy, providing detailed information about the type of action the robot performed. We used a 2-stage stepwise linear discriminant analysis approach, and tested this using brain signals from 8 and 14 participants observing two robot navigation tasks. Classification results were significantly above the chance level, with mean overall accuracy of 44.3% and 36.0% for the two datasets. As a proof of concept, we have shown that it is possible to perform fine-grained, 4-way classification of robot navigational actions, based on the electroencephalogram responses of participants who only had to observe the task. This study provides the next step towards comprehensive implicit brain-machine communication, and towards an efficient semi-autonomous brain-computer interface.