Robotic Sensor Research Articles

RumexWeeds: A grassland dataset for agricultural robotics

AbstractComputer vision can lead toward more sustainable agricultural production by enabling robotic precision agriculture. Vision‐equipped robots are being deployed in the fields to take care of crops and control weeds. However, publicly available agricultural datasets containing both image data as well as data from navigational robot sensors are scarce. Our real‐world dataset RumexWeeds targets the detection of the grassland weeds: Rumex obtusifolius L. and Rumex crispus L. RumexWeeds includes whole image sequences instead of individual static images, which is rare for computer vision image datasets, yet crucial for robotic applications. It allows for more robust object detection, incorporating temporal aspects and considering different viewpoints of the same object. Furthermore, RumexWeeds includes data from additional navigational robot sensors—GNSS, IMU and odometry—which can increase robustness, when additionally fed to detection models. In total the dataset includes 5510 images with 15,519 manual bounding box annotations collected at three different farms and four different days in summer and autumn 2021. Additionally, RumexWeeds includes a subset of 340 ground truth pixels‐wise annotations. The dataset is publicly available at https://dtu-pas.github.io/RumexWeeds/. In this paper we also use RumexWeeds to provide baseline weed detection results considering a state‐of‐the‐art object detector; in this way we are elucidating interesting characteristics of the dataset.

Mimicking of human skin to build wearable sensors.

Yichen Cai creates thin, flexible devices that could have myriad uses, from wearable blood-pressure monitors to touch sensors for robots. Yichen Cai creates thin, flexible devices that could have myriad uses, from wearable blood-pressure monitors to touch sensors for robots.

Review on challenges for robotic eye surgery; surgical systems, technologies, cost-effectiveness, and controllers.

In recent decades, a number of surgical systems have been developed and are applied for a growing variety of surgeries. This review will consider the significant challenges of robotic surgery for the eye. These challenges take into account the different eye diseases, available technologies, and costs in different surgical systems for the eye. The conditions of a suitable controller will be discussed with consideration of relevant control engineering concepts. Comparison is made between the different characteristics of surgical robots for the eye. In this review, some comparisons will be made in eye surgical robots, control algorithms, sensors in surgical robots, communication protocols, and actuators.

An Interventional Surgical Robot Based on Multi-Data Detection

Vascular interventional surgery is the most common method for the treatment of cardiovascular diseases. Interventional surgical robot has attracted extensive attention because of its precise control and remote operation. However, conventional force sensors in surgical robots can only detect the axial thrust pressure of the catheter. Inspired by the function of insect antennae, we designed a structure with a thin-film force sensing device in the catheter head. Combined with the pressure sensor in the catheter clamping device, multiple sensor data were fused to predict and classify the current vascular environment using the LSTM network with 94.2% accuracy. During robotic surgery, real-time feedback of current pressure information and vascular curvature information can enhance doctors’ judgment of surgical status and improve surgical safety.

Ultra-stable and tough bioinspired crack-based tactile sensor for small legged robots

For legged robots, collecting tactile information is essential for stable posture and efficient gait. However, mounting sensors on small robots weighing less than 1 kg remain challenges in terms of the sensor’s durability, flexibility, sensitivity, and size. Crack-based sensors featuring ultra-sensitivity, small-size, and flexibility could be a promising candidate, but performance degradation due to crack growing by repeated use is a stumbling block. This paper presents an ultra-stable and tough bio-inspired crack-based sensor by controlling the crack depth using silver nanowire (Ag NW) mesh as a crack stop layer. The Ag NW mesh inspired by skin collagen structure effectively mitigated crack propagation. The sensor was very thin, lightweight, sensitive, and ultra-durable that maintains its sensitivity during 200,000 cycles of 0.5% strain. We demonstrate sensor’s feasibility by implementing the tactile sensation to bio-inspired robots, and propose statistical and deep learning-based analysis methods which successfully distinguished terrain type.

Creating Stretchable Electronics from Dual Layer Flex-PCB for Soft Robotic Cardiac Mapping Catheters

The authors present in this study the development of a novel method for creating stretchable electronics from dual-layer flex printed circuit boards (flex-PCBs) as a platform for soft robotic sensor arrays (SRSAs) for cardiac voltage mapping applications. There is a crucial need for devices that utilize multiple sensors and provide high performance signal acquisition for cardiac mapping. Previously, our group demonstrated how single-layer flex-PCB can be postprocessed to create a stretchable electronic sensing array. In this work, a detailed fabrication process for creating a dual-layer multielectrode flex-PCB SRSA is presented, along with relevant parameters to achieve optimal postprocessing with a laser cutter. The dual-layer flex-PCB SRSA’s ability to acquire electrical signals is demonstrated both in vitro as well as in vivo on a Leporine cardiac surface. These SRSAs could be extended into full-chamber cardiac mapping catheter applications. Our results show a significant contribution towards the scalable use of dual-layer flex-PCB for stretchable electronics.

Triangular lattice formation in robot swarms with minimal local sensing

AbstractThe problem of triangular lattice formation in robot swarms has been investigated extensively in the literature, but the existing algorithms can hardly keep comparative performance from swarm simulation to real multi‐robot scenarios, due to the limited computation power or the restricted field of view (FOV) of robot sensors. Eventually, a distributed solution for triangular lattice formation in robot swarms with minimal sensing and computation is proposed and developed in this study. Each robot is equipped with a sensor with a limited FOV providing only a ternary digit of information about its neighbouring environment. At each time step, the motion command is directly determined by using only the ternary sensing result. The circular motions with a certain level of randomness lead the robot swarms to stable triangular lattice formation with high quality and robustness. Extensive numerical simulations and multi‐robot experiments are conducted. The results have demonstrated and validated the efficiency of the proposed approach. The minimised sensing and computation requirements pave the way for massive deployment at a low cost and implementation within swarms of miniature robots.

Ardosia: Simulating Circuits and Robotic Systems in a Single Learning Platform

Circuit simulation, together with practical sessions, is considered a fundamental tool to assist the teaching of electronics. Robotics offers rich real-world applications that foster student interest in developing engineering projects. Despite the potential benefits of uniting these two realities, software tools that combine circuits and robotic systems in a single simulation platform are scarce. This paper presents Ardosia, a simulation software that combines electronics and robotics in a single simulation arena. The electronics aspects of the simulation are handled by the Falstad Circuit Simulator, a software with proven success in promoting student interest and learning outcomes. Importantly, this software features significant interactive and visualization characteristics that are deemed valuable to enhance conceptual understanding. Non-electrical physics simulation is performed by a special purpose engine, designed to simulate sensor and actuator behavior in a 2D world. In combination, the two simulation engines yield a model where electronic circuits are embodied in the world through robot sensors and actuators interacting together and with the surroundings. Such a simulator is intended to promote the use of robotics projects in electronics education, since students will be equipped with a software to conveniently design and evaluate robotic systems, through simulation, before applying their ideas in an experimental setting. The paper presents an overview of Ardosia, complemented with examples of use in electronics education and a validation experiment comparing simulation with real-world data.

Simultaneous Online Registration-Independent Stiffness Identification and Tip Localization of Surgical Instruments in Robot-assisted Eye Surgery.

Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing of surgery states (e.g. instrument tip localization and tool-to-tissue interaction forces). Many of the existing tool tip localization methods require preoperative frame registrations or instrument calibrations. In this study using an iterative approach and by combining vision and force-based methods, we develop calibration- and registration-independent (RI) algorithms to provide online estimates of instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model based on the forward kinematics (FWK) of the Steady-Hand Eye Robot (SHER) and Fiber Brag Grating (FBG) sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the deflected instrument tip position estimations during robot-assisted eye surgery. The conducted experiments demonstrate that when the online RI stiffness estimations are used, the instrument tip localization results surpass those obtained from pre-operative offline calibrations for stiffness.

Wearable sensor network for lower limb angle estimation in robotics applications

In human-robot interaction, sensors are relevant in guaranteeing stability and high performance in real-time applications. Nonetheless, accuracy and portable sensors for robots usually have high costs and little flexibility to process signals with free software. Therefore, we propose a wearable sensor network to measure lower limb angular position in human-robot interaction systems. The methodology employed to achieve the aim consisted in implementing a wireless network using low-cost devices, verifying design requirements, and making a validation via a proof of concept. The requirements to design the network include low loss of information, real-time communication, and sensor fusion to estimate the angular position using a gyroscope and accelerometer. Hence, the sensor network developed has a client-server architecture based on ESP8266 microcontrollers. In addition, this network uses the standard 802.11 b/g/n to transmit angular velocity and acceleration measures. Furthermore, we implement the user datagram protocol (UDP) protocol to operate in real-time with a sample time of 10 ms. Finally, we implement a proof of concept to show the system’s effectiveness. Thus, we use the Kalman filter to estimate the angular position of the foot, shin, thigh, and hip. Results indicate that the implemented sensor network is suitable for real-time robotic applications.

A railway track reconstruction method using robotic vision on a mobile manipulator: A proposed strategy

Autonomous robot integration in railways infrastructure maintenance accelerates the digitization and intelligence of infrastructure survey & maintenance, providing high-efficiency and low-cost execution. This paper proposes a health assessment based on 3D reconstruction technology for railway track maintenance using a mobile robotic sensing platform. By combining multiple sensing and taking advantage of a robotic manipulator, a digital model of the target track components is built by a robot-actuated vision system which provides better 3D structural and surface condition reconstruction. Global geo-location and surrounding laser scanning are integrated to reinforce the digital completeness of the model for intelligent management. The new method consists of the following steps: First, according to scheduled maintenance tasks, a Robotics Inspection and Repair System (RIRS) navigates to the task location and uses the onboard depth camera for positioning. Then robot-mounted vision system is guided with an automated trajectory to build the 3D reconstruction of the track or repair object using the vision modeling technique. Finally, the 3D reconstructed model is fused with surrounding mapping of depth vision and Lidar scanning. Both laboratory tests and a realistic track test validated the feasibility of the proposed method by creating an accurate 3D reconstructed model. The modeled rail steel section size is quantitively compared with the ground truth in dimension, demonstrating good accuracy with a size error of less than 0.3 cm. The main contribution includes: (1) unmanned automatic 3D reconstruction by a robotic mobile manipulator, (2) the technique trims the reconstruction details & data to the specific maintenance goal or components, which supports the infrastructure maintenance towards the high-detailed & target-oriented digital management. This combination strategy of robotic automation and sensor fusion lies down a promising foundation for automated digital twin establishment for railway maintenance with autonomous RIRS, and upgrades technology readiness and digital intelligence for maintenance management

Pose accuracy improvement in robotic machining by visually-guided method and experimental investigation

Industrial robots have been widely used in the industries of automotive, machining, electrical and electronic, rubber and plastics, aerospace, food, etc., owing to their high efficiency and flexibility in contrast to large scaled machining centers. However, the poor accuracy resulted from the serial configuration of industrial robots has restricted their applications to high-precision machining for several decades. In this paper, an error compensation technique is proposed using the visual guidance to effectively improve the pose accuracy of industrial robots. Firstly, the effect of the establishment method of tracking coordinate system for a visual sensor on the pose measurement error is analyzed and the establishment method is optimized. Next, a fuzzy PID controller is designed and integrated with a KUKA robot controller KRC via the KUKA robot sensor interface to guide the robot to the desired pose in real-time. Finally, experimental tests are implemented to validate the effectiveness of the proposed approach.

Energy Harvesting and Task-Aware Multi-Robot Task Allocation in Robotic Wireless Sensor Networks

In this work, we investigate an energy-aware multi-robot task-allocation (MRTA) problem in a cluster of the robot network that consists of a base station and several clusters of energy-harvesting (EH) robots. It is assumed that there are M+1 robots in the cluster and M tasks exist in each round. In the cluster, a robot is elected as the cluster head, which assigns one task to each robot in that round. Its responsibility (or task) is to collect the resultant data from the remaining M robots to aggregate and transmit directly to the BS. This paper aims to allocate the M tasks to the remaining M robots optimally or near optimally by considering the distance to be traveled by each node, the energy required for executing each task, the battery level at each node, and the energy-harvesting capabilities of the nodes. Then, this work presents three algorithms: Classical MRTA Approach, Task-aware MRTA Approach, EH and Task-aware MRTA Approach. The performances of the proposed MRTA algorithms are evaluated under both independent and identically distributed (i.i.d.) and Markovian energy-harvesting processes for different scenarios with five robots and 10 robots (with the same number of tasks). EH and Task-aware MRTA Approach shows the best performance among all MRTA approaches by keeping up to 100% more energy in the battery than the Classical MRTA Approach and keeping up to 20% more energy in the battery than the Task-aware MRTA Approach.

A Skin-Like Soft Compression Sensor for Robotic Applications.

We present a compression sensor based on a strain-sensitive carbon black-silicone composite cast on top of a printed circuit board with interdigitated electrodes. This results in a very sensitive and soft capacitive compression sensor not requiring a structured dielectric or compliant electrodes. We show how the optimal loading of carbon black to maximize the sensitivity depends on the type of carbon black and the stiffness of the silicone matrix. The optimal quantity of carbon black leads to a high sensitivity of 252% for an input force of 10 N (this corresponds to an input pressure of 17 kPa), without stiffening the silicone matrix or increasing the viscoelastic losses noticeably. The fabrication process of the sensors is much simpler than that of other soft capacitive sensors, and unlike carbon black-silicone resistive sensors, these capacitive sensors do not exhibit time-dependent impedance creep. They can be made thick without affecting their base capacitance or sensitivity, leading to compliant and conformable sensing interfaces suitable for a variety of applications, such as robotic tactile sensors.

Mosaic Charge Distribution-Based Sliding and Pressing Triboelectrification under Wavy Configuration

As high-voltage output and fast response devices, triboelectric nanogenerators (TENGs) are widely used for sensors with fast and high-sensitivity performance. As a primary electrical signal, the waveform output provides an accurate and rapid response to external stimulus parameters such as press and slide. Here, based on mosaic charging and residual charge theories, the contact charging principle of TENGs is further discussed. Moreover, a wavy structure is obtained in the vertical contact separation and lateral sliding modes to further study the influence of external parameters applied to TENGs, which thus helps further the understanding of the output waveforms. The experimental results show that wavy TENGs have output properties that are excellent compared to those of TENGs with flat structures, such as longer charging and discharging times and more complex waveforms. By researching the waveform in depth, our work will provide new prospects for application in various sensors of interactive wearable systems, intelligent robots, and optoelectronic devices based on TENGs.

Perspective for soft robotics: the field’s past and future

Since its beginnings in the 1960s, soft robotics has been a steadily growing field that has enjoyed recent growth with the advent of rapid prototyping and the provision of new flexible materials. These two innovations have enabled the development of fully flexible and untethered soft robotic systems. The integration of novel sensors enabled by new manufacturing processes and materials shows promise for enabling the production of soft systems with ‘embodied intelligence’. Here, four experts present their perspectives for the future of the field of soft robotics based on these past innovations. Their focus is on finding answers to the questions of: how to manufacture soft robots, and on how soft robots can sense, move, and think. We highlight industrial production techniques, which are unused to date for manufacturing soft robots. They discuss how novel tactile sensors for soft robots could be created to enable better interaction of the soft robot with the environment. In conclusion this article highlights how embodied intelligence in soft robots could be used to make soft robots think and to make systems that can compute, autonomously, from sensory inputs.

High-Precision FBG 3-D Force Sensor Based on Spatial Hierarchical Sensing Structure

A 3-D force sensor based on fiber Bragg grating (FBG) is proposed in this article. A spatially layered elastomer structure is designed, and a single-mode fiber inscribed with four FBGs is laid at each place of the structure to realize the 3-D force sensing of the robot and compensate for temperature interference. The principle of FBG sensing and force measurement is investigated to reveal the mapping relationship between the wavelength drift and force of this 3-D force sensor. Based on the finite-element simulation analysis, the FBG package position is optimized, and the sensor prototype is tested. Experimental results show that the linearity of the 3-D force sensor force sensing is good, and its sensitivity is 3.104, 3.077, and 1.890 pm/N. Based on sparrow search algorithm optimization extreme learning machine (SSA-ELM), a nonlinear decoupling algorithm is used to construct the neural network model, and the SSA is used to optimize the model and obtain the best initial weights and thresholds of the network. The maximum I error after decoupling is 1.58%, the maximum II error is 0.93%, and the decoupling training time is 1.44 s. The algorithm can construct the interdimensional coupling relationship of 3-D force more effectively and improve the sensor measurement accuracy. In addition, the sensor has strong resistance to electromagnetic interference and noise, fewer sensitive components, a simple lineup, a small structure and lightweight, and easy processing, and meets the requirements of miniaturization, lightweight, and refinement of multidimensional force sensors for collaborative robots.

Noise characterization of unmanned ground vehicles for infrasound array deployment

Deployment of infrasound arrays can present humans with challenges including harsh environments and dangerous chemicals. Infrasonic monitoring is the use of subaudible acoustics to monitor something of interest. Sources of infrasound include natural and manmade sources such as volcanic eruptions, large forest fires, and explosions. Robotic infrasound array deployment is being investigated to mitigate the potential challenges associated with human deployment. Robotic deployment would also allow for remote sensor repositioning to improve array response for specific sources. Unfortunately, robotic sensor emplacement will induce noise associated with the robotic platform, and this noise has a potential to mask sources of interest under certain conditions. Understanding to which conditions robotic deployment is applicable is vital to exploring infrasonic monitoring in hazardous environments in the future. This presentation will describe the characterization of noise generated by multiple robotic platforms in both mobile and stationary states. Analysis during each of these states allows for the determination of when robotic mounted infrasonic monitoring is viable.

Thermally Drawn Elastomer Nanocomposites for Soft Mechanical Sensors.

Stretchable and conductive nanocomposites are emerging as important constituents of soft mechanical sensors for health monitoring, human-machine interactions, and soft robotics. However, tuning the materials' properties and sensor structures to the targeted mode and range of mechanical stimulation is limited by current fabrication approaches, particularly in scalable polymer melt techniques. Here, thermoplastic elastomer-based nanocomposites are engineered and novel rheological requirements are proposed for their compatibility with fiber processing technologies, yielding meters-long, soft, and highly versatile stretchable fiber devices. Based on microstructural changes in the nanofiller arrangement, the resistivity of the nanocomposite is tailored in its final device architecture across an entire order of magnitude as well as its sensitivity to strain via tuning thermal drawing processing parameters alone. Moreover, the prescribed electrical properties are coupled with suitable device designs and several fiber-based sensors are proposed aimed at specific types of deformations: i) a robotic fiber with an integrated bending mechanism where changes as small as 5° are monitored by piezoresistive nanocomposite elements, ii) a pressure-sensing fiber based on a geometrically controlled resistive signal that responds with a sub-newton resolution to changes in pressing forces, and iii) a strain-sensing fiber that tracks changes in capacitance up to 100% elongation.

Unified Robot and Inertial Sensor Self-Calibration.

Robots and inertial measurement units (IMUs) are typically calibrated independently. IMUs are placed in purpose-built, expensive automated test rigs. Robot poses are typically measured using highly accurate (and thus expensive) tracking systems. In this paper, we present a quick, easy, and inexpensive new approach to calibrate both simultaneously, simply by attaching the IMU anywhere on the robot's end effector and moving the robot continuously through space. Our approach provides a fast and inexpensive alternative to both robot and IMU calibration, without any external measurement systems. We accomplish this using continuous-time batch estimation, providing statistically optimal solutions. Under Gaussian assumptions, we show that this becomes a nonlinear least squares problem and analyze the structure of the associated Jacobian. Our methods are validated both numerically and experimentally and compared to standard individual robot and IMU calibration methods.