On-board Computer Research Articles

SpaceDrones 2.0—Hardware-in-the-Loop Simulation and Validation for Orbital and Deep Space Computer Vision and Machine Learning Tasking Using Free-Flying Drone Platforms

The proliferation of reusable space vehicles has fundamentally changed how assets are injected into the low earth orbit and beyond, increasing both the reliability and frequency of launches. Consequently, it has led to the rapid development and adoption of new technologies in the aerospace sector, including computer vision (CV), machine learning (ML)/artificial intelligence (AI), and distributed networking. All these technologies are necessary to enable truly autonomous “Human-out-of-the-loop” mission tasking for spaceborne applications as spacecrafts travel further into the solar system and our missions become more ambitious. This paper proposes a novel approach for space-based computer vision sensing and machine learning simulation and validation using synthetically trained models to generate the large amounts of space-based imagery needed to train computer vision models. We also introduce a method of image data augmentation known as domain randomization to enhance machine learning performance in the dynamic domain of spaceborne computer vision to tackle unique space-based challenges such as orientation and lighting variations. These synthetically trained computer vision models then apply that capability for hardware-in-the-loop testing and evaluation via free-flying robotic platforms, thus enabling sensor-based orbital vehicle control, onboard decision making, and mobile manipulation similar to air-bearing table methods. Given the current energy constraints of space vehicles using solar-based power plants, cameras provide an energy-efficient means of situational awareness when compared to active sensing instruments. When coupled with computationally efficient machine learning algorithms and methods, it can enable space systems proficient in classifying, tracking, capturing, and ultimately manipulating objects for orbital/planetary assembly and maintenance (tasks commonly referred to as In-Space Assembly and On-Orbit Servicing). Given the inherent dangers of manned spaceflight/extravehicular activities (EVAs) currently employed to perform spacecraft maintenance and the current limitation of long-duration human spaceflight outside the low earth orbit, space robotics armed with generalized sensing and control and machine learning architecture have a unique automation potential. However, the tools and methodologies required for hardware-in-the-loop simulation, testing, and validation at a large scale and at an affordable price point are in developmental stages. By leveraging a drone’s free-flight maneuvering capability, theater projection technology, synthetically generated orbital and celestial environments, and machine learning, this work strives to build a robust hardware-in-the-loop testing suite. While the focus of the specific computer vision models in this paper is narrowed down to solving visual sensing problems in orbit, this work can very well be extended to solve any problem set that requires a robust onboard computer vision, robotic manipulation, and free-flight capabilities.

Vision-guided unmanned aerial system for rapid multiple-type damage detection and localization

Unmanned aerial systems (UASs) are increasingly applied for bridge inspection. A vision-guided UAS with a lightweight convolutional neural network is developed to detect and locate bridge cracks, spalling, and corrosion. The contributions are as follows: (1) To address the problem that traditional UASs are global positioning system (GPS) required while GPS signals under bridge bottom generally are weak. A vision-guided UAS is designed and applied, in which a stereo vision-inertial fusion method is used to provide position data instead of GPS and an ultrasonic ranger is applied to avoid obstacles. (2) Most of the deep learning-based damage detection methods are offline detection, which is unsuitable for UAS-based inspection because the endurance time is limited. To solve this problem, a lightweight end-to-end object detection network is proposed, by replacing the backbone of the original You Only Look Once v3 network with MobileNetv2, and the proposed network of much faster inference speed can be transplanted to the onboard computer of the designed UAS so that real-time edge computing can be performed during inspection. (3) A damage location method based on vision positioning data and simultaneous localization and mapping is also proposed to meet the urgent needs of locating damage in the whole structure. Finally, the proposed system is applied to inspect a long-span bridge to detect and locate the most common damages: crack, spalling, and corrosion with high accuracy and efficiency, which verified the practicability of the system.

DESIGN OF STABILIZED INFORMATION AND MEASURING SYSTEM PLATFORM FOR INTELLIGENCE OPERATIONS

The article presents the results of the design of a stabilized platform of information and measurement system for reconnaissance operations based on robotic mobile tracked platform. The analysis of existing mobile robots for reconnaissance operations was carried out and it was established that all of them are practically effective only in the conditions of work in the open area. A number of ideas, approaches and working hypotheses for the design of a new information and measurement system for reconnaissance operations based on a robotic mobile tracked platform. It has been established that this system is necessary to warn of possible threats of explosions or gas poisoning, rescue operations, etc. The developed design of the robotic caterpillar platform is described, the classical scheme of caterpillar wheels is shown and its movements are described. Attention is paid to the position of the drive, because depending on the orientation in the space of the drive, depends on the direction of rotation of its output shaft, and accordingly the direction of movement of tracked tracks. The expediency of using caterpillar platforms for reconnaissance operations is substantiated. All possible angles of deviation of the stabilized platform from zero position are described. The principle of operation of the stabilized platform of the information-measuring system for conducting reconnaissance operations is mathematically described. It is established that the stabilization of the platform is realized with the help of only two linear accelerometers and motors controlled by an on-board digital computer. The block diagram of the stabilized platform of the information-measuring system for carrying out reconnaissance operations is developed and the admissible error of its stabilization in vertical position is defined.

Design and verification of small solid rockets navigation algorithm test platform

Aiming at the navigation system of small solid rockets, a navigation algorithm test platform with real time simulator and flight control computer is designed. The inertial navigation algorithm is taken as an example to test the effectiveness of the platform. The flight dynamic model of a small solid rocket is established in a real time simulator, and the test is carried out in the working environment of a small test rocket to simplify the model. The acceleration and angular velocity data generated by the simulation is sent to the onboard computer to calculate the attitude of the rocket in real time, which verifies the correctness of the data communication link, real time solution and navigation algorithm between the onboard computer and the real time simulator. The test results show that the test platform is easy to use and can test the rocket navigation algorithm quickly and effectively.

A Modular Open-Source Continuum Manipulator for Underwater Remotely Operated Vehicles

Abstract Continuum manipulators are a class of robots with many degrees-of-freedom, leading to highly flexible motion with inherent compliance. These attributes make them well suited for manipulation tasks and physical interaction with the environment. A high impact yet challenging field for exploring continuum robot designs is free-floating underwater manipulation with a remotely operated vehicle (ROV). In this article, we propose a modular, reconfigurable, cable-driven continuum arm for free-floating underwater manipulation and present a corresponding kinematics, control, and computation framework. The mechanical design consists of a continuum arm, an actuation unit, and a waterproof enclosure. The kinematics model is introduced as two mappings between three spaces: the joint space, the configuration space, and the task space. The differential kinematics for each mapping is also derived. An electronics system design is proposed for underwater applications, including the communication framework between the topside computer (above surface), on-board computer, and manipulator mechatronics. Experimental validation is presented to demonstrate the robot’s underwater functionality, test the limits of its articulation, and evaluate the arm’s stiffness. Future work includes field testing with an ROV platform and development of advanced controls and planning for manipulation tasks.

Исследование информационного взаимодействия между различными подсистемами на основе стандартов CCSDS при идентификации материальных объектов

Due to the increasing complexity of information processes, there is a need to develop models and methods of interaction between various subsystems, segments and, in relation to them, to develop methods for identifying ob-jects. The object of research is the process of accounting for the movement of material objects between different segments of complex systems, for example, between different segments of the international space station, between different segments of the material flow system or other segments of the Earth-Space system. Instruments and instrument complexes were chosen as material objects. Based on the performed study of data structure complexity levels, the necessity of development of identification systems for different segments of the material flow transportation system was substantiated. A new method is proposed and the results of practical experiments on the identification of material objects on the basis of CCSDS standard (CCSDS RFID Tag-Encoding Yellow Book) are considered. The hardware platform for accounting of tangible objects and information collection was selected radio-frequency readers and on-board computer networks represented by CCSDS 881.1-B-1 (Spacecraft onboard interface services - RFID tag encoding specification). RFID tags are used to identify tangible objects. To solve the problem of transferring information about material objects the developed model of information exchange and method of identification of material objects are presented, necessary fields and tag's memory bank structure are presented. In the process of describing the identification of material objects it is proposed to use a special alphabet ECMA-113 for data representation in the identifiers placed in the RFID tags. As part of the study, the simulation of information exchange between different subsystems during the transfer of information about material objects was conducted. The results of the experiments are presented, confirming that the universality of the alphabet lies in the possibility of simultaneous use of Latin and Cyrillic coding. For an estimation of efficiency of the offered models and modelling of a situation of an exchange of the information on material objects in wireless on-board networks separate independent subprogrammes of coding, decoding of identifiers of material objects have been developed and practical modelling of transfer of the big information volume between two various agencies (segments) is spent. The programmes were developed by the participants independently 
 of each other. On the basis of fulfilled modeling of information interaction and information transfer on material objects it was found that the error of data recognition does not exceed 6-7% and it does not increase with the increasing number of objects. The development results allowed to formulate a conclusion about the success of the experiment on the application of object identifiers based on CCSDS for airborne wireless networks, for the use of radio-frequency identification systems.

Onboard Orbit Determination for Deep-Space CubeSats

In this work, an orbit determination algorithm suitable for CubeSats onboard implementation is developed, which simulates optical autonomous navigation accomplished by a stand-alone platform. An extended Kalman filter featuring line-of-sight acquisitions of planets is selected as the state estimator, and its performances are tested on a Raspberry Pi, whose characteristics are comparable to a miniaturized onboard computer. An improvement of the solution accuracy is performed by correcting the planetary light-time and aberration effects as well as by exploiting the optimal beacons selection strategy to acquire the external observations. Moreover, the numerical precision of the estimator is improved through the implementation of factorization techniques and nondimensionalization strategies. The results are presented for a sample Earth–Mars transfer, where the time slot for the navigation campaign involves 2 h every 10 days. At final time, the probe position and velocity are estimated with a accuracy of 360 km and 0.04 m/s, respectively.

A Shadowcasting-Based Next-Best-View Planner for Autonomous 3D Exploration

In this letter, we address the problem of autonomous exploration of unknown environments with an aerial robot equipped with a sensory set that produces large point clouds, such as LiDARs. The main goal is to gradually explore an area while planning paths and calculating information gain in short computation time, suitable for implementation on an on-board computer. To this end, we present a planner that randomly samples viewpoints in the environment map. It relies on a novel and efficient gain calculation based on the Recursive Shadowcasting algorithm. To determine the Next-Best-View (NBV), our planner uses a cuboid-based evaluation method that results in an enviably short computation time. To reduce the overall exploration time, we also use a dead end resolving strategy that allows us to quickly recover from dead ends in a challenging environment. Comparative experiments in simulation have shown that our approach outperforms the current state-of-the-art in terms of computational efficiency and total exploration time. The video of our approach can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://www.youtube.com/playlist?list =PLC0C6uwoEQ8ZDhny1VdmFXLeTQOSBibQl</uri> .

White Whale, Black Swan

White Whale, Black Swan

AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation

In this letter, we present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerial vehicles (UAVs) with on-board RGB cameras and computation. Existing methods are limited by calibrated cameras and off-line processing. Thus, we present the first method (AirPose) to estimate human pose and shape using images captured by multiple extrinsically <b>uncalibrated</b> flying cameras. AirPose itself calibrates the cameras <i>relative to the person</i> instead of relying on any pre-calibration. It uses distributed neural networks running on each UAV that communicate <i>viewpoint-independent</i> information with each other about the person (i.e., their 3D shape and articulated pose). The person&#x2019;s shape and pose are parameterized using the SMPL-X body model, resulting in a compact representation, that minimizes communication between the UAVs. The network is trained using synthetic images of realistic virtual environments, and fine-tuned on a small set of real images. We also introduce an optimization-based post-processing method (AirPose<inline-formula><tex-math notation="LaTeX">$^{+}$</tex-math></inline-formula>) for offline applications that require higher MoCap quality. We make our method&#x2019;s code and data available for research at <uri>https://github.com/robot-perception-group/AirPose</uri>. A video describing the approach and results is available at <uri>https://youtu.be/xLYe1TNHsfs</uri>.

Integrated decision and control at multi-lane intersections with mixed traffic flow

Autonomous driving at intersections is one of the most complicated and accident-prone traffic scenarios, especially with mixed traffic participants such as vehicles, bicycles and pedestrians. The driving policy should make safe decisions to handle the dynamic traffic conditions and meet the requirements of on-board computation. However, most of the current researches focuses on simplified intersections considering only the surrounding vehicles and idealized traffic lights. This paper improves the integrated decision and control framework and develops a learning-based algorithm to deal with complex intersections with mixed traffic flows, which can not only take account of realistic characteristics of traffic lights, but also learn a safe policy under different safety constraints. We first consider different velocity models for green and red lights in the training process and use a finite state machine to handle different modes of light transformation. Then we design different types of distance constraints for vehicles, traffic lights, pedestrians, bicycles respectively and formulize the constrained optimal control problems (OCPs) to be optimized. Finally, reinforcement learning (RL) with value and policy networks is adopted to solve the series of OCPs. In order to verify the safety and efficiency of the proposed method, we design a multi-lane intersection with the existence of large-scale mixed traffic participants and set practical traffic light phases. The simulation results indicate that the trained decision and control policy can well balance safety and tracking performance. Compared with model predictive control (MPC), the computational time is three orders of magnitude lower.

Coming Electric Vehicle Era

Coming Electric Vehicle Era

Lightweight Challenge-Response Authentication in SDN-Based UAVs Using Elliptic Curve Cryptography

Unmanned aerial vehicles (UAVs) (also known as drones) are aircraft that do not require the presence of a human pilot to fly. UAVs can be controlled remotely by a human operator or autonomously by onboard computer systems. UAVs have many military uses, including battlefield surveillance, effective target tracking and engagement in air-to-ground warfare, and situational awareness in challenging circumstances. They also offer a distinct advantage in various applications such as forest fire monitoring and surveillance. Surveillance systems are developed using advanced technologies in the modern era of communications and networks. As a result, UAVs require enhancements to control and manage systems efficiently. Network security is a critical concern with respect to UAVs due to the risk of surveillance information theft and physical misuse. Although several new tools have been introduced to secure networks, attackers can use more advanced methods to get into a UAV network and create problems that pose an organizational threat to the entire system. Security mechanisms also reduce the performance of systems because some restrictive measures prevent users from accessing specific resources, but a few techniques and tools have overcome the problem of performance reduction in various scenarios. There are many types of attacks, i.e., denial of service attacks (DOS), distributed denial of service attacks (DDOS), address resolution protocol (ARP) spoofing, sniffing, etc., that make it challenging to maintain a UAV network. This research paper proposes a lightweight challenge-response authentication that can overcome the previously mentioned problems. As security is provided by utilizing a minimum number of bits in memory, this technique offers the same security features while using fewer network resources, low computing resources, and low power consumption.

Modelling Diversity in Redundant Systems: Case Study of Satellite Systems

Diversity plays an essential role in increasing reliability of redundant systems, however in safety and mission critical applications. The onboard computer of satellites and the flight computer of aircrafts, which are ultra-reliable system, utilizes various hardware platforms for their redundant architecture to resolve the common cause failure problem. Furthermore, the software is also developed by separate teams based on different software platforms to mitigate the specification and design flaws, and implementation mistakes. This paper focuses on modelling diversity of redundant systems using Markov reliability analyzing method. The proposed scheme is explored in two types of applications: mission critical applications (with long mission time) and safety critical applications (with short mission time). Analytical and simulation results show the effectiveness of diversity in increasing the reliability of these systems. Since about ten percent of all failures appear as common cause failures, which restrict reliability improvement through similar redundant modules, achieving ultra-reliable necessitate to consider diversity in these systems

A Comparative Study of Autonomous Object Detection Algorithms in the Maritime Environment Using a UAV Platform

Maritime operations rely heavily on surveillance and require reliable and timely data that can inform decisions and planning. Critical information in such cases includes the exact location of objects in the water, such as vessels, persons, and others. Due to the unique characteristics of the maritime environment, the location of even inert objects changes through time, depending on the weather conditions, water currents, etc. Unmanned aerial vehicles (UAVs) can be used to support maritime operations by providing live video streams and images from the area of operations. Machine learning algorithms can be developed, trained, and used to automatically detect and track objects of specific types and characteristics. EFFECTOR is an EU-funded project, developing an Interoperability Framework for maritime surveillance. Within the project, we developed an embedded system that employs machine learning algorithms, allowing a UAV to autonomously detect objects in the water and keep track of their changing position through time. Using the on-board computation unit of the UAV, we ran and present the results of a series of comparative tests among possible architecture sizes and training datasets for the detection and tracking of objects in the maritime environment. We tested architectures based on their efficiency, accuracy, and speed. A combined solution for training the datasets is suggested, providing optimal efficiency and accuracy.

Optimal guidance for lunar soft landing with dynamic low-resolution image sequences

To relieve the burden of obtaining high-precision lunar surface information during lunar soft landing, a new guidance algorithm based on dynamic low-resolution image sequences is proposed. The integrated procedure of dynamic descent stage considering obstacle-avoiding requirement without the hover measurement is designed, which enables the spacecraft to optimize the final landing site by rolling in orbit. A small number of points are used to obtain the variance, slope and roughness of the field of vision subarea, and the shortest distance spiral search algorithm is used to determine the safest landing point. In order to reduce unnecessary control, an acceptance domain algorithm of alternative landing sites is designed to judge whether or not to accept the change of landing sites in dynamic low-precision image sequences. Assuming that the control acceleration is adjustable, the non-singular fast terminal sliding mode (NFTSM) controller based on terminal attractor is adopted and improved, and the stability of the closed-loop system is also proved. The validity, reliability and effectiveness of the proposed guidance algorithm are verified by numerical simulation on the case with disturbance and large sample data. The results demonstrate that the image information required for stable lunar soft landing is far less than that of the traditional methods, which can greatly reduce the burden of the onboard computer.

Towards new frontiers in mobile manipulation: Team CTU-UPenn-NYU at MBZIRC 2020

In this paper we present an autonomous robotic system for picking, transporting, and precisely placing magnetically graspable objects. Such a system would be especially beneficial for construction tasks where human presence is not possible, e.g. due to chemical or radioactive pollution. The system comprises of two primary components – a wheeled, mobile platform and a manipulator arm. Both are interconnected through an onboard computer and utilize various onboard sensors for estimating the state of the robot and its surroundings. By using efficient processing algorithms, data from the onboard sensors can be used in a feedback loop during all critical operational sections, resulting in a robust system capable of operating on uneven terrain and in environments without access to satellite navigation. System functionality has been proven in Challenge II of the MBZIRC 2020 competition. The Challenge required a ground robot to build an L-shaped structure of colored bricks laid in a predefined pattern. Such a mission incorporates several demanding subchallenges, spanning multiple branches of computer science, cybernetics, and robotics. Moreover, all the subchallenges had to be performed flawlessly in rapid succession, in order to complete the Challenge successfully. The extreme difficulty of the task was highlighted in the MBZIRC 2020 finals, where our system was among the only two competitors (out of 32) that managed to complete the task in autonomous mode.

Autonomous Aerial Robot for High-Speed Search and Intercept Applications

In recent years, high-speed navigation and environment interaction in the context of aerial robotics has become a field of interest for several academic and industrial research studies. In particular, Search and Intercept (SaI) applications for aerial robots pose a compelling research area due to their potential usability in several environments. Nevertheless, SaI tasks involve a challenging development regarding sensory weight, onboard computation resources, actuation design, and algorithms for perception and control, among others. In this work, a fully autonomous aerial robot for high-speed object grasping has been proposed. As an additional subtask, our system is able to autonomously pierce balloons located in poles close to the surface. Our first contribution is the design of the aerial robot at an actuation and sensory level consisting of a novel gripper design with additional sensors enabling the robot to grasp objects at high speeds. The second contribution is a complete software framework consisting of perception, state estimation, motion planning, motion control, and mission control in order to rapidly and robustly perform the autonomous grasping mission. Our approach has been validated in a challenging international competition and has shown outstanding results, being able to autonomously search, follow, and grasp a moving object at 6 m/s in an outdoor environment.

Bayesian Shape Reconstruction and Optimal Guidance for Autonomous Landing on Asteroids

Construction of the precise shape of an asteroid is critical for spacecraft operations as the gravitational potential is determined by spatial mass distribution. The typical approach to shape determination requires a prolonged “mapping” phase of the mission over which extensive measurements are collected and transmitted for Earth-based processing. This paper presents a set of approaches to explore an unknown asteroid with onboard calculations, and to land on its surface area selected in an optimal fashion. The main motivation is to avoid the extended period of mapping or preliminary ground observations that are commonly required in spacecraft missions around asteroids. First, range measurements from the spacecraft to the surface are used to incrementally correct an initial shape estimate according to the Bayesian framework. Then, an optimal guidance scheme is proposed to control the vantage point of the range sensor to construct a complete 3D model of the asteroid shape. This shape model is then used in a nonlinear controller to track a desired trajectory about the asteroid. Finally, a multi resolution approach is presented to construct a higher fidelity shape representation in a specified location while avoiding the inherent burdens of a uniformly high resolution mesh. This approach enables for an accurate shape determination around a potential landing site. We demonstrate this approach using several radar shape models of asteroids and provide a full dynamical simulation about asteroid 4769 Castalia.

RPEOD: A Real-Time Pose Estimation and Object Detection System for Aerial Robot Target Tracking

Pose estimation and environmental perception are the fundamental capabilities of autonomous robots. In this paper, a novel real-time pose estimation and object detection (RPEOD) strategy for aerial robot target tracking is presented. The aerial robot is equipped with a binocular fisheye camera for pose estimation and a depth camera to capture the spatial position of the tracked target. The RPEOD system uses a sparse optical flow algorithm to track image corner features, and the local bundle adjustment is restricted in a sliding window. Ulteriorly, we proposed YZNet, a lightweight neural inference structure, and took it as the backbone in YOLOV5 (the state-of-the-art real-time object detector). The RPEOD system can dramatically reduce the computational complexity in reprojection error minimization and the neural network inference process; Thus, it can calculate real-time on the onboard computer carried by the aerial robot. The RPEOD system is evaluated using both simulated and real-world experiments, demonstrating clear advantages over state-of-the-art approaches, and is significantly more fast.