Online assessment of computer vision and robotics skills based on a digital twin

Abstract

This paper presents online simulation along with physical environments employing digital twins to develop robotic and computer vision applications, such as bin picking, by providing a virtual testing environment with real conditions for robotic systems and their surroundings. Particularly for situations such as the current COVID-19 pandemic, this approach permits the integration of knowledge and practice remotely without having the need of requiring a physical robot, increasing equity and inclusivity in the assessment, by reducing the gap between students from different institutions due to the lack of resources to buy the required robotic devices and even enabling remote connection to the classrooms. The proposed task can be performed as long as the educational center counts with basic computer equipment and internet connection. This paper presents a bin picker application based on mono and stereo vision to make the classification of cylinders of different colors and sizes and their arrangement in a base by using image processing and depth estimation algorithms. The performance of the classification was measured in terms of the error in millimeters from the difference of the desired and the obtained position. The images used for classification were taken in real life considering the environment conditions, and a digital twin of the bin picker and objects was made. The image classification and the overall instructions that the robot performed were generated on MATLAB, where the data from the real environment captured by cameras was sent. From here, the instructions were sent to URsim, which serves as the robot controller, and the final virtual simulation with the models of the base and cylinders was performed on Kuka-Sim Pro. This approach can then be utilized to implement different assessments based on personal results from simulation activities. As this assessment requires specialized knowledge regarding robotics, computer vision and classification algorithms, this methodology is intended to be applied for university students, at the level that is adequate according to the educational programs of each university.