The force feedback technology has a crucial impact on the precise control of teleoperation system. Robots in fields such as minimally invasive surgery and nuclear waste cannot integrate force sensors to obtain force feedback due to their end size and harsh working environment. In order to obtain end-effector force feedback in sensorless situation, a sensorless force estimation model based on artificial neural network is defined in this paper. For improving the model prediction ability, a new Multi-layer Depth Extreme Learning Machine (MDELM) is proposed. Firstly, the network structure of the extreme learning machine is redesigned, and the smoothing function and Gauss-Laplacian function are constructed as the feature extraction layer and enhancement layer of the MDELM model. Then, to further improve the prediction performance of MDELM, an improved Sooty Tern Optimization Algorithm (STOA) is introduced to optimize the parameters of the model. The results show that the proposed force estimator outperforms the existing model with an MAE of less than 0.205, which is at least 18.65% lower than the existing model. In addition, when the acceleration of the remote operating system is a fixed constant or the information is not available,a sensorless force feedback system can be constructed by combining motor torque, joint position and joint velocity parameter. In general, this study provides an effective force estimation solution for teleoperation system without force sensor.