While master-slave teleoperated robotic systems have extensive applications in practice, the physiological tremors can easily affect the control accuracy and even destroy the stability of the closed-loop control systems during operation. Hence, the development of some effective approaches for counteracting physiological tremors is of both theoretical and practical importance. In this paper, a broad learning network-based filter integrating a deep learning network and modified incremental learning algorithms is proposed to reconstruct and compensate for tremor signals. To strengthen the recognition of correlations between different moments, the lateral connectivity structure is adopted to obtain multi-scale feature maps. Each feature window is obtained from multi-scale feature maps generated by the convolutional neural network, which has an advantage that makes the feature nodes fuse the feature information of long time series and short time series by the lateral connection. The broad learning network is a unique construction, which only needs to obtain the input and the output to conveniently calculate the connection weights by the pseudo-inverse without involving backpropagation. It is known that the relation between the data X and the label Y can be represented as XW=Y, and the solution W can be obtained by the pseudo-inverse W=X+Y. In addition, to guarantee the ill-posed problem, a ridge regression algorithm is used for the pseudo-inverse calculation. The effectiveness of our raised network architecture is illustrated by comparative simulation and experiment results.