Piezoelectric actuators are widely used for precise micro-positioning. In most applications, there is contact with the environment. Sensing external forces is a challenging problem in controlling mechanical systems, especially in micro scales. The external force could cause dual hysteresis non-linearity effects, which will disrupt the actuator positioning accuracy. Therefore, an effective controller should be designed to guarantee the trajectory tracking in the presence of external forces and perturbations. In this paper, a modified Prandtl–Ishlinskii operator and its inverse is utilized for both identification and real time compensation of the hysteresis effect. A variable structure controller is proposed for trajectory tracking of the actuator position. A sliding-based observer would estimate full states from the only measurable position trajectory. An adaptive perturbation estimation approach is proposed so that external forces can be separated from disturbances. The stability of the controller in the presence of estimated states and external forces is demonstrated analytically. The experimental results depict that the proposed approach achieves precise trajectory tracking and good external force estimation.