At microscale, thin films with passivation layer exhibit strong size effect during loading and prominent Bauschinger effect during reverse loading. It is commonly recognized that the hardening, which is caused by geometrically necessary dislocations (GNDs) near the passivation layer, is responsible for these phenomena. However, GNDs may contribute to hardening either by slip resistance via a generalized Taylor equation or by back stress originating from elastic interaction between dislocations. Which one dominates the thin film plasticity has not been well understood yet. In this paper, a dislocation-based crystal plasticity model is proposed, which includes both slip resistance and back stress hardenings. The model is applied to simulate passivated thin films with different thickness and the results are compared with experimental data. In the case where slip resistance hardening is considered, the predicted size effect of flow stress is much smaller than that in experiments and there is no obvious Bauschinger effect. While in the case where back stress hardening is considered, the predicted size effect matches well with experiments and the Bauschinger effect is clearly revealed. Also, it is found that the reverse plastic flow during unloading comes from the heterogeneous distribution of stress caused by the pile up of dislocations near the passivation layer.