It is still a challenging task to objectively evaluate the fabric smoothness appearance due to the complexity and variety of fabrics. Innovative methods that can accurately quantify the surface smoothness in a practical and repeatable way are desideratum. To address this challenge, we put forward a new mechanical testing method, that is, mechanical test system for fabric shape retention (MTS-FSR). The design details and testing principle of the MTS-FSR were presented. The measured force-displacement curves were fitted and eight characteristic indices related to fabric wrinkling were determined and interpreted by their physical meanings. The repeatability and accuracy of the system were studied to verify the reliability of the method. Moreover, a prediction model for fabric smoothness appearance was established based on decision tree algorithm that was proved superior to other common used learning algorithms (e.g. support vector machine (SVM), random forest (RF) and Back-Propagation neural network (BPNN)) based on comparison analysis. The experimental results show that the designed instrument provides a feasible and effective measurement method for the objective evaluation of fabric smoothness appearance with good accuracy and resolution.