The unique rigid-flex connection between the fin-rays and fin-surface in a bionic undulatory fin robot endows the fin-surface with both active flexibility and load-bearing capacity, enabling this robot to perform amphibious motions in underwater, terrestrial, and even marshy environments. However, investigations into dynamic modeling problems for the undulatory fin robot, considering the impact of nonlinear deformation and frictional contact on ground locomotion performance, are scarce. Given this, based on the absolute nodal coordinate formulation (ANCF), this paper presents an efficient and accurate nonlinear dynamic model for this robot to elucidate the fin's flexible deformation and motion law. This model considers material, geometric, and boundary nonlinearities, utilizing ANCF thin plate elements and reference nodes to individually describe the fin-surface and fin-rays of the undulatory fin. Then, by using the master-slave technique, a frictional contact formulation for the fin and the ground is proposed. Furthermore, we conduct in-depth research and analysis on the formation and undulatory motion of the undulatory fin, encompassing its static deformation, static contact deformation, and frictional contact motion, and successfully obtain its responses under various conditions. Research indicates that during fin-surface motion, longitudinal sliding or a tendency for sliding at the contact points results in the undulatory fin moving in a crawling gait. The proposed theoretical model correctly captures the fin's complex nonlinear deformations and frictional characteristics and reveals its ground locomotion mechanism, whose effectiveness and superiority are validated through numerical examples and experiments.