To fully understand the polymer-filler interfacial interaction mechanism, we use a coarse-grained molecular dynamics simulation to mainly investigate the interfacial dynamic properties by tuning the polymer-filler interaction, temperature, chain length, volume fraction of filler, and size and shape of filler. The polymer beads around the filler exhibit an obvious layering behavior and a gradient of polymer dynamics is observed for systems filled with three kinds of fillers (spherical, rod-like and sheet-like). By analyzing the dynamics of the interfacial beads in the first adsorbed layer, we find that the mobility of interfacial beads becomes greater at lower polymer-filler interaction strength and higher temperature. The adsorption/desorption dynamics of interfacial beads decreases with the increase of the chain length, and then becomes nearly unchanged when the chain length exceeds the entanglement molecular weight. It is found that the influence of the different size of nanospheres on the mobility of interfacial beads is just induced by the curvature. However, for systems filled with nanorods and nanosheets, besides the curvature, the force exerted on the polymer beads also plays an important role. For systems filled with three kinds of fillers, the mobility of interfacial beads is the slowest for the nanosheet filled system and only in this case the "glassy layer" exists for strongly attractive interfacial interaction. By comparing the dynamics of the adsorbed polymer beads for three different shapes of filler, it is concluded that it is the total force exerted on the polymer beads by the filler that determines the mobility of the interfacial beads. In short, this work could provide valuable information on the fundamental understanding of polymer-filler interfacial behavior, especially for systems filled with fillers of different shapes.