A theoretical study on the vortex image is carried out near a vortex core in the presence of the anisotropic superconducting gap or the anisotropic Fermi surface with fourfold symmetry by solving Bogoliubov-de Gennes (BdG) equations. Both results show the anisotropic vortex core with fourfold symmetry, but detailed structures are rather different. At zero energy, the local density of states (LDOS) is always larger along curvature minima than along curvature maxima of the Fermi surface at the same radial distances from the vortex center in the presence of the anisotropic Fermi surface. In contrast, in the presence of the anisotropic superconducting gap, the LDOS is larger along gap maxima than along gap minima at short distances, but with an inversion at larger distances. With increasing energy, the evolutions of vortex images in these two situations also behave quite differently. Our results provide an effective method to distinguish these two contributions from the anisotropic superconducting gap and Fermi surface to the vortex image.