We show the chirality dependent dynamics of the single magnetic vortex in a submicron permalloy rectangle due to the symmetry breaking by a local magnetic field. For the clockwise (counter-clockwise) chirality, the local positive (negative) field leads to the softening of the gyrotropic mode of vortex core relative to the case of the homogeneous field, resulting in an asymmetric dependence of the gyrotropic frequency which depends on the vortex chirality. The gyrotropic frequency has strong correlation with the distance from the vortex core equilibrium position to the edge of the rectangle. We enlighten that the measurement of the gyrotropic frequency of single vortex under the local field can be an alternative way to determine the vortex chirality at room temperature.