Based on the goal of improving the rock breaking ability when drilling asteroid rock samples under limited energy conditions, this paper proposes a method of drilling rock using self-vibratory drilling head and develops a small self-vibratory drilling head for drilling rock. In order to analyze the dynamic characteristics of drilling rock with self-vibratory drilling head, a rock model hypothesis that the rock consists of a series of micro-segment uniform continuous media is put forward. Base on the hypothesis, the model of drilling rock is constructed, the stability lobes diagram of self-vibratory drilling head drilling basalt is plotted, and the energy mechanism of drilling rock system with self-vibratory drilling head is discussed. The comparison tests of drilling basalt by three kinds of self-vibratory drilling head with different spring stiffness and the conventional method are carried out. The rotational speed is a variable in the comparative tests. The test results show that under the specific rotational speed and spring stiffness, self-excited vibration is produced in self-vibratory drilling head drilling basalt. When self-excited vibration drilling is conducted, although the amplitude fluctuation range of the drilling thrust force is wider than that of conventional drilling, the average drilling thrust force is smaller than the conventional drilling. The amplitude of drilling thrust force increases with an increase in spring stiffness. The method of drilling rock by self-vibratory drilling head is proved to be a potential method for viable drilling of asteroid rock samples.