Wireless networks with directional antennas, like millimeter wave (mmWave) networks, have enhanced security. For a large-scale mmWave ad hoc network in which eavesdroppers are randomly located, however, eavesdroppers can still intercept the confidential messages, since they may reside in the signal beam. This paper explores the potential of physical layer security in mmWave ad hoc networks. Specifically, we characterize the impact of mmWave channel characteristics, random blockages, and antenna gains on the secrecy performance. For the special case of uniform linear array (ULA), a tractable approach is proposed to evaluate the average achievable secrecy rate. We also characterize the impact of artificial noise in such networks. Our results reveal that in the low transmit power regime, the use of low mmWave frequency achieves better secrecy performance, and when increasing transmit power, a transition from low mmWave frequency to high mmWave frequency is demanded for obtaining a higher secrecy rate. More antennas at the transmitting nodes are needed to decrease the antenna gain obtained by the eavesdroppers when using ULA. Eavesdroppers can intercept more information by using a wide beam pattern. Furthermore, the use of artificial noise may be ineffective for enhancing the secrecy rate.