The vision of Internet of Vehicles (IoV) is to make the dream of zero traffic accident come true. However, the problems of drivers’ poor fitness and distraction go from bad to worse on account of the increase of chronical patients and the prosperity of mobile phones. Most of the security-related literatures aimed to offer a better efficiency, protection, or scalability on safety-related message verification, but few security protocols were suitable for the transmission of the drivers’ abnormal reports because of low latency requirement. In this paper, we propose a secure notification scheme [driver-abnormal notification scheme (DANS)] based on edge-fog computing to offer real-time response. After passing two-factor authentication, vehicles serve as the edge nodes to download and to real-time compare the sensing parameter for reducing the long-distance communication latency. By means of hash-chain-based public key cryptosystem, DANS can omit the certificate overhead to keep short latency, and also holds the benefits of driver authentication, mutual authentication, integrity for abnormal reports, conditional privacy preservation, non-repudiation, fast verification, and good scalability. Performance evaluations and simulation demonstrate that DANS earns outstanding results in terms of verification delay and communication overhead. To the best of our knowledge, this paper is the first attempt to design a notification scheme to real-time detect the driver-abnormal issues over IoV.