Autonomous Vehicles (AVs) are a highly discussed topic owing to their great performance and convenience. However, some requirements limit their wider deployment. Specifically, AV should be controlled by remote users during emergencies. It may lead to AV’s system facing the risk of being intruded on by a malicious party, resulting in unreasonable decisions. We, therefore, design the Internet of autonomous vehicle (IoAV) model to mitigate the problems arising from these limitations. To promote a secure remote control of the AV, a reliable authentication scheme, which can be used in IoAV, must be performed. Our proposed chaotic map-based authenticated key agreement (CMAKA) method provides secure remote control features for AVs. In this method, users, data centers, and AV establish a secure communication channel through the negotiation of three independent session keys. Furthermore, a physical unclonable function (PUF) is employed to produce a trusted private key during the authentication. The security of our scheme is evaluated using game hopping through the widespread Real-or-Random (ROR) model. Compared with other existing three-factor authentication schemes, the performance of our protocol is higher in both security requirements and total cost.