In Internet of Vehicles (IoV), the secure data transmission between vehicles and transportation infrastructure effectively ensures the safety and fast driving of vehicles, while authentication and key agreement protocols between vehicles and transportation infrastructure (V2I) play an important role in ensuring data security and user privacy. However, how to design a provably secure and lightweight V2I authentication protocol for IoV is a challenge. Recently, Kumar et al. proposed a conditional privacy-preserving and desynchronization-resistant authentication protocol for IoV, but we find that Kumar et al.’s scheme is vulnerable to identity guessing attacks, impersonation attacks, and a lack of session key secrecy once the attacker obtains data stored in smart card through side-channel attacks. We also point out that Kumar et al.’s protocol is vulnerable to Roadside Unit (RSU) captured attacks and lacks perfect forward secrecy. Therefore, we propose an improved V2I authentication protocol for IoV, which uses the Physical Unclonable Function (PUF) to resist RSU captured attacks, and designed a three-factor secrecy strategy to resist side-channel attacks; a conditional privacy-preserving strategy was also adopted to achieve anonymity and malicious user tracking. Furthermore, the proposed protocol is provably secure under the random oracle model and has low computation and communication costs.