Internet of Vehicles (IoV) being a subdivided application of the Internet of Things, is considered as one of the most prominent and emerging technologies for model transportation systems. However, security and privacy remain two key requirements for IoV networks, as communications between vehicles and other Internet-connected things are generally carried out over public channels. Some of the most typical attack issues for the IoV networks include hardware tampering, unauthorized data access, message modification, tracking vehicle locations, etc. Although there have been a number of solutions (e.g., mutual authentication and key agreement protocols) proposed to ensure the secure communication for IoV, most of them still suffer from some vulnerabilities, such as linkability, server spoofing, and replay attacks, in violation of the security requirements of IoV. Hence, it remains challenging to design secure and efficient solutions. In this article, we first take a recently proposed authentication protocol as an example and analyze the weaknesses of it with simple mathematical analysis. We then propose an improved multiserver-based authentication and key agreement protocol for IoV (called SeMAV), which applies the password and smart card to hide the private keys. We also present both formal and informal security proofs to confirm the robustness against those commonly known attacks. The theoretical comparative summary and simulation results also show that SeMAV can achieve a good performance when compared with some other related protocols in the literature.