Recent considerable state-of-the-art advancements within the automotive sector, coupled with an evolution of the promising paradigms of vehicle-to-everything communication and the Internet of Vehicles (IoV), have facilitated vehicles to generate and, accordingly, disseminate an enormous amount of safety-critical and non-safety infotainment data in a bid to guarantee a highly safe, convenient, and congestion-aware road transport. These dynamic networks require intelligent security measures to ensure that the malicious messages, along with the vehicles that disseminate them, are identified and subsequently eliminated in a timely manner so that they are not in a position to harm other vehicles. Failing to do so could jeopardize the entire network, leading to fatalities and injuries amongst road users. Several researchers, over the years, have envisaged conventional cryptographic-based solutions employing certificates and the public key infrastructure for enhancing the security of vehicular networks. Nevertheless, cryptographic-based solutions are not optimum for an IoV network primarily, since the cryptographic schemes could be susceptible to compromised trust authorities and insider attacks that are highly deceptive in nature and cannot be noticed immediately and are, therefore, capable of causing catastrophic damage. Accordingly, in this article, a distributed trust management system has been proposed that ascertains the trust of all the reputation segments within an IoV network. The envisaged system takes into consideration the salient characteristics of familiarity, i.e., assessed via a subjective logic approach, similarity, and timeliness to ascertain the weights of all the reputation segments. Furthermore, an intelligent trust threshold mechanism has been developed for the identification and eviction of the misbehaving vehicles. The experimental results suggest the advantages of our proposed IoV-based trust management system in terms of optimizing the misbehavior detection and its resilience to various sorts of attacks.