Routing protocols play an important role in the communication and information distribution within an Internet of Things (IoT) system. RPL is one such popular routing protocol for IoT devices and systems. However, security in RPL is an afterthought, and it does not meet the demands of today’s complex cyberthreat landscape. Focusing on sybil attack detection in RPL-based IoT, we first propose a lightweight Bloom filter and physical unclonable function (PUF) based sybil attack detection mechanism (hereafter referred to as liteSAD). Our approach is designed to minimize memory cost as well as detection latency, without affecting the detection accuracy. Specifically, in liteSAD, Destination-Oriented Directed Acyclic Graph (DODAG) root generates a Bloom filter array through hashing each legitimate node’s identifier and PUF response, and distributes it through a new packet named BF-DAO. Upon receiving the BF-DAO packet, each legitimate node retrieves the Bloom filter array, updates its local copy, and employs it to detect sybil attack. We also propose a probabilistic DIO reply mechanism (i.e., proDIO) to reduce the number of broadcasted DIO packets in response to attack DIS packets. We investigate the setting of Bloom filter parameters that minimize the probability of false positive and time complexity while meeting the requirement of memory constraints in IoT devices. We also evaluate the performance of our mechanism liteSAD+proDIO through extensive simulation experiments, where the results demonstrate that liteSAD+proDIO can provide better performance in terms of detection rate, detection latency, miss detection rate, DIO Trickle timer, number of broadcasted DIO packets, and energy consumption. In summary, our major contributions are twofold: (i) the comprehensive analysis of RPL routing protocol, Trickle algorithm, and the impact of sybil attack; and (ii) the proposal of lightweight Bloom filter and PUF based sybil attack detection mechanism.