AbstractWhile IPSec standard is largely used to protect real time network applications, it unfortunately consumes more processing time, cause packet delay and impede QoS enforcement. The QoS level that a flow receives depends on the value of the Type of Service (ToS) field; the later is set by the ‘Multi‐Field’ (MF) packet classifiers according to the IP source and destination addresses and ports as well as the transport layer protocol. The last three fields are encrypted by the IPSec ESP, and thus ESP prevents network control devices from providing preferred treatment for time critical applications. To solve this problem, we propose a QoS‐friendly Encapsulated Security Payload (Q‐ESP) as a new IPSec security protocol that provides QoS supports while enforcing the same security services assured by IPSec ESP and AH used jointly. Basically, Q‐ESP allows network elements to inspect all the needed fields to perform classification adequately. In this paper, we present details about Q‐ESP design, processing and kernel implementation. Moreover, we give analytical as well as experimental evaluation of our protocol to measure its impact on real time VoIP; we also compare it to IPSec ESP and AH according to QoS and security metrics. Finally, we present and discuss some application scenarios in which the use of the Q‐ESP protocol has many advantages. Copyright © 2010 John Wiley & Sons, Ltd.