The performances of cognitive radio system have been studied with primary user emulators existing. In this study, the authors use the theory of effective capacity to study the performances under quality-of-service (QoS) constraints in the link layer. New decision rules are proposed to decide the channel states. For a cognitive radio channel, the secondary users send data at two different rates regarding the channel states of being idle or occupied by primary users. When the channel is sensed as being occupied by a primary user emulator, which is performed by a secondary user, the secondary user stops the data transmission. To model the transmission channel, an 18-state model of transition is constructed. The effective capacity expression of such cognitive system is also derived and analysed with different transmission rates. Besides, the impact on the effective capacity from the system parameters is investigated. It is shown from the numerical results that under the proposed decision rules, sensing durations and thresholds affect the detection probabilities greatly. For the effective capacity, it is shown that QoS imposes the most effect on the effective capacity, sensing durations and transmission rates can impact the effective capacity in certain regions.