In cloud environment, the communication between virtual machines (VMs) residing on the same physical system is inevitable, which brings convenience for malicious software to attack other VMs through the communication channels between VMs. In order to reduce the security risk of the VMs in cloud environment, a novel hybrid security model (HSMVM) is presented, which combines the features of confidentiality, integrality, and organisation isolation of generic security model. The limitations and the advantages of the existing security models are thoroughly analysed in this paper. Afterwards, the basic definitions of HSMVM, judgment method for the system security state, and the state transformation rules are described in detail. The prototype system of HSMVM is developed and tested on Xen system. Experimental results indicate that the security model can effectively prevent the attacks between the VMs with reasonably low resource utilisation.