In this paper, we introduce a new family of lattices, namely QC-MDPC lattices, which are a special case of LDPC lattices, and an improved bit flipping algorithm for decoding of these lattices. Encoding and decoding implementations of QC-MDPC lattices are practical in high dimensions. Indeed, to take advantage of practical decoding, we use “Construction-A” lattices which makes a tight connection between the structure of lattices and codes. Using these features, we design a lattice-based public key encryption scheme enjoying linear encryption and decryption complexities. The proposed scheme has a reasonable key size due to the sparseness of the parity-check matrix, and the quasi-cyclic structure of the parity-check and generator matrices. Besides, the message expansion of the proposed scheme is smaller than other lattice-based and code-based cryptosystems with comparative parameters. All these features provide a lattice-based public key encryption scheme with reasonable key size, linear encryption, and decryption algorithms and small message expansion. On the other hand, we show that the cryptosystem is resistant against all known attacks both on lattice-based and code-based cryptosystems for different levels of security.