A new fast public key cryptosystem is proposed, which is based on two dissimilar number-theoretic hard problems, namely the simultaneous chaotic maps (CM) problem and quadratic residue (QR) problem. The adversary has to solve the two hard problems simultaneously to recover the plaintext according to their knowledge about the public keys and the cipher-text. Cryptographic quadratic residue and chaotic system are employed to enhance the security of our cryptosystem scheme. The encryption, and decryption are discussed in details. Several security attacks are proposed to illustrate the system shield through chaotic maps and quadratic residue problems. The performance analysis of the proposed scheme show a much improved performance over existing techniques.