When embedding secret message into image by steganography with matrix encoding, there are still no effective methods to recover the stego key because it is difficult to statistically distinguish the stego coefficient sequences selected by true and false keys. Therefore, this paper proposes a method for recovering the stego key of a typical JPEG (Joint Photographic Experts Group) image steganography—F5 which composes of the check matrix and shuffling key. Firstly, the check matrix is recovered based on the embedding ratio estimated by quantitative steganalysis. The shuffling key is then recovered based on the distribution difference between the bit sequences extracted by the true and false shuffling keys. Additionally, the cardinality of the shuffling key space is significantly reduced by examining the extracted encoding parameter and message length. Experimental results show that the proposed method can recover the stego key accurately and efficiently, even when the existing Xu’s method fails for the high or very low embedding ratio.