The frequency-hopping communication system has been widely used in anti-jamming communication due to its anti-interception and anti-jamming performance. With the increasingly complex electromagnetic environment, the frequency-hopping communication system needs more flexible frequency-hopping patterns to deal with interferences, which brings great challenges to the communication receiver. In this paper, an intelligent receiving scheme of frequency-hopping sequences is proposed, which combines time–frequency analysis with deep learning to realize an intelligent estimation of frequency-hopping sequences. A hybrid network module is designed by combining a convolutional neural network (CNN) with a gated recurrent unit (GRU). In the proposed network module, the combination of a residual network (ResNet) and squeeze and extraction (SE) improves the feature extraction and expression capabilities of the CNN network. The GRU network is proposed to solve the problem of dealing with signals with variant input lengths. A transfer learning scheme is further proposed to deal with communications systems with different frequency-hopping sets. Simulation results show that the proposed method has strong generalization ability and robustness, and the bit error rate (BER) performance of intelligent receiving is close to the receiving performance under ideal conditions.