Infrared small target segmentation plays an important role in infrared guidance systems. In this paper, a fine-grained guided model fusion network with attention mechanism (FAMNet) is proposed for improving the performance of the infrared small target segmentation. An autonomous traditional feature extraction algorithm based on information entropy and four-directional gradient contrast is proposed for solving the difficulty of feature extraction from small targets. The deep features are extracted from an improved U-Net. The two kinds of features are fused by channel shuffle, which could enhance the communication efficiency between the channels of the two features. More context information from the target and background is introduced into the network by cross layer parallelization convolutional block attention module (CLPCBAM). Compared with some state-of-the-art techniques, the proposed FAMNet performs significantly better in terms of IoU, nIoU, and F1-score comprehensively.