Information on sow nursing behavior plays an important role in improving the survival rate of weaned piglets. To obtain more information on sow nursing processing, it is necessary to introduce beginning- and end-stage nursing behavior recognition. This is defined as fine-grained nursing behavior recognition. In this paper, we attempt to automate fine-grained nursing behavior recognition with surveillance video. First, nursing behavior is divided into three subbehaviors: before piglet sucking, piglet sucking and end piglet sucking. Then, a deep neural network model with a dual-stream structure, named SlowFast, is introduced to realize preliminary sow nursing behavior recognition by extracting spatiotemporal features. Finally, for some erroneous results in behavior recognition and the recognition ambiguity in behavior transition, a hidden Markov model (HMM) is introduced to modify the behavioral label sequence by using the Viterbi algorithm. To verify the effectiveness of our method, all experiments are carried out on sow nursing videos collected in an actual breeding environment. The results show that our method achieves a 90.50% sequence coincidence degree for the recognition results on long videos containing the whole nursing behavior, and the accuracy of behavior transition times reaches 87.05%. In summary, this study refines the automatic sow nursing behavior monitoring content to provide more information for precision livestock farming.