• A method for individual identification of cows in unconstrained barn was developed. • The extracted and selected features have strong adaptability to pattern deformation. • A method involving fusion of Mask R-CNN and SVM was used to identify cows. • To reduce the running time and number of model parameters, Mask R-CNN was modified. The identification of individual dairy cows is an important prerequisite for dairy cow behaviour analysis and disease detection. Computer vision-based cow recognition is a noncontact and stress-free approach. In a free environment in a barn, due to changes in camera position and angle, recorded cow patterns are often deformed, making individual cow identification difficult. For cows in an unconstrained barn environment, this paper proposes a method for individual cow identification. First, a top-view image of a cow is obtained, and an improved Mask R-CNN is used to segment this image and extract the shape features of the cow’s back. Then, a Fisher approach is used to select the best feature subset, and a support vector machine (SVM) classifier is applied to identify individual cows. To verify the effectiveness of the target detection algorithm, the proposed method is compared with the traditional Mask R-CNN model, and the precision, recall, F1 score, average run time per image and average precision of the improved Mask R-CNN model are 98.21%, 96.48%, 97.34%, 1.02 s, and 97.39%, respectively. An SVM classifier trained based on the obtained shape features is used for individual cow identification. The proposed method achieves a 98.67% cow identification accuracy based on a dataset containing top-view images of 48 cows. The results demonstrate the effectiveness of the proposed cow identification method and its significant potential for use in precision dairy cow management.