In recent years, the convolutional neural network (CNN) has been widely used in the field of intelligent fault diagnosis. However, each convolutional layer of CNN cannot take the overall and local information into account, and the feature extraction ability of CNN with fewer layers is weak. These circumstances lead to poor performance of CNN in practical fault diagnosis with variable operating conditions. To solve these problems, this paper proposes a multiscale holospectrum CNN (MH-CNN) based on the methods of two-dimensional multiscale feature fusion and decision-level feature fusion. First, the continuous wavelet transform is used to map the time-domain signal to the time-frequency plane to fully reflect the complex information contained in the signal. Then the two-dimensional multiscale feature fusion is introduced to extract features at different scales, which can take both overall and local information into account. Finally the decision-level feature fusion is introduced to fuse the features from signal in X, Y directions in the decision-level of CNN, which serves to enhance the features. By combining these methods, the proposed MH-CNN can extract more distinguishable features with a shallow structure, which can ensure the classification capability while avoiding the overfitting problem caused by overly complex networks. The effectiveness of the MH-CNN is verified using complicated data sets consisting of 16 rolling bearings with four different health conditions, two speeds and three loads. Results show that the proposed MH-CNN achieves a correct rate of 99.8% for rolling bearing fault diagnosis under variable operating conditions, which is much higher than other comparative methods.