Network traffic classification is the basis of many network security applications and has received significant attention in the field of cyberspace security. Existing research on deep traffic analysis typically involves converting traffic data into images to extract spatial traffic features using Convolutional Neural Networks (CNNs). However, this approach ignores the semantic differences and details in the various packet structures. In this paper, we propose an MLP-Mixer based multi-view multi-label neural network for network traffic classification. Compared with the existing CNN-based methods, our method adopts the MLP-Mixer structure, which is more in line with the structure of the packet than the conventional convolution operation. In our method, one packet is divided into the packet header and the packet payload, together with the flow statistics of the packet as input from different views. We utilize a multi-label setting to learn different scenarios simultaneously to improve the classification performance by exploiting the correlations between different scenarios. We conduct experiments on three public datasets, and the experimental results show that our method can achieve superior performance. Code is available at https://github.com/ZxuanDang/MV-ML-traffic-classification.