With the rapid development of wireless technologies, many researchers use WiFi signals for human action recognition. Most of the previous methods are based on traditional machine learning algorithms, requiring researchers to manually extract time-domain or frequency-domain features from the obtained WiFi Channel State Information (CSI), but they can not effectively represent the CSI continuous features of human actions. In this paper, we propose an attentional temporal convolutional network (ATCN) for action recognition using CSI. By combining causal convolution and dilated convolution, the proposed ATCN guarantees no CSI features leakage from the future to the past and effectively expands the receptive field to maintain a longer memory size. Meanwhile, the self-attention mechanism is leveraged to assign more weights for the CSI data that is representative of human actions. To better verify and evaluate our proposed model, we build the Njtech-CSI-Action data set and make comparative experiments. The experimental results demonstrate that our Wi-ATCN outperforms previous methods and achieves the state-of-the-art performance.