High accuracy and fast face alignment algorithms play an important role in many face-related applications. Generally, the model speed is inversely related to the number of parameters. We construct our network based on densely connected encoder-decoders, which is an efficient method to balance the parameter number and localization results. In each encoder-decoder, we introduce stacking depthwise convolution and depthwise feature fusion within the same channel, which greatly improves the performance of depthwise convolution and reduces the number of model parameters. In addition, we enhance the mean square loss function by assigning different penalty weights to each coordinate according to the distance to the position corresponding to the maximum value in the label heatmap. Experiments show that the model with the improved loss function obtains better localization results. In the experiment, we compare our method to state-of-the-art methods based on 300W and WFLW. The localization error is 2.76% with the common subset of 300W and the model size (0.7M) is small and even utilizes approximately 1% of the number of parameters of the other models. The dataset and model based on WFLW are publicly available at https://github.com/iam-zhanghongliang/DC-EDN .