Face recognition in unconstraint surveillance is a complicated problem on account of motion blur, expression variations and low resolution. Recent works have demonstrated that patch-attention is strictly more powerful than convolution in recognition models. In this study, we investigate the task of unconstraint surveillance face recognition. First, a Patch-Attention Generative Adversarial Network (PA-GAN) model is devised to aggregate some robust features on behalf of a set of raw surveillance frames, which not only increases the recognition accuracy but also reduces the computational costs of face matching. Second, an improved center loss function combined with abundant unlabeled surveillance faces is utilized to accurately classify the known identities. With the proposed method, the discriminativeness of the face representations is largely enhanced. Finally, the proposed method is verified in two widely used datasets, IJB-A dataset and QMUL-SurvFace dataset to demonstrate the effectiveness. Evaluation of the algorithm performances in comparison with other state-of-the-art methods indicates that the proposed design can achieve competitive accuracy on both the verification and identification protocols.