Most methods only use normal samples to learn anomaly detection (AD) models in an unsupervised manner. However, these samples may be noisy in real-world applications, causing the models to be unable to accurately identify anomaly objects. In addition, there are a small number of anomaly samples in real industrial production that should be fully utilized to help model discrimination. Existing methods of introducing anomaly samples still have bottlenecks in model identification capabilities. In this paper, by introducing both normal and a few abnormal samples, we propose a novel semi-supervised learning method for anomaly detection, named RobustPatch, which can improve the model discriminability through a self-cross scoring mechanism and the learning of feature AutoEncoder. Our approach contains two core designs: Firstly, we propose a self-cross scoring module, calculating the weights of normal and anomaly features extracted from corresponding images using a self-scoring and cross-scoring manner, respectively. Secondly, our approach proposes a fully connected feature AutoEncoder to rate the extracted features, which is trained with the supervision of the scored weights. Extensive experiments on the MVTecAD and BTAD datasets validate the superior anomaly boundaries discriminability of our approach and superior performance in noise-polluted scenarios.