In industrial manufacturing sites, defect detection is crucial to improve reliability and lower inspection costs. Though prior anomaly detectors have shown success, they rely on large amounts of labeled data. Now, we aim to answer “how to cost-effectively utilize limited data for defect detection?”. Inspired by positive and unlabeled learning and few-shot learning, we propose a Positive Unlabeled learning based Few-shot Anomaly Detection model (PUFAD) that builds a representative memory bank of patch features using the large unlabeled set and few normal samples. Frequently co-occurring patch features in the unlabeled set, and cycle-consistent features between normal and unlabeled samples are regarded as pseudo normal features used for classifier training, including memory bank updates. Given test samples during inference, abnormal samples are detected by comparing the features in the memory bank via density estimation. Our method achieves state-of-the-art performance compared to existing few-shot anomaly detection methods on two benchmarks. This work addresses the real-world challenges of collecting large datasets for visual anomaly detection by proposing a practical solution that requires only a few normal samples and unlabeled data.