X-ray imagery security screening is an essential component of transportation and logistics. In recent years, some researchers have used computer vision algorithms to replace inefficient and tedious manual baggage inspection. However, X-ray images are complicated, and objects overlap with one another in a semi-transparent state, which underperforms the existing object detection frameworks. To solve the severe overlapping problem of X-ray images, we propose a foreground and background separation (FBS) X-ray prohibited item detection framework, which separates prohibited items from other items to exclude irrelevant information. First, we design a target foreground and use recursive training to adaptively approximate the real foreground. Thereafter, with the constraints of X-ray imaging characteristics, a decoder is employed to separate the prohibited items from other irrelevant items to obtain the foreground and background (FB). Finally, we use the attention module to make the detection framework focus more on the foreground. Our method is evaluated on a synthetic dataset with FB ground truth and two public datasets with only bounding box annotations. Extensive experimental results demonstrate that our method significantly outperforms state-of-the-art solutions. Furthermore, experiments are performed in the case where only a small number of images contain the FB ground truth. The results indicate that our method requires only a small number of FB ground truths to obtain a performance equivalent to that of all FB ground truths.