Herein, marble tailings (MTs) were used as mineral filler to prepare polyvinyl chloride (PVC) based composite plates. In order to improve the toughness of MTs/PVC composites, chlorinated polyethylene (CPE), methyl methacrylate butadiene styrene (MBS), ethylene–vinyl acetate copolymer (EVA), and polyolefin elastomer (POE) were adopted to investigate the toughening effects on the performance of MTs/PVC composites. The results show that incorporation of tougheners in MTs/PVC composite enhances restraint on the movement of PVC molecular chain segments, leading to the decrease of energy storage modulus and loss factor. Although these four tougheners decrease tensile strength and flexural properties of MTs/PVC composite plates, they significantly increase notched impact strength. In particular, the impact strength of MTs/PVC composite is improved by nearly 179% via using CPE toughener. Taken together, CPE not only presents the relatively most excellent rheological properties for MTs/PVC composites, but also increases toughness to the maximum extent without significantly reducing the tensile strength and flexural properties of MTs/PVC composites. Hence, a pilot testing was carried out to produce CPE-MTs/PVC composite hollow flooring. This work points out an effective resource utilization direction for the marble tailings, and it provides a reference for industrial application of MTs/PVC composite in construction and building engineering.