Clothing can absorb fine particulate pollutants from the atmosphere outdoors and then release them after entering the room in a form similar to three-handed smoke. In this study, the experimental site in Zhengzhou, a typical city with heavy pollution in central China, was used to investigate the adsorption and diffusion patterns of three different fabrics of polyester, polyester-cotton, and cotton clothing for fine particle pollutants, to guide people’s behavior patterns and indoor air distribution, and to reduce indoor pollution exposure. The results showed that when garments of three different fabrics were exposed to moderate and heavy PM2.5 pollution for 1 h, 2 h, and 3 h, respectively, and then transferred to the diffusion chamber, polyester always released the highest PM2.5 concentration to indoor air during the same release cycle, followed by cotton and polyester-cotton. In addition, the concentration of indoor air pollutants will be periodically affected by the diffusion of fine particulate adsorbed by clothing fabrics. With the increase in outdoor pollution and exposure duration, the indoor PM2.5 concentration takes longer to stabilize at certain levels after the clothing is transferred to the indoor side. Finally, the comparison of natural ventilation experiments proved that it is not feasible to rely solely on natural ventilation to improve the influence of clothing pollution sources on indoor air quality.