Salinity is one of the most critical challenges for crop production and soil and water management in arid and semi-arid regions, such as a large area of Morocco. These regions are characterized by low rainfall with an erratic distribution, long drought periods, and high evaporation, resulting in salt accumulation in the superficial layer of the soil and soil and water degradation. Therefore, phosphogypsum (PG) could be a promising amendment to reduce the salinity effect and improve soil quality in salt-affected soils. In this context, the present study aimed to evaluate the effect of PG on the physical properties of Luvisols and Cambisols collected from salt-affected soils in four regions in Morocco: Chichaoua, Ras El Ain, Sidi Zouine, and Sed El Masjoune. The treatments consisted of different rates of PG (15, 30, and 45 t/ha), natural Gypsum (G) (15 t/ha), and control. Our findings revealed that PG application improved soil structure by promoting flocculant action provided by calcium. Linear regression indicated that Water Aggregate Stability (WAS) and PG doses were strongly correlated with a high coefficient of determination (R2 = 93.41%, p value < 0.05). Compared to the control, the overall efficiency of 45 t/ha of PG amendment reached 53%, 95%, and 36%, respectively, in Chichaoua, Ras El Ain, and Sed El Masjoune soils. PG application presented a positive effect on other soil physical properties (soil hydraulic properties, total porosity, and bulk density), especially for the soils of Chichaoua and Ras El Ain regions. The total porosity was increased by 8% with 45 t PG/ha in Ras El Ain soil, and in Chichaoua soil, the bulk density was 5% lower in the pot treated with 45 t PG/ha compared to the control. This study supports the use of PG as an amendment for reclaiming salt-affected soils through monitoring agronomic and environmental impacts.