The unclear influence of recycled glass fiber reinforced plastic (GFRP) on concrete induced by high source diversity and complex material nature prohibited its wide application as fine aggregates. Three types of recycled GFRP powders were applied to cement-based materials, and the effects of these powders on the workability, physical-mechanical properties and microstructure of cement mortar were systematically investigated for the first time. Four types of modification methods were applied to alleviate the detrimental effects of G-Dust on mortar. It was revealed that the crushed GFRP powder and dust had different influences on mortar. The workability and mechanical strength were improved by GP1 and GP2, and the microstructural compactness was consistently increased with the reduction of large pores and formation of honeycomb-featured C–S–H gels. On the other hand, volumetric expansion was induced to mortar by the reaction of G-Dust in cement paste, which significantly decreased the strength and increased the pore volume of mortar. Fortunately, the addition of 10 wt% silica fume can effectively solve the expansion problem, which promoted the hydration degree, reduced the formation of fibrous C–S–H gels, AFm and Ca(OH)2, and dramatically decreased the large pore proportions of mortars. As a result, the compressive strength and flexural strength were increased by 47.8% and 15.7%, respectively.