Compression molding is a lightweight technology that allows to preserve fiber length and retain better mechanical properties compared to injection molding. In compression molding development, a suitable material such as glass fiber mat thermoplastics (GMT) is often used. However, because of the complicated micro-structure of the fibers and the fiber–resin matrix interactions, it is still quite challenging to understand the mechanism of compression molding and it is very difficult to obtain a uniformly compressed GMT product. In this study, we propose a method to measure the rheological properties of GMT through a compression system. Specifically, we utilized a compression molding system to record the relation between the loading force and the displacement. This quantitative information was used to estimate the power-law index and viscoelastic parameters and predict viscosity. Moreover, the estimated viscoelastic parameters of GMT were implemented into Moldex3D to evaluate the flow behavior under compression. The results showed that the trend of the loading force variation was consistent in numerical simulation and experiments. However, at the final stage of compression molding, the experimental loading force was much higher than that estimated by simulation. To find out the mechanism causing this deviation, a series of studies were performed. Through TGA measurement, we found that the fiber content of the center portion of the compressed part increased from 63% to 85% during compression. This was expected, as a result of the fiber–polymer matrix separation effect. This fiber–polymer matrix separation effect influenced the power-law index and rheological parameters of GMT, making them fluctuate. Specifically, the power-law index changed from 1.0 to 0.62. These internal changes of the rheological properties further induced a much higher loading force in the real experimental GMT system. We further verified the rheological properties variation using pure polyamide (PA) and found that since there is no fiber–polymer matrix interactions the power-law index and curve-fitting rheological parameters were almost constant. The mechanism causing the deviation was therefore validated.