During the manufacturing processes of the thin-film transistor liquid-crystal display (TFT-LCD) panel, red, green, and blue (RGB) color filter coatings undergo the layer-adding process which causes the rough surfaces between the layers. The rough surfaces make acquiring an accurate measurement of the optical properties and thickness (n, k, d) much more difficult because the scattering effects occur. The effective layer-included model is considered in determining the (n, k, d) by including effective layers to reside between and above the multilayer (ML). To show the feasibility of the effective layer-included model, we examined the model by fitting the (n, k, d) for different virtual systems which contain different kinds of scatters reside between and above it. Our findings show that the fitted (n, k, d) can be closer to the assumed (n, k, d) by using the effective layer-included model rather than the standard model. Also, the tolerance of initial assigned (n, k, d) regions to obtain the accurate results are investigated. Further, both models are used to determine the (n, k, d) of the fabricated RGB color filter samples. In the experimental measurements, all reflection and transmission signals are measured by utilizing the in-house variable angle spectroscopic ellipsometry (VASE) system. Consequently, the thicknesses determined from effective layer-included model are closer to the thicknesses measured from profilometry (Alpha-step 100). Also, the transmissions under 0 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> , 15 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> , and 30 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> illuminations calculated from the fitted (n, k, d) through the effective layer-included model are closer to the VASE measurements rather than the standard model for each sample. We conclude that the effective layer-included model can be used to determine the accurate (n, k, d) of RGB color filter coatings with rough surface.