The exact solution of the modal dispersion relation of multilayer optical fibers is very critical and complicated, especially in the case of complex refractive indices of some layers added to the fiber. In this paper, a different methodology is proposed to solve the complex dispersion relations for cladding modes, based on the well-defined proper expressions of electromagnetic fields in the different layers of optical fibers. An optical fiber, coated by a dielectric nonlinear layer, is analyzed using the exact four-layer model, and the results obtained are compared with those analyzed in the literature based on the approximate three-layer model, where the effect of the coating layer is neglected when solving the dispersion relation. The results obtained show a remarkable difference between the exact and the approximate values of the effective refractive indices of the cladding modes. Inappropriate values of the effective refractive indices strongly affect phase matching and coupling between modes, which are required in different applications such as second-harmonic generation. The proposed approach for solving general dispersion relations is also employed to obtain complex values of the effective refractive indices of the cladding modes for a five-layer optical fiber with a metallic thin film inserted between the nonlinear layer and the fiber cladding. Using the appropriate expressions that describe the electric field in the different dielectric and metallic layers of optical fibers, field distributions are displayed for some cladding modes.