The mathematical model for description of pulse propagation in optical fiber for the generalized anti-cubic nonlinearity with arbitrary refractive index is considered. The Cauchy problem for this partial differential equation cannot be solved by the inverse scattering transform and the analytical solutions of this mathematical model are found taking into account the traveling wave reduction. First integral of nonlinear ordinary differential equation is presented. Analytical solutions in the form of periodic and solitary waves are found using some transformations. Partial cases of the mathematical model for the generalized anti-cubic nonlinearity are studied.