For a better understanding of optical interactions in magnetic layered structures we establish general expressions of both polar Kerr and Faraday first-order magneto-optical (MO) effects. The approach presented here enables the MO response of an arbitrary magnetic layer system to be analytically expressed with an accuracy practically equal (less than ${10}^{\mathrm{\ensuremath{-}}4}$ deg) to that of the full matrix calculations usually employed for modeling MO effects in magnetic multilayers. The formulas for any particular structure are obtained from general MO expressions by introducing associated reflection and transmission coefficients. This procedure is justified here in the simple case of an ultrathin magnetic film sandwiched between two nonmagnetic layers. These accurate expressions are further simplified for ultrathin magnetic films and the limitations of this approximation discussed. The simplified formulas are then used to analyze MO polar Kerr and Faraday data in well-defined Au (5 nm)/Co/Au (25 nm) structures, grown on float glass substrates. The Co layer thicknesses are chosen in the range 0.4--2.0 nm to preserve a perpendicular magnetic anisotropy. The simplified expressions, as well as the numerical modeling of the trends in MO spectra with a variation of the overlayer or buffer layer thicknesses, show that the shape of the polar Kerr effect spectrum is mainly dependent upon the optical properties of the Au buffer layer whereas the observed MO amplitudes are reduced consistently with the absorption of the Au overlayer.