Optical soliton solutions in a magneto-optical waveguide and other exact solutions are investigated for the coupled system of the nonlinear Biswas–Milovic equation with Kudryashov’s law using the extended F-expansion method. Various types of solutions are extracted, such as dark soliton solutions, singular soliton solutions, a dark–singular combo soliton, singular combo soliton solutions, Jacobi elliptic solutions, periodic solutions, combo periodic solutions, hyperbolic solutions, rational solutions, exponential solutions and Weierstrass solutions. The obtained different types of wave solutions help in obtaining nonlinear optical fibers in the future. Furthermore, some selected solutions are described graphically to demonstrate the physical nature of the obtained solutions. The results show that the current method gives effectual and direct mathematical tools for resolving the nonlinear problems in the field of nonlinear wave equations.