In this research, the new auxiliary equation method (NAEM) for higher order nonlinear fractional Huxley equation is being employed to extricate the novel soliton solutions using Beta and M-Truncated fractional derivatives. For waves of finite amplitude, the Huxley equation demonstrates a substantial transfer of spectrum energy. A comparison of the solutions of the model with both fractional derivatives is also included in this research. Various kinds of solitary traveling wave solutions, such as trigonometric, hyperbolic, exponential, rational functions, etc., are found. These types of solutions demonstrate the superiority of the novelty of the method. This method’s key advantage over others is that it provides more broad solutions with certain flexible parameters. 3D and 2D graphs are used graphically to demonstrate the dynamical structures of the solutions. The results are presented in a way that demonstrate the usefulness and competence of the approach used to handle various nonlinear fractional partial differential equations. Lastly, we investigate the comparison of the gain spectra for modulation instability and the depiction of certain noteworthy outcomes by illustratively depicting the 2D figures produced by carefully considering the parameters