Abstract This paper discusses the dynamical behavior of a new 3D chaotic system of integer and fractional order. To get a comprehensive knowledge of the dynamics of the proposed system, we have studied competitive modes and Hamilton energy for different parameter values. In order to get the ultimate bound set for the proposed system, we employed the Lagrange coefficient approach to solve the optimization problem. We have also explored the use of the bound set in synchronization. Furthermore, we have examined the Hamilton energy, time series, bifurcation diagrams, and Lyapunov exponents for the fractional version of the proposed chaotic system. Finally, we looked at the Mittage-Leffler positive invariant sets and global attractive sets by merging the Lyapunov function approach with the Mittage-Leffler function. Numerical simulations have shown the obtained bound sets and other analytical outcomes.