In this article, the complex dynamics of susceptible-infected-treated-recovered (SITR) model describing the COVID-19 transmission is explored. Two types of transmission are included here, such as bi-linear with unaware susceptible individuals and nonlinear with aware susceptible individuals. Modified saturation recovery in treated individuals and a difference self-quarantined parameter in susceptible individuals have been considered. Sensitivity indices of the parameters associated with the reproduction number (R0) have been derived using the normalized forward sensitivity index to recognize the significant parameters in changing the disease dynamics. The criteria of existence and local stability of equilibrium points are discussed. The backward bifurcation, forward bifurcation and Hopf bifurcation are demonstrated analytically and numerically, indicating rich dynamics of the proposed model. Chaotic dynamics and supercritical Hopf bifurcation also have been investigated by computing the Lyapunov exponent and the first Lyapunov coefficient respectively. It is observed that, difference self-quarantined rate and psychological or awareness effect ensure the existence of only disease-free equilibrium for R0<1. Optimal control theory and Pontryagin’s maximum principle are applied to compute the efficiency of vaccination and isolation controls in lowering infection while minimizing the associated costs. According to the simulation results, applying both controls together is the most efficient way to prevent the spread of the disease, compared to the situation where a single control is used to achieve the same.