Abstract This work aims to investigate the outage and throughput performance of non-orthogonal multiple access assisted cooperative relay system (CRS-NOMA) considering the realistic impairments caused due to in-phase and quadrature-phase imbalance (IQI), channel estimation errors (CEE), and successive interference cancellation (SIC) errors. More specifically, we investigate a model in which two-phase downlink transmission is carried out in two different modes: (i) CRS-NOMA without direct links and (ii) CRS-NOMA with direct links (CRS-DLNOMA). In CRS-NOMA mode, the source broadcasts a composite NOMA signal to destination users with the assistance of a decode-and-forward (DF) relay. In contrast, in CRS-DLNOMA, direct and cooperative links are available for transmission. We derive the analytical expressions of outage probability and throughput for both the NOMA destinations to evaluate the system performance of both CRS-NOMA and CRS-DLNOMA modes of transmission. Furthermore, numerical simulations also study and validate the influence of IQI, CEE, and SIC errors on the outage and throughput performance. The simulation results verify that realistic impairments degrade the system performance, but the presence of direct link has a positive impact on outage and throughput. Additionally, we use the golden search method to optimize the power allocation factor (PAF) and transmission rate to maximize the throughput at the near user while ensuring the throughput constraint at the far user.