The integration of multiple-input multiple-output (MIMO) and non-orthogonal multiple access (NOMA) technologies is a hybrid technology that overcomes a myriad of problems in the 5G cellular system and beyond, including massive connectivity, low latency, and high dependability. The goal of this paper is to improve and reassess the bit error rate (BER), spectrum efficiency (SE) of the downlink (DL), average capacity rate, and outage probability (OP) of the uplink (UL) in a 5G network using MIMO. The proposed model utilizes QPSK modulation, four users with different power location coefficients, SNR, transmit power, and two contrasting bandwidths 80 and 200 MHz under selective frequency Rayleigh fading channels. The proposed model’s performance is evaluated using the MATLAB software program. The DL results found that the BER and SE against transmitted power showed the MIMO-NOMA enhanced the BER performance for the best user U4 from 10−1.7 to 10−5.2 at 80 MHz bandwidth (BW), and from 10−1.5 to 10−5 at 200 MHz for transmitting power of 40 dBm. In contrast, the SE performance for the best user U4 is enhanced from 24 × 10−3 to 25 × 10−3 bits/second/Hz at 80 MHz BW and from 19.8 × 10−3 to 20 × 10−3 bps/Hz at 200 MHz BW. Although the outcomes for the UL were obtained in terms of average capacity rate and OP versus SNR at 80, and 200 MHz BW, the MIMO-NOMA result showed that the average capacity rate for the best user U4 performance improves by 12 bps/Hz for 1 dB SNR and the OP is reduced by 15 × 10−3 for 80 MHz BW and by 12 × 10−3 for 200 MHz BW at an SNR of 0.17 dB. As the BW increased the BER, the average capacity rate increased while the SE and OP decreased. For both DL/UL NOMA with and without MIMO, closed-form expressions for BER, SE, average capacity rate, and OP were obtained. All users’ performance, even those whose connections were affected by interference or Rayleigh fading channels significantly improved, when MIMO-NOMA was implemented.