Systematic Luby transform (SLT) codes constitute rateless codes, which are capable of adaptively adjusting their code rate depending on the channel quality without any explicit channel state information (CSI) at the transmitters. SLTs are also suitable for space time collaboration-aided relay networks. In this study, an iterative decoding-aided SLT scheme is combined with 16 quadrature amplitude modulation (QAM) transmission in a wireless relay-aided network. The bit error ratio (BER) results and extrinsic information transfer (EXIT) charts are provided to evaluate the performance of the proposed scheme. The simulation results show that the proposed scheme using amplify-and-forward (AF) relaying achieves a 2.5 dB gain at a BER of 1025, while the attainable improvement is nearly 6 dB for decode-and-forward (DF) relaying, compared to the non-iterative detection scheme operating without the relay’s assistance. Moreover, the AF relaying-aided SLT coded 16-QAM scheme is more beneficial, when the relay station is close to the source. By contrast, the DF relaying performs best near the mid-point between the source and the destination. Additionally, a modified Hybrid Automatic-Repeat-reQuest (HARQ) protocol using incremental redundancy is applied along with the SLT-coded 16-QAM scheme in order to enhance the achievable throughput and energy efficiency of cooperative networks. This arrangement reduces the total transmit power by about 5%, compared to the classic HARQ scheme.