Speech reception in the presence of competing sounds declines in middle age. Here we examined how energetic and informational masking degrades speech reception in middle-aged adults, and further explored how the performance degradation depends on temporal processing efficiency and working memory capacity. We measured the intelligibility of a target phrase masked by noise or a competing masker phrase, which was spoken by the same talker, same-sex talker, or different-sex talker. Temporal processing was assessed by interaural phase difference thresholds and frequency modulation detection limen. Working memory capacity was assessed by the listening span test. Speech reception performance was lower in middle-aged adults than young adults, regardless of the type of disturbing sounds. The performance differences between the same-talker and different-sex talker conditions, which reflects the effects of informational masking, was larger in middle-aged adults. These results suggest that both energetic and informational masking contributes to the degradation of speech reception in middle-aged adults. The correlation analysis for the middle-aged adults further revealed that the contribution of working memory capacity was larger in the same-talker condition, where informational masking was strong, whereas the contribution of temporal processing efficiency was larger in the noise condition, where energetic masking was dominant.