Time-frequency (TF) filtering of analog signals has played a crucial role in the development of radio-frequency communications and is currently being recognized as an essential capability for communications, both classical and quantum, in the optical frequency domain. How best to design optical time-frequency (TF) filters to pass a targeted temporal mode (TM), and to reject background (noise) photons in the TF detection window? The solution for 'coherent' TF filtering is known-the quantum pulse gate-whereas the conventional, more common method is implemented by a sequence of incoherent spectral filtering and temporal gating operations. To compare these two methods, we derive a general formalism for two-stage incoherent time-frequency filtering, finding expressions for signal pulse transmission efficiency, and for the ability to discriminate TMs, which allows the blocking of unwanted background light. We derive the tradeoff between efficiency and TM discrimination ability, and find a remarkably concise relation between these two quantities and the time-bandwidth product of the combined filters. We apply the formalism to two examples-rectangular filters or Gaussian filters-both of which have known orthogonal-function decompositions. The formalism can be applied to any state of light occupying the input temporal mode, e.g., 'classical' coherent-state signals or pulsed single-photon states of light. In contrast to the radio-frequency domain, where coherent detection is standard and one can use coherent matched filtering to reject noise, in the optical domain direct detection is optimal in a number of scenarios where the signal flux is extremely small. Our analysis shows how the insertion loss and SNR change when one uses incoherent optical filters to reject background noise, followed by direct detection, e.g. photon counting. We point out implications in classical and quantum optical communications. As an example, we study quantum key distribution, wherein strong rejection of background noise is necessary to maintain a high quality of entanglement, while high signal transmission is needed to ensure a useful key generation rate.