This paper investigates balanced detection schemes for optical duobinary communication systems, and compares duobinary with differential phase-shift keying (DPSK) modulation. The balanced detection schemes are evaluated in duobinary and DPSK systems limited by amplified spontaneous emission noise, narrow optical filtering, and chromatic dispersion. We also consider the optimization of duobinary generating low-pass filter. Experimental data at 10 Gb/s show that balanced detection yields a significant advantage in receiver sensitivity compared with direct detection for duobinary systems, allowing duobinary systems to approach within ~1.5 dB the performance of DPSK. We also employ Monte Carlo simulations to evaluate balanced detection for 40 Gb/s duobinary transmission systems with narrow optical filtering designed for 50 GHz channel spacing. The simulations confirm our experimental results, and provide a more general investigation of balanced detection schemes. In particular, we propose a novel balanced detection scheme for duobinary systems designed to improve the tolerance to narrow optical filtering.