Optically preamplified receiver performance according to the vestigial sideband (VSB) filtering has been numerically investigated for 40-Gb/s optical signals modulated with nonreturn-to-zero, duobinary nonreturn-to-zero (NRZ), return-to-zero (RZ), carrier-suppressed RZ, and duobinary carrier-suppressed RZ formats. The VSB filtering enables the spectral widths of NRZ, duobinary NRZ, and RZ signals to be reduced without severe power penalties at the receiver. On the other hand, carrier-suppressed RZ and duobinary carrier-suppressed RZ signals have no large advantages over VSB filtering because of the characteristics of their signals. Our results suggest that RZ signals are the most suitable modulation format for VSB filtering, without considering the filter loss, because of the tolerance of the intersymbol interference and a large spectral width. However, duobinary NRZ signals are the most suitable modulation format for VSB filtering, considering the filter loss, because of their narrow spectral width.