Optical vector analyzer (OVA) based on optical single-sideband (OSSB) modulation is attractive due to its potential sub-Hz resolution. However, the accuracy and dynamic range are limited by the unwanted sidebands of the OSSB signal, including the high-order sidebands and the unwanted first-order sideband. Previously, the influence of the high-order sidebands has been studied and several methods to suppress the errors induced by the high-order sidebands have been proposed. However, the investigation of the influence of the unwanted first-order sideband is still insufficient. In this paper, an analytical model is established and a numerical simulation is performed to comprehensively study the impact of the unwanted first-order sideband on the accuracy and dynamic range of the OSSB-based OVA. Then the simulation results are verified by experiment, which indicates that the unwanted first-order sideband has evident influence on the measurement accuracy and places a restriction on the dynamic range. Given that some prior information of the transmission responses is known, certain parameters of the device under test may still be achievable with high accuracy even when large unwanted first-order sideband is presented. This study may provide guidance in design and application of the OSSB-based OVA.