Effective treatment of shadows generated by the obstruction of trees and buildings is an inevitable task for extracting detailed spectral and spatial information from urban high-resolution images. Object-based shadow detection methods can take full advantages of spatial features in the urban very high resolution (VHR) images. However, the effect of different segmentation parameters for detecting shadows has not been well studied. In this study, we proposed an object-based method for shadow detection on urban high-resolution image and addressed quantitative assessment of segmentation. In proposed object-based method, a multi-scale segmentation method, known as fractal net evolution approach (FNEA), was employed to generate primitive objects; then, three spectral properties of shadows were fused based on Dempster–Shafer (D–S) evidence theory to identify shadows. In quantitative assessment, a method for ordering significance of parameters and deriving optimal parameters based on orthogonal experimental design was proposed to evaluate the impact of different segmentation variables on the accuracy of shadow detection. Experimental results indicate that the best overall accuracy (OA) for shadow detection of our method was 89.60% after segmentation parameters’ optimization and scale is the most influential parameter of FNEA segmentation parameters in determining the performance of shadow detection.