To improve evaluations of cortical and subcortical diffusivity in neurological diseases, it is necessary to improve the accuracy of brain diffusion tensor imaging (DTI) data segmentation. The conventional partial volume segmentation method fails to classify voxels with multiple white matter (WM) fiber orientations such as fiber-crossing regions. Our purpose was to improve the performance of segmentation by taking into account the partial volume effects due to both multiple tissue types and multiple WM fiber orientations. We quantitatively evaluated the overall performance of the proposed method using digital DTI phantom data. Moreover, we applied our method to human DTI data, and compared our results with those of a conventional method. In the phantom experiments, the conventional method and proposed method yielded almost the same root mean square error (RMSE) for gray matter (GM) and cerebrospinal fluid (CSF), while the RMSE in the proposed method was smaller than that in the conventional method for WM. The volume overlap measures between our segmentation results and the ground truth of the digital phantom were more than 0.8 in all three tissue types, and were greater than those in the conventional method. In visual comparisons for human data, the WM/GM/CSF regions obtained using our method were in better agreement with the corresponding regions depicted in the structural image than those obtained using the conventional method. The results of the digital phantom experiment and human data demonstrated that our method improved accuracy in the segmentation of brain tissue data on DTI compared to the conventional method.