White matter hyperintensities (WMHs) on MRI are associated with cognitive dysfunction, particularly slow processing speed and executive dysfunction. However, it is not clear whether WMHs burden affects isolated executive function independent of aging when WMHs are assessed separately in periventricular hyperintensity (PVH) and deep and subcortical white matter hyperintensity (DSWMH). To assess the relationship between the degree of WMHs and the performance on the Trail Making Test (TMT), which can evaluate isolated ability of set-shifting and working memory. 74 participants who visited our memory clinic and underwent the TMT subtests (TMT-A and TMT-B) and the Mini-Mental State Examination (MMSE). All subjects performed the TMT within the time limits and their MMSE scores were 24 or higher, and they were diagnosed as having normal cognition or mild cognitive impairment. The extent of PVH and DSWMH was graded from 0 to 3 using the Fazekas scale. We obtained testing time to complete the TMT-A and TMT-B, and calculated TMT-B minus TMT-A. We performed correlation analyses between the degree of WMHs and the time measures of the TMT subtests with adjustment of age. Average scores of the MMSE were not different among the groups either by PVH grade or by DSWMH grade. In contrast, average time required for the TMT-A, TMT-B, and TMT-B minus TMT-A increased along with exacerbation of PVH and DSWMH grade. After the adjustment of age we found significant association between only DSWMH grade and the time difference of TMT-B minus TMT-A. Exacerbation of PVH and DSWMH differentially affected isolated executive functions assessed by the TMT subtests independent of age and general cognitive function.