This study sought to determine the relationship between urban and transportation environments and health among older adults who have returned their driver’s licenses. Many elderly people who have given up their driver’s license consider it a good thing. The reasons given were that they no longer had to worry about causing traffic crashes and the stress of driving. Factors that influenced whether or not they were glad to give up their driver’s license included the shift in transportation from driving to walking, and the health benefits of walking. Returning one’s driver’s license at the appropriate time may contribute to the health of elderly people. The health status of elderly people was assessed using the Tokyo Metropolitan Institute of Gerontology Index of Competence and a frailty index. Structural equation modeling was used to structure the relationship between the urban and transportation environment and health of elderly people who had given up their driver’s licenses. The results showed that the layout of facilities (e.g., supermarkets, hospitals, and bus stops) around the home influenced the age at which the driver’s license was returned. It was also shown that the younger the age of returning the motor vehicle driver’s license, the less frailty and the higher activity capacity was experienced by individuals. This may be because returning a driver’s license at an earlier age makes it easier to adjust to a new life and to new modes of transportation.