Observations of the flight response of ungulates are commonly used to test behavioral responses to predation risk. In gregarious ungulates with sexual body-size dimorphism such responses are likely to be stronger in situations where individuals have perceptions of less security as well as among more-sensitive individuals, such as female groups or female groups with offspring which are understood to use safety habitats more often than males do. However, little is known about these behaviors in solitary ungulates with little sexual dimorphism. Therefore, we examined the flight response to human presence and the habitat use of Japanese serow (Capricornis crispus), a solitary ungulate with little sexual dimorphism, in relation to its physical and social environments, based on direct observations conducted over approximately four years. Based on a total of 335 sightings, serows took flight less often when in steep terrain, in closed habitat, or in low-visibility seasons, and they selectively used steep–closed habitat. These findings suggest that steep slopes and low-visibility conditions provide the species with greater security, and that serows select safer habitats to decrease predation risk. There were no significant differences in the flight responses and habitat use of solitary males and solitary females; the absence of differences is likely related to the serow’s habit of monogamy and its underdeveloped sexual dimorphism. Females with kids more frequently fled than other group types; this difference is thought to be linked to females’ priority to secure the survival of their offspring. This result shows the key role of offspring presence in affecting flight response in the serow. Conversely, there were no differences in habitat use between females with kids and other group types; here, the absence of such differences may be associated with intra-sexual territoriality of the serow, since all types of territory holders (including females with kids) need to continuously use the entire home range to maintain a territory.