The representational image of the city has attracted people’s long-term attention. Nevertheless, the mechanism of interaction between the image and the built environment (BE) and image studies at the town scale have not been fully explored. In this study, we collected multi-source data from 26 characteristic towns in Tianjin, China. We explored a deep learning approach to recognize social media data, which led to the development of quantifiable town uniqueness image (UI) variables. We studied the influence of the BE on the town UI and the moderating effects of positive emotions on the relationship between the two. The results showed that positive emotions had significantly positive moderating effects on the water system ratio’s effect on UI, but weakened sidewalk density and tourist attraction density. They also inhibited the negative effects of road connectivity but could strengthen the negative effects of the sky view factor and points of interest (POI) mix. The moderating effects on other variables are relatively mediocre. This study helps to reveal the inner mechanism of BE and town image. It is conducive to accurately coordinating the relationship between planning policies and design strategies, optimizing resource allocation, and promoting sustainable town development.