In the development of the cultural and creative industry, popular cultural and creative attractions are reviewed for integrating local architecture, history, and culture into products to promote local economy and culture. We conducted a field investigation at the Xiaojia ancient house in Jiadong Township, Pingtung County, and found that there was a cultural cognitive gap between local cultural and creative products and consumers, which was a dilemma in promoting the development of the local cultural and creative industry. Through the design thinking of cultural and creative products, the spirit of Hakka culture can be delivered and expanded into local specialty products. In the design of cultural and creative products, we used the KJ method to analyze, summarize, and extract the most representative architectural features as the source of design elements. At the same time, combined with the principles of design semiotics, deep cultural symbols were integrated into the product design of the pen holder with richer local spiritual meaning. The traditional and modern elements of the Xiao family’s ancient house were combined to design a multi-functional cultural and creative pen holder. It brought a unique cultural experience and promoted the local cultural and creative industries and the Hakka culture of Xiaojia Ancient House to develop related culture and tourism. Semiotics and the KJ method in theory and practice were useful in product design and programming to develop local cultural and creative products. In the future, we apply this design method to different cultures and cross-domains to achieve wider research and application.