With the growing global awareness of sustainable development, the impact of “fast fashion” on the environment has drawn much attention. This study investigated 365 fast fashion consumers with higher-than-average spending on fast fashion garments in Taiwan and analyzed the data with the analytic network process. From the perspective of social responsibility theory, the triple bottom line framework, and the concept of cradle-to-cradle, this study conducted in-depth interviews with 10 experts in the field of clothing. Based on the results, this study constructed a model with 4 core dimensions and 12 key criteria of sustainable clothing design, with the enterprise, design, and consumer sides as the decision-making options. Moreover, it was found that consumers believe that the first decision-making key that affects sustainable design is the enterprise side, and the most important green item that affects sustainable design is “cherishing”. The findings of this study can provide an important reference standard for sustainable design in the apparel industry.