Sustainability analysis should include the assessment of the environmental, social, and economic impacts throughout the life cycle of a product. However, the social sustainability performance assessment is seldom carried out during materials selection due to its complex nature and the lack of a social life cycle assessment tool. This study presents a single score-based social life cycle assessment methodology, namely social sustainability grading model, for assessing and comparing the social sustainability performance of construction materials using a case study on recycled and natural construction materials. The proposed method is developed based on the methodological framework provided by the United Nations Environment Programme/Society of Environmental Toxicology and Chemistry guidelines published in 2009 and the methodological sheets published in 2013, the indicators and sustainability reporting guidelines provided by the Global Reporting Initiatives and ISO 26000 for social responsibility of products, and the indicators provided by the Hong Kong Business Environment Council Limited for construction sustainability. A twofold research approach is proposed in this model: the first one is the qualitative research based on expert interviews to identify, select, and prioritize the relevant subcategories and indicators, and the second one is the operational research based on the case-specific survey to collect the required data. A social sustainability index was proposed for the interpretation of the results effectively. A case study on construction materials was conducted to illustrate the implementation of the method using case-specific first-hand data. The major outcome of this study is the systematic development of a social sustainability assessment tool based on the established standards and guidelines. The case study showed that four subcategories are crucial social concerns for construction materials (i.e., health and safety issues of the materials, health and safety of workers, company’s commitment to sustainability, and company’s policies on energy and water consumption). Based on the sustainability index proposed, using recycled aggregates from locally generated waste materials scored higher (about 31–34%) social sustainability than using imported natural aggregates. In addition, recycled aggregates and natural aggregates achieved “sustainable” and “neutral” rating sustainability levels, respectively. However, several subcategories (e.g., health and safety, working hour, forced work, training and social benefits of workers, and quality of the materials and information disclosing to public) are still needed to improve the social sustainability performance of recycled aggregates. An integrated social life cycle assessment method is presented in this study for assessing the social sustainability of construction materials. In addition, the reported case study in this paper is one of the first attempts for social sustainability assessment of recycled construction materials, and the method can be applied to other recycled materials/products for comparative analysis. However, several critical factors, such as integration in other life cycle methods and software, sensitivity analysis, and more case studies, are still needed for further improvement of the developed method.