The structural lightweight concrete is gaining significant interest in the research community due to its reduced dead load. The reduction in the self-weight not only lowers the gravity load but can also lower the seismic demands for the structural members and hence can potentially reduce the overall cost. This study examines the feasibility of developing a sustainable structural lightweight concrete by utilizing rigid polyurethane foam waste as coarse aggregates (5–10 mm). The silica fume was incorporated as a cement replacement to enhance the compressive strength of the mix. A total of three composite mix formulations consisting of polyurethane foam, cement-coated polyurethane foam, and pumice (control specimens) as coarse aggregates were tested. The testing was performed for workability, compressive strength, flexural strength, elastic modulus, absorption, permeable voids, chloride-ion penetration, freeze and thaw resistance, and drying shrinkage. The results achieved for elastic modulus, chloride ion penetrability, and drying shrinkage, were meeting the minimum requirements of the standards making this composite suitable for the structural application. The results revealed that uncoated polyurethane based lightweight aggregate concrete resulted in an air-dry density of 1829 kg/m3 with the corresponding compressive strength of 19.5 MPa and flexural strength of 2.78 MPa at 28 days. Besides, the polyurethane-based lightweight concrete also showed lower internal structure damage against freeze and thaw action, along with improved thermal conductivity.