This research aimed to determine compressive stress and response time of using refractory brick waste as alternative natural coarse aggregates combined with blended cement in manufacturing sustainable concrete. The experiment was conducted by adding 15–50% aggregates volume of refractory bricks coarse aggregate (RBCA) to concrete mix in order to determine the suitability as coarse aggregates. The results showed that the slump values for the combination of standard concrete (SC) with all RBCA satisfied the design value of ± 20 cm and this showed easy handling and workability of the fresh concrete produced. Moreover, the peak stress and toughness were reduced as RBCA increased from 15 to 50% at both w/c of 0.52 and 0.49. The relationship between compressive stress and response time also showed that the travel time at the elastic, peak, and ultimate stages decreased as RBCA increased from 15 to 50% at both w/c. It was concluded that concrete mix with 50% RBCA did not meet the required criteria at w/c of 0.49 but those with 15–50% RBCA satisfied the minimum requirements at w/c of 0.52.