Use of industrial by-products as raw materials in concrete is becoming necessary to address the sustainability of both the concrete and the industrial growth. The present study assesses the potential of imperial smelting furnace (ISF) slag as in concrete, considering the presence of toxic elements (lead and zinc) and their detrimental effects on the early hydration of cement. Equivalent volume of sand was replaced by ISF slag in different percentages. Concrete specimens were prepared at different water-to-cement ratios. Compressive, flexural, and pull-off strength, along with abrasion resistance, were examined. Leaching potentials of toxic lead, zinc, and cadmium from ISF slag concrete mixtures were also analyzed to evaluate environmental viability. Results are encouraging because sign of delay in setting was not observed. Improvement in compressive and pull-off strength, comparable flexural strength and abrasion resistance, and leaching of toxic elements within safe limits assures the potential of future use of ISF slag as sand in concrete.