Geopolymer concrete is a sustainable and eco-friendly substitute for traditional OPC (Ordinary Portland Cement) based concrete, as it reduces greenhouse gas emissions. With various supplementary cementitious materials, the compressive strength of geopolymer concrete should be accurately predicted. Recent studies have applied deep learning techniques to predict the compressive strength of geopolymer concrete yet its hidden decision-making criteria diminish the end-users’ trust in predictions. To bridge this gap, the authors first developed three deep learning models: an artificial neural network (ANN), a deep neural network (DNN), and a 1D convolution neural network (CNN) to predict the compressive strength of slag ash-based geopolymer concrete. The performance indices for accuracy revealed that the DNN model outperforms the other two models. Subsequently, Shapley additive explanations (SHAP) were used to explain the best-performed deep learning model, DNN, and its compressive strength predictions. SHAP exhibited how the importance of each feature and its relationship contributes to the compressive strength prediction of the DNN model. Finally, the authors developed a novel DNN-based open-source software interface to predict the mix design proportions for a given target compressive strength (using inverse modeling technique) for slag ash-based geopolymer concrete. Additionally, the software calculates the Global Warming Potential (kg CO2 equivalent) for each mix design to select the mix designs with low greenhouse emissions.