Geopolymer concrete, also known as an earth-friendly concrete, has been under continuous study due to its environmental benefits and a sustainable alternative to conventional concrete construction. The supplies of many source materials, such as fly ash (FA) or slag (SG), to produce geopolymer concrete (GPC) may be limited; however, quarry rock dust (QRD) wastes (limestone, dolomite, or silica powders) formed by crushing rocks appear virtually endless. Although significant experimental research has been carried out on GPC, with a major focus on the mix design development, rheological, durability, and mechanical properties of the GPC mixes; still the information available on the structural behavior of GPC is rather limited. This has implications in extending GPC application from a laboratory-based technology to an at-site product. This study investigates the structural behavior of quarry-rock-dust-incorporated fiber-reinforced GPC columns under concentric and eccentric loading. In this study, a total of 20 columns with 200 mm square cross-section and 1000 mm height were tested. The FA and SG were used as source materials to produce GPC mixtures. The QRD was incorporated as a partial replacement (20%) of SG. The conventional concrete (CC) columns were prepared as the reference specimens. The effect of incorporating quarry rock dust as a replacement of SG, steel fibers, and loading conditions (concentric and eccentric loading) on the structural behavior of GPC columns were studied. The test results revealed that quarry rock dust is an adequate material that can be used as a source material in GPC to manufacture structural concrete members with satisfactory performance. The general performance of the GPC columns incorporating QRD (20%) is observed to be similar to that of GPC columns (without QRD) and CC columns. The addition of steel fibers considerably improves the loading capacity, ductility, and axial load–displacement behavior of the tested columns. The load capacities of fiber-reinforced GPC columns were about 5–7% greater in comparison to the CC columns. The spalling of concrete cover at failure was detected in all plain GPC columns, whereas the failure mode of all fiber-reinforced GPC columns is characterized with surface cracking leading to disintegration of concrete cover.