High ductile concrete (HDC) is a kind of cement-based composites containing discontinuous short polyvinyl alcohol (PVA) fibers exhibiting strain-hardening performance and high ductility based upon the bridging effect of short fibers. The study reported here investigated the axial compressive performance of HDC-masonry composite columns/walls, which can be used in new masonry constructions. Three different type of masonry specimens, namely short columns, full-scale columns, and walls, were prepared and tested to failure under monotonic concentric axial loads. The experimental results were reported and analyzed in terms of axial load–displacement response, failure mode, strength, stiffness, and energy dissipation capacity of tested members. Overall, the composite effect of HDC improved both load-carrying capacity and peak axial displacement of masonry columns/walls, with effectiveness factors ranging from 1.37 to 2.29 for the peak load and from 1.09 to 1.27 for the peak displacement. Due to the high slenderness ratio of full-scale masonry columns, the full-scale HDC-masonry composite columns showed lower increases in peak load, stiffness, and energy dissipation capacity, compared with short column and wall specimens. The negative influence of high aspect ratio on the axial capacity of HDC-masonry composite members will be weakened by the increased elastic modulus and cross-sectional area. Therefore, the test results obtained from the present study allowed the development of the stability coefficient of unreinforced masonry (URM) structures, using equivalent cross-section approach, for the prediction of the axial load-carrying capacity of composite columns/walls. The analytical model provided a better approximation for the column specimens while gave a conservative prediction for the wall specimens.