Sustainability, safety and service life expansion in the construction sector have gained a lot of scientific and technological interest during the last few decades. In this direction, the synthesis and characterization of smart cementitious composites with tailored properties combining mechanical integrity and self-sensing capabilities have been in the spotlight for quite some time now. The key property for the determination of self-sensing behavior is the electrical resistivity and, more specifically, the determination of reversible changes in the electrical resistivity with applied stress, which is known as piezoresistivity. In this study, the mechanical and piezoresistive properties of mortars reinforced with carbon nanotubes (CNTs) and carbon micro-fibers (CMFs) are determined. Silica fume and a polymer with polyalkylene glycol graft chains were used as dispersant agents for the incorporation of the CNTs and CMFs into the cement paste. The mechanical properties of the mortar composites were investigated with respect to their flexural and compressive strength. A four-probe method was used for the estimation of their piezoresistive response. The test outcomes revealed that the combination of the dispersant agents along with a low content of CNTs and CMFs by weight of cement (bwoc) results in the production of a stronger mortar with enhanced mechanical performance and durability. More specifically, there was an increase in flexural and compressive strength of up to 38% and 88%, respectively. Moreover, mortar composites loaded with 0.4% CMF bwoc and 0.05% CNTs bwoc revealed a smooth and reversible change in electrical resistivity vs. compression loading—with unloading comprising a strong indication of self-sensing behavior. This work aims to accelerate progress in the field of material development with structural sensing and electrical actuation via providing a deeper insight into the correlation among cementitious composite preparation, admixture dispersion quality, cementitious composite microstructure and mechanical and self-sensing properties.