To reveal the toughening mechanism of polymers and fibers in composite applications, a combination of physical tests (uniaxial compression, flexural fracture, single fiber pull-out, scanning electron microscopy, and X-ray diffraction (XRD)) and molecular dynamics simulations have been carried out. Three polymers, i.e. ethylene-vinyl acetate copolymer (EVA), styrene-acrylate copolymer (SAE), and styrene-butadiene copolymer (SB), and two fibers, i.e. polypropylene fiber (PP) and polyvinyl alcohol fiber (PVA) are considered. Their composite effect mechanisms are explained in the viewpoint of three scales. At the macro-scale, the pull-out tests of single fiber show that the polymer increased the equivalent bond strength between the fiber and the mortar matrix. At micro-scale, it was observed from the SEM experiments that the fiber and polymer film inhibited the crack extension at different scales, besides the polymer could absorb on the fiber surface to improve the interfacial transition zone (ITZ) density around the fiber and increase the roughness of the fiber surface. At nano-scale, MD simulations demonstrate that three polymers facilitated the bond strength between PP fiber and C–S–H at the nano-scale, mainly because of the formation of Ca–O coordination bond and H-bonds between the polymers and C–S–H, and the presence of van der Waals forces between the polymers and PP fiber. However, SAE facilitated the bonding between the PVA fibers and the C–S–H, which originated from the coordination bonds formed between Ca ions on the surface of SAE and O atoms in PVA. In addition, the large number of H-bonds formed between SAE and PVA.