AbstractThermoplastic composites are broadly used in many industries. Excellent mechanical characteristics are frequently required to achieve the appropriate properties for the applications. Therefore, the objectives of this study were (1) presenting an in‐depth analysis of the effects of styrene butadiene styrene (SBS) and halloysite nanotubes (HNTs) on the tensile and fracture properties of the thermoplastic composites based on acrylonitrile butadiene styrene (ABS), and (2) examining the relationship between the microstructure of the thermoplastic composites and their mechanical behavior. For these purposes, HNTs were utilized with four levels (0, 1, 3, and 5 wt%) and SBS was used with two levels (15 and 30 wt%). The tensile strength and modulus were reduced by adding SBS to 30 wt%; however, the elongation at break was increased by 30%. Furthermore, tensile strength and modulus, and elongation at break were improved by increasing HNTs up to 3 wt%. The results indicated the elastic work (we) and plastic work (βwp) rose by 29% and 10%, respectively, when 30 wt% of SBS was added to ABS. Furthermore, we and βwp rose by the addition of HNTs up to 3 wt%. To examine fracture mechanisms, field emission scanning electron microscope (FESEM) images of the fracture surface of the essential work of fracture (EWF) specimens were taken. Several voids are present in the compounds containing SBS, activating the plastic deformation mechanism. Based on the optimization process, the best strength, stiffness, and toughness balance was attained for the compound consisting of 15 wt% SBS and 1 wt% HNTs.Highlights The effects of HNTs and SBS on the fracture properties of polymer composites. The addition of SBS increased we, βwp, and the elongation at break. HNTs had a positive effect on the tensile strength and modulus, we, and βwp. The important mechanisms include plastic deformation and cavitation.