Thermoplastic composites are broadly used in many industries. Excellent mechanical characteristics are frequently required to achieve the appropriate properties for the applications. Therefore, the objectives of this study were (1) presenting an in‐depth analysis of the effects of styrene butadiene styrene (SBS) and halloysite nanotubes (HNTs) on the tensile and fracture properties of the thermoplastic composites based on acrylonitrile butadiene styrene (ABS), and (2) examining the relationship between the microstructure of the thermoplastic composites and their mechanical behavior. For these purposes, HNTs were utilized with four levels (0, 1, 3, and 5 wt%) and SBS was used with two levels (15 and 30 wt%). The tensile strength and modulus were reduced by adding SBS to 30 wt%; however, the elongation at break was increased by 30%. Furthermore, tensile strength and modulus, and elongation at break were improved by increasing HNTs up to 3 wt%. The results indicated the elastic work (w_e) and plastic work (βw_p) rose by 29% and 10%, respectively, when 30 wt% of SBS was added to ABS. Furthermore, w_e and βw_p rose by the addition of HNTs up to 3 wt%. To examine fracture mechanisms, field emission scanning electron microscope (FESEM) images of the fracture surface of the essential work of fracture (EWF) specimens were taken. Several voids are present in the compounds containing SBS, activating the plastic deformation mechanism. Based on the optimization process, the best strength, stiffness, and toughness balance was attained for the compound consisting of 15 wt% SBS and 1 wt% HNTs.