In this research, the microstructure and mechanical properties of glass fiber reinforced unsaturated polyester (GFRP) composites modified by styrene-butadiene rubber (SBR) and fumed silica were investigated. The fumed silica particles were dispersed uniformly in the nano-scale into the unsaturated polyester resin. Then, it was mixed with SBR and chopped glass fiber by bulk molding compound (BMC) process, and molded by compression molding. A full factorial design and analysis of variance (ANOVA) were utilized to study the effect of factors and their interactions on the mechanical properties. The addition of fumed silica nano-filler improved tensile, flexural and impact strength, tensile and flexural modulus but reduced the elongation at break. On the other hand, adding SBR content decreased flexural strength and modulus and tensile modulus of composites, but, increased elongation at break, impact and tensile strength. Based on optimization results, the SBR with 10 wt% and 3 phr for fumed silica has been selected as the optimum composite formulation providing the best strength-stiffness-toughness balance among the samples.