Increasing contamination of the environment by toxic compounds such as endocrine disrupting chemicals (EDCs) is one of the major causes of reproductive defects in both sexes. Estrogen/androgen pathways are of utmost importance in gonadal development, determination of secondary sex characteristics and gametogenesis. Most of the EDCs mediate their action through respective receptors and/or downstream signaling. The purpose of this review is to highlight the mechanism by which EDCs can trigger antagonistic or agonistic response, acting through estrogen/androgen receptors causing reproductive defects that lead to infertility. In vitro, in vivo and in silico studies focusing on the impact of EDCs on estrogen/androgen pathways and related proteins published in the last decade were considered for the review. PUBMED and PUBCHEM were used for literature search. EDCs can bind to estrogen receptors (ERα and ERβ) and androgen receptors or activate alternative receptors such as G protein-coupled receptors (GPCR), GPR30, estrogen-related receptor (ERRγ) to activate estrogen signaling via downstream kinases. Bisphenol A, dichlorodiphenyltrichloroethane, dichlorodiphenyldichloroethylene, polychlorinated biphenyls and phthalates are major toxicants that interfere with the normal estrogen/androgen pathways leading to infertility in both sexes through many ways, including DNA damage in spermatozoids, altered methylation pattern, histone modifications and miRNA expression.