DNA wraps around histone octamers to assemble nucleosomes, which pack into high-ordered chromatin structure in the nucleus, and block access to the genetic material. The SWI/SNF family chromatin remodeling complexes are multi-subunit, ATP-dependent molecular machines that slide and evict nucleosomes. They play a major role in control of the chromatin structure and regulate gene transcription in eukaryotic cells. Mutations of the human complexes are often found in cancers and neurodevelopmental disorders. Here, we review recent studies of the structures of the yeast complexes (SWI/SNF and RSC) and the human homologs (BAF and PBAF). We discuss how the subunits interweave into functional modules and submodules. We discuss the conserved mode of nucleosome recognition across evolution, and propose the mechanism of chromatin targeting through interaction with transcription factors and histone modifications. We also discuss the implications of disease-associated mutations in the regulation of chromatin remodeling. These findings shed light on the mechanism and function of the SWI/SNF family complexes, and provide the structural basis for drug development.