As computer science (CS) education expands at the K-12 level, we must be careful to ensure that CS neither exacerbates existing equity gaps in education nor hinders efforts to diversify the field of CS. In this paper, we discuss structural and social barriers that influence Blacks, Hispanics, and girls, based on surveys of 1,672 students, 1,677 parents, 1,008 teachers, 9,805 principals, and 2,307 superintendents in the United States. We find that despite higher interest in CS among Black and Hispanic students and parents, these students experience greater structural barriers in accessing computers and CS classes than White students. And while girls have the same access as boys, social barriers exist with girls reporting lower awareness of CS opportunities outside of classes, less encouragement from teachers and parents, and less exposure to CS role models in the media. It is critical for expanding CS opportunities to address the unique issues for each group.