The underlying structure of working memory (WM) in young children with and without specific language impairment (SLI) was examined. The associations between the components of WM and the language abilities of young children with SLI were then analyzed. The Automated Working Memory Assessment and four linguistic tasks were administered to 58 children with SLI and 58 children without SLI, aged 4–5 years. The WM of the children was best represented by a model with four separate but interacting components of verbal storage, visuospatial storage, verbal central executive (CE), and visuospatial CE. The associations between the four components of WM did not differ significantly for the two groups of children. However, the individual components of WM showed varying associations with the language abilities of the children with SLI. The verbal CE component of WM was moderately to strongly associated with all the language abilities in children with SLI: receptive vocabulary, expressive vocabulary, verbal comprehension, and syntactic development. These results show verbal CE to be involved in a wide range of linguistic skills; the limited ability of young children with SLI to simultaneously store and process verbal information may constrain their acquisition of linguistic skills. Attention should thus be paid to the language problems of children with SLI, but also to the WM impairments that can contribute to their language problems.