The human brain grows quickly during infancy and early childhood, but factors influencing brain maturation in this period remain poorly understood. To address this gap, we harmonized data from eight diverse cohorts, creating one of the largest pediatric neuroimaging datasets to date focused on birth to 6 years of age. We mapped the developmental trajectory of intracranial and subcortical volumes in ∼2,000 children and studied how sociodemographic factors and adverse birth outcomes influence brain structure and cognition. The amygdala was the first subcortical volume to mature, whereas the thalamus exhibited protracted development. Males had larger brain volumes than females, and children born preterm or with low birthweight showed catch-up growth with age. Socioeconomic factors exerted region- and time-specific effects. Regarding cognition, males scored lower than females; preterm birth affected all developmental areas tested, and socioeconomic factors affected visual reception and receptive language. Brain–cognition correlations revealed region-specific associations.