H3K4 methylation is associated with active transcription and in combination with H3K27me3 thought to keep genes regulating development in a poised state. The contribution of enzymes regulating trimethylation of lysine 4 at histone 3 (H3K4me3) levels to embryonic stem cell (ESC) self‐renewal and differentiation is just starting to emerge. Here, we show that the H3K4me2/3 histone demethylase Jarid1b (Kdm5b/Plu1) is dispensable for ESC self‐renewal, but essential for ESC differentiation along the neural lineage. By genome‐wide location analysis, we demonstrate that Jarid1b localizes predominantly to transcription start sites of genes encoding developmental regulators, of which more than half are also bound by Polycomb group proteins. Virtually all Jarid1b target genes are associated with H3K4me3 and depletion of Jarid1b in ESCs leads to a global increase of H3K4me3 levels. During neural differentiation, Jarid1b‐depleted ESCs fail to efficiently silence lineage‐inappropriate genes, specifically stem and germ cell genes. Our results delineate an essential role for Jarid1b‐mediated transcriptional control during ESC differentiation.