Identification of structural connections between neurons is a prerequisite to understanding brain function. Here we developed a pipeline to systematically map brain-wide monosynaptic input connections to genetically defined neuronal populations using an optimized rabies tracing system. We used mouse visual cortex as the exemplar system and revealed quantitative target-specific, layer-specific and cell-class-specific differences in its presynaptic connectomes. The retrograde connectivity indicates the presence of ventral and dorsal visual streams and further reveals topographically organized and continuously varying subnetworks mediated by different higher visual areas. The visual cortex hierarchy can be derived from intracortical feedforward and feedback pathways mediated by upper-layer and lower-layer input neurons. We also identify a new role for layer 6 neurons in mediating reciprocal interhemispheric connections. This study expands our knowledge of the visual system connectomes and demonstrates that the pipeline can be scaled up to dissect connectivity of different cell populations across the mouse brain.