The segmentation of building from satellite and airborne images is necessary for high‐resolution buildings maps generation and it is still challenging. On annotated pixel‐level images, trained deep convolutional neural networks (CNNs) were used to improve segmentation of building. The cost of labelling training data is high, which reduces their usage. Human labelling efforts can be significantly reduced using weakly supervised segmentation techniques. Here, a novel weakly supervised framework is introduced for building semantic segmenting that relies on deep seeds to construct a superpixels‐CRF model over superpixels segmentation in order to generate high‐quality initial pixel‐level annotations, as the initialization step. Then, the segmentation network is trained using the initial pixel‐level annotations. Next, the CRF model is used to refine the segmentation masks, and the segmentation network is retrained to achieve accurate pixel‐level annotations while iteratively optimizing the segmentation. The experimental results on three public building datasets demonstrate that the proposed framework significantly improved the quality of building semantic segmentation while remaining computationally efficient.