Building detection from panchromatic (PAN) and multispectral (MS) images is an essential task for many practical applications. In this article, a dual-stream asymmetric fusion network is proposed, named DAFNet. DAFNet can achieve effective information fusion at the feature level. It obtains better building detection performance from the following three perspectives: a two-stream network structure is designed to guarantee the ability to extract information from PAN and MS images; an asymmetric feature fusion module is proposed to fuse features efficiently and concisely; and two consistency regularization losses, i.e., PAN information preservation loss and cross-modal semantic consistency loss are applied to further explore the consistency between features for better fusion. The experiments are conducted on a challenging building detection dataset collected from GaoFen-2 satellite images. Comprehensive evaluations on 12 popular detection methods demonstrate the superiority of our DAFNet compared with the existing state-of-the-art fusion methods. We reveal that feature-level fusion is more suitable for building detection from PAN-MS images.