Accurate breast lesion segmentation is a great help in the initial stage of breast cancer treatment planning. Ultrasound is considered the safe and cheapest method for the breast screening process. However, ultrasound images inherently contain speckle noise, unclear boundaries, and complex shapes, making it more challenging for automatic segmentation methods. This work proposes a multiscale dual attention-based network (MDA-Net) for concurrent segmentation of breast lesions images. The multiscale fusion (MF) block is introduced that addresses the classical fixed receptive field issues, and helps to extract more semantic features and aims to achieve more features diversity. A dual-attention (dA) is also proposed, which is a hybrid of channel-based attention (cA) and lesion attention (lA) blocks that improves the feature representation capability and adaptatively learns a discriminative representation of high-level features. As a result, a combination of two attention blocks helped the proposed network to concentrate on a more relevant field of view of targets. The MDA-Net is extensively tested on both self-collected private datasets and two public UDIAT, BUSIS datasets. Furthermore, our method is also evaluated on MRI datasets to observe the broad applicability of our method in a different imaging modality. The MDA-Net has achieved the DSC of 87.68%, 91.85%, 90.41%, 83.47% on UDIAT, BUSIS, Private, and RIDER breast MRI datasets (p-value < 0.05 with paired t-test). Our MDA-Net implementation code and pretrained models are released at GitHub: https://github.com/ahmedeqbal/MDA-Net.