Breast cancer is considered the most commonly diagnosed cancer globally and falls second to lung cancer. For the early detection of breast tumors in women, breast cancer analysis using Ultrasound, Mammography, and MRI modalities as the initial screening process. Due to the random variation, irregular shapes, and blurred boundaries of tumor regions, the accurate segmentation of breast tumors is still a tricky task. The existing convolutional neural networks (CNNs) inherit their limitation by extracting global context information and, in most cases, proved less efficient in obtaining satisfactory results. As a solution, we proposed the BTS-ST network, a novel solution for breast tumor segmentation and classification that Swin-Transformer (ST) inspires. The BTS-ST network incorporates Swin-Transformer into traditional CNNs-based U-Net to improve global modeling capabilities. To improve the feature representation capability of irregularly shaped tumors, we first introduced a Spatial Interaction block (SIB), encoding spatial knowledge in the Swin Transformer block by developing pixel-level correlation. The segmentation accuracy of small-scale tumor regions is increased by building a Feature Compression block (FCB) to prevent information loss and compress smaller-scale features in patch token down sampling of Swin-Transformer. Finally, a Relationship Aggregation block (RAB) is developed as a bridge between dual encoders to combine global dependencies from Swin-Transformer into the features from CNN hierarchically. Extensive experiments are performed on breast tumor segmentation and classification tasks using multimodality Ultrasound, Mammogram, and MRI-based datasets. The results demonstrate that our proposed solution is comparatively better than other state-of-the-art methods.