The integrated terrestrial and non-terrestrial networks in 5G and beyond 5G are envisioned to support dynamic, seamless, and differentiated services for emerging use cases with stringent requirements. Such service heterogeneity and rapid growth in network complexity pose difficulties in network management and resource orchestration. Network slicing paves the way for delivering highly customized services and enabling service-oriented resource allocation. In this context, artificial intelligence (AI) becomes a key enabler for network slicing management. However, AI-based approaches encounter critical challenges in adapting to dynamic and complex wireless environments. In this article, first, we aim to provide a comprehensive understanding of these challenges, open issues, and future research opportunities. Second, we highlight the investigations on dynamic-adaptive AI solutions for dealing with the effect of concept drift. Third, we identify typical dynamic scenarios in case studies and provide numerical results to illustrate the effectiveness of the discussed AI solutions.