The dynamic nature of stock market styles, referred to as concept drift, poses a formidable challenge when applying deep learning to stock prediction. Models trained on historical data often struggle to adapt to the latest market styles, as the patterns they have learned may no longer hold true over time. To alleviate this issue, the recently popularized concept of In-Context learning has provided us with valuable insights. In this approach, large language models (LLMs) are exposed to multiple examples of input-label pairs, also known as demonstrations, as part of the prompt before performing a task on an unseen example. By thoroughly analyzing these demonstrations, LLMs can uncover potential patterns and effectively adapt to new tasks. Building upon this concept, we propose a Context-Informed drift-aware method for Stock Prediction (CISP), which continually adjusts to the latest market styles and offers more accurate predictions. Our proposed method consists of two key parts. Firstly, we introduce a straightforward and efficient technique for designing demonstrations that aggregate current market information, thereby indicating the prevailing stock market style. Secondly, we incorporate a prediction module with dynamic parameters, allowing it to appropriately adjust its model parameters based on the market patterns embedded in the aforementioned demonstrations. Through extensive experiments conducted on real-world stock market datasets, our approach consistently outperforms the most advanced existing methods for stock prediction.