Globalization impacts directly pollution, and one of the side effects is the increasing number of forest fires and their fast propagation. To reduce this phenomenon, several studies have tried to model fire propagation and find an adequate method to predict (or follow) this spread to stop it. However, to the best of our knowledge, most of the developed works do not use an adaptive exploration strategy to follow the fire evolution. Instead, they are mainly focused on observing the evolution and trying to find an adequate spread model to predict fire behavior. Due to the lack of all the exact parameters to predict the fire spread, in this paper, we propose a new exploration strategy based on reinforcement learning using an Unmanned Aerial Vehicle (UAV). The learned exploration strategy is able to predict and follow the fire propagation. To follow the generated waypoints by the exploration strategy, we develop a new robust controller, which can be embedded in a UAV. All these modules are encapsulated and interact with each other using a synchronized architecture. To prove the efficiency of our proposal, we present the obtained results on several simulated forest fire scenarios and compare them with other approaches. Then, We validate our entire architecture and show its efficiency using software in the loop environment using the Robot Operating System (ROS).