To avoid fire disaster, the early alarm is very important for an effective fighting since the forest fires' area cannot be controlled, causes a serious threat to not only natural environment but also public safety. In this paper, we particularly study one model of forest fire detection using aerial videos. Different from static images from closed circuit television (CCTV), objects and background in the images captured by always-moving-UAVs are also moving. To overcome this big challenge, two stages of detection are used to enhance the detection rate. Fire-colored pixels are extracted using multi-color feature of fire. Optical flow algorithm is used to examine the motion characteristic of forest fires and extract fire motion pixels from its dynamic background. To combine the results, fire pixels are finally decided using rules. In order to validate the robustness of our model, a large database of videos of forest fires was collected from many sources, videos captured not only by Unmanned Aerial Vehicles (UAVs) but also by helicopters. The results show that the forest fires are point out effectively using our particular model with a high detection rate of 96.09% and a low false alarm rate of 3.91%. Moreover, the computation time also meets the demand of real time application of forest fire detection using video captured by UAVs.