Hand tracking and hand trajectory analysis play important roles for the task of understanding human actions in videos. In this paper, we present a hand tracking method using proposal selection based on temporal information, hand detection and human pose estimation. We use color and motion features for hand tracker, and obtain hand detection proposals using Faster Region-based Convolutional Neural Network(RCNN). Stacked-hour-glasses network is used for human pose estimation to provide possible hand regions based on the spatial relationship between wrist and other joints. We use a selection method to re-initialize hand tracker based on the detected hand and wrist positions to correct the tracker error. The proposed method is evaluated on on a public dataset and on our own human interaction dataset. We achieve improved tracking accuracy for both datasets.