In this work, we present a personalised participant independent pain recognition system based on the video channel. Instead of using an entire annotated dataset to train a classification model that would be later applied to an unseen participant, a similarity metric is used to select the most interesting annotated samples based on the data of the unseen participant. These samples are subsequently used to train a model adapted to the unseen participant. The selection process helps to avoid redundant and irrelevant data samples, thus improves the performance as well as the efficiency of the trained model. From the video channel, several features are extracted and subsequently fed into an hierarchical fusion architecture to further improve the performance of the system.