Nowadays, the increasing complexity of the social environment brings much difficulty in group decision making. The more uncertainty exists in a decision-making problem, the more collective wisdom is needed. Therefore, large scale group decision making has attracted a lot of researchers to investigate. Since the probabilistic linguistic terms have impressive performance in expressing DMs’ opinions, this paper proposes a novel method for large scale group decision making with probabilistic linguistic preference relations. More specifically, (1) a probability k-means clustering algorithm is introduced to segment DMs with similar features into different sub-groups; (2) an integration method is proposed to construct the collective probabilistic preference relation that retains initial information to the most extent; (3) taking the personality of each DM into account, a consensus model is constructed to improve the rationality and efficiency of consensus reaching process. Several simulation experiments are designed to analyze the influence factor in the feedback mechanism and make some comparative analysis with the existing method. Finally, an illustrative example of contractor selection is conducted to verify the validity of the proposed method.