Decision rules have been successfully used in various classification applications because of their interpretability and efficiency. In many real-world scenarios, especially in industrial applications, it is necessary to generate rule sets under certain constraints, such as confidence constraints. However, most previous rule mining methods only emphasize the accuracy of the rule set but take no consideration of these constraints. In this paper, we propose a Confidence-constraint Rule Set Learning (CRSL) framework consisting of three main components, i.e. rule miner, rule ranker, and rule subset selector. Our method not only considers the trade-off between confidence and coverage of the rule set but also considers the trade-off between interpretability and performance. Experiments on benchmark data and large-scale industrial data demonstrate that the proposed method is able to achieve better performance (6.7% and 8.8% improvements) and competitive interpretability when compared with other rule set learning methods.