Building performance design plays a key role in reducing the energy consumption of buildings. However, the widely used simulation-based design is facing several challenges, such as the labor-intensive modeling process and the performance gaps between design stage estimations and operational energy use. For these reasons, artificial intelligent methods are expected by designers to improve the efficiency and reliability of building energy-efficient design. To date, there has not been a practical data-driven design method of envelopes. This study aimed at exploring data-driven building energy-efficient design of envelopes based on their quantified impacts. A feature selection method and a game-theoretic method were applied to quantify the impacts of envelopes on space heating and cooling energy, which were performed on two building datasets, one of which is from the U.S. and the other from China. Random forest classifiers were developed to conduct the study. Based on discovered energy patterns and quantified impacts of envelopes on energy consumption, a rectified linear design method of envelopes was proposed with the idea of improving the performance of high-impact envelopes. Besides, a validation study was conducted on two office buildings in the hot-summer cold-winter region. To design the envelopes of a building, the data-driven analysis was driven by its similar buildings other than the whole dataset. Moreover, a detailed energy simulation was conducted to evaluate the energy performance of different design solutions. The results showed that compared with baseline design solutions, new strategies could save 1.05%–21.2% energy for space heating and cooling for these two case buildings. The proposed method is a general building envelope design approach and allows designers to easily find an energy-efficient configuration of envelopes. This study demonstrated the feasibility and effectiveness of the data-driven energy-efficient design of building envelopes.