Building patterns are crucial for urban form understanding, automated map generalization, and 3 D city model visualization. The existing studies have recognized various building patterns based on visual perception rules in which buildings are considered as a whole. However, some visually aware patterns may fail to be recognized with these approaches because human vision is also proved as a part-based system. This paper first proposed an approach for linear building pattern recognition combining convex polygon decomposition. Linear building patterns including collinear patterns and curvilinear patterns are defined according to the proximity, similarity, and continuity between buildings. Linear building patterns are then recognized by combining convex polygon decomposition, in which a building can be decomposed into sub-buildings for pattern recognition. A novel node concavity is developed based on polygon skeletons which is applicable for building polygons with holes or not in the building decomposition. And building’s orthogonal features are also considered in the building decomposition. Two datasets collected from Ordnance Survey (OS) were used in the experiments to verify the effectiveness of the proposed approach. The results indicate that our approach achieves 25.57% higher precision and 32.23% higher recall in collinear pattern recognition and 15.67% higher precision and 18.52% higher recall in curvilinear pattern recognition when compared to existing approaches. Recognition of other kinds of building patterns including T-shaped and C-shaped patterns combining convex polygon decomposition are also discussed in this approach.