This paper presents a hierarchical method for matching line segments from two images. Line segments are matched first in groups and then in individuals. While matched in groups, the line segments lying adjacently are intersected to form junctions. At the places of the junctions, the structures are constructed called Line–Junction–Line (LJL), which consists of two adjacent line segments and their intersecting junction. To reliably deal with the possible scale change between the two images to be matched, we propose to describe LJLs by a robust descriptor in the multi-scale pyramids of images constructed from two original images. By evaluating the description vector distances of LJLs from the two images, some candidate LJL matches are obtained, which are then refined and expanded by an effective match-propagation strategy. The line segments used to construct LJLs are matched when the LJLs they formed are matched. For those left unmatched line segments, we match them in individuals by utilizing the local homographies estimated from their neighboring matched LJLs. Experiments demonstrate the superiorities of the proposed method to the state-of-the-art methods for its robustness in more difficult situations, the larger amounts of correct matches, and the higher accuracy in most cases.