A multibeam antenna system fed by a compact Rotman lens is presented in multilayer substrate integrated waveguide (SIW) technology. The lens is implemented in two layers using a new transition based on several star-shaped coupling slots and an SIW integrated reflector. Compared to standard rectangular coupling slots, this transition is broadband and maximizes power transfer between the two layers of the lens regardless the position of the beam port along the focal arc of the Rotman lens. The radiating part is an array of 15 slotted waveguides. These waveguides are centrally fed to enhance their bandwidth and reduce the “long line effect.” To validate the proposed concepts, an antenna system has been prototyped at 24.15 GHz. The designed transition leads to a physical size reduction of about 50% for the Rotman lens, and more than 33% for the antenna footprint compared to standard single-layer implementations. The experimental results demonstrate a scanning range of ±33° and a 4% bandwidth for VSWR <; 2; the isolation between input ports is better than -10 dB.