We introduce here a novel dual-layer multibeam antenna with a folded Rotman lens as a compact beam forming network. The antenna is based on substrate integrated (SIW) technology. The objective is to reduce the overall size of the antenna system by folding the Rotman lens on two layers along the array port contour. The Rotman lens is folded by resorting to a novel transition based on an exotic reflector and several coupling vias holes. To validate the proposed concepts, an antenna system has been designed at 24.15 GHz. The radiating structure consists of a SIW slotted waveguide array made of fifteen resonant waveguides. The simulated results show very good scanning performances over an angular range of ±47°. It is also demonstrated that the proposed transition can lead to a size reduction of about 50% for the lens, and more than 33% for the overall size of the antenna.