We present an immersed boundary projection method for flow-structure interaction problems involving rigid bodies with complex geometries. Dynamics of a rigid body interacting with fluid flow and kinematics of other rigid bodies undergoing prescribed motions are coupled implicitly with the incompressible vorticity equations. In particular, the method is formulated in a frame of reference fixed on the rigid body under flow-structure interaction (the target) so that the coupled system can be solved non-iteratively, accurately, and efficiently. The implicit coupling of the fluid solver and dynamics and kinematics of rigid bodies ensures the method being stable for low solid-to-fluid mass ratios. The influence of fictitious fluid inside the rigid body is considered and the spurious oscillations in surface stresses are filtered to impose physically correct rigid body dynamics. Similar to many predecessors of the immersed boundary projection method, the resulting discrete system is solved efficiently using a block-LU decomposition. We then validate the method with two-dimensional test problems of a neutrally buoyant cylinder migrating in a planar Couette flow and a freely falling or rising cylindrical rigid body.