We consider the problem of uplink channel estimation for millimeter wave (mmWave) systems, where the base station (BS) and mobile stations (MSs) are equipped with large antenna arrays to provide sufficient beamforming gain for outdoor wireless communications. Hybrid analog and digital beamforming structures are employed by both the BS and the MS due to hardware constraints. We propose a layered pilot transmission scheme and a CANDECOMP/PARAFAC (CP) decomposition-based method for joint estimation of the channels from multiple users (i.e., MSs) to the BS. The proposed method exploits the intrinsic low-rank structure of the multiway data collected from multiple modes, where the low-rank structure is a result of the sparse scattering nature of the mmWave channel. The uniqueness of the CP decomposition is studied, and the sufficient conditions for essential uniqueness are obtained. The conditions shed light on the design of the beamforming matrix, the combining matrix, and the pilot sequences, and meanwhile provide general guidelines for choosing system parameters. Our analysis reveals that our proposed method can achieve a substantial training overhead reduction by leveraging the low-rank structure of the received signal. Simulation results show that the proposed method presents a clear advantage over a compressed sensing-based method in terms of both estimation accuracy and computational complexity.