An efficient resource management strategy can save network resources and improve network quality of service (QoS). In order to solve the problem of resource management difficulties of multiple SPs sharing the same backhaul network and QoS mismatch in heterogeneous networks, a downstream resource management in 5G virtualized backhaul network is proposed in this paper. Firstly, the wavelength assignment problem is described as a mathematics model with the consideration of minimizing turning overhead and load balancing. The quantum-behaved particle swarm optimization algorithm is used to solve the mathematics model. Secondly, the 5G traffic is mapping to different priority queues in TWDM-PON by the proposed QoS mapping algorithm. Finally, the downstream traffic is scheduled by the proposed resource scheduling algorithm. Numerical results show that the proposed mechanism can maintain high resource utilization, reduce wavelength tuning overhead and improve the delay of traffic.