Digital twin (DT) has emerged as a promising technology to create digital replicas of physical vehicles to facilitate driving in Internet of Vehicles (IoV). However, due to the uneven distribution of vehicles in the networks and the dynamics of heterogeneous wireless links, how to optimize the connection along the vehicle's trip between the vehicle and its DT to achieve the lowest cost ( i.e. , delay and energy consumption) is a fundamental challenge. In this paper, we investigate the joint network selection and power level allocation problem in the data synchronization issue between vehicles and DTs in the heterogeneous access networks. In order to better use the information shared by DT to facilitate the data synchronization process, we propose a learning-based heterogeneous network selection scheme in DT empowered IoV. First, we consider the data divided into blocks and model the data synchronization problem as a Markov Decision Process (MDP) to minimize the long-term cost in terms of the delay and energy consumption. Based on the information obtained from the digital twin communications, we develop an actor-critic learning approach to solve the MDP problem with the block-level state method, and obtain the optimal network selection strategy for the vehicles. In addition, we explore the capability of digital twins by applying transfer learning. The vehicles are divided into two types, i.e. , learner vehicles and expert vehicles, so that the knowledge can be transferred from the expert vehicles to the learner vehicles to effectively improve the learning efficiency and decrease the data synchronization latency. Extensive experiments demonstrate that our scheme can effectively find the optimal network selection policy and achieve the fast convergence and better performance compared with existing related approaches.