This paper presents a new physics-based constitutive model to accurately describe the deformation behaviour of metals during ultrasonic vibration assisted (UVA) forming. Dislocation dynamics and acoustic energy transformation mechanisms in materials under ultrasonic vibration were considered in the modelling. A user defined subroutine was also developed for modelling the UVA forming processes using the finite element method. To assess the proposed model, upsetting forming, press forming, and incremental forming were simulated separately, and the predicted results were compared with their corresponding experiments. It was found that the application of the ultrasonic vibration can significantly decrease the material flow stress, making the material softer in forming. A large vibration amplitude results in a large reduction in flow stress. Relevant experimental measurements showed that the model has captured the material behaviour and the major mechanics in UVA forming. Different forming processes, due to their difference in tool-workpiece contact condition, would have different acoustic energy and stress transformation efficiency.