Translating a nighttime thermal infrared (NTIR) image into a daytime color image, denoted as NTIR2DC, may be a promising way to facilitate nighttime scene perception. Despite recent impressive advances in the field of image-to-image translation, content distortion in the NTIR2DC task remains under-addressed. Previous approaches resort to annotated pixel-level labels to enforce semantic invariance during translation, which greatly limits their practicality. To ameliorate this intractability, we propose a generative adversarial network with Dynamic LAbel Mining called DlamGAN, which encourages a semantically consistent translation without requiring pixel-wise manual annotation. Specifically, a dynamic label mining strategy is proposed to extract the semantic cues of NTIR images online. The obtained cues are served as pseudo labels to constrain the semantic consistency in the translation process. In addition, an adaptive surrounding response integration module is introduced for adaptively integrating contextual information to reduce coding ambiguity of NTIR images. Experimental results on FLIR and KAIST datasets demonstrate the superiority of the proposed model in semantic preservation.