Oral cancer evolves from different premalignant conditions and the key to save lives is through diagnosis of early symptoms. The conventional practice of post biopsy histopathology reporting is dependent on specificity of sampling site and optical coherence tomography (OCT) imaging is clinically used for guidance. Clinicians infer the tissue constitution by interpreting intensity images and are challenged by inter-and intra-observer variability. In this paper we propose transfer learning of tissue specific photon interaction statistical physics in swept-source OCT for characterizing the oral mucosa with the aim of reducing this reporting variability. The source task models statistical physics of ballistic and near-ballistic photons and its intensity attenuation and target task learns the parameters obtained by solving the source task to identify co-located heterogeneity of tissues. Performance is compared with conventional histopathology of healthy, premalignant and malignant oral lesions supporting its use towards in vivo histology of the oral mucosa for pre-biopsy screening.