Neurofibrillary tangle (NFT) pathology is linked to neurodegeneration in the medial temporal lobe (MTL). Using a tailored pipeline, we correlated atrophy rate, as measured from retrospective longitudinal MRI, with NFT burden, measured from 18F-AV-1451 PET, within MTL regions of earliest NFT pathology. In amyloid-ß positive but not amyloid-ß negative individuals, we found significant correlation between 18F-AV-1451 uptake and atrophy rate that was strongest in the transentorhinal cortex, the first region with NFT pathology. This supports the role of NFTs in driving neurodegeneration and the utility of 18F-AV-1451 PET and structural measurement of transentorhinal cortex in tracking early tau-mediated disease progression.