Microbial communities in ancient marine sediments composed of clay and silt obtained from the terrestrial subsurface were phylogenetically analyzed based on their 16S rRNA gene sequences. Chloroflexi and Miscellaneous Crenarchaeotic Group were predominant in bacterial and archaeal clone libraries, respectively. Of 44 operational taxonomic units (OTUs) that had close relatives in the database, 30 were close to sequences obtained from marine environments. Some sequences belonged to the candidate groups JS1, ANME-I, and Marine Benthic Group-C, which are typically found in marine sediments. Low chloride concentrations in the sediments suggest that these marine-affiliated sequences may not reflect currently active microbial communities. Our results indicate the existence of long-term preserved DNA or descendants of ancient oceanic microbial components in subsurface muddy sediments in a temperate region, which may reflect indigenous population of paleoenvironments.