C Olston, S Pandey - Proceedings of the 17th international conference …, 2008 - dl.acm.org
It is crucial for a web crawler to distinguish between ephemeral and persistent content.
Ephemeral content (eg, quote of the day) is usually not worth crawling, because by the time …