All crawling approaches basically need to solve two problems:
* Search Engines must be able to discover URLs, which should be crawled.
* Your server must be able to respond to these URLs, when the Search Engine crawler asks for it.
Let’s first discuss these two problems and possible solutions. Afterwards i will discuss two common approaches adressing both problems.
Via Jan Hesse