2012
2008
2007
2006
marc-o.net » » Wikio
Wikio serait le nom du projet de nouveau moteur de recherche annoncé par Pierre Chappaz.
Yahoo! : How can I reduce the number of requests you make on my web site?
User-agent: Slurp
Crawl-delay: 20
2005
Robots.txt, The Big Crawl
(via)While testing our new robots.txt validator and checker, we needed some test fodder. We found more than 5% of the robots.txt used bad style and up to 2% were so badly formed that they would not be recognized by any spider. The following lists some of the problems we discovered.
