Portal Home > Knowledgebase > Articles Database > Yahoo bot
Yahoo bot
| Posted by Richard XI, 06-25-2008, 02:38 AM |
| For several months the Yahoo bot had been controlled to one visit per 40 seconds with a robots.txt page. Today it is not working and creating several page impressions per minute.
Has anyone seen any new advice on how to control this pesky creature?
This is the file content I have been using:
User-agent: Slurp
Crawl-delay: 40
User-agent: *
Disallow:
|
| Posted by BookMarc, 06-25-2008, 08:39 AM |
| deny access to it... for sometime using .htaccess
|
| Posted by Lsupport, 06-25-2008, 02:45 PM |
| Its worth checking this site http://www.botsense.com/
|
| Posted by RBBOT, 06-25-2008, 03:41 PM |
| This may be the reason for the change, but doesn't give any detail as to exactly why its had this effect on your site:
http://www.ysearchblog.com/archives/000531.html
Although the rollout started 2 months ago, it sometimes takes a while for it to start affecting all sites as the phase the deployment.
|
| Posted by Richard XI, 06-26-2008, 01:53 AM |
| Thanks for your comments.
I have blocked access to the IP and will see if the Yahoo bot behaves itself next month when I'll take the IP block off for a while.
It's strange that no one else seems to have had the same problem.
|
Add to Favourites 
Print this Article
Also Read
