Portal Home > Knowledgebase > Articles Database > Need mod_security rules to prevent Googlebot from crawling one file
Need mod_security rules to prevent Googlebot from crawling one file
| Posted by vectro, 02-07-2012, 12:02 AM |
| I need mod_security rules to prevent Googlebot from indexing any file named browse.php anywhere on the server, while still allowing Googlebot to access anything else. I figured mod_security will do the trick because it can recognize user-agents and set rules accordingly.
Any ideas?
|
| Posted by RoseHosting, 02-07-2012, 04:52 AM |
| You can use robots.txt to disallow search engine crawling of specific directories or pages on your web site.
|
| Posted by Server Management, 02-07-2012, 06:52 AM |
| Beat me to it - Using a robots.txt file is the correct way to do what your wanting to do...
|
| Posted by vectro, 02-07-2012, 10:02 PM |
| robots.txt only affects one site at a time. I need this rule to be server-wide. There are hundreds of domains and it would take forever to use robots.txt.
|
| Posted by vectro, 02-13-2012, 08:09 PM |
| I did some research on creating mod_security rules and figured this out myself. Here is a server-wide mod_security rule for the main Apache configuration which will keep Googlebot off of 1 certain file. This only applies if the file shows up in the root directory of a domain, but it applies to all domains on the server.
Change file.php to the name of the file you want to protect. The part that says "Googlebot" can also be changed to any user-agent. It's a pattern match and not an explicit match. This means the full user-agent simply needs to include the word for the rule to apply.
|
Add to Favourites 
Print this Article
Also Read
