Posts Tagged ‘mod_rewrite’

|

Block Bad Bots Globally In Apache

Permanent Link posted January 15th, 2008

There are plenty of guides around to block bots using mod_rewrite however on a shared system with many domains, if you want to block a spoofed bot which is changing IPs you can block it globally from the user-agent in the apache httpd.conf.

In this example, MajesticBot+ had their bot spoofed for the past several months but the user-agent is different than the current version they themselves use (basically it’s an older version being spoofed).

mod_rewrite .htaccess code:
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot/v1\.0\.8.*$
RewriteRule .* - [F]

httpd.conf code:
SetEnvIfNoCase User-Agent “^MJ12bot(.*)1\.0\.8″ stay_out
<Location />
Order Allow,Deny
Deny from env=stay_out
Allow from all
</Location>

Note it only blocks 1.0.8 version of this bot which has been spoofed:

89.130.142.68 - - [15/Jan/2008:10:55:21 +0000] “GET /somepage HTTP/1.1″ 200 48167 “-” “MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)”

All others will be allowed to crawl your site.

Exact httpd.conf code obtained from this blog entry

Tags: , ,
Posted in Web Hosting | No Comments »

www. and non-www.

Permanent Link posted November 18th, 2007

This makes my blood boil. Why can’t server or website admins assume / expect that users use website addresses with and without the preceding www.? i.e. www.bbc.co.uk vs plain old bbc.co.uk.

I’ve come across two sites this morning alone. Pentagonglasstech.com (www works) and thetraffordcentre.co.uk (www works).

All you need to do, plebians, is alter the DNS slightly to modify the A record so it’s not on the www, but on the domain.com record then set up the www record as a CNAME to point back to the domain.com A record. It works the same as ftp.domain.com, mail.domain.com, these are all CNAME records.

If the admin is worried about the SEO aspect of having both www. and non-www. addresses in the index then use a mod_rewrite .htaccess directive to filter to essentially redirect www. to non-www. or vice versa (totally depending on your preference).

#www to non-www

RewriteCond %{HTTP_HOST} ^www\.domain\.com$ [NC]

RewriteRule ^(.*)$ http://domain.com/$1 [R=301,L]

or

#non-www to www

RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]

RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]

Tags: , ,
Posted in Web Hosting | No Comments »