Register Members List Search Today's Posts Mark Forums Read

Reply
 
Article Options
.htaccess for webmasters
TheSupportForum
Join Date: Jan 2007
Posts: 1,158

by TheSupportForum TheSupportForum is offline 28 Dec 2009

Description:

for many webmasters who log visitors and allow spiders to crawl their site this guide will help you with Bad robots, spiders, crawlers and harvesters


Require the www
(please note this has been tested on vBulletin 4.0.6 Gold)
if this does not work , click remember me before you login
or you could try clearing your browser cache before you test it


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Replace:
www\.ereptalk\.co\.uk
and
www.ereptalk.co.uk
With:
Your domain name

Replace:
http://www.tutorials4you.co.uk/
With:
Your domain name


Loop Stopping Code
Sometimes your rewrites cause infinite loops, stop it with one of these rewrite code snippets.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Fix for infinite loops

An error message related to this isRequest exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.or you may seeRequest exceeded the limit,probable configuration error,Use 'LogLevel debug' to get a backtrace, orUse 'LimitInternalRecursion' to increase the limit if necessary


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Prevent Files image/file hotlinking and bandwidth stealing


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Replace:
http://(www\.)?askapache.com/
With:
Your Domain Name


Stop browser prefetching


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

This module uses a rule-based rewriting engine (based on a regular-expression parser) to rewrite requested URLs on the fly. It supports an unlimited number of rules and an unlimited number of attached rule conditions for each rule, to provide a really flexible and powerful URL manipulation mechanism. The URL manipulations can depend on various tests, of server variables, environment variables, HTTP headers, or time stamps. Even external database lookups in various formats can be used to achieve highly granular URL matching.

This module operates on the full URLs (including the path-info part) both in per-server context (httpd.conf) and per-directory context (.htaccess) and can generate query-string parts on result. The rewritten result can lead to internal sub-processing, external request redirection or even to an internal proxy throughput.


How to prevent or allow directory listing


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

The above line enables Directory listing.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

The above disables directory listing for your web site.



Block Bad robots, Spiders, Crawlers and Harvesters


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.


Last edited by TheSupportForum; 19 Aug 2010 at 11:21.. Reason: change to .htaccess file
Views: 8152
Reply With Quote
Comments
  #2  
Old 01 Jan 2010, 06:59
lazydesis lazydesis is offline
 
Join Date: Sep 2006
very nicely done

thanks
__________________
http://www.lazydesis.com
Reply With Quote
  #3  
Old 02 Jan 2010, 11:13
leodestroy's Avatar
leodestroy leodestroy is offline
 
Join Date: Jul 2008
simonhind, can you help with htacces option?
We can't add site to partner sape.ru if use .htaccess
Reply With Quote
  #4  
Old 02 Jan 2010, 12:47
TheSupportForum TheSupportForum is offline
 
Join Date: Jan 2007
Originally Posted by leodestroy View Post
simonhind, can you help with htacces option?
We can't add site to partner sape.ru if use .htaccess

you need to exlain what you are tring to do

what steps are you taking to add site to partner sape.ru
__________________
http://www.multihunters.co.uk - all your coding needs
Reply With Quote
  #5  
Old 02 Jan 2010, 14:24
leodestroy's Avatar
leodestroy leodestroy is offline
 
Join Date: Jul 2008
I add my site to sape.ru. But it scans only the pages of forums and blogs. CMS does not recognize. It seems to me that problem in redirect
Reply With Quote
  #6  
Old 02 Jan 2010, 16:43
TheSupportForum TheSupportForum is offline
 
Join Date: Jan 2007
Originally Posted by leodestroy View Post
I add my site to sape.ru. But it scans only the pages of forums and blogs. CMS does not recognize. It seems to me that problem in redirect

fist thing you need to check in your .htaccess file is the following


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

this should only appear once

this should be below


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

end result should be


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.


i hope this helps
__________________
http://www.multihunters.co.uk - all your coding needs
Reply With Quote
  #7  
Old 03 Jan 2010, 03:08
leodestroy's Avatar
leodestroy leodestroy is offline
 
Join Date: Jul 2008
simonhind, sape.ru is not my domain name. This is partner program for web master. Its robots crawl only forum and blogs. A CMS is passed
Reply With Quote
  #8  
Old 25 Feb 2010, 17:14
darren1981 darren1981 is offline
 
Join Date: Oct 2008
this sounds like a problem on sape.ru maybe their bot is programed for only blogs and forums
Reply With Quote
  #9  
Old 06 May 2010, 09:28
abdobasha2004's Avatar
abdobasha2004 abdobasha2004 is offline
 
Join Date: Aug 2008
thanks a lot
really useful
__________________
Egypt News website, latest Egyptian news updated instantly.
Reply With Quote
  #10  
Old 25 Sep 2010, 07:34
as7apcool's Avatar
as7apcool as7apcool is offline
 
Join Date: Feb 2009
thianks 4 good work
Reply With Quote
  #11  
Old 05 Oct 2010, 16:11
avsunforum avsunforum is offline
 
Join Date: Feb 2008
thank you
good work 3.8
Reply With Quote
  #12  
Old 23 Oct 2010, 04:07
Tanapangarap's Avatar
Tanapangarap Tanapangarap is offline
 
Join Date: Dec 2007
Real name: Kevin
Originally Posted by simonhind View Post
Prevent Files image/file hotlinking and bandwidth stealing


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Replace:
http://(www\.)?askapache.com/
With:
Your Domain Name
Hi.

Question!

Sorry for my ignorance, but when you say to replace http://(www\.)?askapache.com/ with my URL, does that mean http://(www\.)?myurlhere.com/ or http://myurlhere.com/ In other words, I would like to add my URL without the "www.", but I do not know if I should remove the "(www\.)?"
__________________
Join The Infinity Program, my den of villains and swashbucklers.

My latest article: "The effects of a shoutbox on a forum community."
Reply With Quote
  #13  
Old 26 Oct 2010, 23:58
TheSupportForum TheSupportForum is offline
 
Join Date: Jan 2007
replace
http://(www\.)?askapache.com/
with for example

http://(www\.)?me.com/
http://(www\.)?website.com/

etc....

hope this makes it easier
__________________
http://www.multihunters.co.uk - all your coding needs
Reply With Quote
  #14  
Old 19 Dec 2010, 21:56
GONUMBER6's Avatar
GONUMBER6 GONUMBER6 is offline
 
Join Date: Jan 2010
Real name: Lisa
Hi, I am trying to block the 100+ Baiduspiders that are crawling my site all the time. All I need to do is copy/paste the bottom box to my .htaccess file? Where do I add that code? At the bottom of my existing file?
Reply With Quote
  #15  
Old 27 Dec 2010, 17:23
final kaoss final kaoss is offline
 
Join Date: Apr 2006
anywhere in the .htaccess file
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Article Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 02:04.

Layout Options | Width: Wide Color: