Register Members List Search Today's Posts Mark Forums Read

Mod Options
Old 01 Apr 2008, 08:52
SEOvB's Avatar
SEOvB SEOvB is offline
Join Date: May 2007
Real name: Jarvis
Google will usually only send one spider thats really efficent, its probably that pesky yahoo bot which is killing you.

Use robots.txt to slow the spiders, you can see what should be in a standard vBulletin robots.txt and where to put it on your server, and then just adding something like

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

to the end of it will slow the bots. These changes probably won't take effect for 24hrs, as most spiders only download robots.txt file once a day.
vBulletin Services and vBulletin Hosting
Reply With Quote
Old 01 Apr 2008, 16:31
MikeHolohan's Avatar
MikeHolohan MikeHolohan is offline
Join Date: Mar 2007
Wow thanks for that...

is the crawl delay value in minutes like 120 minutes?

thanks again.. Mike.

--------------- Added 01 Apr 2008 at 20:36 ---------------

worked a charm, I have only 8 spiders on my site now... thanks for the tip...

Last edited by MikeHolohan; 01 Apr 2008 at 20:36. Reason: Auto-Merged DoublePost
Reply With Quote
Old 10 Apr 2008, 09:01
edgecutioner edgecutioner is offline
Join Date: Jul 2007
Location: Qatar
Real name: ???
Originally Posted by MikeHolohan View Post
Ok thanks, is there a recommended hosting company by vbulletin?
Well no.

Originally Posted by edgecutioner's brain
Can Jelsoft run a hosting company for vB just like IPB does? I mean, that would be cool.....
Reply With Quote
Old 15 Apr 2008, 09:55
Hornstar's Avatar
Hornstar Hornstar is offline
Join Date: Jun 2005
Real name: Matt
Personally I would much prefer a million spiders and spend a bit more to be on a better server. Anyway to ensure all my content is being indexed at a faster speed in the hope to attract more members for the content that i'm trying to drive members to is important IMO.

As for a good host, there are many, it does not seem like your forums are that busy or large currently and you can still get away with shared hosting until you start to get quite a lot larger where you start using too many resources for that server/host where it would become unfair to the other people on the server.
Some hosts are good with their tos in this respect, others have things in place where you use x amount cpu you will be suspended and the value is not even that high sometimes.

I run a hosting company (wont release the site here) but I feel if a hosting company is ever suspending or terminating sites without giving adequate warning and advise before hand, then you should never go with that host.

Remember in most cases you are paying for what you get. In some cases you may find a golden treasure but for just how long will it last? Take a look at how long they have been around, look for reviews etc. WHT can be a good place to start looking for hosts as well.

I wish you luck in finding a good host
Reply With Quote

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

New To Site? Need Help?

All times are GMT. The time now is 06:32.

Layout Options | Width: Wide Color: