Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Remove Spiders from Who's Online Details »
Remove Spiders from Who's Online
Mod Version: 1.00, by mfyvie (R.I.P) mfyvie is offline
Developer Last Online: May 2008 I like it Show Printable Version Email this Page

This modification is in the archives.
vB Version: 3.6.x Rating: (9 votes - 4.44 average) Installs: 66
Released: 15 Jul 2007 Last Update: 17 Jul 2007 Downloads: 357
Not Supported DB Changes Uses Plugins  

*** Staff note: The author of this modification has passed away in a diving accident. We wish his family all strength in dealing with this traggic issue. ***

Remove Spiders from Who's Online

The problem

The majority of the the "guests" on your forum are probably spiders. While you might think that it is impressive to show so many guests using your forum, this does not reflect the true number of people online. The who's online page does offer visitors the chance to change the display based on members, guests, or spiders, however the totals (and the record number of users online at one time) do not exclude spiders.

This mod does not remove spiders from your forum, it simply stops them being counted in the totals.

Some search engine spiders such as Yahoo's Slurp can initiate hundreds of connections at the same time (from different IP addresses). Each one of these connections will be included in your totals as a "guest".



What does this modification do?
  • Completely remove all known spiders (spiders found in your includes/xml/spiders_vbulletin.xml file) from the who's online display
  • Updates the true guest totals displayed on the main forum page
  • Resets your maximum users online record on installation
  • Includes changes for vBadvanced's online users module
  • Doesn't offer separate spider statistics or configuration, it just makes it appear like those spiders aren't on your forum (even though they are)
  • Easy access through template conditionals and variables to display different behaviours based on whether the connection is from a spider or not.
Who should use this modification?

Anyone who wants to their forum to show the true number of users online in the statistics

I run a big system, will this slow my board down?

Absolutely not. It's been written with big boards in mind. Even if you use a very large spiders_vbulletin.xml file, this file will only be checked once for each new session created. After that, whether the session is used by a spider is recorded in the session table. For users running vBulletin 3.6.6 or later, it will actually increase the speed of the who's online page, since known spiders are no longer compared against the list of known spider user agents. No new database queries are added, as existing queries are used or modified instead.

Hasn't this been done before?

There are a couple, but I've taken a slightly different approach with this mod - my focus was on simplicity and performance. You might also want to check out some other mods here and here, to see if they suit your purposes better.



Installation instructions
  • If using vBadvanced, see the instructions below for a manual edit of your onlineusers.php file
  • Install the enclosed .xml file via AdminCP -> Plugins & Products -> Manage Products -> Add/Import Product. Remember to click overwrite if upgrading
  • Wait a few minutes - existing sessions must have some activity before they are reclassifed as a genuine member or guest
  • (optional) Edit your admincp quick statistics file (instructions below)
  • (optional) Reset your maximum online statistics again (if you feel the need, but it is done automatically during installation)
Configuration instructions

Nothing to configure! Just see the note below about spiders_vbulletin.xml

How can I make this more effective?

Spiders are identified via the includes/xml/spiders_vbulletin.xml file. However, the version that ships with vBulletin is quite small, and will recognise only the most popular spiders. Anything else will still show up in your statistics as a guest, making it hard to tell real guests from spiders.

Therefore it is recommended to update your spiders_vbulletin.xml if using this mod. You can obtain a much better version of this file from Christian Stadler here. Please contribute new user agent strings directly to him and encourage him to keep providing new updates of this file.

Will using a big spiders_vbulletin.xml slow things down? Theoretically yes, but in practical terms now. Lookups are only one once for each session when it is created. If you are running a version earlier than vBulletin 3.6.6 it will mean that more work must be done on the who's online page, but nowhere else.

Which versions of vBulletin will this work on?

This mod is designed to work for 3.6.6 and above, but there is some additional code included to handle 3.6.0 - 3.6.5 as well.

Optional extras

If you are using vBAdvanced then you'll want to modify your online users module. The following change should work for but versions 2.x and 3.x. Go to your modules/onlineusers.php file and locate this line:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Directly under this line, insert the following line (on a line by itself):

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Please don't forget to include the quotes exactly as they appear above.

If you'd like to also include the quick stats in your admincp with this modification open your admincp/index.php file and locate this line:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

replace it with:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Would you like to reset your maximum online users statistic (the one on the front page or who's online page that tells you the most number of users ever online)? go to the AdminCP -> Maintenance -> Execute SQL Query and run the following query:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

There is an additional file, spider_test.php that can be used to test the operation of this mod, as well as giving you a handy overview of which guests have not been classified as spiders, but probably should be. You can then use the user agent strings to expand your spiders_vbulletin.xml file. You can run spider_test.php from any forum directory, just upload it and then type the path to the file into your browser.

Support

Will be provided to those who click install

Version history

1.0 (15.07.2007) Initial version

Disclaimer: No actual spiders where harmed during the development of this modification.

Download Now

Only licensed members can download files, Click Here for more information.

Screenshots

Click image for larger version

Name:	kill_spiders_stats.gif
Views:	605
Size:	5.7 KB
ID:	67003  

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
Comments
  #2  
Old 15 Jul 2007, 14:59
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Real name: Mark
Reserved
Reply With Quote
  #3  
Old 15 Jul 2007, 16:02
TTG's Avatar
TTG TTG is offline
 
Join Date: May 2004
Yahoo slurp are the biggest PITB but I just use a robots.txt to reduce the times they are allowed to visit. Works well enough for me. Whats the benefit in this compared to a simple robots.txt ?
User-agent: *
Crawl-delay: 30
Disallow:
User-agent: Slurp
Crawl-delay: 1500
Reply With Quote
  #4  
Old 15 Jul 2007, 16:07
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Real name: Mark
Robots.txt works only for those bots that respect it, and identifying every single bot to put into a robots.txt can also be time consuming. There is also a performance overhead associated with robots.txt. I block all my bad bots directly in the webserver, but all the "good bots" to crawl my site.

However, everything that I've written above (and your question) relating to blocking bots from the site.

This mod simply removes the spiders from the totals, allowing you to get a better idea of how many "real" guests you have on your forum. If you keep statistics and graphs as I do this gives you a clearer picture of growth, etc.

It also saves questions from users saying "Wow, there were 800 guests online last night, this forum is really popular".
Reply With Quote
  #5  
Old 15 Jul 2007, 17:14
iogames's Avatar
iogames iogames is offline
 
Join Date: Jan 2007
Real name: Josh Arcadia
What can I say???
__________________
. █│║▌│█│║▌║│█║▌│║║█║
Reply With Quote
  #6  
Old 15 Jul 2007, 18:15
FleaBag's Avatar
FleaBag FleaBag is offline
 
Join Date: Dec 2001
Hi Mark! A lovely little hack, when you said you have a bunch o' "why didn't anyone else think of that" hacks, you were right!

I will install sometime soon! And on another note, apologies I have not replied to your email (I'm not ignorant, honest) - I just forgot I'd emailed you from my domain account (and not my Gmail account) - I couldn't work out where your email had gone!
__________________
AndroidETC.com :: Android, and other stuff.
Reply With Quote
  #7  
Old 15 Jul 2007, 18:15
keyness keyness is offline
 
Join Date: Oct 2006
You know there are some good mods, for example in my forum it seems like this: (14 members & 16 guests & 1 logged out & 32 bots) But I really would like to remove bots from who's online without reducing number in forumhome. Could it be possible?
Reply With Quote
  #8  
Old 15 Jul 2007, 18:24
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Real name: Mark
Originally Posted by keyness View Post
You know there are some good mods, for example in my forum it seems like this: (14 members & 16 guests & 1 logged out & 32 bots) But I really would like to remove bots from who's online without reducing number in forumhome. Could it be possible?
You could selectively disable the plugins from this mod to remove spiders from who's online, but leave them on the main forum page if you want.
Reply With Quote
  #9  
Old 15 Jul 2007, 18:47
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Real name: Mark
Some of you might be wondering what is so different about this mod, so I'll take a little bit of time to explain how you might use it to do some different stuff.

This mod actually classifies each session as spider or normal user when the session is created and stores the result in the session table. Why? Because it makes it very easy for us to check the "spider status" on any session from any page, WITHOUT having to compare the user agent against a list of known user agents every time.

Why is this better? Two reasons, the first is speed and the second is flexibility. By default the mod will suppress the spiders from the statistics (it won't block anything). But we could use this mod to do more if we wanted.

If you look in the session table you'll see an extra column called "spider", the values are -1 (normal user), 0 (not yet determined - you'll rarely see this) and 1 (spider). Now you can easily change things depending on whether a spider is viewing the page or not. You could put something in a template like this:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Now that text would appear only to spiders. Along the same lines you could show text only to real users like this:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

You could also use it inside any plugin simply by testing the value of $vbulletin->session->vars['spider'] like this:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Putting a variable into the session table to use elsewhere is the same kind of approach used in GLA (some of you might have seen this recently).

So with a little bit of imagination some of you might also be able to see that this mod could help you do all sorts of new things (I've seen many requests for example from people who want to display different links or pages to spiders than normal users - this would be simple using the above techniques).

What if you wanted to get some quick statistics to show how many users or guests are online. Easy - now you can use a simple SQL query like this:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

This would show the number of non-spider guests in the connection table (but you'd have to remember to add something onto the end to limit the last activity to the last 15 minutes (or whatever you have it configured to), so the final code in a plugin might look something like this:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Again you could vary it to something like userid > 0 for members, or spider = 1 for the spiders. This way you can get the totals quickly from any page with just a single line of code and a single query.
Reply With Quote
  #10  
Old 15 Jul 2007, 21:14
Quantnet Quantnet is offline
 
Join Date: Nov 2006
thanks
Reply With Quote
  #11  
Old 15 Jul 2007, 21:33
ERuiz ERuiz is offline
 
Join Date: Feb 2005
I installed this mod and made the corresponding changes to the onlineusers.php file, but when I access my homepage, it says there are some guests on the who's online module block but if I go to the forum who's online display, it says there are no guests. What could be wrong?
Reply With Quote
  #12  
Old 15 Jul 2007, 21:44
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Real name: Mark
Originally Posted by ERuiz View Post
I installed this mod and made the corresponding changes to the onlineusers.php file, but when I access my homepage, it says there are some guests on the who's online module block but if I go to the forum who's online display, it says there are no guests. What could be wrong?
Point 3 of the installation instructions - wait a few minutes

When you first install the mod you will get different values for the numbers of guests in your vbadvanced block, the who's online total, and the total listed on the main forum index. Once your cookie timeout has expired (15-30 minutes) these values should converge to be the same. The reason is that you may be seeing old sessions which (because they are not active) have not been reclassified to either spider or non-spider in the session table.
Reply With Quote
  #13  
Old 15 Jul 2007, 21:52
ERuiz ERuiz is offline
 
Join Date: Feb 2005
Originally Posted by mfyvie View Post
Point 3 of the installation instructions - wait a few minutes

When you first install the mod you will get different values for the numbers of guests in your vbadvanced block, the who's online total, and the total listed on the main forum index. Once your cookie timeout has expired (15-30 minutes) these values should converge to be the same. The reason is that you may be seeing old sessions which (because they are not active) have not been reclassified to either spider or non-spider in the session table.
Thanks for the quick response. Ok, I will give it 1 hour, LOL. Thanks again...
Reply With Quote
  #14  
Old 15 Jul 2007, 22:01
ERuiz ERuiz is offline
 
Join Date: Feb 2005
I did the edit for the control panel option and I get an error... It says it can't find the session table.

I had to manually add vb3_ (my prefix) to the session.spider portion of the code you provided and this got rid of the error.

I am using 3.6.7 and VBA RC1
Reply With Quote
  #15  
Old 15 Jul 2007, 22:32
mfyvie mfyvie is offline
 
Join Date: Mar 2007
Real name: Mark
Originally Posted by ERuiz View Post
I had to manually add vb3_ (my prefix) to the session.spider portion of the code you provided and this got rid of the error.
Thanks for pointing that out - I checked my example above and changed it slightly. I dropped off the "sessions." part which means it should work without a prefix. I also noticed a small problem with that example - the value to test for was 0, it should have been -1. You should change yours accordingly otherwise it will report your number of guests as 0.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 07:16.

Layout Options | Width: Wide Color: