Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Nexia's Web Crawlers Bundle Details »»
Nexia's Web Crawlers Bundle
Mod Version: 1.00, by vbenhancer (Coder) vbenhancer is offline
Developer Last Online: Nov 2012 I like it Show Printable Version Email this Page

vB Version: 3.8.x Rating: (2 votes - 4.50 average) Installs: 24
Released: 05 Mar 2012 Last Update: Never Downloads: 82
Not Supported DB Changes Uses Plugins Additional Files  

A bundle of features for the Spiders access. Hits tracking, complete listing for info, Specific usergroup for Spiders, etc

It's globally a merge with some small addons i wrote in the past, and as i did not want to release a ton of minimal tools that just fit together, i make a real bundle, 4 or 5 tools together, with activation and permissions settings when needed.

Spiders List: a little spiders tracker for your forum. It's not tracking each page the engine is viewing, because this is pointless. Instead, It is listing the name of the spiders that visit your sites, the last date of a visit, the number of unique visits and the number of pages viewed. That information is not very important for the indexation of your site, but it helps to see why your site may be occupied or not. You can then take action if a crawler is visiting and still giving no result on search engines.

Specific Usergroup for Spiders: i released this addon on vb.org long time ago, and it was copied in source, but this version is updated and have more flexibility. You simply have to choose the proper usergroup in the settings so when a spider/crawler visit your site, it is considered having some permissions... it's useful if you do not want to fill your robots.txt file with strange access blocks. This let you give access to crawlers for profiles but not visitors messages, etc...

Also remember to follow the TOS of the search engines you are registered to. Google until lately was blocking sites that were ghosting their content.

Display Spiders in WOL: and in any page showing "Currently Active Users" (showthread, forumdisplay, etc) ... that way, you see where these beasts are visiting..

... some other tools are to be decided to join in the bundle, i'll see later!

CRON JOB:
to make it easier on the server, there is a cronjob storing the hourly stats about the crawlers... once the cronjob is done once (it's the cron named Hourly #1), the stats appear in the right place...:

...update: may 1st, 10:50, a small change, the Crawlers listing will now update the spiders list in cache if the file changed, so you can update it when needed.


a note... this engine use the Spiders List from Dream... with his permission.
http://www.wolfshead-solutions.com/spiders-list

someone said to me that vbSEO Googlemap was doing something similar... hum... yeah, similar, tracking 3 web crawlers -- you have to generate new entries by hand if you want to track each crawler...

this engine is compatible with Paul M's Guest tracking... it's not doing exactly the same thing, but you know, you choose... this engine is not tracking guests activity, just crawlers page hits. (adding a small query per page, but useful when you REALLY need to know what web crawlers are doing on your site) -- really good for site owners showing potential for paid Ads!

Also, there will be a newer version with more settings soon... permissions and blocking, etc

...

as a success story, i must say i had a great visitor this week, that i would never been able to track without this engine... the "Majestics MJ12bot" crawler make 20 times the hits on my site that Google was able to do in months... i checked their site, and it was obvious they were trying to leech the site, not crawl it...

Download Now

Only licensed members can download files, Click Here for more information.

Screenshots

Click image for larger version

Name:	nex_crawlers_bundle_info.jpg
Views:	139
Size:	11.7 KB
ID:	136864Click image for larger version

Name:	nex_crawlers_bundle_list.jpg
Views:	215
Size:	62.5 KB
ID:	136865Click image for larger version

Name:	nex_crawlers_bundle_online.jpg
Views:	175
Size:	48.8 KB
ID:	136866Click image for larger version

Name:	nex_crawlers_bundle_settings.jpg
Views:	147
Size:	55.3 KB
ID:	136867Click image for larger version

Name:	nex_crawlers_bundle_wol_ug_markup.jpg
Views:	163
Size:	19.4 KB
ID:	136868

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
Similar Mod
Mod Developer Type Replies Last Post
Nexia's Web Crawlers Bundle nexialys Modification Graveyard 25 13 May 2009 14:24
[Release] No Crawlers Tip fraqheart vBulletin 2.x Template Modifications 8 08 Jun 2003 18:20

Comments
  #2  
Old 05 Mar 2012, 22:02
vbenhancer's Avatar
vbenhancer vbenhancer is offline
 
Join Date: Dec 2009
Real name: nexia
this version was tested on more than 250 websites, so it's supposed to work properly...
__________________
... retired... stop bothering, GTFO!
Reply With Quote
  #3  
Old 05 Mar 2012, 22:18
Mosh's Avatar
Mosh Mosh is offline
 
Join Date: Aug 2004
Location: Melbourne, Australia
Real name: Mosh Shigdar
Actually Nexia, the spiders list you are pointing to is my spider's list, not Dreams And, yes I do give permission
__________________
-- The WS Spiders List is still downloadable from vBulletin.com.
-- As of Monday 1st July 2013, I will no longer be supporting my hacks as I am leaving the vBulletin community due to medical and life-style reasons.
Reply With Quote
  #4  
Old 05 Mar 2012, 22:20
vbenhancer's Avatar
vbenhancer vbenhancer is offline
 
Join Date: Dec 2009
Real name: nexia
oh, i didn't see the change of ownership, sorry.. rofl

and thanks...

... am i the only coding making any usage of the spider engine yet?!

looks like it was added in vB 3.6 and nobody ever used it...
__________________
... retired... stop bothering, GTFO!
Reply With Quote
  #5  
Old 05 Mar 2012, 22:27
Mosh's Avatar
Mosh Mosh is offline
 
Join Date: Aug 2004
Location: Melbourne, Australia
Real name: Mosh Shigdar
Originally Posted by vbenhancer View Post
oh, i didn't see the change of ownership, sorry.. rofl

and thanks...
Dream stopped his, I wrote mine from scratch.

Originally Posted by vbenhancer View Post
... am i the only coding making any usage of the spider engine yet?!

looks like it was added in vB 3.6 and nobody ever used it...
Boofo & Paul M have hacks that make use of the spiders list as well.
__________________
-- The WS Spiders List is still downloadable from vBulletin.com.
-- As of Monday 1st July 2013, I will no longer be supporting my hacks as I am leaving the vBulletin community due to medical and life-style reasons.
Reply With Quote
  #6  
Old 28 May 2012, 05:43
AFMichael AFMichael is offline
 
Join Date: Sep 2004
Excellent mod! Installed!
Reply With Quote
  #7  
Old 01 Jun 2012, 00:57
matrex722's Avatar
matrex722 matrex722 is offline
 
Join Date: Jan 2007
Excellent mod! Installed!
Reply With Quote
  #8  
Old 03 Jun 2012, 08:26
Fashel.Net Fashel.Net is offline
 
Join Date: May 2012
thanks for this mod

if the crawlers activity could be shown only by Administrators it will fantastic, i hope you add this option to the mod because i really liked it

Installed
__________________
There is no God but Allah and Muhammad is his Messenger
المنتدى
Reply With Quote
  #9  
Old 04 Jun 2012, 10:06
incisor incisor is offline
 
Join Date: May 2006
Thumbs up

very impressed

thank you for the effort!
Reply With Quote
  #10  
Old 12 Jul 2013, 03:22
WebMonster2013 WebMonster2013 is offline
 
Join Date: May 2013
i cant get this working at all.



as you can see the bot is viewing a no perm's message.
Reply With Quote
  #11  
Old 12 Jan 2014, 13:59
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Aug 2009
Real name: Chris
I have decided to update the spiders list, I added 220 spiders to it, you can now get the list from here, vBulletin Spiders List Updated

Note: You must be logged in in order to download the list.
__________________
My Mods also available at OzzModz.com

New vBulletin Spider Definitions, vBulletin Spiders List Hits 1000 Spiders!
Stop Spam Now The era of Big Spam is over.



Vote to bring back hooks in vB5, http://tracker.vbulletin.com/browse/VBV-10333
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
Forum Jump


New To Site? Need Help?

All times are GMT. The time now is 21:09.

Layout Options | Width: Wide Color: