Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Ban Spiders by User Agent Details »
Ban Spiders by User Agent
Mod Version: 3.1.2, by Simon Lloyd (Coder) Simon Lloyd is offline
Developer Last Online: Aug 2019 I like it Show Printable Version Email this Page

vB Version: 4.x.x Rating: (64 votes - 4.86 average) Installs: 483
Released: 09 Aug 2011 Last Update: 18 Dec 2014 Downloads: 1910
Supported Uses Plugins  

What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent!

How to install
Simply import the product ban_spider, the mod is active by default but none of the other options are turned on.

What is a UserAgent?
http://en.wikipedia.org/wiki/User_agent

Understanding a UserAgent string
http://user-agent-string.info/parse

Genuine User Getting Blocked?
http://www.vbulletin.org/forum/showp...&postcount=105

Tools to help
http://whatsmyuseragent.com/SwitchingUserAgents.asp
http://www.botsvsbrowsers.com/SimulateUserAgent.asp

FAQ
http://www.vbulletin.org/forum/showp...&postcount=137

How does it work?
http://www.vbulletin.org/forum/showp...&postcount=381

What's a bot?
http://en.wikipedia.org/wiki/Spambot

How do i ban a bot?
http://www.vbulletin.org/forum/showp...&postcount=318
http://www.vbulletin.org/forum/showp...7&postcount=51

Where's output.txt located?
http://www.vbulletin.org/forum/showp...&postcount=216

Bad bot lists
http://www.vbulletin.org/forum/showp...&postcount=259
http://www.vbulletin.org/forum/showp...&postcount=224
http://www.vbulletin.org/forum/showp...&postcount=281

Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version.

____________________________________________________________________
Special thanks to:
Lior
KH99
BoP5
for helping me sort out a few issues

...and beta testers

ForceHSS (Special thanks to Force for latest testing)
ozzy47
GreyHost

If you use this please mark as INSTALLED

History
9th June 2011 Orginal xml added
12th June 2011 Added both email notification and text file logging
22nd June 2011 Version 2.0.0, Added create thread on activity
  1. Added match facility you can now use something like Yandex and it will match MOZILLA/5.0 (COMPATIBLE; YANDEXBOT/3.0; +HTTP://YANDEX.COM/BOTS)
  2. Added clickable link to visited thread
22nd September 2011 added user redirect url selection
08th October Beta testing started for thread creation.
20th October Beta testing started for emailing.
21st October Beta testing complete Ver 3.0.0 uploaded
29th October minor fix added to cope with empty userid on thread creation
30th October Beta testing automatic redirection to spiders/bots IP
31st October New xml uploaded with automatic redirect to IP
25th November Minor fix for blank forumid fixed
26th November 2011 Fixed version check & create thread Off by default
17th December 2014 Version 3.1.0 uploaded, Hook changed extra logging and statistics added by Ozzy47 (Chris)
18th December 2014 Version 3.1.1 uploaded, prevented spiders being counted when mod turned off.
17th December 2014 Version 3.1.2 uploaded, due to rogue code from another mod
The Bad Bots list is now included in the product
Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" :

Support will now only be given to those who have this mod marked as INSTALLED

Download Now

Only licensed members can download files, Click Here for more information.

Supporters / CoAuthors

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
Similar Mod
Mod Developer Type Replies Last Post
Miscellaneous Hacks Ban Spiders by User Agent Simon Lloyd vBulletin 3.8 Add-ons 188 20 Jul 2015 12:34

  #811  
Old 01 Nov 2016, 15:26
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
That isn't an issue with this modification as it doesn't use glowhostspamomatic. You also have a blank in your list, this could effectively ban everyone and everything!!
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #812  
Old 01 Nov 2016, 15:33
RichieBoy67's Avatar
RichieBoy67 RichieBoy67 is offline
 
Join Date: Apr 2004
Real name: Richie
Originally Posted by grafbyte View Post
hi

after the install ...
Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

mit dis list
Add this to your config under <?php:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

__________________

Let us take care of your forum, seo, seo reports, maintenance, what ever you need.

Reply With Quote
  #813  
Old 28 Dec 2016, 20:37
imported_dboogie2288 imported_dboogie2288 is offline
 
Join Date: Apr 2004
I have a dumb question. I'm up on running a forum and such, but not on SEO and what these spiders and bots are actually doing. Since installing this product, I continually notice that the blocked spiders number keeps going up...at an alarming rate. First it was a few hundred, then it was a thousand, and now its about 8100 blocked spiders at a given moment. That was about 3-4 weeks ago. What's going on? Is this something along the lines of, "block them and they will come?" I fear the day that this plug in gets disabled by accident or whatever because the floodgates might come bursting open.
Reply With Quote
  #814  
Old 03 Jan 2017, 19:53
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Which bots are showing as increasing in numbers? it's more than likely the same bot that keeps changing IP address, don't worry it will settle down.
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #815  
Old 05 Jan 2017, 18:06
w00fy w00fy is offline
 
Join Date: May 2011
I just wanted to thank you for this. My site had slowed to a crawl with page load times of 15 seconds and I did not know why. I installed this and my page loads are now mostly under a second.

I had no idea these bots and spiders were doing this to me..after a week I see this.

Spiders/Bots Activity Blocked From Site: 102,285

Thanks again, much appreciated.
Reply With Quote
  #816  
Old 08 Jan 2017, 15:25
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
I suspect that you have some other process issues that are affecting load times but I'm glad this has helped.

Do take a look at your images that your site has, try to use a CDN to deliver them or maybe create a sprite to contain them all and deliver the images from sprite coordinates, also look at which scripts are called on page load, sometimes its best to try and minimize these and minify your JS scripts, this might help https://hostway.com/blog/10-ways-to-...e-load-faster/

Its a bit of work but well worth the effort.
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
  #817  
Old 25 Apr 2017, 09:14
conaero conaero is offline
 
Join Date: Feb 2010
I also have been using this product for over year and initially it wiped 90% of the traffic, as they were bots but over time the number has gone back up again.

I would assume what we need is a new list of bots?

Have had a search around but cant find one???

I am using this updated list as per a few posts back:

baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6
w3m

almaden
Anarchie
Artabus
ASPSeek
attach
autoemailspider
BackWeb
Baidu
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
GNIP
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Ichiro
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Magpie
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
omgili
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
speedy
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus
Reply With Quote
  #818  
Old 11 Oct 2017, 09:54
WOLF-G4MIN-X WOLF-G4MIN-X is offline
 
Join Date: Oct 2017
Question

I just installed this on 4.2.5, and as soon as I have it installed, this is showing up on the left upper side of my header where the logo is.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Could anyone tell me if that can be fixed, and if, how ?

Thanks in advance.

Wolf
Reply With Quote
  #819  
Old 23 Aug 2019, 10:39
furnival furnival is offline
 
Join Date: Mar 2008
Seems to work fine for me on VBulletin 4.2.5 so the error above in @WOLF-G4MIN-X's post must be the Glowhost Spamomatic mod incompatibility mentioned elsewhere in the thread.
Reply With Quote
  #820  
Old 24 Aug 2019, 12:50
Simon Lloyd's Avatar
Simon Lloyd Simon Lloyd is offline
 
Join Date: Aug 2008
Real name: Simon
Originally Posted by WOLF-G4MIN-X View Post
I just installed this on 4.2.5, and as soon as I have it installed, this is showing up on the left upper side of my header where the logo is.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Could anyone tell me if that can be fixed, and if, how ?

Thanks in advance.

Wolf
I havent seen this before and it doesn't use the bootstrap, This loads at every page call which prevents the spiders actually loading a page, so it could be a third party hack that doesn't like it being loaded, you could change the load order in the products page of admincp, normally most hacks are set at 5.
__________________
Kind regards,
Simon Microsoft Office Help
My Mods: Find my modifications here
Please do not pm me for support unless i have invited you to!
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 00:00.

Layout Options | Width: Wide Color: