 |
|
|
Mod Options |
Ban Spiders by User Agent Details »»
|
|||||||||||
|
Ban Spiders by User Agent
Developer Last Online: May 2013 
What this mod does
With this mod you can enter User Agents to watch or ban, you can also recieve emails or have an Output.txt created and updated with time and date of visits. It doesn't just have to be spiders, you can watch, log or ban any useragent! How to install Simply import the product ban_spider, the mod is active by default but none of the other options are turned on. What is a UserAgent? http://en.wikipedia.org/wiki/User_agent Understanding a UserAgent string http://user-agent-string.info/parse Genuine User Getting Blocked? http://www.vbulletin.org/forum/showp...&postcount=105 Tools to help http://whatsmyuseragent.com/SwitchingUserAgents.asp http://www.botsvsbrowsers.com/SimulateUserAgent.asp FAQ http://www.vbulletin.org/forum/showp...&postcount=137 How does it work? http://www.vbulletin.org/forum/showp...&postcount=381 What's a bot? http://en.wikipedia.org/wiki/Spambot How do i ban a bot? http://www.vbulletin.org/forum/showp...&postcount=318 http://www.vbulletin.org/forum/showp...7&postcount=51 Where's output.txt located? http://www.vbulletin.org/forum/showp...&postcount=216 Bad bot lists http://www.vbulletin.org/forum/showp...&postcount=259 http://www.vbulletin.org/forum/showp...&postcount=224 http://www.vbulletin.org/forum/showp...&postcount=281 Tested on vb3.7.x, vB3.8.x , vB4.x.x but should work on any version. ____________________________________________________________________ Special thanks to: Lior KH99 BoP5 for helping me sort out a few issues ...and beta testers ForceHSS (Special thanks to Force for latest testing) ozzy47 GreyHost If you use this please mark as INSTALLED History 9th June 2011 Orginal xml added 12th June 2011 Added both email notification and text file logging 22nd June 2011 Version 2.0.0, Added create thread on activity
08th October Beta testing started for thread creation. 20th October Beta testing started for emailing. 21st October Beta testing complete Ver 3.0.0 uploaded 29th October minor fix added to cope with empty userid on thread creation 30th October Beta testing automatic redirection to spiders/bots IP 31st October New xml uploaded with automatic redirect to IP 25th November Minor fix for blank forumid fixed 26th November 2011 Fixed version check & create thread Off by default The Bad Bots list is now included in the product  Please prune out all those that you wish to be able to see your site (i suggest you definately prune out "DA" and "Custo" : Support will now only be given to those who have this mod marked as INSTALLED Download Now Only licensed members can download files, Click Here for more information. Show Your Support
The following members like this post: tricksodave, rootsxrocks, MAD--DOG, ForceHSS, BlueCheri and 3 others.
|
|||||||||||
About Developer
Visit Web Site
Supported|
Similar Mod
|
||||
| Mod | Developer | Type | Replies | Last Post |
| Ban Spiders by User Agent | Simon Lloyd | vBulletin 3.8 Add-ons | 110 | 01 May 2013 17:07 |
|
#211
05 Nov 2011, 00:16
|
|||
|
|||
|
Originally Posted by Simon Lloyd
OK now I feel a right idiot, it never occurred to me to just create the needed file in my forum root directory and CHMOD just the file for read/write access. I can't see the woods for the trees!
On a side note due to me making a slight error while setting up the hack I have threads made by the hack in all sorts of places, the threads don't actually exist and just make the forum look a mess. Any idea on how to remove them? Somebody mentioned about a SQL thing to do but I have no idea about that. Thanks for the support. Oh and I forgot to mention that I am running v3.8.5 of vBulletin. gigawiz. EDIT - I currently have a specific forum for the threads created by this hack and if I don't put them there then they end up everywhere. How do I set it so as no threads are made at all and just the output.txt file is made? No members have liked this post.
Last edited by gigawiz : 05 Nov 2011 at 02:52. 
|
|
#212
05 Nov 2011, 06:32
|
||||
|
||||
|
Firstly this is the vb4 thread so specific version questions should be in the thread for that version, however, to clean up just go to admincp>maintainance>Update Counters then update forum information.
To NOT create threads (which of course was my recommendation) then simply uncheck the radio button for "Create Thread"
__________________
Kind regards, Simon Microsoft Office Help My Mods: Prevent Number Only Registrations, Ban Spiders by User Agent, WhoIsIp Info from any post, Insert Objects/ads anywhere using php, Floating Notice Box in Forumhome, Ban IP's Easily No members have liked this post.
|
|
#213
05 Nov 2011, 12:24
|
|||
|
|||
|
Originally Posted by Simon Lloyd
You sir are a gentleman and a scholar! That cleanup bit did just the trick, sorry for posting in the wrong version thread I will look for the other one? Should I re-post my previous questions over in that thread don't want to be seen as double posting type thing.
Thanks again for your continued support! gigawiz. No members have liked this post.
|
|
#214
05 Nov 2011, 16:47
|
||||
|
||||
|
gigawiz, no need to post in the other thread now, the mods are the same but different versions can give different erros which is why i have versions of this for vb3.7 and vb3.8
__________________
Kind regards, Simon Microsoft Office Help My Mods: Prevent Number Only Registrations, Ban Spiders by User Agent, WhoIsIp Info from any post, Insert Objects/ads anywhere using php, Floating Notice Box in Forumhome, Ban IP's Easily No members have liked this post.
|
|
#215
06 Nov 2011, 18:56
|
|||
|
|||
|
Awsome mod, thanks so much for this.
Spent hours trying to get rid of Baiduspider via htaccess and robots.txt then found this. Twenty minutes after having it turned on no crappy unwanted bots. Voted for MOTM One quick question. I turned on the logging to the output.txt file that shout be in my forum root but I didn't see it generated in my httpdocs folder once the bots were removed? I then turned logging off after just twenty minutes of having the mod installed. Does the log take a while to generate? Thanks for this mod, it is really helpfull No members have liked this post.
|
|
#216
06 Nov 2011, 19:05
|
||||
|
||||
|
The output.txt is generated as bots found in your list attempt to call a forum or thread, there's no time lag and the file should be created straight away. If you have no cms then the file should be available at www.mysite.com/output.txt if forum is in a folder then something like www.mysite.com/forum/output.txt
Any issues post back and i'll deal with them for you
__________________
Kind regards, Simon Microsoft Office Help My Mods: Prevent Number Only Registrations, Ban Spiders by User Agent, WhoIsIp Info from any post, Insert Objects/ads anywhere using php, Floating Notice Box in Forumhome, Ban IP's Easily No members have liked this post.
|
|
#217
06 Nov 2011, 19:40
|
|||
|
|||
|
Ah ok, I will turn the logging back on and let you know. Should be fine though.
Thanks No members have liked this post.
|
|
#218
07 Nov 2011, 15:06
|
||||
|
||||
|
Just installed this, very cool, thank you!
I'm using the full list but I see many are not. I don't care about most Asian traffic. Actually, I only care about the main bots, the rest can go you know where. Is the full list recommended then? I don't need a log, notifications or to create threads etc. I just want to turn this on and have it work without having to dump logs etc. I've set it to the top 3 and pointing to www.klikhierniet.net Is this enough? Thanks again! No members have liked this post.
Last edited by bigtree : 07 Nov 2011 at 16:55.
|
|
#219
07 Nov 2011, 17:55
|
||||
|
||||
|
Firstky glad you like it
You don't have to have any logging of any sort, that stuff was added by request so folk could monitor things...etc. The FULL list isn't exhaustive and there are many missing off it, denying bots/spiders is a personal thing, just use the names of those that you don't want to see your site (if you are using the full list remove DA and Custo as these may cause issues with real users), remember you are banning bots/spiders by user agent and what you see in WOL isn't necessarily in the UA, if you go to WOL and then chose the option for displaying useragents aswell it will help you. I personally ban: Yandex Yeti Youdao Sogou SoSo Baidu spinn3r psbot SBIder exabot speedy omgili wget Amongst a few others, like i said, you can ban agressively as you like EDIT: You can use the automatic option of redirecting each spider/bot to their own IP address instead of redirecting to a site!
__________________
Kind regards, Simon Microsoft Office Help My Mods: Prevent Number Only Registrations, Ban Spiders by User Agent, WhoIsIp Info from any post, Insert Objects/ads anywhere using php, Floating Notice Box in Forumhome, Ban IP's Easily No members have liked this post.
|
|
#220
07 Nov 2011, 22:26
|
||||
|
||||
|
This is such a great Mod! You are king!
No members have liked this post.
|
|
#221
07 Nov 2011, 23:50
|
|||
|
|||
|
Great mod, Simon.
I'm loving the difference it makes. Today I noticed the Baidu spider on my site, despite it being in the ban list. Any ideas? No members have liked this post.
|
|
#222
08 Nov 2011, 05:23
|
||||
|
||||
|
Originally Posted by bigtree
I have no idea
, i originally built this to get rid of the chinese bots/spiders from my site as they were using up a lot of bandwidth and cpu time.
Originally Posted by spillage
I'll bet you are using Paul M's mod track guest visits or something like that, if so read back a page or two of this thread
Glad you're both happy with it!
__________________
Kind regards, Simon Microsoft Office Help My Mods: Prevent Number Only Registrations, Ban Spiders by User Agent, WhoIsIp Info from any post, Insert Objects/ads anywhere using php, Floating Notice Box in Forumhome, Ban IP's Easily No members have liked this post.
|
|
#223
08 Nov 2011, 10:38
|
|||
|
|||
|
I installed the update "31st October New xml uploaded with automatic redirect to IP" a few days ago and I noticed that by visitors number seemed to jump and be much higher afterwards. It used to work fine with the previous version.
I took the advice here and waited, but even after a few days, I'm still seeing "Baidu" spiders appearing and active in the "Who's On-line", even though this mod is active and Baidu is in the list of banned spiders? What am I missing? My ban list says ... Yandex Yeti Baidu soso sogou ichiro speedy spinn3r mlbot psbot SBIder Ezooms snap shots metauri YoudaoBot youdao Regards, Doug No members have liked this post.
|
|
#224
08 Nov 2011, 12:07
|
||||
|
||||
|
try this list works well for me
Baidu almaden Anarchie ASPSeek attach autoemailspider BackWeb Bandit BatchFTP BlackWidow Bot\mailto:craftbot@yahoo.com Buddy bumblebee CherryPicker ChinaClaw CICC Collector Copier Copyscape Crescent DIIbot DISCo DISCo\Pump dotbot Download\Demon Download\Wonder Downloader Drip DSurf15a eCatch EasyDL/2.99 EirGrabber EmailCollector EmailSiphon EmailWolf Express\WebPictures ExtractorPro EyeNetIE FileHound FlashGet FrontPage GetRight GetSmart GetWeb! gigabaz Go\!Zilla Go!Zilla Go-Ahead-Got-It gotit Grabber GrabNet Grafula grub-client HMView HTTrack httpdown .*httrack.* ia_archiver Image\Stripper Image\Sucker Indy*Library Indy\Library InterGET InternetLinkagent Internet\Ninja InternetSeer.com Iria JBH*agent JetCar JOC\Web\Spider JustView larbin LeechFTP LexiBot lftp Link*Sleuth likse //Link LinkWalker Mag-Net Magnet Mass\Downloader Memo Microsoft.URL MIDown\tool Mirror Mister\PiX Mozilla.*Indy Mozilla.*NEWT Mozilla*MSIECrawler MS\FrontPage* MSFrontPage MSIECrawler MSProxy Navroad NearSite NetAnts NetMechanic NetSpider Net\Vampire NetZIP NICErsPRO Ninja Nutch Octopus Offline\Explorer Offline\Navigator Openfind PageGrabber Papa\Foto pavuk pcBrowser Ping PingALink Pockey psbot Pump QRVA RealDownload Reaper Recorder ReGet Scooter Seeker Siphon sitecheck.internetseer.com SiteSnagger SlySearch SmartDownload Snake sogou Soso SpaceBison Spinn3r sproose Stripper Sucker SuperBot SuperHTTP Surfbot Szukacz tAkeOut Teleport\Pro URLSpiderPro Vacuum VoidEYE vBSEO Web\Image\Collector Web\Sucker WebAuto [Ww]eb[Bb]andit webcollage WebCopier Web\Downloader WebEMailExtrac.* WebFetch WebGo\IS WebHook WebLeacher WebMiner WebMirror WebReaper WebSauger Website Website\eXtractor Website\Quester Webster WebStripper WebWhacker WebZIP Wget Whacker Widow WWWOFFLE x-Tractor Xaldon\WebSpider Xenu Yandex Yeti YOUDAOBOT Zeus.*Webster Zeus No members have liked this post.
|
|
#225
08 Nov 2011, 12:52
|
||||
|
||||
|
Originally Posted by BadgerDog
hi Doug, are you using any visitor tracking mods ?
__________________
Kind regards, Simon Microsoft Office Help My Mods: Prevent Number Only Registrations, Ban Spiders by User Agent, WhoIsIp Info from any post, Insert Objects/ads anywhere using php, Floating Notice Box in Forumhome, Ban IP's Easily No members have liked this post.
|
|
«
Previous Mod
|
Next Mod
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) |
| Mod Options | |
|
|
| New To Site? | Need Help? |
All times are GMT. The time now is 05:40.