vBulletin.org Forum - Integration with vBulletin

Quote:

Originally Posted by thenetbox (Post 2182478)

Things seem to work so far :) Thanks!

When I enter the http:BL API Key, the entire forum goes blank though. White screen, no error.

Odd. I've been testing the http:BL for some time, works fine on my end. Any errors in your error log?

Quote:

Originally Posted by thenetbox (Post 2182481)

Is there a way to whitelist a few IP addresses or user agents?

Thanks again.

Edit /includes/whitelist.ini

Gone for the install today and all went very easy

Looking through my logs, google and bing seem to get stopped a lot
Im going to white list these in the top level whitelist and see if it cures the problem

It looks like there is also a complete google range missing 74.125.0.0/16

This is the message I get on most Google bot hits

f1182195

HTTP Response: 403
Explanation: An invalid request was received. You claimed to be a major search engine, but you do not appear to actually be a major search engine.
Log Message: User-Agent claimed to be Googlebot, claim appears to be false.

A bit more playing around and it looks like google gets blocked when reverse proxy is enabled

Quote:

Originally Posted by Eric (Post 2182486)

Odd. I've been testing the http:BL for some time, works fine on my end. Any errors in your error log?

No. There are no errors in the errorlog when the white screen happens. Removing the API key makes everything go back to normal again.

Quote:

Originally Posted by thenetbox (Post 2182664)

No. There are no errors in the errorlog when the white screen happens. Removing the API key makes everything go back to normal again.

What PHP version are you using?

Quote:

Originally Posted by Lee G (Post 2182629)

Hmm, odd. This appears to be with the Bad Behavior core - you can either add that range to the whitelist.ini file, or edit /includes/bad-behavior/searchengines.inc.php

I'll add this in the next release.

Quote:

Originally Posted by Eric (Post 2182716)

What PHP version are you using?

Thanks for following up :)

Using PHP Version 5.2.5

I have been through and found what option kills the google connections
When you put a tick in the "Reverse Proxy" and leave everything below it as is on install, it blocks Google with the f1182195 error and from what I could see also bing

That still occurred with all the bots ip ranges white listed

Once I found that, it has been working well

Version 1.0.2, 04/10/2011

Updated /includes/functions_vb_badbehavior.php to:
- disable Reverse Proxy if Reverse Proxy Addresses are empty
- distinguish SQL queries using "SET", for example: SET @@session.wait_timeout = 90 - which is used by BB
- set "offsite_forms" to false by default, as it's not really needed in vB IMHO, and it can cause problems with certain setups
- cleaned up the bb2_read_settings() function and fixed a typo in one of the vbulletin options calls
Updated /includes/whitelist.ini to include the following GOOGLE ranges:
- 74.125.0.0/16
- 216.239.32.0/19
- 209.85.128.0/17
- 66.102.0.0/20
Updated /admincp/vb_badbehavior.php
- Log pruning was pruning all logs, despite what was entered for number of days

Nice smooth upgrade
I cant believe how much junk this stops without adding any extra user agents
Just over 3500 log entries since I have been running it

Thanks for all the work your putting into this Eric

I think the explanation of what BB is should include that BB also blocks a large number of content scrapers and malicious bots. This saves bandwidth costs and increases security.

I find these very important aspects of BB and adding this will increase the number of sites that install BB.

Thanks for this great mod, Eric!

Quote:

Originally Posted by Eric (Post 2183010)

Version 1.0.2, 04/10/2011[*]Updated /includes/whitelist.ini to include the following GOOGLE ranges:

74.125.0.0/16
216.239.32.0/19
209.85.128.0/17
66.102.0.0/20

Does the whitelist also include Google's MediaBot (for AdSense)?

I have enabled Bad Behavior again. It immediately freed up my server from an insane server load. Server load went from 38 to 0.7 almost instantly. :)

I do see a valid members blocked. Details:

A very large number of these:

Quote:

Key: HTTP Response: 403
Explanation: You do not have permission to access this server. Before trying again, close your browser, run anti-virus and anti-spyware software and remove any viruses and spyware from your computer.
Log Message: POST more than two days after GET
User agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
URI: /forum/ajax.php
Entity: security token present.
Headers: POST /forum/ajax.php HTTP/1.1
Host: www.my-forum.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.my-forum.com/forum/forumdisplay.php?f=398
Content-Length: 82
Cookie: bb2_screener_= [omitted by Alfa1]
DNT: 1
Pragma: no-cache
Cache-Control: no-cache

I dont understand how it is possible that a large number of valid user post more than 2 days after GET.

A large number of these:

Quote:

Key: HTTP Response: 403
Explanation: An invalid request was received from your browser. This may be caused by a malfunctioning proxy server or browser privacy software.
Log Message: Required header 'Accept' missing

UserAgent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
URI: /forum/misc.php?do=page&template=ncode_opensearch
Entity:
Headers: GET /forum/misc.php?do=page&template=ncode_opensearch HTTP/1.1
Host: www.my-forum.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: bb2_screener_= [omitted by Alfa1

I find this one worrisome because its in the 2b021b1f key.

Quote:

Key: HTTP Response: 403
Explanation: You do not have permission to access this server. Before trying again, run anti-virus and anti-spyware software and remove any viruses and spyware from your computer.
Log Message: IP address found on http:BL blacklist
UserAgent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
URI: /forum/ajax.php
Entity: securitytoken: xxxxxxxxxxxxxxxx
do: securitytoken
ajax: 1
Headers:POST /forum/ajax.php HTTP/1.1
Host: www.my-forum.com
Content-Length: 82
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.my-forum.com/forum/search...archid=2679481
Cookie: bb2_screener_=xxxxxxxxxxxxx
Pragma: no-cache
Connection: keep-alive

I see these valid members are using proxies like TOR and similar.

Quote:

Key:
UserAgent:
URI:
Entity:
Headers:

Feature request 1: for the log: filter per key, so that it is possible to see all entries except those with key 00000 and key 2b021b1f. Or just view all entries with a certain key. That makes it much easier to see the similarities of the entries with the same key.

Feature request 2: Alert the admin which members have been blocked by BB and why. This makes it easier to detect problems with BB and forum accounts registered by bots. I think the optimal way to notify the admin is by PM.

Feature request 3: Trace IP directly from the log.

Feature request 4: related to FR 2. If bbuserid is present in headers then show link to user profile in the log. This makes it easy to check if the blocked members was a valid user or not.

Running in debug mode and checking out the queries exposes this error on forum home:

Quote:

Warning: Cannot modify header information - headers already sent by (output started at /private_html/forum/global.php(355) : eval()'d code:166) in /private_html/forum/includes/bad-behavior/screener.inc.php on line 8
End call of global.php: 0.19540810585

Just been through my last 450 denies and it looks like a Yahoo bot got the cold shoulder

Bot ip 67.195.112.41

Full ip range 67.195.0.0/16
http://whois.domaintools.com/67.195.112.41

Apart from that, its been working like a dream

vBulletin Mods