Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
vbArchive - Search Engine Indexer for vBulletin Details »
vbArchive - Search Engine Indexer for vBulletin
Mod Version: 1.00, by TECK (Member) TECK is offline
Developer Last Online: Aug 2013 I like it Show Printable Version Email this Page

This modification is in the archives.
vB Version: 2.2.x Rating: (11 votes - 5.00 average) Installs: 177
Released: 13 Jan 2003 Last Update: Never Downloads: 26
Not Supported  

vbArchive v1.3 Released
240,000 Google pages (and counting...) indexed so far. Congratulations to all users!

WHAT'S NEW IN VERSION 1.3:
I added the number of threads and posts for each forum.
The problem with a static page like the main archive page (as well the category ones) was that it never changed.
Now, every time a crawler visit the page, it will see new elements changed, since the number of threads and posts will always change.
Take a look at my archive, until FireFly updates the vBulletin.org one.

I also added a new meta tag for crawlers:
<meta name="robots" content="index,follow">

To upgrade from 1.2, simply upload the new archive.txt and forumdisplay.txt files.
Then REVERT to original all templates and run the NEW installer script to un-install/install the templates.

IMPORTANT
If you already installed version 1.3, check the "archive_forumtitle" template and see if you have there a variable called "$archiveurl".
If you do, clean your browser temp files and redownload the package, it was a mistake I made in the code.
The file is now updated with the new code.

Not convinced this is a good script? A picture and it's Google results is worth 1000 words (thanks xiphoid).
Is funny that few people rated 1 my script, we wonder who they might be?


DEMO WEB SITE: vBulletin.org Archive (Google Results for TeckWizards.com Archive)
ESTIMATED INSTALL TIME:
2 minutes

Script Information
IF YOU WANT TO READ fastforward's EVALUATION ABOUT THE TECHNIQUE USED, READ MORE HERE.


This script will install the Search Engine Indexer Add-On for vBulletin.
Is the little brother of vbHome (lite) Archive Add-On.

NOTE
The script uses Apache's ForceType directive. Most of Apache servers have it installed by default.
Check with your host to make sure the module is installed.
If you use another server then Apache, the script will NOT work.
The only solution I found for IIS is the ISAPI Rewrite module.

The script uses only 6-12 queries, depending on what page you view, and it works with any 2.2.x vBulletin version.

Some of it's cool features are:
- vBulletin 3 style
- listings based on forum/thread permissions
- forums architecture followed [example]
- classic .html extension usage
- dynamic meta tags (unique for each forum/thread/post, extremely important for good indexing)
- no broken links while using no_permission functions
- navigation bar
- multiple pages (200 threads or 100 posts per page) [example]
- template based, so you can edit it's look easy
- installer included

You will ask: So what the script does?
It makes all your forum URL's as search engine friendly, so they can be easy indexed by all search engines.
For example, the URL:
forum/forumdisplay.php?forumid=9&daysprune=365&sortorder=&sortfield=lastpost&perpage=2 5&pagenumber=2
will look like:
forum/forumdisplay/f-9-p-2.html
That will allow any search engine to index properly all your forum contents, in no time.

IMPORTANT
Do NOT get "creative" and start adding crazy stuff (popups, etc.) and links to the actual templates.
The most 2 important things for your pages are:
1. good meta tags
2. clean html code that won't upset the crawlers

The script was optimized to perform at it's best the way it is now, so crawlers gravitate only onto the archives files, not outside.
You can edit the archive_homekeytag to enter your web site key words.

The link to forums page is needed (image logo), because some search engines might consider this as URL cloacking, if you don't link it back to your actual forums. Don't worry about the rest, if you performed the first 3 steps in Forums Optimizations (listed below), they will go back and forward to the archives, without any problems.
Also, follow the readmefirst.htm instructions carefully.

Upgrade from previous version (lower then 1.2)
Estimated time for uninstall-install process: 5 minutes
Follow these steps (clear your browser temporary files before you download the new file):
1. Revert to original all archive templates.
2. Run the OLD Installer script and un-install the script components.
3. Follow the NEW instructions in the readmefirst file, included in v1.2 package.
NOTE: Overwrite the OLD code with the NEW one, in functions.php file.

Other similar scripts
Those scripts are alternatives to my code. Have your pick for the one it suit better your taste or forum performance.
SkuZZy's vB Easy Archive - another script coded by Xenon
fastforward's Spider Friendly URL's - it uses the mod_rewrite

Forums Optimizations
You MUST perform also some the mods listed below if you want your forums optimized properly for search engines indexing.
Steps 1 to 3 are vital, the rest is optional.

1. TO STRIP THE sessionhash FROM TEMPLATES (ONLY FOR CRAWLERS), READ MORE HERE.
2. TO BLOCK CRAWLERS GO TO CERTAIN PAGES, READ MORE HERE.
3. TO LINK EACH FORUM/THREAD DIRECTLY TO ARCHIVE FILES, READ MORE HERE.
4. TO DISPLAY NICE LOCATIONS, THE FIX FOR online.php FILE IS HERE.
5. TO DISPLAY CRAWLER NAME INSTEAD OF GUEST ON FRONTPAGE AND ONLINE PAGE, READ MORE HERE. (mod by Inphinity and xiphoid)
6. TO DISPLAY CRAWLER NAME INSTEAD OF GUEST ON ONLINE PAGE only, READ MORE HERE.
7. IF YOU WANT THE MAIN ARCHIVE FILE TO HAVE A .php EXTENSION, READ MORE HERE.
8. TO CHANGE THE threads/posts per page NUMERIC VALUES, READ MORE HERE.
9. TO DISPLAY THE SMILIES AS image parsed, READ MORE HERE (mod by Logician).

IMPORTANT
Kill crawler918.com! READ MORE HERE.


Other Users Demo's
Feel free to post your archive link, so I can display it here.

TeckWizards.com Archive
eva2000's Anime Boards archive
overgrow's Edge Forums archive
glenvw's Yes-Its-Free archive
codeweb's Code Webs archive
xiphoid's Open Forum archive
Hwulex's Xaprief archive
BiggieSwolls' Steroidology archive
GearedUp's FitnessGeared archive
saint_seiya's VG City archive

Search Engine Submission
You should follow these guidelines to get listed in every major search engine:
(also visit those forums for more information)

DO NOT CHEAT
- do not use URL cloaking
- do not use automatic search engine submitters, do it manually
- do not use 1 pixel images to link your archive file
- do not make invisible your link text, by masking it with the same background color
- do not use 1-4 pixels text at the top of your page, to display the site contents
- do not link your archives file to an image without using the alt="" tag

GOOGLE INDEX STATUS
To see how your site is doing, related to links, go to Google Web Site and type:
site:yourwebsite.com archive

GOOGLE FACTS
1. Google uses a crawler named Googlebot which crawls the web approximately every thirty days.
2. It is not necessary to submit any page to Google. If you do submit, submit only your most important page to this search engine.
3. Googlebot is a deep crawler and should crawl all of your pages.
4. Google supplies ranking results for placement in Netscape Search, the ODP, Anzwers, Yahoo! and Ilor.
5. Google can crawl pages in ASP, JSP, CFM, PHP, Excel, Microsoft Word, newsgroups, PDF and PostScript files, Power Point and Rich Text formats.
6. Google loves sites with a high number of legitimate, relevant incoming links.
7. Google hates spam.

GOOGLE TECH SUPPORT E-MAIL [LINKS ARE "DROPPED"? NO]
If your site is new, or hasn't shown up in Google for long, it may because our "fresh crawl" (which runs each day) was finding your site instead of our main crawl (which runs about once a month). Our "fresh crawl" is a newer feature, and we're still experimenting with which pages to crawl, how deeply to crawl, etc. We even reserve the right to (gasp!) not do a fresh crawl on some days because we're doing tests or reviewing new code. Someone wrote in recently and said "my site got in Google three weeks ago, and you've dropped me four times!" Nope, it's just that we don't always crawl the same pages in our fresh crawl, and we don't always crawl to the same depth. As we do a full crawl of the web, we find most of the sites from our fresh crawl and put them in our regular index. My advice on our fresh crawl is to view it as a nice "bonus" on top of Google's deep index. Users can always search our full index, but sometimes we can serve up even fresher pages as an extra nicety.

What does this mean for the average webmaster? In the word of the great Hitchhiker's Guide, "Don't Panic."
Just do the normal things you should do:
1. Create a great site.
2. Submit your site to Google on our "add url" form.
3. Get a link from the Open Directory Project or other directories (Yahoo, etc.).
4. Don't panic if your site takes a little while to show up in Google. Be patient, and start to look around the web--there's lots of great advice about improving your site for users and search engines.

Hope this helps,
xxxxxxxxxx

RECOMMENDED SEARCH ENGINES
1. Google - The largest and best index at the time.
Submit your link here.

2. Inktomi - This is the database that feeds iWon, 4anything, AOL Search, HotBot, GoTo, ICQ, LookSmart, MSN Search & Snap.
Submit your link here and here.

3. Fast / AllTheWeb - This Norwegian index is almost as good as Google.
Submit your link here.

4. AltaVista - Still one of the big guns, despite its temperamental ranking system.
Submit your link here.

5. Walhello - Mysterious new index with great results. Get listed, it is on its way up.
Submit your link here.

6. Non-English Indexes - The people of these countries use their own search engines. It helps if your site is in their language, because they will be searching for keywords in their own language.
Caloweb France - Submit your link here.
Caloweb Germany - Submit your link here.
Caloweb Spain - Submit your link here.

DEAD ENGINES
There is no point trying to submit to these search engines:
Excite - dead, now uses pay-per-click results
Direct Hit - will be retired shortly
Northern Light - no longer available to the general public
Lycos - now use AlltheWeb's index

Once you done all this, watch the incoming traffic that will arrive to your site.
Good luck.


Copyright Permissions
1. You ARE NOT allowed to REMOVE or MODIFY the copyright text at the bottom of the page.

The copyright MUST be in a distinctive color and easy readable by visitors.
2. You ARE NOT allowed to ALTER in any way the URL links listed in the copyright text.
The Search Engine Indexer link pointing to TeckWizards.com MUST stay intact.
You can remove ONLY the vBulletin version or replace the direct link to vBulletin site with your referral link.
3. You ARE NOT allowed to DISTRIBUTE the contents of downloaded .zip file.
4. You ARE NOT allowed to COPY ANY PARTS of the code and use it for distribution.

Download Now

Only licensed members can download files, Click Here for more information.

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
Similar Mod
Mod Developer Type Replies Last Post
Search Engine Indexer - vbHome (lite) add-on TECK vBulletin 2.x Full Releases 14 01 Feb 2004 15:47

  #361  
Old 08 Feb 2003, 16:42
Floris Floris is offline
 
Join Date: Jan 2002
Originally posted by TECK
Well, you posted screenshots, so I presumed is done.
Then you should wait before you post anything...

And I don't like to sit down. :banana:
Then we just let you wait another day maybe
__________________
My community; http://wetalknation.net
Reply With Quote
  #362  
Old 08 Feb 2003, 16:51
TECK's Avatar
TECK TECK is offline
 
Join Date: Dec 2001
Real name: Floren Munteanu
/me starts the revolution....
__________________
Floren Munteanu
Axivo Inc.
Axivo Community - Visit the forums to find out more about us
Why Queued - My personal blog
Reply With Quote
  #363  
Old 09 Feb 2003, 01:47
Floris Floris is offline
 
Join Date: Jan 2002
inph - you got the code done! Stop playing wc3 and start posting now EYE even get inpatient
__________________
My community; http://wetalknation.net
Reply With Quote
  #364  
Old 09 Feb 2003, 02:38
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
Originally posted by xiphoid
Stop playing wc3 and start posting
LOL :cheeky: *sorry for spam* :classic:
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.
Reply With Quote
  #365  
Old 09 Feb 2003, 03:05
inphinity's Avatar
inphinity inphinity is offline
 
Join Date: Oct 2001
useragent checking

oi wc3 is important

useragent checking

Works both standalone and as a very nice compliment to TECK's vbarchive hack.

What does it do?
Allows you to match the useragent for Guests in Who's Online and display custom names/urls for recognised useragents such as Google, Teoma, Inktomi etc

You can also use it for matching the useragent anywhere on vb, ie for Currently Active Users on forumhome. expect a jazzed up online.php with icons next to names for which browser people are using sometime.

Why?
Got bored of looking up IPs then digging around the session table trying to find out which guests were really web robots also nosey to see who was reading the archives.

Install
Instructions in the file, should work with 2.2.x
Install time, 3-5mins. level, medium.

List of Detected Web Robots (thanks to TECK for listing the main ones)
Last updated: 08/02/03 10pm GMT
googlebot www.google.com Google
gulliver www.northernlight.com Northern Light
ia_archiver www.archive.org The Internet Archive
internetseer www.internetseer.com Internet Seer
linkalarm linkalarm.com Link Alarm
mercator www.research.compaq.com/SRC/mercator Mercator
openbot www.openfind.com.tw Openbot
pingalink www.pingalink.com PingALink Monitor
psbot www.picsearch.com/bot.html PicSearch
scooter www.altavista.com AltaVista
slurp www.inktomi.com/slurp.html Inktomi
turnitinbot www.turnitin.com/robot/crawlerinfo.html Turnitin
slysearch www.turnitin.com/robot/crawlerinfo.html Turnitin
zeus http://www.waltbren.com/products/zeu...rnet_robot.htm Zeus Internet Marketing
zyborg www.wisenutbot.com WiseNut
teoma www.teoma.com Teoma/Ask Jeeves

-- these last 3 are generic and will display the useragent on who's online with a link to robotstxt.org where you can look up the useragent for obsecure and new bots.

spider Web Spider
spyder Web Spyder
crawl Web Crawler
robot Web Robot

Screenshots?
Who's online:
http://www.vbulletin.org/forum/attac...&postid=351495
http://www.vbulletin.org/forum/attac...&postid=351832
http://www.vbulletin.org/forum/attac...&postid=351533

Currently Active Users
http://www.vbulletin.org/forum/attac...&postid=351831

enjoy,
inph

thanks to floris for screenshots and testing
Attached Files
File Type: txt useragentchecking.txt (9.3 KB, 268 views)
__________________
inphinity

Last edited by inphinity; 09 Feb 2003 at 03:29.
Reply With Quote
  #366  
Old 09 Feb 2003, 03:15
inphinity's Avatar
inphinity inphinity is offline
 
Join Date: Oct 2001
just a quick note if you're using TECK's guest_crawler

you should add the trailing dot to the ip addresses


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

so that you dont match (ie an octet at the beginning):
*216.239.46*
*66.196.72*

with the trailing dot you will only match:
216.239.46.*
66.196.72.*



also a minor point for the vbarchive installer

the templates added are set to templatesetid=-1
which is fine but in vB's upgrade scripts, lines like:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

tend to obliterate peoples templates

i would recommend adding the templates twice once with -1 and then again with the style id's so they appear as custom templates (with default content)
__________________
inphinity

Last edited by inphinity; 09 Feb 2003 at 03:32.
Reply With Quote
  #367  
Old 09 Feb 2003, 07:48
limey's Avatar
limey limey is offline
 
Join Date: Dec 2001
hey is 1000 hits by googlebot in 2 days good?

edited the number from 609 -1000

Last edited by limey; 09 Feb 2003 at 21:26.
Reply With Quote
  #368  
Old 09 Feb 2003, 09:52
TECK's Avatar
TECK TECK is offline
 
Join Date: Dec 2001
Real name: Floren Munteanu
Originally posted by inphinity
just a quick note if you're using TECK's guest_crawler

you should add the trailing dot to the ip addresses


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.
Very good point, I edited the file.
Also, I'm going to add your mod in the first post, with credit of course. Great job.
About the templates, is really easy, simply run again the installer, no need to recustomize the templates because they are automatically saved, if they are edited (not original), so your work is not lost...

EDIT: Link added. Check no. 5 in Forum Optimizations section.
__________________
Floren Munteanu
Axivo Inc.
Axivo Community - Visit the forums to find out more about us
Why Queued - My personal blog

Last edited by TECK; 09 Feb 2003 at 10:05.
Reply With Quote
  #369  
Old 09 Feb 2003, 12:52
Floris Floris is offline
 
Join Date: Jan 2002
Glad you liked the hack teck
__________________
My community; http://wetalknation.net
Reply With Quote
  #370  
Old 09 Feb 2003, 15:42
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
@ inphinity > big thanx for adding that feature to show web robots on home page! and also thanx for releasing this addon! Very nice

PS: I think you should also release it as a hack so people will know it exist and it will be possible to add this hack to the fine collection of vb.org hacks

Cheers!
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.
Reply With Quote
  #371  
Old 09 Feb 2003, 22:39
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
:: 51 members, 46 guests and 32 web robots (Google) on the boards

Nice to see it on the main forum page! Thanx again for the great addon!
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.

Last edited by wooolF[RM]; 09 Feb 2003 at 23:13.
Reply With Quote
  #372  
Old 09 Feb 2003, 22:45
Mike Gaidin's Avatar
Mike Gaidin Mike Gaidin is offline
 
Join Date: Oct 2001
Real name: Mike
Re: useragent checking

Originally posted by inphinity
oi wc3 is important

useragent checking

Works both standalone and as a very nice compliment to TECK's vbarchive hack.

In the instructions for the modification of functions.php it just has a piece of code, but no instructions as to where to put it. Where does it go?
Reply With Quote
  #373  
Old 09 Feb 2003, 22:57
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
Find
Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Add ABOVE this code the code mentioned in the attached file
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.
Reply With Quote
  #374  
Old 09 Feb 2003, 23:13
limey's Avatar
limey limey is offline
 
Join Date: Dec 2001
Looks like those first googlebots were scouts and they sent the deepcrawlers over the past few days. Here they are in action.
Attached Images
File Type: gif googlebotinvasion.gif (110.3 KB, 67 views)
Reply With Quote
  #375  
Old 09 Feb 2003, 23:16
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
They are crawling my forum right now
I have about 120 users online + 32 Google bots
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 18:51.

Layout Options | Width: Wide Color: