Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
Dynamically Linked Source - Speed Up Your Multiple Site Network & Upgrades Details »
Dynamically Linked Source - Speed Up Your Multiple Site Network & Upgrades
Mod Version: 1.00, by Jafo232 (Coder) Jafo232 is offline
Developer Last Online: Nov 2018 I like it Show Printable Version Email this Page

This modification is in the archives.
vB Version: 3.7.2 Rating: (11 votes - 5.00 average) Installs: 51
Released: 14 Aug 2008 Last Update: Never Downloads: 305
Not Supported Uses Plugins Additional Files  

For the purposes of this documentation, "Network" refers to a group of forums on a single server or cluster of servers.

When 3.7.1 PL2 was released, I posted over at vbulletin.com lamenting the fact that I had 15 upgrades to perform and how it would be nice to have a multiple site license to lighten the burden of downloading each install separately. In the end, I spent much more time coming up with this system.

Dynamically Linked Source or DLS allows operators of two or more forums on the same server or cluster, for all intents and purposes, to use one set of source code instead of multiple sets. By using one set of source code, Op code caching is much more efficient as only one copy of the source is cached. This greatly reduces the amount of memory used caching the source code, and also makes it more efficient. It will lead to less fragmentation for systems using APC, XCache, eAccelerator, or other such methods of caching Op code.

The method uses symbolic links to a central repository of the source code. Therefore, upgrades to all sites on the network can be done quickly by just replacing the source. For patch level releases, this means the entire network can be patched with one upload. Upgrades to a new version can be done this way, or a new repository can be created and each site can be upgraded one at a time.

The issue of licensing has also been addressed. Once the product file is installed, and assuming the installation instructions were followed, each installation will report the proper license key when salting cookies, or reporting back to Jelsoft as Vbulletin does from time to time (mostly in the AdminCP).

I contacted the development team at Jelsoft and discussed this method before releasing it, and got the OK from Mike. I was initially concerned that it may break licensing or perhaps give away too much of the license reporting methodology, but now that I look back, it really does not do either.

In tests on a rather large network that contains seven forums with about 7 million posts, hundreds of thousands of registered users, and uses three load balanced servers the load has been reduced by almost 40%. You can see in the graph attached to this thread for the particulars.

I have tried to make the readme as clear as possible, but honestly that was never one of my strong points. I cannot emphasize more that before you implement this method, TEST it first in your sandbox. Also, make sure you backup your software and database before you use this.

I also have included a PDF that explains what is happening in greater detail.

Updates:

8/20/08 - This will work with the latest version of VBSEO (3.2.0 at this time). Add all the files to the DLS except the config file, and run the php-clone.php script.

This product was developed for Andy R from Social Knowledge and released here for others to enjoy.


Please don't forget to click install as I have worked quite a while on this.

Download Now

Only licensed members can download files, Click Here for more information.

Screenshots

Click image for larger version

Name:	Web Server Cacti.jpg
Views:	881
Size:	61.2 KB
ID:	85447  

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • If you like this modification support the author by donating.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
  #31  
Old 21 Aug 2008, 23:28
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Originally Posted by Beermonster View Post
Where do you put the path to the forum in? there is one line in php-clone.php that says Path to the forum not forums, which path is this?

Edit: and also how are you able to upgrade the databases or is it just files this updates?

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

you can use relative or absolute when it comes to the path i.e."

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

This does nothing to the databases.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #32  
Old 23 Aug 2008, 19:17
AWS's Avatar
AWS AWS is offline
 
Join Date: Nov 2001
Real name: Bob
Originally Posted by y2ksw View Post
I've tried it. Shortcuts won't work. So that's a Linux-only solution
What kind of problems are you having with shortcuts?. I use shortcuts to serve up files and resources all the time. Might be a little config tweaking needed.
Reply With Quote
  #33  
Old 24 Aug 2008, 10:30
GoTTi GoTTi is offline
 
Join Date: Jun 2002
great mod. so if i understand whats going on here, this mod will let me upgrade multiple forums on the same server, under different domain names (or accounts on the server), as long as i input the license information properly, it will upgrade all the forums running the software at one time?

seems like a really great mod here. i am old fashioned and enjoy upgrading the forums one at a time, and download each script for my domains individually. not all forums can be upgraded to the same version, due to hacks/addons from here that might be out dated, and forum owners dont wanna loose them...

but i do need a better understanding of this mod. does the server need any pre-req software or modifications to it or is this a out of the box kind of mod because i am not server smart, i understand real basic stuff, and i see things in here like OP code and that throws me for a loop like am i suppose to look for something to make this work right...the link in your PDF for opcode is not working so i cant get information on it.

anyways, if you could answer my question id appreciate it. this mod just lets me upgrade multiple forums that are installed on the server in 1 shot?

thanks and good job, been a while since i seen a mod like this that really benefits the server instead of its users.
Reply With Quote
  #34  
Old 26 Aug 2008, 15:32
Jafo232 Jafo232 is offline
 
Join Date: May 2004
You can use it to upgrade multiple forums at once if you want to, but I would recommend creating a new source repository and upgrading your forums one at a time just for the reasons you laid out.

Opcode caching is basically this. When a PHP script is executed on your server, the server reads the script, compiles it, and then executes the compiled code (op code). It does this EVERY time the script is called. Of course, there is a lot of overhead compiling the source over and over so caching will only compile it once and then the server will use the compiled code saving you A LOT of memory and CPU.

When you run multiple forums on a server, and you are using opcode caching, each forums file will be cached even though for all intents and purposes, they are the same files. By allowing all the forums to share one set of source files, they are only cached once.

So index.php on forum1 and index.php on forum2 are only cached one time when you use the DLS as opposed to twice. This cuts the memory usage for cached files in half.

If you are not comfortable with your server prowess, you may want to test this on some test installs before you try to do it live.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #35  
Old 26 Aug 2008, 16:37
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by AWS View Post
What kind of problems are you having with shortcuts?. I use shortcuts to serve up files and resources all the time. Might be a little config tweaking needed.
I'm using Apache on Windows 2003 Servers. When using a shortcut (actually terminates with a .php.lnk buth shows .php) Apache returns a 404 page not found error. I tried this with the sendmessage.php file in the same domain, but different folder, just to see if it does anything besides a page not found error

It may work with IIS, but I won't use IIS for other reasons
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #36  
Old 26 Aug 2008, 23:12
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Hmm, I know very little about Windows servers, but could you not associate the ink file to run as PHP?
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #37  
Old 26 Aug 2008, 23:46
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by Jafo232 View Post
Hmm, I know very little about Windows servers, but could you not associate the ink file to run as PHP?
I tried that and it failed because PHP attempts to read directly the link file.

If I run the file first (then it's in the cache), the file link works. However after restarting Apache the cache is empty and it would fail with a 404 error.

Ok nevermind, it's a great solution for Linux, and there it'll surely apply
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #38  
Old 27 Aug 2008, 00:18
sockwater's Avatar
sockwater sockwater is offline
 
Join Date: Apr 2008
You can use hard links on windows and it works (just tested). A hard link basically means that if you change either file, the other one will also be changed. You can create a hardlink on windows like this:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Also you might want to try using junctions, which are basically directory symlinks. I don't know if this would work under apache or not.

EDIT: If you try using hardlinks, I'd like to know if the opcode caching sees them as the same file or separate files.

Last edited by sockwater; 27 Aug 2008 at 00:34.
Reply With Quote
  #39  
Old 27 Aug 2008, 01:47
Jafo232 Jafo232 is offline
 
Join Date: May 2004
That would be ideal. Otherwise it is only good for patch releases.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #40  
Old 27 Aug 2008, 09:34
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by sockwater View Post
You can use hard links on windows and it works (just tested). A hard link basically means that if you change either file, the other one will also be changed. You can create a hardlink on windows like this:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Also you might want to try using junctions, which are basically directory symlinks. I don't know if this would work under apache or not.

EDIT: If you try using hardlinks, I'd like to know if the opcode caching sees them as the same file or separate files.
I'll try that tonight ... and if it works I'll make a program which creates those links recursively for Windows Servers
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #41  
Old 27 Aug 2008, 17:03
sockwater's Avatar
sockwater sockwater is offline
 
Join Date: Apr 2008
I just tested with junctions as well. Apache reads them just fine. Use this command, after downloading junction.exe from Microsoft.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

newvbdir will now be a symlink to oldvbdir, including all files and directories recursively. It seems to work the same way as a hard link, if you change the contents of either file it changes the other.

The bottom line is both junctions and hard links should be tested to see if the opcode caching only caches one copy of the file or multiple copies.
Reply With Quote
  #42  
Old 27 Aug 2008, 17:34
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Don't forget what you have to do with the config files.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #43  
Old 27 Aug 2008, 18:14
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
Originally Posted by sockwater View Post
I just tested with junctions as well. Apache reads them just fine. Use this command, after downloading junction.exe from Microsoft.


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

newvbdir will now be a symlink to oldvbdir, including all files and directories recursively. It seems to work the same way as a hard link, if you change the contents of either file it changes the other.

The bottom line is both junctions and hard links should be tested to see if the opcode caching only caches one copy of the file or multiple copies.
I tried this first. It seems as if the cache sees just one copy, thus theorically it would work. Practically however you can't use it for vBulletin because of the configuration file which can't be separated with junctions. junctions would make sense though if you had your board files in a different disk or folder - which may become handy when moving a site.

Now I'll try the fsutil solution.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
  #44  
Old 27 Aug 2008, 19:29
Jafo232 Jafo232 is offline
 
Join Date: May 2004
Originally Posted by y2ksw View Post
I tried this first. It seems as if the cache sees just one copy, thus theorically it would work. Practically however you can't use it for vBulletin because of the configuration file which can't be separated with junctions. junctions would make sense though if you had your board files in a different disk or folder - which may become handy when moving a site.

Now I'll try the fsutil solution.
Well, you probably can if you follow the directions in this mod. You have to create a configs folder in the includes directory. Place your site configs in there and rename them uniquely. Then use the config file in this mod.
__________________
Become a Lampwright today at www.lampwrights.com!
Reply With Quote
  #45  
Old 27 Aug 2008, 20:39
y2ksw's Avatar
y2ksw y2ksw is offline
 
Join Date: Aug 2003
Real name: Giovanni
I've tried now both systems (junction, fsutil) on a clean test environment. Both systems are not cached once, but as if they were separate files. A clean 3.7.0 installation and copy produce each about 2.55 MB of uncompressed eAccelerator cached files on forum home (index.php).
  • The junction feature requires to rename the config.php files and is a one-liner.
  • The fsutil feature replaces the single files with a hardlink to the common files folder, and allows thus to keep some of the files in their original state. Thus, renaming of config.php and other files are not necessary. fsutil requires an utility to execute recursively. Please allow me to test it a little before releasing
  • Both features require to repeat the junction/fsutil calls on each copy. It is not possible to copy and paste those links, and doing so, will just retrieve the original files.
__________________
Giovanni
Italian support: http://www.vbulletin.it/
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 16:50.

Layout Options | Width: Wide Color: