PDA

View Full Version : Cache System Explanation (datastore)


Logikos
17 Mar 2006, 08:20
I'm going to try and explain the datastore system used in vBulltin. This tutorial is based around users who are already familar with PHP and MySQL.What is Cache?
A memory area where frequently accessed data can be stored for rapid access. (View Definitions (http://www.google.com/search?hl=en&lr=&pwst=1&oi=defmore&defl=en&q=define:Cache)). Basicly, you can store information into the database, and grab all this information with just using 1 query. Some may question this and say: "Isn't all information in the database retrived by queires?". Yup, all information in the database is retrived using MySQL queries, though the method used is diffrent when using the datastore system in vBulletin.
I'll show you way using the datastore is a better solution for you and the users who will use your hacks.

Datastore Method
Lets say you want to use a drop down form with 3 options to choose from. Obviously you would need to create alittle script to add, delete, and edit the information; and when you make this script. Where does the info go? Well it would go into the database. Lets call the MySQL table "dropdownoptions" with 3 fields. ID, Name, and Value.

44423
Now everytime you add, edit, or delete your information, it will be changed in the database. Here is an example of the database tree. We have 3 rows of information.
ID = (1) Name = (Dog) Value(Dog)
ID = (2) Name = (Cat) Value(Cat)
ID = (3) Name = (Cow) Value(Cow)44424

After you insert this information into the database, the next step would be to insert this info in the datastore. You can do that like this.


$query = $vbulletin->db->query_read("
SELECT *
FROM " . TABLE_PREFIX . "dropdownmenu
ORDER BY id DESC
");

while ($variable = $vbulletin->db->fetch_array($query))
{
$variable_array[] = $variable;
}

build_datastore('dropmenu', serialize($variable_array));


Let me go ahead and tell you exacly what the above does. What we did was just grab all in the information in the database from the table dropdownmenu. The while() (http://us2.php.net/manual/nl/control-structures.while.php) function will loop the information from the begging to the end. While it's getting all this information, It's storing it all into the variable named $variable_array.

Then you need to use the function called build_datastore(). This function requires the following. build_datastore(1, 2). #1 is the name of your datastore item, and number two is the serialize (http://us2.php.net/manual/nl/function.serialize.php) information to store. This function will create a new datastore item with the name and all your info.

The serialize info will look like this...

a:3:{i:0;a:3:{s:2:"id";s:1:"1";s:4:"name";s:3:"Dog";s:5:"value";s:3:"Dog";}i:1;a:3:{s:2:"id";s:1:"2";s:4:"name";s:3:"Cat";s:5:"value";s:3:"Cat";}i:2;a:3:{s:2:"id";s:1:"3";s:4:"name";s:3:"Cow";s:5:"value";s:3:"Cow";}}

So now we have all our infomation stored in another area of the database, in one row. Heres a picture so you can compair it to the other ones.

44425

Don't let the serialize data freak you out. You don't really NEED to read that data. Though I kinda like doing it, so I'll break it down alittle for you. Here is a tree of the serialize data.


a:3:{
i:0; a:3:{
s:2:"id";s:1:"1";
s:4:"name";s:3:"Dog";
s:5:"value";s:3:"Dog";
}

i:1; a:3:{
s:2:"id";s:1:"2";
s:4:"name";s:3:"Cat";
s:5:"value";s:3:"Cat";
}

i:2; a:3:{
s:2:"id";s:1:"3";
s:4:"name";s:3:"Cow";
s:5:"value";s:3:"Cow";
}
}

In PHP it would look like this.

$data = array(
'0' => array(
'id' => '1',
'name' => 'Dog',
'value' => 'Dog'
),
'1' => array(
'id' => '2',
'name' => 'Cat',
'value' => 'Cat'
),
'2' => array(
'id' => '3',
'name' => 'Cow',
'value' => 'Cow'
),
);

So now we have the info stored, lets get it back out in a readable formate. :p This is the easy part. This is the code I use.


$vbulletin->dropmenu = unserialize($vbulletin->dropmenu);
foreach ($vbulletin->dropmenu AS $dropmenu)
{
echo "(ID: " . $dropmenu['id'] . ") (Title: " . $dropmenu['title'] . ") (Value: " . $dropmenu['title'] . ")<br />";
}

This will print the following:
(ID: 1) (Title: Dog) (Value: Dog)
(ID: 2) (Title: Cat) (Value: Cat)
(ID: 3) (Title: Cow) (Value: Cow)

What your doing is simple. The variable $vbulletin->dropmenu holds the information though it's still serialize. The reason why $vbulletin->dropmenu is the variable is because you need to tell vBulletin which datastore row to get. In this case it is dropmenu because remember we stored it in there like this: build_datastore('dropmenu', serialize($variable_array));

Now in order for vBulletin to know about that specific row, you need to add it to the $specialtemplates array. Example:


$specialtemplates = array(
'dropmenu'
);

So now that you understand where the variable is coming from, we need to unserialize the data, cause it looks all ugly and weird.

So $vbulletin->dropmenu = unserialize($vbulletin->dropmenu); baiscly gets the serialize info and unserialize's it using the function called unserialize() (http://us2.php.net/manual/nl/function.unserialize.php) Once that is finished you want to loop the info using a foreach() (http://us2.php.net/manual/nl/control-structures.foreach.php) function. foreach ($vbulletin->dropmenu AS $dropmenu). Doing that is storing the array info into $dropmenu and you can get each one by using the following vaiables. $dropmenu['id'], $dropmenu['title'], and $dropmenu['value'].

The reason why this is better is because vBulletin allready runs one global query to get all the datatore information. So instead of running a query everytime you want to get the drop down information, you just get it from the datastore and save that query. Some may think this is kinda extreme when you can just run a query, though as your site grows; you want to save as many queires as possiable.

If your interested in saving queries and bandwidth. I would suggest taking a look at Trigunflames (http://www.vbulletin.org/forum/member.php?u=19191#hacks) profile. He has released many hacks to help in these areas. :)

Copyright 2004-2006 vBHackers.com (http://www.vbhackers.com) All Rights Reserved.
This tutorial may not be redistributed in whole or significant part.

Princeton
17 Mar 2006, 13:53
Excellent article!

The only thing I have to say, "Damn, these thumbnail size!"

-=Sniper=-
25 Mar 2006, 18:16
great, now I can understand how it can be used :)

akanevsky
25 Mar 2006, 22:50
Great article. Thank you, LiveWire.

Developer
16 Apr 2006, 14:40
very, very, great tutorial 'ken' ;)

Wild-Wing
13 Sep 2006, 14:45
huh now i get what the a:3 means its the size of the array. damn i feel dumb now.

netwind
07 Jan 2007, 12:38
Nice. Is it useful write my own items to file-based datastore cache?
If yes, how I can do this ?

SDB
12 Jan 2007, 02:00
right..

I'm being thick here, clearly :

I want to show $numbermembers and $totalposts at the top of each page of my forum, from within the header template.

As these two values are not available unless you've already parsed the forumhome stats, I've successfully cached these values just after they are calculated at the end of index.php.

I have added it to special templates on the index.php page.

The problem I'm having is getting the data back out again, and into a useable format so I can insert it into the template.

I'm missing a step somewhere..

If I stick $vbulletin->cachedstats into the header template I see the full serialised data in there, but I can't work out from the examples exactly how (and probably more importantly WHERE) to unserialise the data and turn it into eval variables that I can use in the header template.

If anyone could give me a bit of code and tell me where to put it, I would be eternally greatful!

Thanks in advance.

Simon

netwind
12 Jan 2007, 09:59
You need $vbulletin->cachedstats = unserialize($vbulletin->cachedstats);
for all data (except loaded from file based datastore in my case).
store full parsed html code for maximum speed.
If not, use like this in templates :
Total users : {$vbulletin->cachedstats[totalusers]}

SDB
12 Jan 2007, 10:30
Thank you netwind.

Could you please tell me where I can put the unserialise code?

I've tried lots of places, but wherever I'm placing it seems to be either before the cache memory is populated, or after the header has been built.

cheers

netwind
12 Jan 2007, 11:04
Read beginning from words
Now in order for vBulletin to know about that specific row, you need to add it to the $specialtemplates array.
Try parse_templates hook.

SDB
12 Jan 2007, 11:35
LOL

I'd already added it to $specialtemplates, as stated in my original post :)

Thank you very much for your help, the parse_templates hook worked a treat.

The problem I have now, is having to manually add 'cachestats' to the special templates array of every single page. Is there a way to maybe add to this array from within global.php or something?

Thanks again

Simon

aah..

answered my own question (wonders will never cease!)

I've added :
$specialtemplates[] = 'cachestats';

to the top of the global.php file which works. So will play around to find the right hook to put it in.

--

additional :
It seems that there isn't a hook early enough in global.php
I'm really playing around in the dark here, so if anyone knows of a hook which runs early enough, could you please let me know?

Cheers
Simon

netwind
12 Jan 2007, 11:46
Hook global_start for $specialtempates[] .=
and parse_templates (and later called specific hooks) for unserialize and other jobs

SDB
12 Jan 2007, 11:47
LOL

what a muppet.. thanks mate :)

correction..

global_start appears to be way too late. There are two that seem to run earlier than this, and neither of those are early enough either.

ok.. I've whacked the $specialtemplates[] = ... into global_start..

but the variable is empty when the header template is built.

when I had identical set-up, but instead of global_start, I hacked the line into the start of the global.php file it worked fine.

netwind
12 Jan 2007, 12:01
Sorry, this my tip was for custom stand-alone scripts like this :

$specialtemplates = array(
'banemail',
);
require_once('./global.php');


I think modifying global.php only one way..

SDB
12 Jan 2007, 12:05
ok.

Thanks a lot for your help netwind, it got me working anyway, with only one file edit.

If anyone else knows a hook I can use, it would be great.

Thanks

Simon

netwind
12 Jan 2007, 13:02
I have another related problem: build html in admincp/ not work,
since style not selected. I need example code which build all special vars for function fetch_template() in default style.

bigfoot
07 Feb 2007, 23:48
I must say this is a GREAT article, but I need a little help building the forumcache. If I do as this article says, it just f*cks up my forum and its unable to load correctly :(

thincom2000
11 Feb 2007, 08:59
I must say this is a GREAT article, but I need a little help building the forumcache. If I do as this article says, it just f*cks up my forum and its unable to load correctly :(

I believe forumcache has its own function.

Like build_options(), build_datastore(), build_birthdays(), build_maxloggedin(),
perhaps build_forumcache() ?

I think this thread only applies to datastore objects you have created yourself.

bigfoot
15 Feb 2007, 19:24
I can't find any similar functions that work for the forumcache. Actually build_datastore() works fine saving the data, but I think the input is wrong :(

dwh
28 Sep 2007, 16:23
This was a great article and it explained the concept very well. I do have a minor quibble in your example. Rather than name= dog, value=dog, it might help people understand better if the var name and var value were different, e.g. name=dog, value=terrier.

Something I'd add to the discussion is that cron and serialize go hand in hand. The best place to use serialize is when accessing data frequently, therefore using vb's cached datastore w/o adding a query...but the updating of data is relatively infrequent. A classic way to update that data periodically is cron.
http://www.vbulletin.org/forum/showthread.php?t=141695&highlight=cron

vertigo jones
09 Nov 2007, 20:34
I really hope they add a hook that will allow us to add to $specialtemplates before it's actually used. It's kind of a bummer to have to hack up a file over something so simple.

Not to suggest that adding the hook is simple. I think there's some initialization for the hook system in the datastore, or something along those lines. It's just a pain and it would be cool if they stumble across some fix in the future.

Logikos
09 Nov 2007, 21:53
I really hope they add a hook that will allow us to add to $specialtemplates before it's actually used. It's kind of a bummer to have to hack up a file over something so simple.

Not to suggest that adding the hook is simple. I think there's some initialization for the hook system in the datastore, or something along those lines. It's just a pain and it would be cool if they stumble across some fix in the future.
I think the trick around that is with the "array_merge()" function. It's been awhile so I'm not completely sure, but I believe it's due to the $specialtemplates array being called before global.php.

vertigo jones
10 Nov 2007, 03:14
It's because the query that uses $specialtemplates and populates the datastore variables comes before global_start. I don't really understand why, but I think it has something to do with the hook system not being able to work until the datastore info is available.

Opserty
10 Nov 2007, 10:31
Have you tried the init_startup hook? There is another Datastore fetch executed after it I think. This isn't tested but you could maybe try:

Hook Location: init_startup

$datastore_fetch[] = "'dropmenu'";

vertigo jones
10 Nov 2007, 22:45
Awesome, thanks man. I haven't tested it either, but it looks like that should work.

--------------- Added 1194736320 at 1194736320 ---------------

Ok, it did work. It added the extra query, but it works.

You have to be careful and make sure it's like this though:

$datastore_fetch[] = "'dropmenu'";

Doing it without both sets of quotes makes everything (including the admincp) inaccessbile until you go in and correct it in both the plugin table and the datastore entry. They try to warn against it in the code, but I wasn't exactly sure what it meant.

Opserty
11 Nov 2007, 11:03
Ooops yup you need to put it in quotes because otherwise MySQL will read it as a column name instead of a value.

I'll correct it in my original post.

Antivirus
13 Feb 2008, 03:51
The Function build_datastore also has a 3rd parameter which should be mentioned, as it seems all serialized data in the datastore table requires a value of 1...


#######################################################
/**
* Saves the specified data into the datastore
* @param string The name of the datastore item to save
* @param mixed The data to be saved
* @param integer 1 or 0 as to whether this value is to be automatically unserialised on retrieval
*/

function build_datastore($title = '', $data = '', $unserialize = 0)



By setting the 3rd param to true, it's no longer necessary to run unserialize like this:
$vbulletin->dropmenu = unserialize($vbulletin->dropmenu);

There'fore we can just access the info from the datastore like this:


$vbulletin->dropmenu['id']

RobDog888
05 Jun 2008, 19:11
What if I have my own class object and I want to build cache acessible from it only and not use the $vbulletin object?

Like ...
$myclass->mycache['something']

instead of $vbulletin->mycache['something']

How would I change it so its like that?

briansol
06 Jun 2008, 07:50
With a couple tweaks (probably because this is 2 years old...), i've successfully got my first datastore plugin working :)

Thanks for the write up Ken. Never experienced with the datastore much.

seko3
06 Jun 2008, 20:17
With a couple tweaks (probably because this is 2 years old...), i've successfully got my first datastore plugin working :)

Thanks for the write up Ken. Never experienced with the datastore much.

Can you please share it with us. I am so tired of trying to make this work. It has been 5 days but I couldn't cache anything yet. It would be great to know how to cache sql queries.

RobDog888
06 Jun 2008, 22:10
With a couple tweaks (probably because this is 2 years old...), i've successfully got my first datastore plugin working :)

Thanks for the write up Ken. Never experienced with the datastore much.
What was different from the posted solutions?

briansol
06 Jun 2008, 22:15
i'm still trying to figure out how to export this entire thing (with all my crons/ phrases etc) but no one is helping me out on .com
http://www.vbulletin.com/forum/showthread.php?t=274552
otherwise, i'd just post my entire product.


basically, this is what i did.

1) I made a cron job to populate a table from live post/thread table data. (Perhaps this table is useless now altogether, but i like having a table i can reference during testing. next version probably won't include this cache_ table at all)


<?php

error_reporting(E_ALL & ~E_NOTICE);
if (!is_object($vbulletin->db))
{
exit;
}

//clean up...
$vbulletin->db->query_write('TRUNCATE TABLE ' . TABLE_PREFIX . 'cache_activetopics');


$vbulletin->db->query_write('insert into ' . TABLE_PREFIX . 'cache_activetopics
/// my big select query here
');

// now, this loads that new data into an array, which is then shoved into the build datastore function:

$query = $vbulletin->db->query_read('SELECT * from ' . TABLE_PREFIX . 'cache_activetopics');

while ($variable = $vbulletin->db->fetch_array($query))
{
$variable_array[] = $variable;
}

build_datastore('skm_active_cache', serialize($variable_array), true);


log_cron_action('', $nextitem, 1);

?>


Again, you can skip truncating and building the table... just run the big query here as the $query = part to get your array/data.


Set up the cron job to run this script as you like.
Run it once manually to make sure you have data in the datastore row.

there will actually be a row with the name skm_active_cache in the datastore table.


2) from there, i have 4 plugins in my product

Product : SKM - Active threads from past 24 hours
SKM Active Threads - Cache output templates cache_templates
SKM Active Threads - Cache Special Templates (datastore row) init_startup
SKM Active Threads - Parse and Print forumhome_complete
SKM Active Threads - Template Group template_groups

a) cache_templates hook : contains the simple caching of my custom template which prints 1 row of data (ie, <tr><td>1</td><td>2</td></tr> ) similar to the 'threadbits' template

it contains a simple line:
$globaltemplates[] = 'skm_activethreads24_forumhome';

b) init_startup hook : special templates is probably a mis-nomer at this point, as i'm NOT using the special templates array merge style.
This file pulls the datastore row into memory:
$datastore_fetch[] = "'skm_active_cache'";

note the quotes are important. This name should match what you ran in your cron under the build_datastore() function.

c) forumhome_complete hook : parse the data and print it to the template step. obviously, forumhome_complete should match the area you are trying to hook into. in my case, it was on the forum home.


foreach ($vbulletin->skm_active_cache AS $mycache)
{
$output['threadid'] = $mycache['tid'];

eval("\$mythreadbits .= \"".fetch_template('skm_activethreads24_forumhome')."\";");
}


d) the template will look for the $output array to print, ala:
<a href="showthread.php?t=$output[threadid]&amp;goto=newpost">

e) the template group is not required... just makes things easier to group when you have a lot of templates.


That's about it.

basically, you need to fill the datastore with your query, and pull with a foreach of the array once the data has been initialized.

gabrielt
02 Jan 2009, 22:21
Hello,

Thank you so much for your tutorial. This was exactly what I was looking for. I write my own products/plugins (our forums is huge) and I was wondering how to cache stuff around.

However I faced one problem: I couldn't read data back.

The solution was to add a "1" parameter on the store command:

build_datastore('name',serialize($variable),1);

When retrieving you don't need to "unserialize" it.

Also I was trying to call my cached array inside a class that hasn't had $vbulletin as a global variable, so I needed to add:

global $vbulletin;

On the top of my plugin.

And I also needed to add a plugin for the init_startup hook, adding:

$datastore_fetch[]="'name'";

I hope this feedback helps other coders.

Cheers,
Gabriel.

Jafo232
11 Feb 2009, 17:20
And I also needed to add a plugin for the init_startup hook, adding:

$datastore_fetch[]="'name'";



This should really be added to the first post. Took forever to figure this out. :)

bananalive
21 Apr 2009, 19:42
Is there a datastore per user?

mokujin
21 Apr 2009, 23:23
If you have this, now How can I delete one of the data (red below) ???
a:3:{
i:0; a:3:{
s:2:"id";s:1:"1";
s:4:"name";s:3:"Dog";
s:5:"value";s:3:"Dog";
}

i:1; a:3:{
s:2:"id";s:1:"2";
s:4:"name";s:3:"Cat";
s:5:"value";s:3:"Cat";
}

i:2; a:3:{
s:2:"id";s:1:"3";
s:4:"name";s:3:"Cow";
s:5:"value";s:3:"Cow";
}
}

rob01
22 Apr 2009, 00:59
it could be possible to fetch more than 1 value

for example

(ID: 1) (Title: Dog) (Value: Dog , dog2, dog4)

and be able to delete dog4

(ID: 1) (Title: Dog) (Value: Dog , dog2)

or add more


any ideas?

maybe first check if its empty or :S?

solita_ugc
11 May 2009, 07:22
Hello,

I seem to be unable to persist the values in datastore. Variables built and stored to datastore in global_start plugin are lost in subsequent requests.

I wish the purpose, usage, lifecycle, limitations and performance of datastore would be discussed more in the article.

-Pete

IdanB
22 May 2009, 11:58
And I also needed to add a plugin for the init_startup hook, adding:

$datastore_fetch[]="'name'";


Thank you for this, was pulling my hair out my head trying to get this to work.
That's what i was missing :)

mokujin
23 Dec 2009, 22:43
How does it work in vb4? Thanks

squishi
29 May 2010, 10:58
Old, but still excellent article.

The Function build_datastore also has a 3rd parameter which should be mentioned, as it seems all serialized data in the datastore table requires a value of 1...

That is not correct (or no longer correct).

On my vbulletin v3.8.4, I see values of 0, 1, and 2 for the "unserialize" column in the datastore database table.

What does this mean? Where is it used?
I am inclined to think that
0 = no unserialization, 1 = unserialize data, 2= unserializa and wakeup

A wakeup call is automatically added by php if data is an object, though.
So what does this column mean?