Register Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
  #766  
Old 19 Jan 2010, 17:24
eoc_Jason's Avatar
eoc_Jason eoc_Jason is offline
 
Join Date: Dec 2001
Originally Posted by mute View Post
We have 2 post indexes, one or our live post table, and one for our archived post table. They each have 30 million posts each. I don't see a point in sharding the post indexes aside from being able to take advantage of multiple CPUs when indexing.

The way I see it, if I can keep the old indexes online while I do a full reindex, I don't really care how long the full reindex takes since (at least in our case), the search server is just a slave database server and not our primary.
Splitting up the posts index into several sources has more advantages that just re-indexing. As kmike eluded to, you can setup "agents" on your server so that when a person does a search it will search all the sources in parallel utilizing one source per CPU.

For most people though, even large indexes we are all probably just using 1 CPU without realizing it and still maintaining less than one second search times...

This week I'm going to be swamped working on some new AMD processor reviews, so work on the new sphinx code will have to be set aside until Tuesday next week.

I'm super pumped though, after diving into the documentation and code, I realized it's really not that bad at all...
__________________
My Site: EXTREME Overclocking

Do not PM me with your iTrader problems or asking for the code. I will just delete your PM without reading it.
Reply With Quote
  #767  
Old 19 Jan 2010, 18:45
kris kris is offline
 
Join Date: Nov 2001
Splitting up the posts index into several sources has more advantages that just re-indexing. As kmike eluded to, you can setup "agents" on your server so that when a person does a search it will search all the sources in parallel utilizing one source per CPU.
eoc_Jason, can you share how to split post table index into several sources ?
Reply With Quote
  #768  
Old 26 Jan 2010, 11:52
kmike kmike is offline
 
Join Date: Oct 2002
FYI, realtime index updates in Sphinx are near:
http://twitter.com/sphinxsearch/statuses/8179531738
Looks like so far the realtime indexes are only accessible via SphinxQL.
Reply With Quote
  #769  
Old 25 Feb 2010, 01:30
weeno weeno is offline
 
Join Date: Jan 2002
hey guys,

Can someone post their sphinx.conf file for sphinx version 0.9.9? I'm still running into a few problems so wanted to compare.

arn
Reply With Quote
  #770  
Old 26 Feb 2010, 03:43
boggseric's Avatar
boggseric boggseric is offline
 
Join Date: Sep 2009
Was post number 1 supposed to contain some XML mods? Or am I missing something here.

edit: Never mind, there are no xml files, you have to create the plugins manually and copy and paste the code into them.

Last edited by boggseric; 27 Feb 2010 at 02:01.
Reply With Quote
  #771  
Old 01 Mar 2010, 22:42
ivanp ivanp is offline
 
Join Date: Sep 2007
Any update for Sphinx for vB4?
Reply With Quote
  #772  
Old 01 Mar 2010, 23:12
Lea Verou Lea Verou is offline
 
Join Date: Jul 2005
Real name: Lea Verou
Has anyone used Sphinx with languages != english?
Does it work correctly?
FULLTEXT had issues with greek...
Reply With Quote
  #773  
Old 13 Mar 2010, 18:18
boggseric's Avatar
boggseric boggseric is offline
 
Join Date: Sep 2009
Originally Posted by kerplunknet View Post
Has anyone figured out the "Find More Posts by UsernameHere" issue? It results in these error messages, but the results seem to work properly:


Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

I really don't want to put Sphinx live until this is fixed... any advice would be welcome.

I am wondering if it has something to do with using an older 3.6 version of vBulletin.
This is a pretty old post but it is still relevant. The from looking at the line it's refereing to it looks like he problem is with the function that it is trying to find the search text in the post result to highlight it. Since there is no search text the value is null and strpos does't like that.


edit: I think I got it fixed. The problem is with the process_quote_removal function.
Keep in mind I am running 3.8.4pl2
I edited includes/functions_search.php and made the following changes

Search for this block of code:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

This line is the culprit for me:
Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

it appears that $word ends up being null which causes strpos to puke the errors.

so I changed the function to this:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

You can see I just added an if statement to check if $word is null. If $word is null it skips over the code since there is obviously nothing to highlight.

I have tested the normal search and the search term is still highlighted and I am no longer getting errors during search for more posts by this user.. YMMV.

Last edited by boggseric; 13 Mar 2010 at 19:36. Reason: I think I fixed it
Reply With Quote
  #774  
Old 14 Mar 2010, 01:11
WoodiE WoodiE is offline
 
Join Date: May 2002
Is this real or just an info about Sphinx? Someone told me that it costs $2,000 to have it setup on vBulletin - no?
Reply With Quote
  #775  
Old 14 Mar 2010, 01:57
TechGuy TechGuy is offline
 
Join Date: Nov 2001
Real name: Mike
Unfortunately, no... you can install Sphinx yourself for free, but it is no small task and there are some issues to deal with.

The $2k is likely referring to the "Axivo Searchlight" product, though I hesitate to use the word product as it's been "coming soon" for more than a year. They finally released a closed beta, so I suspect it's still being developed, but that was over three months ago and still no product.

Last edited by Marco van Herwaarden; 14 Mar 2010 at 10:54.
Reply With Quote
  #776  
Old 14 Mar 2010, 01:59
boggseric's Avatar
boggseric boggseric is offline
 
Join Date: Sep 2009
Originally Posted by WoodiE View Post
Is this real or just an info about Sphinx? Someone told me that it costs $2,000 to have it setup on vBulletin - no?
I just set it up using the info in post 1 and the files from post 582

I used the info from this post to make sure it stayed running and this post to setup the crons for delta and nightly.

Other than that the only changes I made were from my previous post to fix the "find more posts by this user" issue.

It is now up and running on my 3.8.4pl2 forums with:
Threads: 799,861, Posts: 11,892,282, Members: 24,815

I set the max amount of of results to 20,000 and it's working great. Searches no longer take my forum to it's knees.

The things I haven't done yet are to remove the indexes from post and convert my tables to innodb.


Total cost? My time reading through this never ending thread and compiling the data above.
Reply With Quote
  #777  
Old 14 Mar 2010, 15:10
WoodiE WoodiE is offline
 
Join Date: May 2002
TechGuy - I didn't realize there was two different options for using Sphinx on vBulletin. Though by the sounds of it neither are very easy to setup. One either very pricey, two according to boggseric a lot of files and editing to do - both still have issues and nothing everything working.

Does anyone know of any other solutions to making vBulletin search easier on the server and sill maintain all of vB's features?
Reply With Quote
  #778  
Old 14 Mar 2010, 20:24
adnoid adnoid is offline
 
Join Date: May 2008
Originally Posted by boggseric View Post
I just set it up using the info in post 1 and the files from post 582

I used the info from this post to make sure it stayed running and this post to setup the crons for delta and nightly.

Other than that the only changes I made were from my previous post to fix the "find more posts by this user" issue.

It is now up and running on my 3.8.4pl2 forums with:
Threads: 799,861, Posts: 11,892,282, Members: 24,815

I set the max amount of of results to 20,000 and it's working great. Searches no longer take my forum to it's knees.

The things I haven't done yet are to remove the indexes from post and convert my tables to innodb.


Total cost? My time reading through this never ending thread and compiling the data above.
Thank you for putting this all together. I've been looking at the alternatives for our board, now approaching 5,000,000 posts. I'm not a heavyweight coder, but I'll be giving your method a shot.
Reply With Quote
  #779  
Old 15 Mar 2010, 13:39
boggseric's Avatar
boggseric boggseric is offline
 
Join Date: Sep 2009
Originally Posted by WoodiE View Post
TechGuy - I didn't realize there was two different options for using Sphinx on vBulletin. Though by the sounds of it neither are very easy to setup. One either very pricey, two according to boggseric a lot of files and editing to do - both still have issues and nothing everything working.

Does anyone know of any other solutions to making vBulletin search easier on the server and sill maintain all of vB's features?
You'd probably be best off if you did it yourself, that way if you need to make changes in the future you know what you are doing. It's not overly complicated.

I don't know squat about linux or vB but I know computers and some programming (other than PHP). I managed to fumble my way through it.

If you can afford the 2k solution, good. If not maybe you'd be better off paying someone to set sphinx up for you.
Reply With Quote
  #780  
Old 15 Mar 2010, 13:42
TechGuy TechGuy is offline
 
Join Date: Nov 2001
Real name: Mike
My point is that even if you can afford the $2k solution, it doesn't matter -- that solution doesn't exist.

Without reading through the entire thread, what's the status of using advanced search fields using the solution you mentioned earlier? (It was a long time ago when we last tried, but I remember we had to drop it because we couldn't sort by most of the fields.)
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 09:01.

Layout Options | Width: Wide Color: