PDA

View Full Version : Quad Opterons + FreeBSD + Lighttpd + FastCGI/PHP + APC = Fast vB :)


Erwin
03 May 2006, 00:37
Yup, I migrated my forums to new servers (dual xeon to quad opterons) using completely new software - FreeBSD, Lighttpd instead of Apache. Lighttpd is really amazing, and I recommend that everyone switch from Apache to Lighttpd. APC works fine on the setup too. Just sharing.

Brad
03 May 2006, 04:32
Good to hear everything is going well. ;)

Erwin
03 May 2006, 07:37
FreeBSD is quite a robust OS with a very good upgrade system. I highly recommend it as a good basis for a server.

kmike
03 May 2006, 19:28
Well, all benchmarks I saw suggest otherwise - FreeBSD scales worse than Linux in multiprocessor setup. YMMV, of course, but I also can't see how the FreeBSD ports system is superiour to the yum/apt repositories. And if you're into recompiling everything to tune up for your system, Gentoo's ebuilds are just for that.

edit: I see that you didn't make it to SoftLayer...

FlyBoy73
03 May 2006, 21:30
Sounds good, Erwin. Thanks for sharing your experience with this setup.
I'm wondering if I could get away with something like this. My new Adaptec SAS RAID controller card doesn't have the widest range of drivers available yet, thus I'm using CentOS 4.1 rather than the latest version.
I'm always down for going faster ;)

mathias
03 May 2006, 21:43
how is converting apache over to lighhtpd

adalren
03 May 2006, 21:50
Erwin, did you notice less load on the servers vs apache? Any optimizations? I converted to lighttpd but didn't see a noticable drop in load but the pages are generated slightly faster.

DJ XtAzY
03 May 2006, 21:55
hmm i guess im gonna switch from apache =]

but i like gentoo as my OS =]

Erwin
03 May 2006, 23:40
Well, all benchmarks I saw suggest otherwise - FreeBSD scales worse than Linux in multiprocessor setup. YMMV, of course, but I also can't see how the FreeBSD ports system is superiour to the yum/apt repositories. And if you're into recompiling everything to tune up for your system, Gentoo's ebuilds are just for that.

edit: I see that you didn't make it to SoftLayer...
Only if you compile the wrong kernel. :) FreeBSD 6.0 using AMD64 SMP kernel on a Quad Opteron just plain flies.

Ports are better than yum/rpm because ports automatically fixes all broken dependencies and installs everything automatically using 1 command: portinstall apache or portinstall mysql5

Can't get any easier than that. ;)

Erwin
03 May 2006, 23:41
Sounds good, Erwin. Thanks for sharing your experience with this setup.
I'm wondering if I could get away with something like this. My new Adaptec SAS RAID controller card doesn't have the widest range of drivers available yet, thus I'm using CentOS 4.1 rather than the latest version.
I'm always down for going faster ;)
The only downfall of FreeBSD is its lack of driver support - I had to change my RAID card and go with a RAID 10 rather than a 0+1 setup. But it's all good.

Erwin
03 May 2006, 23:41
how is converting apache over to lighhtpd
On FreeBSD, it's this: portinstall lighttpd.

;)

Erwin
03 May 2006, 23:42
Erwin, did you notice less load on the servers vs apache? Any optimizations? I converted to lighttpd but didn't see a noticable drop in load but the pages are generated slightly faster.
Maybe because your webserver wasn't badly overloaded? I can tell you that I was able to consolidate my multiple webservers into a single more powerful one which makes maintaining files and the datastore cache much easier.

Zachery
04 May 2006, 00:19
Gentoo's portage system does all of that too, and i personally would have gone with Gentoo over FreeBSD.

Erwin
04 May 2006, 01:13
Yeah but Gentoo is still Linux and it basically copied the BSD ports system. BSD is the oldest *nix around AFAIK. In any case, there is no one OS that suits everyone. :) They are all good in their own way. I just wanted to try something different. :)

Zachery
04 May 2006, 01:21
I'm glad its working for you, what is your current setup now, and would you mind sharing who you went with?

Erwin
04 May 2006, 02:58
For the database server, it's a dual processor/dual core Opteron 270 with 8Mb RAM, 8 x 15k SCSI PCI-X Raid 10 array, FreeBSD 6, MySQL 5 - runs like the wind. Loads never hit 0.5. :) The main webserver is the Quad Opteron with similar specs. We also have image server, irc, attachment, dns, and a slave search server.

We went with webnx in the end - best prices in town - look for Dan Paultz and tell him Erwin sent you - he'll look after you well. :)

kmike
04 May 2006, 06:21
Only if you compile the wrong kernel. :) FreeBSD 6.0 using AMD64 SMP kernel on a Quad Opteron just plain flies.
Well, but you haven't compared it to Linux, have you?
I already posted some links to FreeBSD/Linux comparisons in MySQL performance, here're some more:
FreeBSD 5.x: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2005-06/0331.html
FreeBSD 6.x:
http://www.archivesat.com/FreeBSD_Performance_tuning/thread412787.htm
http://www.archivesat.com/FreeBSD_Performance_tuning/thread420280.htm


Ports are better than yum/rpm because ports automatically fixes all broken dependencies and installs everything automatically using 1 command: portinstall apache or portinstall mysql5

Can't get any easier than that. ;)
I see some confusion here: yum does resolve the missing dependencies! I'm not sure what do you mean by "broken" dependencies, probably 3rd party rpm packages? But the same goes for the ports - if you install some 3rd party program manually, outside the ports system (doing usual "./configure; make; make install"), you have to resolve its dependencies manually.

There's one fundamental difference between the ports and rpm-based package managers - rpm's are all binary, and you build your ports from the source when installing them. It could be considered as an advantage as you're supposedly tuning the package for your system when building it (with the right parameters), but some time ago I saw the tests of Gentoo (which is touting the same approach as an advantage) vs some binary-based distribution didn't reveal any substantial performance advantages.

And finally, you can easily rebuild RPM package from the source rpm with options tailored for your architecture, too (that's what I did for our install, rebuilt php rpms with Opteron optimizations). It's as easy as doing "rpmbuild --target=athlon --rebuild yourpackage.src.rpm"

Erwin
04 May 2006, 06:28
Thanks for the links, it's very specific stuff there. Interesting.

I know about yum - I ran RHEL servers for 4 years. :)

I'm not here to start an OS-debate. :) I'm just posting my experiences which mirrors the experience of many others. :)

T.S.
04 May 2006, 12:29
For the database server, it's a dual processor/dual core Opteron 270 with 8Mb RAM, 8 x 15k SCSI PCI-X Raid 10 array, FreeBSD 6, MySQL 5 - runs like the wind. Loads never hit 0.5. :) The main webserver is the Quad Opteron with similar specs. We also have image server, irc, attachment, dns, and a slave search server.

We went with webnx in the end - best prices in town - look for Dan Paultz and tell him Erwin sent you - he'll look after you well. :)
I would have went with a little more ram. :p

Erwin
05 May 2006, 01:58
My swap is still zero. :) Loads are still less than 1. I think it's okay for now considering all it has is the one database as a dedicated database server, MINUS any search or IP searches - we have another dedicated slave database Dual Xeon with 4 MB or RAM to do just search or IP searches. :)

T.S.
05 May 2006, 02:12
My swap is still zero. :) Loads are still less than 1. I think it's okay for now considering all it has is the one database as a dedicated database server, MINUS any search or IP searches - we have another dedicated slave database Dual Xeon with 4 MB or RAM to do just search or IP searches. :)
Hopefully you mean GB rather than MB. ;)

Erwin
05 May 2006, 04:32
LOL!

Yeah, Gb. The last time I had 4 Mb RAM was on a 386... My 286 had 1 Mb RAM and my XT had 64kb.

eva2000
05 May 2006, 14:12
For the database server, it's a dual processor/dual core Opteron 270 with 8Mb RAM, 8 x 15k SCSI PCI-X Raid 10 array, FreeBSD 6, MySQL 5 - runs like the wind. Loads never hit 0.5. :) The main webserver is the Quad Opteron with similar specs. We also have image server, irc, attachment, dns, and a slave search server.

We went with webnx in the end - best prices in town - look for Dan Paultz and tell him Erwin sent you - he'll look after you well. :)
Those cpu loads are very nice :D

Xorlev
07 May 2006, 18:55
I recommend giving Litespeed Webserver a try as well...I much prefer it over Lighttpd.

Zachery
07 May 2006, 20:30
Theres that whole cost thing.
Enterprise Edition (4 CPU) $1199 ;)

Xorlev
07 May 2006, 22:38
The standard version is free and has no limits as long as it keeps to one CPU (or a dual core) and 300 concurrent users. But the benchmarks speak for themselves when you're under high load.

I do love both however. :)

Erwin
08 May 2006, 01:30
Lighttpd works very well if the config is tweaked. I'm using the Quake.net config example - works great.

kerplunknet
08 May 2006, 02:36
Lighttpd works very well if the config is tweaked. I'm using the Quake.net config example - works great.
Where would a person find this "Quake.net config" ?
:)

Erwin
08 May 2006, 04:26
Someone on vB.com posted a link to it in the large forums thread. :) Should still be there.

bang
08 May 2006, 18:20
gotta back up Zachery here. Gentoo is plain awesome. upgrading just doesn't get easier than Gentoo has it - execute "emerge -u world" in screen and forget about it. it's very stable - I have a single-CPU Gentoo box that has 260 days uptime and handles ~15k uniques a day. and needless to say, Gentoo logo is the best. Pacman forever! :D

kerplunknet
08 May 2006, 23:00
Ah, thanks ;)

http://nopaste.php-q.net/206712

Erwin
08 May 2006, 23:11
Mind you, my lighttpd is causing server errors at the moment, and I'm trying to troubleshoot it. It's possible APC doesn't want to work it. I can't get eA to work with it.

Zachery
09 May 2006, 00:23
You are trying to run both at the same time erwin? eA and APC accomplish the same thing.

I've got lighttpd running on a new server just for serving images, and large downloads (E3 is a killer on the site in question). Has made a huge differnece offloading just the images to another server. Loads on the three servers are below 1 and its running great.

Erwin
09 May 2006, 01:09
No, not at the same time - one or the other.

Erwin
09 May 2006, 01:41
I had to remove the quake.net tweaks in the end. :) I was getting server errors with them for some reason...

Erwin
09 May 2006, 01:44
By the way, my servers are with www.webnx.com and they are by far the cheapest providers of top end hardware I've found. We're talking Quad Opterons or Dual/Dual Opterons with 15k SCSI hardware RAID arrays for a few hundred bucks - and their service has been great so far - unmanaged servers but they help out wherever they can. I highly recommend them. :) You can contact Dan Paultz, who's a great guy, if you're interested in some high-end boxes to lease. The provisioning of the boxes take a while but once they're up they're good.

Zachery
09 May 2006, 03:37
Erwin, which method did you decide to go for when doing virtial hosts? If you have any?

Erwin
09 May 2006, 06:15
Virtual hosts are flexible in lighttpd - I don't do them at the moment but if I did I would be IP based.

Zachery
09 May 2006, 06:20
I was refering to the simple version or the conditional based version.

adalren
17 May 2006, 22:34
Mind you, my lighttpd is causing server errors at the moment, and I'm trying to troubleshoot it. It's possible APC doesn't want to work it. I can't get eA to work with it.

What kind of problems are you having with lighttpd?

We're using lighttpd + phpfcgi + 64bit CentOS and lately experiencing segfaults from the php processes. I see a few bug listings regarding segfaults on APC (http://pecl.php.net/bugs/search.php?cmd=display&status=Open&package_name[]=APC) and have a hunch it might be that.

BTW, eA does not work on a 64bit OS.

Zachery
17 May 2006, 23:00
What kind of problems are you having with lighttpd?

We're using lighttpd + phpfcgi + 64bit CentOS and lately experiencing segfaults from the php processes. I see a few bug listings regarding segfaults on APC (http://pecl.php.net/bugs/search.php?cmd=display&status=Open&package_name[]=APC) and have a hunch it might be that.

BTW, eA does not work on a 64bit OS.
I think it was digitalpoint on vB.com who commented about segfaults and APC under high load.

Dan
17 May 2006, 23:03
I've had a recent experience with FreeBSD servers and Erwin is right, it's really a good distro to use.

But with Lighttpd reference I haven't really had a chance to play with it much in a production enviroment, I'm currently playing around with it on my localhost. Hopefully one day my host can switch it or I could get my own server and use it.

Erwin
17 May 2006, 23:38
Ahhh... eA doesn't work with 64bit... didn't know that... that explains it.

adalren
18 May 2006, 02:00
I think it was digitalpoint on vB.com who commented about segfaults and APC under high load.
That can't be the case since the load is < 1. It seems like a child php process segfaults and serves error pages until the parent process kills it after the # of PHP_FCGI_MAX_REQUESTS. At least it recovers.

Erwin
18 May 2006, 07:27
Anyone want to share their lighttpd.conf? I can't use the quake.net one as my server crashes with it (server 500 error).

I'll share the tweaks I've made to mine:

Add these lines to the top:

server.max-keep-alive-requests = 0
server.max-fds = 2048
server.stat-cache-engine = "fam"


Basically, switch off keep-alive (well, it helps to speed things up on my webserver), double the maxed opened files from 1024 to 2048 (if you have a busy server) and use the stat cache system.

It seems to work well for me so far - let's see how it goes at peak times (it's off-peak now, only 1,000 users online).

More info on optimizing lighttpd here:

http://www.lighttpd.net/documentation/performance.html

kmike
18 May 2006, 20:39
Hmm, keep alive should help multiplexing web servers like lighttpd. Number of open files should not be an issue if you're not on the shared hosting, and there're no other downsides to enabling keep-alive.

adalren
18 May 2006, 21:42
How do you get fam/gamin to work? I installed gamin (yum install gamin) and there wasn't any daemons or init scripts to run.

Here is my conf file. It runs fine with apc 3.0.8 but segfaults with 3.0.10.

server.event-handler = "linux-sysepoll"
server.max-fds = 4096
server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.max-read-idle = 20
server.max-write-idle = 180

Erwin
18 May 2006, 23:45
Hmm, keep alive should help multiplexing web servers like lighttpd. Number of open files should not be an issue if you're not on the shared hosting, and there're no other downsides to enabling keep-alive.
I thought keep alive is only useful for static files? I may be wrong obviously, I'm no expert. No. of files may be significant if there are lot of attachments, again from what I've read, so it's probably more relevant for an attachment server. Enabling keep-alive just means connections are kept longer - I have to say that when I was using apache, I never had keep alive on, even with my image server.

kmike
19 May 2006, 09:44
Well, I'm going to get to the basics on that.
The beauty of multiplexed web servers (lighttpd, thttpd, Zeus, nginx, etc) is in that number of connections isn't a choke point, their performance is usually disk (for static file serving) or CPU (for dynamic content) bound.
So there's _no_ reason not to enable keep-alive as it eliminates the need to re-open connection for the next request(s), and managing idle connections does not take up any resources for a multiplexed web server utilizing one of the advanced solutions to c10K problem (http://www.kegel.com/c10k.html) like epoll, rtsig or kqueue.

Back to Apache, everyone's disabling keep-alive in Apache because every connection, even an idle one, requires one active Apache process or thread to manage it, taking up memory.

I hope it answers your questions.

Erwin
19 May 2006, 14:05
For some reason, the default

server.max-write-idle = 360

is too short... I've increased that to 720 and it's still too slow according to the error log.

I'll give:

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4

a try then. :) I'll report back.

In relation to kqueue, I'm using FreeBSD so I have this line:

server.event-handler = "freebsd-kqueue"

That covers that yeah?

Xorlev
19 May 2006, 17:50
Probably should bump up the keep-alive idle time, as not everyone is able to load, read, and make a new request in that time, unless you count the several requests for images and the like. I keep it at about 15 seconds myself.

adalren
19 May 2006, 20:20
Isn't 6minutes kind of long for server.max-write-idle? I see them in the logs but most of them are errors on transfering very small 10kbyte files. The server tries to send the file over and if it idles for that long, it's a good assumption that the user has canceled.

For some reason, the default

server.max-write-idle = 360

is too short... I've increased that to 720 and it's still too slow according to the error log.

I'll give:

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4

a try then. :) I'll report back.

In relation to kqueue, I'm using FreeBSD so I have this line:

server.event-handler = "freebsd-kqueue"

That covers that yeah?

Erwin
20 May 2006, 04:22
Has anyone figured out how to switch logging off in lighttpd? :)

adalren
20 May 2006, 09:41
Removing mod_accesslog in your server.modules should disable logging.

Erwin
20 May 2006, 14:37
Removing mod_accesslog in your server.modules should disable logging.
Good one.

So far lighttpd is going well with the short keep-alive time. My logs are filled with the write-idle timeout thing but otherwise it's fine.

Mmm... I disabled mod_accesslog but logging still occurred. Any other ideas?

adalren
21 May 2006, 00:59
Are you trying to disable error logging because I don't think you can unless you specify /dev/null as the file output. Removing mod_accesslog only disables accesslog and that works fine for me.

Erwin
21 May 2006, 06:31
Yeah, disabling error logging is what I'm trying to achieve. But I'm rotating the logs at the moment so it's not a big deal. Funny how you can't do it in the config.

turkforum
22 May 2006, 10:27
Erwin may i ask who configured the server , is this steven... cause i also work with dan for almost 6 months. Bu t i have to switch from 2 server system to 1 server system as my hits dropped almost 1000% after we have been banned by the freaking nonsenseTurkish Internet Laws and had to change domain name...
Reason being i am asking whether it was steven i can give your site as a reference to himm..
thnx

Erwin
22 May 2006, 11:08
Dan Paultz helps a lot but he also gets Steven Ciaburri from rack911 to assist him - Steven helped me set up my irc server on one of my boxes. Sure, you can give Steven a reference to my site or to me, they'll know.

DirkW
03 Jul 2006, 21:59
Are you using Lighttpd on a multiple CPU server? If so, are you running just one instance? Are you using a PHP optimizer? What is your memory usage like?

nso
27 Nov 2006, 02:44
Erwin, you could experiment with (or implement) cpu_polling in your kernel to decrease the time it takes for the CPU to communicate with your network card(s) -which can be a bottleneck on busy sites. Fortunately FreeBSD has this option.

Add the following to your kernel and rebuild it, I've been using it with great success on my Dual Xeon system;
#Device polling
options DEVICE_POLLING
options HZ=1200