Note: this article was from a series running on PerlMonth from 1999. PerlMonth.com seems to have disappeared, so this article is reposted with some slight modifications to make it current for recent releases of Apache::ASP ( versions >= 2.29 ).

Apache::ASP Site Tuning

by Joshua Chamas <chamas at alumni.stanford.org>

Last month, we used Apache::ASP to build a simple MyBookmarks web application. Now we are going to tune the Apache::ASP web application and Apache web server, taking a mild mannered web app running at 250,000 pages per day, and boosting it up 4 times to a 1,000,000 pages per day powerhouse.

Are you ready? Let's Tune!

Methodology

The application was taken as it was built last week, and incrementally benchmarked, changing one configuration setting at a time, and showing the performance changes below. The URL tested is the main URL of the application, .../bookmarks/index.asp.

The application is run on a Solaris x86 box, PII300 512K cache with 2 4G 7200 RPM SCSI drives in a software RAID 1 configuration. The following numbers are not meant to be compared against other web application environments, or systems, but show relative performance improvements when tuning Apache::ASP and Apache on this particular system.

The web server software being tested is apache 1.3.4, with the testing client ab, or ApacheBench, run locally on the server. Because ApacheBench does not support cookies, a new Apache::ASP $Session is created in StateDir for every request, so that the benchmark numbers are worse than what you would see in a production setting. Obviously testing locally does not take into account many slow client connections over the internet, which would likely be offset by a reverse proxy accelerator in production (see mod_perl guide).

The ab program was run with 5 concurrent clients for 30 seconds with the following command: 

  ab -c 5 -t 30 http:// $HOST /bookmarks/index.asp

Tuning

After running the first bench, and watching the system under top, it becomes strikingly clear that disk i/o is the bottleneck, with 50%+ of the time spent waiting for disk. The reason is that state management for $Session and $Application uses SDBM_File, or optionally DB_File, databases on disk to store data, and is highly i/o intensive.

The trick then is to relocate the StateDir to a fast cached file system. On Solaris, this happens to be /tmp/... by default, but on Linux and WinNT, the file systems seem to be cached automatically, so you may not need to do anything except locate StateDir to a secure, non-browsable, location. Using a cached or RAM file system will also spare your disk, which is good as it is often the first thing to go on your box.

Hits/sec Before After
+116% 6.5 h/s 14.1 h/s
Configuration
AP.htaccess yes yes
APMaxClients 5 5
APMaxRequestsPerChild 50 50
Debug 1 1
Global . .
GlobalPackage My::Bookmarks My::Bookmarks
SessionSerialize 0 0
SessionTimeout 15 15
StateDir .state /tmp/bookmarks
Comments Disk i/o activity takes 50% of the time, because the StateDir is pointed at a non-caching file system. Disk i/o activity is nearly 0%, with roughly 75% of the time spent in the user space, and 25% in the kernel
Legend Configuration items that start with AP are Apache configuration options, the rest are Apache::ASP. The .htaccess one means that configurations are stored in the .htaccess if yes, if no in a configuration file. The configuration changes between benchmarks are highlighted in pink.

More straightforward is turning off debugging. The Apache::ASP code has been streamlined for production when debugging is turned off, and there is a significant penalty for leaving it on, as well as a bunch of clutter in your error_log.

Hits/sec Before After
+12% 14.1 h/s 15.8 h/s
Configuration
AP.htaccess yes yes
APMaxClients 5 5
APMaxRequestsPerChild 50 50
Debug 1 0
Global . .
GlobalPackage My::Bookmarks My::Bookmarks
SessionSerialize 0 0
SessionTimeout 15 15
StateDir /tmp/bookmarks /tmp/bookmarks
Comments Disk i/o activity is nearly 0%, with roughly 75% of the time spent in the user space, and 25% in the kernel
Legend Configuration items that start with AP are Apache configuration options, the rest are Apache::ASP. The .htaccess one means that configurations are stored in the .htaccess if yes, if no in a configuration file. The configuration changes between benchmarks are highlighted in pink.

Those were the two big basic tunes for Apache::ASP. Let's now go to the Apache web server. So far, our configuration information has been stored in a .htaccess file, which must be reparsed every request by Apache. So, we move the configuration information into one of the *.conf configuration files, to avoid that per request overhead. Also make sure to disallow .htaccess file parsing on up the file tree by setting an 
  AllowOverride None
wherever your Apache::ASP application is located.

Hits/sec Before After
+9% 15.8 h/s 17.3 h/s
Configuration
AP.htaccess yes no
APMaxClients 5 5
APMaxRequestsPerChild 50 50
Debug 0 0
Global . .
GlobalPackage My::Bookmarks My::Bookmarks
SessionSerialize 0 0
SessionTimeout 15 15
StateDir /tmp/bookmarks /tmp/bookmarks
Comments Couldn't believe my eyes when I saw there to be so little difference when not using .htaccess. My guess is that the Apache people optimized this at some point.
Legend Configuration items that start with AP are Apache configuration options, the rest are Apache::ASP. The .htaccess one means that configurations are stored in the .htaccess if yes, if no in a configuration file. The configuration changes between benchmarks are highlighted in pink.

The MaxRequestsPerChild is commonly overlooked in web server setup. What this does is control the number of requests that a child httpd server may serve before exiting, and the parent reforking another one. With mod_perl library loading and script recompilations, a fork is no small thing, and avoiding the fork penalty is important.

We had the MaxRequestsPerChild pretty low before at 50 but we up it now to 500 which should be fine for production. You don't want to set this too high, or the mod_perl httpds will often take up too much memory because of leaks, and shared forked code becoming unshared as it gets dirtied.

Hits/sec Before After
+31% 17.3 h/s 22.8 h/s
Configuration
AP.htaccess no no
APMaxClients 5 5
APMaxRequestsPerChild 50 500
Debug 0 0
Global . .
GlobalPackage My::Bookmarks My::Bookmarks
SessionSerialize 0 0
SessionTimeout 15 15
StateDir /tmp/bookmarks /tmp/bookmarks
Comments Couldn't believe my eyes when I saw there to be so little difference when not using .htaccess. My guess is that the Apache people optimized this at some point. This bench only had 684 requests, so there was no parent httpd forking during this time, whereas before there may have been reforking every few seconds during the 30 second bench.
Legend Configuration items that start with AP are Apache configuration options, the rest are Apache::ASP. The .htaccess one means that configurations are stored in the .htaccess if yes, if no in a configuration file. The configuration changes between benchmarks are highlighted in pink.

Back to Apache::ASP, and some finer tuning. By default SessionSerialize is 0, and we are going to turn it on. What this does is lock $Session for exclusive use during the course of the script being run, so that any reads or writes to $Session don't have to lock it every time. Because of the i/o requirements of SDBM_File, each time $Session is read from or written to, it is freshly tied to the database and locked, so to avoid these ties can save much.

The reason why SessionSerialize is not enabled by default is that one could easily deny service to a user with a long running script, such that no other scripts for that same user $Session could be run, and this requires some expertise. Also SessionSerialize is probably not a good thing for framed sites, where there is greater concurrency for the same $Session.

Hits/sec Before After
+12% 22.8 h/s 25.74 h/s
Configuration
AP.htaccess no no
APMaxClients 5 5
APMaxRequestsPerChild 500 500
Debug 0 0
Global . .
GlobalPackage My::Bookmarks My::Bookmarks
SessionSerialize 0 1
SessionTimeout 15 15
StateDir /tmp/bookmarks /tmp/bookmarks
Comments This bench only had 684 requests, so there was no parent httpd forking during this time, whereas before there may have been reforking every few seconds during the 30 second bench.
Legend Configuration items that start with AP are Apache configuration options, the rest are Apache::ASP. The .htaccess one means that configurations are stored in the .htaccess if yes, if no in a configuration file. The configuration changes between benchmarks are highlighted in pink.

From start to finish, we have nearly quadrupled the speed of the web application, and thus the site's ability to serve up web pages. This one should handle just about 1,000,000 page views per day. :)

Hits/sec Before After
+296% 6.5 h/s 25.74 h/s
Configuration
AP.htaccess yes no
APMaxClients 5 5
APMaxRequestsPerChild 50 500
Debug 1 0
Global . .
GlobalPackage My::Bookmarks My::Bookmarks
SessionSerialize 0 1
SessionTimeout 15 15
StateDir .state /tmp/bookmarks
Comments Disk i/o activity takes 50% of the time, because the StateDir is pointed at a non-caching file system.
Legend Configuration items that start with AP are Apache configuration options, the rest are Apache::ASP. The .htaccess one means that configurations are stored in the .htaccess if yes, if no in a configuration file. The configuration changes between benchmarks are highlighted in pink.

For more information on Apache::ASP tuning, see the tuning docs online.