Increasing the Performance of WURFL (for at least one case)
I’ve been using the PHP interface for WURFL on a few projects recently and always had some nagging questions about how the thing works in terms of caching and performance. So today I really dug into the stuff that’s there to find out why WURFL seems to take longer than what I expect it to. First I should mention that I might have a non-standard workload going on here. The site that I’m trying to optimize gets traffic from all over the place, and WURFL seems to be optimized for a small working set of maybe two dozen or so active sessions.
My test was realistic however, I grabbed a stream of traffic from the website I wanted to optimize, snagged the user agent from each of 500 requests, and then replayed those user agents in the order they came into the site. My test script just loaded up WURFL and did a GetDeviceCapabilitiesFromAgent(), which is pretty standard behavior from what I was able to tell looking at what other folks do. The starting values hovered around 81 seconds to fulfill 500 requests.
The way the caching works in WURFL the scripts end up including a lot of PHP that’s actually data var_dump()ed by the parse routines. So the obvious place to start was trying to speed up loading of files, and APC seems to be the favored accelerator cache these days. It made a pretty decent difference actually, with the average time dropping down to about 67 seconds after installing APC and doing nothing else.
Matching was the next obvious place to look, so I started checking out the logic in GetDeviceCapabilitiesFromAgent() that does the matching against agent strings. I was surprized to see that the code seemed to be doing a linear scan through the agent strings trying to find one that matched, including an optimization to subgroup by a match against the first four characters of the agent, and relooping with sucessively smaller substrings until one of the agents matches. About a dozen different ideas popped to mind about how to fix that up some, but the quickest to implement was turning the agent array used as the main cache element (cache.php, which creates wurfl_agents) from a flat array indexed by full agent strings into a two leveled array indexed by the first four letters of the agent strings (which is naturally very sparse) and then holding a subarray of agents. That paid off, but only a little bit, the average time was now about 60 seconds.
If a major change like that to the matching logic didn’t improve the performance drastically there must be something in there that I’m missing. So I figured maybe this fast cache agent2id thingie is too small. So I up the number of entries in that cache to 60 from 30 and retry the tests. The benchmark goes UP this time. Interesting, very interesting. So I actually take a look at the file and realize that the thing is hundreds of kilobytes long. And with the frequent switches of user agent forcing rewrites of that file, for my workload at least that cache file is causing tons of thrashing. I turned it off completely and reran my tests. 10 seconds to process the 500 agents. Now that’s more like it!
Going back and fooling around with it some it looks like I can get down to 25 seconds with simple APC and removing the use of the WURFL_AGENT2ID_FILE. If you’re seeing a lot of load from scripts using WURFL I suggest checking that out, playing with the config file if you’re not comfortable hacking the code. It looks like there’s a lot of savings that can be wrung out of that code if your workload is outside the expected.
June 21st, 2006 at 3:40 pm
[...] I had another one of those WURFL hacking episodes aimed at getting WURFL to work better for sites with large groups of active devices. And here it is. I’ve started calling it WURFL lightweight. Everyone looks up device info by user agent, so it’s optimized for that. It also assumes a few things: [...]
June 22nd, 2006 at 3:30 am
Hi Mike,
thanks for the good post on WURFL. It’s always good to see people’s experiments. It’s just a pity that you took this time hacking the PHP library when you could have found the exact same conclusions on the WURFL website. Take a look at this page and paragraph:
http://wurfl.sourceforge.net/php/index.php#caching
There is also another performance improvement that you might consider, that is to reduce the number of files in the directory. You could:
a) generate symbolic links when a device in WURFL doesn’t have any specific capability instead of a real file. This breaks a little bit the fall_back tree, but if performance is your concern, this should help
b) store the tiny cache files in a directory tree instead of a single directory. Using the first 3-4 characters should be enough.
A final optimization that I considered but never implemented was to explode every device’s capabilities in every single cache file. This will mean that when a user-agent hits your site you will find all the device information in a single cache file and not load 3, 4 or 5 tiny files. Of course APC helps you in your current configuration as it has all the most common files in shared memory which is even faster the loading a file. You could do this in the update_cache.php script, because I can tell you it will take time.
Some performance tests showed that depending on the file size, sometimes var_dump is better performing, some other times serialize is better. I’ll leave to you these tests, if you’d like. wurfl_parser.php already has the lines, you just need to adapt them to the multicache.
- Andrea
June 22nd, 2006 at 9:18 am
Hey Andrea,
Actually, I had read the stuff about caching at wurfl.sourceforge.net, but I found a lot of them weren’t true for the environment I was working in. I think it’s because I’m working with a much bigger active device list than most sites do. The churn in the agent2id cache was actually generating a lot more load than it was saving. I made a few other changes I think might be interesting for folks running sites with frequent turnover in active devices:
http://www.thisismobility.com/blog/?p=141
I haven’t done the exploding of every device’s capabilities into every cache file, but I think that might be a good one. Thanks for the reply!
- Mike