FRAMINGHAM (04/10/2000) - Because the average user can only stand to wait a maximum of eight seconds for a Web page to load onto his desktop, performance becomes everything. And Web caching has emerged as one of the best techniques for improving Web response times and for reducing bandwidth requirements for corporate WANs and ISP networks.
IRCache, a research group funded by the University of California at San Diego, recently tested 20 Web-caching products in a two-week bake-off in Cypress, Texas, at space donated by Compaq Computer Corp.
Using IRCache's Web Polygraph proxy benchmark, each product was tested in five key areas:
* Peak throughput. This measures the maximum number of HTTP requests per second that the cache can handle at peak load. The results are presented in a price/performance metric that shows how much throughput can be purchased for $1,000.
* Hit ratio. This measures the percentage of times the cache successfully finds a requested Web page on its hard drive. The higher the hit ratio, the better the cache is performing.
* Cache age. Because caches have a finite amount of storage space and are programmed to allow new objects to bump out old objects, cache age measures the average age of the oldest object in the cache. If you have objects many hours old still in the cache, that means you have an ample amount of storage and your hit rate will probably be high.
* Response time. The time it takes for a packet to be delivered under peak load.
* Downtime. The time it takes for the cache to recover from an outage.
Fourteen vendors participated, which is three times as many as took part in the first bake-off last summer. The participants were Cisco, Compaq, Dell, IBM, iMimic, InfoLibria, Lucent, Microbits, Network Appliance, Stratacache, Pionex, Quantex, Squid and Swell Technologies. Cacheflow and Eolian registered for testing but failed to actually participate. Cacheflow ignored our request for an explanation of its absence. Eolian participated with its shipping product in the first bake-off, and officials report that the next-generation product was not ready for public performance testing at the time of the bake-off. Inktomi is currently boycotting all bake-off tests. Also absent from these tests were Cobalt, Entera, Microsoft and Netscape.
While Network World does not typically pit beta products against shipping products, the IRCache bake-off rules allow vendors to submit beta units if they will be generally available within three months of testing. The products tested in beta were Cisco's Cache Engine 590 and Cache Engine 7300, iMimic's DataReactor 5000, InfoLibria's DynaCache 30i Version 3.0 and a cluster of three DynaCache X1 boxes, Lucent's IPWorX WebCache 100 Version 2.0 and IPWorX WebCache 150a Version 2.0, Swell's Tsunami CPX-1000 and Network Appliance's NetCache Appliance 4.1. Most of these products are on schedule to ship by May 1 with the exception of Cisco's Cache Engine 7300, whose scheduled ship date has slipped to some time this summer.
Most of the products tested can be considered caching "appliances." That is, you buy them as preconfigured hardware and software bundles designed specifically for one task - Web caching. The exception is Squid, which is a pure software cache that ran FreeBSD as its underlying operating system. The authors acknowledge they have participated in the development of Squid software; however, neither the rules nor the method of testing were changed in any way to benefit the Squid product.
Ten entries ran Novell's Internet Caching System (ICS) software. These vendors - Compaq, Dell, IBM, Microbits, Stratacache, Pionex and Quantex - sell their caching products under an OEM agreement with Novell. Cisco, InfoLibria and Network Appliance have proprietary operating systems for their products. The entries from iMimic, Lucent and Swell Technologies all run on open source operating systems - either Linux or FreeBSD.
All the products we tested use either Intel Pentium, Celeron or compatible CPUs. There is relatively little customized hardware in these systems. If you buy one of these caching products, and for whatever reason the caching configuration doesn't work out, you can probably still use the hardware for some other purpose on your network.
Our testing rules require that vendors provide all necessary network equipment with their entry. We found that all products employ commodity Ethernet switches or hubs for network connections. One of InfoLibria's entries also uses a proprietary device, called DynaLink, which bypasses the cache when the power or any of the equipment fails.
Getting to the numbers
Price/performance is an important factor in most buying decisions. Instead of charting raw data, or comparing a $5,000 device with a $50,000 device, we normalized the throughput results to arrive at a number that answered this simple question: "How much throughput can $1,000 buy?".
The price is defined as the sum of the cache's list price, plus the cost of all the network gear required to achieve the peak request rate. Network gear costs usually contribute less than 20 percent to the total price.
IMimic'S DataReactor 5000 showed the best return per dollar, followed by the low-end IBM 3500 M10 in second place and the Quantex 4000 and Dell 130B in a tie for third. Cisco's Cache Engine 7300 came in last; its raw throughput was actually a strong second among all vendors, but its total price of nearly $131,000 dragged down its "per $1,000" rating. Cisco's $44,470 Cache Engine 590 came in next to last, followed by Pionex PCA-1000.
Products showing good return on a dollar can be found in all market segments.
For example, the $2,300 Swell Tsunami CPX-1000, the $20,000 Lucent IPWorX WebCache 100 and the $50,500 InfoLibria DynaCache X1 cluster all show similar price/performance returns.
Products from a single vendor tend to yield similar performance/price ratios, implying a consistent, performance-dependent pricing policy. Take InfoLibria, for example. The high-end DynaCache X1 cluster is a little more than twice as expensive as the mid-tier DynaCache 30i and gets a peak throughput that is two and a half times higher. Because the normalized throughput rating stays relatively the same, a buyer can justify the higher price if he needs the added throughput.
Hitting the ratios
A second important measurement is the hit ratio. This is a standard measurement of a cache's performance, and it quantifies the bandwidth savings on the outbound network link. For example, with a 50 percent hit ratio, half of the objects requested by the clients will be served from the cache and will not use the outbound bandwidth.
Hit ratio is also related to request response time of the proxy cache because misses take longer to complete than hits. Misses are fetched from the origin or Web servers, but hits come directly from the cache itself, which in a corporate setting is typically located as close as possible to the clients.
With our simulated Web traffic workload (see www.nwfusion.com, DocFinder:
7623), a cache cannot achieve a hit ratio higher than 55 percent. The actual measured hit rate may be lower due to a whole host of reasons, such as server overload, insufficient disk space and deficiencies in an object replacement policy. While some real-world installations can see hit ratios approaching 70 percent, most correctly configured caches will report ratios close to 55 percent.
In our first bake-off, we found that most vendors were sacrificing other performance factors to achieve close-to-ideal hit ratios. But in this round, many vendors chose to sacrifice hit ratio in exchange for higher throughput.
For example, a box may be able to achieve 100 requests per second throughput with a 50 percent hit ratio. Cache misses force delays, so if a product's hit ratio goes down, its response time goes up. But that same box could probably get 125 requests per second with a 35 percent hit ratio.
Squid Version 2.4 and Swell's Tsunami CPX-1000 - a Squid-based product - scored perfect document hit ratios. Cisco's Cache Engine 590 was a close second at 54.8 percent and Cisco's Cache Engine 7300 was third at 54.5 percent. Also scoring above 52 percent were Pionex, iMimic, Compaq's TaskSmart C2500 and Lucent's IPWorX WebCache 150 products.
The Stratacache F100 and InfoLibria DynaCache 30i finished last with 32 percent hit ratios.
There is no universally acceptable hit ratio, and there are a host of reasons a product could fall below the pack. For example, some products may be designed to stop caching data if the load is too high. Some simply may not have enough disks to store the amount of data required to reach the 55 percent level.
The fact remains, however, that with fewer hits, you will need more outbound network bandwidth and more proxy resources to handle the same load. Your users may also notice slower response times.
You have to decide what's best for your network. If a product does not meet your criteria, you can work with a vendor to add more disks or adjust other configurations. However, such adjustments may not always be possible (a product may not be able to handle more disks due to RAM, CPU or even licensing constraints) and are often costly.
Cache age
One way around the issue of insufficient disk space is to make sure that the cache purges objects on an appropriate basis to make way for incoming fill traffic. Our Cache Age measurement shows an estimated maximum age an object reaches before it gets dumped from a cache. A higher score means a device can store "older" objects and hence have a better chance of producing a hit.
Most of the products can store no more than a few hours of peak fill traffic.
In an effort to improve throughput/price ratios, many vendors designed or configured their products to deliver the highest request rate at minimum disk storage costs. Consequently, their boxes do not have enough storage space to keep objects longer than a few hours. The boxes that fell into this category include Stratacache's F100, IBM's 3500M10, Compaq's TaskSmart C900 and Dell's PowerEdge 130b.
Other caches - such as Squid, Swell, Lucent's IPWorX WebCache 150a and both Cisco boxes - can hold objects eight to 18 hours.
Many cache administrators believe that a production cache should store two to three days of traffic. That roughly corresponds to five to 10 hours of peak fill traffic.
The cache capacity requirement depends on your environment. When configuring a caching system based on our performance reports, make sure you get enough disk storage to keep sufficiently "old" traffic. You may have to increase the price and recompute performance/price ratios if a product you are considering does not have enough storage. You should also check that the product is actually available with the additional disk space. These adjustments may significantly affect the choice of price-conscious buyers.
Cache age is not a measurement of how stale or fresh objects sitting in a cache may be. In our workload, every cacheable response is static. However, in a production system, the cache would use standard HTTP features to validate and refresh cached objects. We are working on a model that will allow us to integrate a staleness/validation in the PolyMix-2 workload for future caching bake-offs.
Response time
In order to measure the response times of the products under real-world peak load conditions, the PolyMix-2 workload introduces an artificial delay on the server side. The server delays are normally distributed with a 2.5-second mean and a 1-second deviation. The delays play crucial roles in creating a reasonable number of concurrent virtual sessions in the cache.
To simulate WAN server side connections, we introduce packet delays of 80 msec round trip and a .05 percent packet loss. These delays increase miss response times and, more importantly, reward caches for using persistent connections.
(The TCP connection setup phase includes sending several packets that also incur the delay.)The delays, along with the hit ratio, affect transaction response time. Ideal mean response time is impossible to calculate precisely because the Web model is too complex. With that said, we estimate the ideal mean response time at about 1.3 seconds based on 55 percent hit ratio, zero delay hits, and 2.6-second misses, factoring in some WAN delays and packet loss.
IMimic scored the best mean response time: 1.35 seconds. Six other products - Swell's Tsunami CPX-1000, NetAppliance's NetCache Appliance 4.1, Squid Version 2.4, Compaq's TaskSmart 2500 and both Cisco products - showed good response times below the 1.5-second mark. Apparently, most vendors have concentrated on keeping response time close to ideal levels, minimizing request processing overheads for both miss and hit paths. The results for both InfoLibria products are noticeably worse than average, ringing in at more than 3 seconds for DynaCache 30i and more than 2 seconds for three-node DynaCache X1 cluster.
InfoLibria officials say the products performed poorly on this particular test because of a bug in how the beta software schedules disk I/O requests. They say the bug should be fixed in the shipping product.
As with hit ratio, the absolute value of measured response time does not mean much, as it depends on the artificial delay on the server side. What's really important is the relative difference between measured response time and ideal response time.
Downtime
The downtime test is designed to estimate the time it takes a product to recover from an unexpected condition, such as power outage or software failure.
Polygraph measures the time until the first miss (TTFM) and the time untilthe first hit (TTFH). Lower values are better for both measurements.
From a user's perspective, the TTFM is somewhat more important. As soon as caching systems can deliver misses, users can access the Web again. Delivering hits is important in reducing outbound bandwidth usage, as hits usually do not consume outbound bandwidth, and from a quality-of-service point of view, as users get pages "faster."
All products tested with the exception of Swell's Tsunami CPX-1000 were able to complete the test. The Swell product required manual intervention to reboot (pushing the power button) because the BIOS would not allow for an automatic boot after the power has been turned back on. Such an intervention was prohibited by the bake-off rules.
InfoLibria Dynacache 30i scored a near perfect rating, which can be attributed to the DynaLink component of the product. In case of power failure, DynaLink switches into hard bypass mode, essentially replacing a cache with a piece of wire.
InfoLibria's three-node X1 cluster, the only product using a Layer 4 switch as its base included in these tests, finished second. TTFM for a Layer 4-based cluster is the time it takes for the switch to boot and detect that a proxy is not yet ready to respond.
For systems with large amounts of RAM and/or many disks, TTFM depends primarily on how much various housekeeping operations are optimized or completed in parallel. For example, scanning 4G bytes of RAM twice or spinning all 10 SCSI disks sequentially significantly increases TTFM readings. Most entries show negligible delay between the time until the first miss and the time until the first hit.
In addition to InfoLibria's device, Network Appliances' and Microbits' scored well on this test. The two Compaq devices scored the worst, followed by Cisco's Cache Engine 590.
This review presents a performance snapshot of the caching industry as of January 2000.
Naturally, all bake-off participants are working to fix the problems revealed by our testing and on improving the products' overall performance. The products may be retested at the vendors' request, and you can expect to see the new results published on the IRCache site. And we're planning a third bake-off this summer.
The authors are employed by the University of California, San Diego, and together they comprise the IRCache group, which finds its roots in National Laboratory for Applied Network Research. They work in Boulder, Colo.
Rousskov is the author of Web Polygraph, and he can be reached at rousskov@ircache.net. Wessels is the creator of the IRCache group and the author of Squid caching proxy, and he can be reached at wessels@ircache.net.
Chisholm works on various IRCache projects, including Polygraph tools and Squid performance optimization, and he can be reached at chisolm@ircache.net.