Uncovering Database Benchmarks

SAN MATEO (04/18/2000) - Common sense suggests that if you want unrestricted scalability for your applications then the Unix platform is mandatory. When open-ended performance is a requirement, most companies prefer the Unix platform, which is a more expensive and complicated solution than Intel-based hardware running Windows. But as we've all learned, sometimes technology turns common sense on its ear.

Recent benchmarks conducted by Compaq Computer Corp. returned results that contradict Unix's accepted superiority as a platform for optimal database performance. Adding spice to this surprise is the fact that the two highest-performing system configurations charted were running not Oracle Corp. or Sybase, but Microsoft Corp.'s SQL Server 2000.

The InfoWorld Test Center wanted to dig a little deeper in hopes of providing a real-world business context with which to compare benchmarks. In this Test Center Analysis, we critically examine these benchmark results and explore the possible implications on the corporate landscape.

How do I measure thee?

Predicting the performance of business applications remains among the most controversial subjects in IT today. A number of software tools measure the throughput of the various system's components, such as processor power, database access, or the responsiveness of a Java virtual machine. However, the results these tools return usually garner dubious reactions from business-savvy individuals; the results are sometimes rejected as being biased or irrelevant.

Benchmarks often meet similar criticism. One of the arguments regularly put forth is that benchmarks measure results produced in an artificial lab environment. A real-world scenario, naysayers aver, could produce a quite different outcome.

Indeed, the performance of a computer system depends on the combined capability of three factors: the hardware components, the operating system, and the applications. A comprehensive benchmark must take all these into consideration and simulate an application load to predict how those hardware and software components will perform -- such as the number of users that will be able to access a system concurrently to enter transactions. Obviously, in order to achieve comparable results, the simulated transaction load should be consistent across different systems.

Industry standard

The benchmarks that are the subject of our discussion were performed using the industry-standard TPC-C benchmark created by the Transaction Processing Performance Council (TPC). TPC, a nonprofit organization, defines specifications that measure database and transaction-processing performance.

Based on these specifications, TPC supporting members develop different benchmarks.

When a company decides to run a benchmark, it must agree to have the results published on the TPC Web site and must provide a comprehensive disclosure of the test environment for TPC members to review and criticize. The results are also scrutinized by a TPC-certified auditor.

Probably the most important aspect of the TPC specifications is that they prescribe business application scenarios that both remain consistent across different platforms and closely mimic a real-world environment. For example, the specifications for TPC-C, an order-entry system, prescribe strict requirements for the application load, such as closely simulating human-driven input, satisfying orders with quantities from different warehouses, and using a consistent, relational database structure.

Over the years, numerous companies have invoked the TPC-C using various combinations of hardware, operating systems, and databases. Though Compaq is the most recent company to do so, TPC makes the results of all TPC-C benchmarks available in several comparative formats on its Web site. (We have reproduced the results for the best performers in our chart on page 64, but you can view the originals at www.tpc.org/new_result/tpcc_results.html.)TPC-C measures the number of order-entry transactions per minute (as defined in the TPC specifications) and estimates the cost of each transaction for a given combination of hardware, operating system, TP (transaction processing) monitor, and database.

However, it is important to remember that the actual results on your system could be differ greatly from those of the benchmark. For example, your applications could behave differently than those in the TPC model, or your environment may not be properly optimized for performance. Nevertheless, the benchmark offers an immediate snapshot of what could be achieved from each system scenario.

Unix challenged?

Until recently, TPC-C results consistently placed Unix systems in the top-performing ranks. In fact, until November 1999, Unix hardware from Bull, Compaq, Hewlett-Packard, IBM, and Sun Microsystems filled the top positions, running Unix OS software primarily from IBM and Sun. Intel-based systems invariably fell significantly behind the performance of these RISC-based machines. Looking at those results, one could conclude that Unix systems possessed an inherent advantage that Intel machines (and, it follows, Microsoft software) would not overcome.

On the database side, Oracle8 and 8i dominated; they were used in eight of the top 10 best-performing systems benchmarked, challenged only by Sybase Adaptive Server Enterprise 11/12. IBM DB2 running AS/400 and Microsoft SQL Server, respectively, followed far behind.

But the most recent TPC-C benchmarks, those performed by Compaq on its new line of ProLiant 8500 high-end Intel-based servers running Windows 2000 Advanced Server and SQL Server 2000, puts the Unix/Oracle reign to an end -- at least in the admittedly narrow milieu of this benchmark. In clusters of 12 and eight servers (96 and 64 processors), the ProLiant 8500 took the first-and second-ranking positions, respectively, on the TPC-C benchmark, outperforming Unix platforms and moving SQL Server to the top of the chart over Oracle and Sybase databases.

The Compaq results are still being reviewed by TPC members. But unless bias is proven, the findings open a range of interesting interpretations. A possibility high on the list is that Windows-heavy sites can breathe a little easier knowing they have a true world-class migration path. But scratch below the surface, and you may find that the prospects are less promising: In order to reap these advantages significant new hardware and software investments must be made. Even though the estimated cost per transaction on the new systems appears to be sensibly lower than the immediate Unix contenders (see chart below) the cost of migrating to Windows 2000 -- which, according to analysts, could be high -- is not factored into the TPC-C's reported estimate for total system cost. (Migration costs include hardware, software, and maintenance.)These results also raise important questions about hardware. TPC-C measures the combined performance of all the system components, but it is fair to assume that the role of the eight-way Compaq ProLiant 8500 was not minor in achieving that performance level. The eight-node cluster scored better than any Unix system, to say nothing of the 12-node cluster, and both are sensibly priced in relation to the other contenders.

But to duplicate such a system in your environment is not a task to be taken lightly. A cluster configuration such as the ones used in the benchmark can be more difficult to set up than a single system. Many sites will undoubtedly favor less complex architectures, such as the IBM RS/6000 S80 or the Sun Starfire Enterprise 1000, both of which deliver similar processing power in a single box.

Compaq plans to introduce new servers with 32 processors by the end of the second quarter, and, subsequently, systems that will hold as many as 256 processors. (Pricing and release dates for these new servers were not available at press time.) However, it is reasonable to assume that such machines would take the benchmark successes earned here even further and begin to earn some respect for Intel-based platforms in the corporate environment.

As for the database software, Microsoft can be proud of the performance of SQL Server 2000 in this benchmark, but I expect these results could be challenged soon. It is difficult to imagine, even if unchallenged, that this will generate a mass migration to SQL Server due to the rigors of database migration.

Nevertheless, it reassures existing users and could attract new customers as well.

No company should base a corporate decision as important as choosing the platform to support their business applications solely on benchmark results.

Far more important considerations should play into your decision, such as platform familiarity and compatibility with existing systems. Nevertheless, the recent TPC-C scoring is too important to be ignored, and it delivers the surprising message that Microsoft SQL Server, Windows, and Intel-based hardware are on equal ground with Unix and OS/400 machines as well as Oracle and Sybase software.

Mario Apicella is a senior analyst for the InfoWorld Test Center. You can reach him at mario_apicella@infoworld.com.

Understanding benchmarks

In years past, companies that evaluated databases with an eye toward a purchase usually examined the TPC benchmarks as part of their cost/benefit analysis.

During this period, databases were typically hosted on a legacy system, a single database server, or, at most, a two-node database server cluster because those were the limitations at the time.

The parameters uncovered during TPC benchmarking were fairly straightforward, given the simplistic hardware configurations available and database support at the time. However, the relevance of external database benchmarks has always been debated as not representing the real world. Today, the computing architecture we use to support our database infrastructures is vastly different from the database configurations of old.

Database clustering is now commonplace, and we're well beyond the two-node limits of just a few years ago. Clustering configurations can be set up in a myriad of ways and can be tweaked for greater initial or long-term performance.

The database clustering configuration you choose for your company could vary widely from those used in the TPC tests. You can further shape your performance by tweaking cluster configurations and database parameters to meet your needs.

Further detracting from the meaningfulness of database benchmarks is the fact that relational technology is fairly mature now. When TPC benchmarking began, relational technology was still new, and the market demand for the software and the benchmarks was intense.

Today, it is less likely that a company with an existing investment in a particular database will abandon that database for another based merely upon TPC benchmarks. The need to compete aggressively while maintaining an extremely short time to market means the majority of companies have a slim chance of effecting a major change to their database infrastructure.

In years past, companies also invested in multiple database technologies for specific projects. These days, those same companies are more concerned with integrating their multiple databases than with jumping ship when a hot benchmark comes along.

The motivation behind this latest benchmark may well be driven by Compaq's desire to increase interest in its high-end server platforms. Moreover, Microsoft's SQL Server 2000 is on the verge of coming to market.

Of course, that's not to say that the TPC's benchmarks are entirely without value. The numbers do highlight the strength of Compaq server clusters, so if you are currently in the market to expand your database infrastructure, you would be wise to evaluate this cluster option.

One notable problem I perceived with the testbed is the use of beta software.

Using a database release that is still in beta and making the results publicly available seems questionable. Database vendors can and should measure the effectiveness of their database performance as a part of the release cycle.

However, the benchmarking numbers achieved during a beta period may actually play out quite differently once the software is released.

Finally, per-transaction costs shown in this latest TPC benchmark cannot be considered completely accurate. The investment needed to upgrade to Windows 2000 is high, and the cost of running SQL Server 2000 is not yet fully known.

In order to cast away such doubts, perhaps another benchmark that includes the total investment picture and uses the production version of SQL Server 2000 would be advisable.

Ultimately, the best database benchmarking can be found by running the numbers against your own real-world configuration. Performance continues to be a key part of database infrastructure, but with many more options available to boost your speed, the business relevance of external benchmarks will remain questionable.

THE BOTTOM LINE

Compaq/Windows/ SQL Server 2000

Business Case: According to the TPC-C benchmarks, Compaq ProLiant 8500 clusters, Microsoft Windows 2000, and SQL Server 2000 offer unrivaled performance for transaction processing systems at a far lower cost than Unix-based platforms.

Technology Case: Interpreting the results of the benchmarks can be complicated and misleading. Companies should also consider the cost of migrating to Windows 2000.

Pros:

+ Best performance and cost/performance ratio in the OLTP segment to date+ Modular, open, and flexible hardware configurationCons:

- Unproven in real-world environment

- Discouraging migration costs and complexity.