SAN FRANCISCO (05/01/2000) - Speed ratings for today's color ink jet printers look impressive. But PC World tests often show a big gap between rated speeds and real-life results. We look inside the ratings game.
Color ink jet printers seem to live in two worlds.
First there's the world of printer company test laboratories, where machines whip out documents and photos at mach speed. Then there's the world the rest of us live in, where printing out a simple letter or photo can test the patience of a stoic.
What Gives?
We analyzed two years' worth of published PC World tests of the top color ink jets. We discovered that the printer companies are waging a "specmanship" contest in which the speeds consumers get rarely match the units' rated speeds.
Instead, in PC World tests, the ink jets average between 40 and 73 percent of rated text speeds, depending on the vendor.
We arrived at these numbers by taking the published performance data for the 24 printers that earned a spot on PC World's printer charts between February 1998 and April 2000. The data includes text and graphics speed ratings provided by the printer companies, and speeds for text and graphics printing as determined by PC World tests. We looked at the four companies whose machines appeared most often on the charts: Canon, Epson, Hewlett-Packard, and Lexmark.
In PC World's text-printing tests, Hewlett-Packard Co.'s printers came closest to their rated speeds. Its models tested, on average, at 73 percent of the company's rated speeds. The Epsons averaged 53 percent of rated speeds in our tests, and Canon Inc.'s and Lexmark's printers checked in at just 40 percent.
In graphics tests, results ranged from 22 percent of rated speed for Epson to a mere 12 percent for Lexmark. HP's printers averaged 18 percent of rated speed, while Canon's averaged 14 percent.
Why such wide discrepancy between vendor ratings and PC World results? The industry has no standard for rating color ink jet printer speed, so vendors make their own rules. For example, some vendors test in draft mode. HP's printers come closest to PC World text results partly because the company provides ratings for normal-mode text printing. Some vendors test by printing from DOS instead of from everyday applications.
"We try and make our tests as fair and representative as possible," says Carolyn Ticknor, president of HP's Imaging and Printing Systems. "But what customers, who use a variety of applications, experience may well be different.
[PC World's] tests may-- and I emphasize may--be more representative of what a user sees," Ticknor says. "In almost every test, we are the most conservative in the industry." HP and the other vendors face the dilemma of a lack of uniform standards, she adds. "We want to be fair to the customer, of course, but also be fair to ourselves."
Standard Unlikely
Lexmark engineer Rhenzi Keys predicts "there would be significant hurdles" in creating a standard. Vendors would have to agree on the PC platform, applications, size and complexity of job, and driver settings. HP and Epson told us they'd welcome a standard but said it would be tough to get the companies to agree on details.
Tom Miller, senior product delivery manager at Canon Computer Systems, notes also that each company might want to capitalize on its own technology.
Given today's situation, explains Charles LeCompte, president of Lyra Research, a printer market-research firm, "print speeds are completely subjective. That doesn't mean there's no test that a given printer performed at [a quoted] speed, but the test is completely arbitrary."
Aggressive ratings aren't a new tactic in the computer industry. For years, monitors carried confusing specifications; you could find two "17-inch" monitors with different amounts of screen real estate. But after enough protest from consumers, manufacturers began to report actual viewable area in ads, so shoppers could understand the specs and make comparisons. For color ink jet shoppers, there's no common, easily understood speed standard.
Performance Gap
Without question, color ink jet printers run faster than they did in 1998. They also produce higher-quality output. But the divide between PC World performance results and the vendors' rated speeds hasn't narrowed much over time. Why did we consider only the records of printers that reached PC World's printer charts? Having already identified these as the best products on the market, we felt they represented a fair, relevant sample. Consider Hewlett-Packard's DeskJet 812C, which captured first place on our April 2000 Top 10 Printers chart. HP rates the 812C at 5.1 text pages per minute; PC World tests clocked it at only 3.7 pages per minute. With graphics documents, HP claims 3.1 pages per minute; but in PC World's tests, the 812C produced only a half-page per minute.
Lexmark's Z51 Color Jetprinter placed third on that same April 2000 chart. Like the HP, it's a good value among its peers. Lexmark says it prints text at 10 ppm; PC World clocked it at 3.9 ppm. One reason: Lexmark's rated text speeds reflect tests run in draft mode--a lower-resolution setting that prints faster.
It's an approach Canon and Epson also take in the quest for high speed ratings.
We Test, They Test
To understand any rating, you need to understand the testing process--a printer can be tested in many more ways than you might realize. And many testing scenarios don't mimic everyday work.
PC World's color ink jet testing methodology is designed to approximate real-world tasks. We use a series of 11 documents of different lengths, some containing text and graphics, others containing only graphics. We create and print these documents from apps including Microsoft Excel and Word, Adobe Photoshop, and CorelDraw. We print one Word document in both draft and normal modes; all others we print in normal mode only. From a Pentium III-450 PC, we print each document twice, reboot the system to clear any caching, run two more rounds of tests, and average the four scores. The combined document scores create test results for both text and graphics.
PC World times from the moment the user clicks Print to the moment the paper hits the output tray--a method known familiarly as "click to clunk." But the big-four printer vendors measure time "feed to clunk"--beginning when the feed mechanism picks up a sheet of paper. Printers generally delay feeding paper until the driver software is already rasterizing a document (converting it into dots); as a result, the feed-to-clunk method shaves some time off recorded speed.
There are technically sound reasons for taking this approach, says Canon's Tom Miller: "A faster processor will rasterize faster; or more-efficient cache management [might affect print speed].... From an engineer's perspective it's very difficult to reproduce that data." Feed to clunk times are reproducible with a range of PCs, Miller adds.
That's true, but of course the ordinary user has to wait out the whole time.
"If the manufacturers are basing their claims on such [feed-to-clunk] methods, then they should include a caveat," says Wayne Hubbell, president of CompuMetric Labs, an independent company that tests printers for Lyra Research and for a number of printer vendors. "It is not unreasonable to measure performance this way, but they should point out that real-world users will obtain slightly lower performance."
Canon and HP follow another procedure that does not mimic routine printing.
Instead of printing from an app like Word, they prerasterize a document by saving it as a print (.prn) file. Then they issue a DOS command to shoot the file from the system's parallel or USB port to the printer. This procedure bypasses the application, the driver software, and Windows.
Don Okerlund, HP's DeskJet Division product manager, says that the company tests this way because the testing is replicable, even with older PCs. "This methodology takes some of the variables out of the testing; it does not rely on the operating system, the computer, the application."
In contrast, Lexmark and Epson print from applications, as PC World does.
The Draft Card
Print resolution affects speed too--whether you print from applications or whether you start with prerasterized files. Canon, Epson, and Lexmark all quote speeds recorded in low-resolution draft mode for text and for graphics--something that you might not realize. (One additional note: Epson rates its printers' speed in "fine mode" for photos.)For example, a consumer comparing printers might try looking up their specifications on the appropriate vendors' Web sites. We checked out the specs for four printers that ranked on PC World's April 2000 Top 10 Printers chart--the Canon BJC-6000, the Epson Stylus Color 760, the HP DeskJet 932C, and the Lexmark Z51 Color Jetprinter--at the respective companies' Web sites. We could not find the words "draft mode" listed anywhere in the Canon, Epson, or Lexmark specifications. We did see phrases such as "up to 10 ppm." (Epson notes that it does specify "economy mode" on the customer tear pads that you find near printers in retail stores. ) As many of you know, draft-quality printing is acceptable for personal documents but is generally not desirable for producing anything you'd share with others. That's why PC World runs its tests almost exclusively in normal mode.
Hewlett-Packard, unlike the other three vendors, publishes ratings for draft, normal, and best-quality output (As we discussed earlier, we base our comparisons on the company's normal-mode ratings). But those numbers appear only on Hewlett-Packard's detailed specification sheets and on its Web site.
The company's advertising and its packaging reflect the draft-mode ratings.
Consumer Catch
Unfortunately, manufacturers of color ink jet printers find themselves in an increasingly difficult spot, Charles LeCompte says. New models must carry faster ratings than old ones, or they won't sell.
"The speeds they quote keep going up, so I think they're getting themselves into a bind," says LeCompte. "The customers definitely care about speed. And it's leading to some preposterous things. Maybe at 12 pages per minute people will still believe you, but what do you do when you introduce another printer that's faster? Do you say it's 14 pages per minute? Or say 12 ppm, even though it really is faster than the other one you already called 12 ppm?"
For now, there's simply no easy way for consumers to decipher ink jet speed ratings. Based on what we've found, you should use rated speeds only to compare color ink jet printers from the same vendor. You should also consider test results from independent sources, including PC World, before making your purchase.
If enough consumers complain, vendors may change their ways. At the least, they could provide more information for comparison shopping. Above all, when you shop for a color ink jet, don't take rated speeds at face value--or the printer may seem a world apart from what you expected.