FRAMINGHAM (01/28/2000) - Site logs are useful, but the numbers they generate can tell you the wrong things.
With apologies to Mark Twain, there are three kinds of lies: lies, damned lies and site logs. Site logs contain data about how users traverse a Web site, and various tools will mine the data for you. But Jared Spool, founding principal at User Interface Engineering (UIE), advises caution. "Site analysis tools tell you things, but they don't tell you the right things." For instance, he says, many tools will tell you the total time the average user spends on your site but can't distinguish whether the user is happy, lost or stuck for 20 minutes on the credit-card verification screen before logging off in disgust.
How quickly users browse a site is another metric many tools measure. Many site managers associate browsing speed with successful task completion. Another lie, says Spool. "Our experience is that the speed with which people move around the site has nothing to do with their usage patterns. I mean, we're not dealing with 7-11s here. You're not just running in and buying a carton of milk."
One large e-commerce site, a client of UIE's that prefers anonymity, lost significant development time and site revenue because it relied solely on site log data to diagnose a problem. The logs said 60% of customers who made it to the credit-card authorization screen didn't complete their purchases. It must be an issue of trust, the client thought, so it invested heavily in secure transaction servers and redesigned the site to tout its security.
Still, 60% of customers wouldn't go past the credit-card screen, so the client asked UIE to perform usability tests. The first of several users went shopping for a heavy, expensive laser printer. After finding it at an excellent price, the now-excited user went to the checkout and then through a series of screens where he typed in a billing address and shipping information. Then he got to the credit-card screen and aborted the transaction.
As it turned out, the user said he didn't feel comfortable giving his credit-card number out before seeing the total price, including shipping. An issue of trust, perhaps, but one that had to do with bad site architecture, not distrust of the seller. The company's decision to put shipping information after the credit-card screen was simple: It had saved the designers 30 minutes of programming. Testing, obviously, would have saved them even more.
Just how subjective a medium is the Web? Extremely. In another test, Spool found that users rated Amazon.com Inc.'s site faster than About.com Inc.'s - a major reason why they liked the Amazon.com site better. Paradoxically, Amazon.com's pages took an average of 36 seconds to download over a 56K-bit modem, while About.com's pages loaded in only 8 seconds. Spool's conclusion from watching how users traversed the site: Speed equals ease of information retrieval. "If you want to improve the perception of how fast your site loads," says Spool, "get users more quickly to the information they're looking for."