"At the end of the day, if you're going to be cheap, you're going to have to be clever," he said. And "in a high-transaction-volume world, you may spend a hideous amount of money and still have a poor-performing Web site."
Day has brought a technique similar to the one he used at Walmart.com over to his Web music start-up, MediaMaster.
For DBAs who like what they're reading but don't want to rebuild their infrastructure from scratch, Day says that open-source Java clustering software from Terracotta, created by the former chief architect for Walmart.com, Ari Zilka, effectively uses many of the same tricks.
"Terracota provides a really sophisticated and intelligent cache for data that is used across the application. Just as importantly, it figures out how to get that caching to work seamlessly in code below your application layer, so it happens transparently," he said.
When upgrades go bad
For Amy Stuemky, longtime DBA at Globus, the problem was in the travel company's mission-critical Web site.
"The site would go down and nobody knew why," Stuemky said. "We weren't sure whether it was the database or the Web servers or applications. It was driving us crazy."
Globus has 50 Oracle databases ranging from Versions 8 to 10g, and 30 Microsoft SQL Server 2000 databases. Stuemky is the sole DBA.
That made it harder for Stuemky to devote a lot of time to figuring out what was wrong with Web site, which travel agents and an increasing number of consumers use to book travel tours.
Stuemky even did what she knew wouldn't work: add two more CPUs to the SQL Server databases.
"We did it because our database licensing allowed it, but the databases just took up all of it [utilization] as well," she said.
After looking at several database tools, including Quest's Spotlight (which Stuemky pronounced "excellent but not within our budget"), Globus also turned to Confio's Ignite.
It bettered SQL Server's built-in performance monitor tool, which let Stuemky check the CPU availability but not drill down to see what the users and queries were actually doing.
After installing Ignite on her desktop PC, Stuemky let it run. Within a day, it came back with a diagnosis: a "huge locking problem" in SQL Server. Armed with the accompanying data, Stuemky was able to obtain a hot fix from previously unhelpful Microsoft support staffers.
She was also able to diagnose the main issue: session state tables created by SQL Server to keep track of user visits that needed to be rebuilt from scratch because of an upgrade in Globus' .Net drivers. Because rebuilding the tables was never mentioned in Microsoft's documentation, Stuemky said it may have taken many more months or even years to figure out that problem.
In the past two and a half months, the site has been "much more stable" and up 24/7, Stuemky said.