The toughest part of the project wasn't the technology, however. "It was new for us to work with event streaming," he says. "We were used to batch processing."
Stream processing tools first arrived on the scene several years ago, having grown out of academic research at several universities, including professor emeritus David Luckham's seminal work on complex event processing (CEP) at Stanford University. Some vendors were spawned from those academic exercises. For example, Stonebreaker is both an adjunct professor of computer science at MIT and CTO at StreamBase. And Celequest worked with researchers at Stanford in developing its product.
Most players in the market are small, and the technology is still maturing, says Philip Howard, an analyst at Bloor Researchin Towcester, England. "None of the big boys have entered this space yet," he says, but that's changing.
Many of the start-ups have only a few customers today, says Gavin Little-Gill, an analyst at TowerGroup. He expects many of the vendors to disappear over the next two years as the market consolidates, new vendors jump in and stream processing capabilities are integrated into databases and other tools. "A bunch of these guys are going to get gobbled up by the Oracles and Microsofts of the world," Little-Gill says.
Already, Tibco Software Inc.has launched its own product, Business Events, and Progress Software acquired startup Apama last year. IBM is working on two CEP projects: one called Active Middleware Technology and another, from its Tivoli Software group, called Active Correlation Technology.
In addition to using stream processing to detect and react to events, some tools also feed real-time updates to a dashboard. "If there's anything that's driving this, it's dashboards," says Rogge.
At first blush, the dashboards presented by stream processing tools sound a lot like business activity monitoring (BAM) tools, but the latter lack the ability to analyze data and perform event processing, says Jeff Wooton, vice president of product strategy at Aleri Labs, a Chicago-based vendor of stream processing technology. Nonetheless, he sees the two areas converging. "Either BAM products will make use of event-processing products, or they will start competing with them," he predicts.
Maja Tibbling, lead enterprise architect at Con-way, says that unlike BAM tools, stream processing can measure what is not happening, which she says is just as important as knowing what is happening. The transportation company uses Tibco's BusinessEvents to track and plan pickups and deliveries and the activities of inbound and outbound trucks to ensure that transportation planners are working with the most up-to-date information.
"In our rules engine, we need to know who cares about [an event] and how long they are allowed not to get [the information]. You can always figure out if something has happened. That absence of events is what's difficult to capture," Tibbling says.
Con-way recently moved toward an SOA as part of an effort to better integrate its systems, and BusinessEvents plays a key role in monitoring some 8 million events a day on the company's enterprise service bus. Events are published on the ESB, where processes that need to know about specific events can subscribe to them in parallel. "All of them, not just one stream, need to complete within a two-minute service-level agreement. If one doesn't complete, we need to know which one," Tibbling says. BusinessEvents triggers that.
Cendant is deploying Celequest's Analytics Server to keep up with hotel reservation requests coming in from channel partners such as Orbitz. "We wanted to know exactly what was happening with the business in real time," says Nick Forte, director of application architecture. With more than 6,500 hotels to book, Cendant's systems handle up to 500 transactions per second during peak times through various channels. Forte uses dashboards to monitor activity. Having an ESB architecture facilitated integrating the tool with data streams. "An ESB makes it much easier to pluck that information off and look at it," says Forte.