Hammering out Web services links

By 12:50 p.m. Central time on Feb. 28, Verizon Communications had already logged 1.7 million Web services transactions for tasks like looking up customer addresses and selling new services.

The New York-based telecommunications company averages about 2.5 million to 3 million Web services transactions a day, anchored by its mostly homegrown service-oriented architecture (SOA), a platform that was two and a half years in the making. Dubbed the IT Workbench, the SOA supports the design, deployment and management of Web services. It went operational early last year and has helped the company slash its IT budget by 50 percent by eliminating redundant systems inherited from the merger of Bell Atlantic and GTE, which spawned Verizon.

Verizon has also tackled some of the most vexing hurdles associated with Web services as part of the IT Workbench project, such as managing and securing the services, charging for reuse and monitoring the performance of service-enabled transactions.

IT Workbench

The project was born in 2002 as executives began looking to reduce inefficiencies in software development, says Shadman Zafar, Verizon's senior vice president of architecture and services. Consolidating application development was key for Verizon, which found itself with multiple groups often duplicating efforts after the merger.

Executives focused on the 250 most-important business transactions the company performed, such as verifying customer credit histories and looking up customer information. On average, each transaction had been developed five to 25 times; one was deployed 45 different times, Zafar says. The duplication was draining developer productivity and created needless ongoing maintenance costs.

The company decided to use Web services to expose the application programming interfaces of common transactions as XML, which could be consumed by the Simple Object Access Protocol (SOAP) standard and used by multiple lines of business.

Zafar says he spent much of 2002 and 2003 evangelizing to developers about the potential benefits of Web services -- especially to the two "religious" camps of .Net and Java developers. Despite some initial resistance, these groups warmed to Web services once project leaders demonstrated that the standards could allow .Net developers to consume Web services-enabled Java applications and Java developers to use .Net Web services, Zafar says.

His efforts to promote the use of Web services were backed by Verizon's CIO, who included target metrics for Web services usage as performance measurements for company vice presidents. "Verizon took a very aggressive view of Web services," Zafar says. "We were not toying with it. We took it as a business metric, and we had to meet a very tough business metric."

Zafar set an initial target in 2004 of building 10 applications and 10 transactions on the IT Workbench as Web services. The company instead built 57 applications and 200 transactions. At the beginning of the year, Verizon was supporting 10,000 Web services transactions per day; by the end of 2004, the daily average had skyrocketed to 2.5 million to 3 million per day.

Anne Thomas Manes, an analyst at Burton Group in Midvale, Utah, says this number of transactions per day qualifies as one of the largest corporate deployments of Web services.

"Verizon appears to have embraced 'true' SOA," she says. "Their goal was to reduce redundancies by building shared reusable services. A few other companies have also embraced SOA, but most companies that I've worked with are still more focused on using Web services for integration as opposed to a real architecture of applications based on SOA design principles."

Now, Verizon is filing patents for some of the technology it built to anchor its use of Web services, and both BEA Systems and IBM have approached the company about licensing its homegrown platform so they can offer it as part of their application server software stacks, Zafar says.

Institutionalizing Web services

With the CIO's mandate in hand, Zafar's group in 2004 set about institutionalizing the use of Web services. After an initial test project to demonstrate that Web services would allow Verizon's billing and wholesale departments to pass a customer record between systems using SOAP, Zafar worked on a process to ensure that development projects would focus on Web services.

At the beginning of 2004, Verizon began funneling each development project for key business transactions through two groups. One ensured that new applications represented useful transactions that could be used by more than one group in the organization, and the other identified which transactions could be deployed as Web services. In addition, Zafar required all new development projects to get his approval before going into production.

"Key transactions have to be developed as Web services . . . so it is a common platform so other consumers of the same services . . . can use them," Zafar says. "If developers want to realize the dream of getting an application into production one day, they have to pass through this gate."

Verizon has limited access to the Web services to development projects to further reduce duplication. Web services transactions can't be added to applications already in production

"Each development group is a publisher of services and a consumer of other people's services," Zafar notes. "You are almost forcing a reduction of duplicate effort and leveraging cross-portfolio development."

On the technology side, project managers began designing a framework to handle the cataloging, security and management of the Web services.

The top priority was making the framework easy to use, says Ruchir Rodrigues, Verizon's executive director of architecture and eServices. To keep maintenance costs low, Verizon chose a decentralized, agent-based approach to its SOA rather than a broker-based system where all service requests are handled by a server or set of servers, Rodrigues says. "We didn't want to get into the maintenance hassle of maintaining these servers with fail-over," he adds.

Verizon custom-developed agents or small code libraries that sit on all its application servers to intercept various pieces of information about the services, like usage, response time and IP address, as they flow back and forth. This information is put into a log file and shipped to the IT Workbench portal to be processed.

The agents support the subscription, management and dashboard layers of Verizon's SOA. The subscription layer allows developers to publish and consume Web services and to set up service-level agreements regarding their use of the services. By linking developers through the portal, Verizon supported its goal of making the use of Web services as nonintrusive as possible, Rodrigues says. "If I didn't have an IT Workbench platform and I wanted to use somebody else's Web service, I would have to call him," he says. "We wanted to start reducing that interaction and making it easier for people to get to the Web services."

IT Workbench project manager Mehul Shah, who helped design the management layer, says he relied on his experience designing network management systems. The agents sitting on the application servers act as "sniffers" to log different management data at the Web services endpoints, he says.

The third layer of the SOA is a dashboard application that allows Verizon to track Web services transaction volumes by line of business on a daily basis. IT has until recently used this information to tweak hardware resources to support load levels. However, early this year, Verizon IT management decided that because the system is now so big, the company will move responsibility for the operation of the IT Workbench to the data center, Rodrigues says.

External integration

The IT Workbench is focused on internal integration using Web services. For external integration with business partners, Verizon knew it wanted a hardware gateway to insulate its intranets from the outside world. But it struggled to find a vendor that could meet its performance demands -- evoking a service in less than 10 milliseconds, according to Rodrigues.

Finally, Verizon tested a management gateway from SOA Software (formerly Digital Evolution) that the company now uses for managing and securing Web services between Verizon and telemarketing partners to exchange customer data.

Verizon is moving deeper into the transactions used by each line of business and plans to support 33 million Web services transactions per year, says Zafar. But the SOA has already helped the company to break down many of the walls separating its 7,000 developers, he adds.

"This has almost become a virtual meeting place of developers," Zafar says. Developers talk a lot more across organizations because of the platform. "When they have questions," he says, "they will call another developer as opposed to developing something else."