SOA deployments: What actually works

SOA may have seemed the savior of bad software architecture and poor development project planning, but the reality is that it's a complex and difficult venture. Thus, the number of failed SOA projects is about equal to the successful ones. In other words, you have a 50 percent chance of failing, and the odds of failure are even greater if you work within a larger Global 2000 organization or within the government.

But key patterns are emerging from the successful SOA efforts, patterns that can help you determine whether your SOA is a failure or a success.

The most important lesson from these patterns is that SOA is as much about old-school IT disciplines as it is about new, inventive technology. Moreover, it's about changing an organization from the people down to the technology, driving a systemic and valuable change. The patterns of success likewise follow that change from the people on down to the technology.

People: From leadership to staff, the right aptitude and commitment is critical

The fundamental cause of SOA failure is the lack of good people working the architecture, from leadership to staff. This lack is not of numbers, but of knowledge and the vision to drive change.

People-based SOA failure begins at the top. A recent Burton Group study determined that the arrival of a new CIO typically spelled success for SOA. In essence, innovative leadership and the ability to change the culture and then the architecture is a clear critical success factor.

Moreover, critical to SOA success is the presence of a leadership team that values an investment in infrastructure and understands the long-term value of being more agile and efficient, and, better yet, is willing to make the investment. SOA is not cheap. Indeed, it's a huge systemic change in how you build, deploy, design, test, and architect your information technology. It's an investment that goes well beyond the millions of dollars required to acquire the necessary training, consulting, and then the technology required to make the changes.

The SOA investment cannot be treated as a one-time "transformation" project. Instead, you need to consider SOA as a journey, not a project, and -- for the purposes of budgeting -- treat it as a series of projects that make up the journey. The SOA effort does need its value clearly defined, and the investment and commitment to achieve that value made up front.

Because SOA is really about a journey, not a project, enterprises need to take a long-term view of SOA. It's a multi-year and typically multi-million-dollar commitment to drive a systemic change to the core IT mechanism. But most SOA "projects" are stopped at some point, typically due to budget issues or to the need to redirect resources at some tactical short-term need. Thus, the SOA never had a chance; there was no follow-through. Executive and IT management must not allow this.

SOA also involves two fundamental business changes that IT historically doesn't have the clout to make happen: One is the sharing of processes across political fiefdoms that fear loss of control. The other is that SOA requires a rethinking of fundamental processes, which is not only hard but also challenges established practices, resource allocations, political power, and so on.

Page Break

At the management and staff levels, the people requirements for successful SOA are even more fundamental. Although many would love to have the existing team drive them to the new world of SOA, the harsh truth is that many in that team won't have the skills or the aptitude. You need to make some hard decisions up front as to who will take you forward. This means you must replace people or you must augment staff. Either approach is costly.

Many companies employ mentors or consultants to help learn the new ways of SOA. Others invest in SOA training, or even bring in entirely new teams as either consultants or employees to assist in "the change."

No matter what you do, never, ever try to drive SOA with the wrong people; it just won't work.

Processes: SOA requires a change in how you develop, manage, and test applications

Approaching SOA means changing how you approach architecture and systems development. In the past, many companies have approached systems by dragging whatever seemed cool at the time into the enterprise to solve a tactical problem. Of course, tactical problems lead to other tactical problems, and these companies kept digging a deeper hole in terms of architectural complexity and efficiency.

Thus, you need to approach SOA using a well-thought-out, practical process that lets you break the architectural domain down to a functional primitive, then build it back up again as a SOA. Unless you're the smallest of enterprises, you also need to break these domains into achievable chunks that can be worked on in sequence, or later in parallel.

Then it's time to do some real work, including looking at the information within the problem domain, meaning application semantics and metadata. This takes a ton of work, because most enterprises typically don't have a good semantic-level understanding of their enterprise systems. Thus, this effort is usually not a process of reviewing common artifacts, but of creating new ones. Many SOA efforts skip this step, which cripples what they end up delivering.

From the information step, you move to the services: defining existing and new services that will make up your SOA. This is all about figuring out what you have, determining what you need, and then normalizing the services into a usable and well-defined grid. You need to make sure you define and document the services, and then link them back to the metadata model you've created.

Keep in mind that most of your services will be existing services, externalized through some sort of middleware mechanism. Most people think SOA is about building all-new services, but that's almost never the case. Many SOA efforts fail because they become excuses to reinvent the wheel, not solve pressing business needs.

You then need to define and create processes that bind the services together into business solutions. There are three approaches here you can use. First, you can create service-oriented business applications, a way to programmatically bind services together into processes or applications. Second, you can leverage an orchestration layer, such BPEL, to bind services together to form business solutions. Third, you can leverage traditional process integration tools to bind services into solutions. No matter what approach you use, you need to make sure to include requirements as well as design.

Page Break

Of course, there are other process things to do, including the creation of test plans and selection of testing tools for the services. Also, you need to create a SOA governance strategy, determine the governance process based on that strategy, and select tools to support the creation and enforcement of governance policies.

Architecture: The core of SOA is usually given short shrift

The "A" in SOA is architecture. Most people forget that, opting instead to jump right into the technology. Why? Because architecture requires a huge effort and is boring, whereas playing with cool technology is fun and fast. Unfortunately, you really have to spend the time to plan and lead the enterprise in the right directions. Create a strategy, processes, architecture, links to business realities, budgets, and then -- the most important action -- lead the technology teams in the proper directions.

Although architecture is fundamental to SOA, it is often neglected in so-called SOA efforts. That's because most enterprise architects within the larger enterprises don't have the authority and reach they need to drive the technology in the right directions. In most cases, they try only to influence and inform, and are more "ivory tower" guys when you get right down to it. This unrespected, powerless approach to architecture won't drive change in the long-term unless there is true dynamic leadership to back them up.

What typically happens due to the low respect and profile of most enterprise architects is that tactical short-term needs trump any long-term strategy, as people try to run their business from one technology fire drill to another, dragging in all sorts of technology into the company with no clear plan as to how all of this stuff will work together.

What are called enterprise architectures are often no more than layers upon layers of these disjointed tactical efforts. Those kinds of "architectures" are too complex and too difficult to change as new business needs arise.

But if you focus on the "A" in SOA, you are much likelier to succeed. Being architecture-focused does require driving changes in how the IT department thinks about technology -- having it do so in the context of architecture, and never with the technology being the de facto architecture. This is a significant cultural shift that most IT staff won't make unless forced to. Making this change happen comes down to the leadership within the company, and its support of the strategic activities. Here's why management should support the change: The current approaches waste millions of dollars a month right now. The challenge is that this waste has become the normal way of operating, so its costs are hidden within the "standard" overhead and lost opportunity costs.

Page Break

Technology: Your technology should follow the requirements, not drive them

There is no one-size-fits-all SOA technology solution for all enterprises. You have to understand your requirements and match the requirements with the right technology. It's that simple.

But most companies that build SOA mistakenly lead with technology. They go out and purchase an ESB or some other SOA-in-a-box solution with the hopes that it will solve all their problems. The end result is a technology solution that's mismatched to their real issues and thus does not work. The SOA effort fails.

The process of matching the right technology with the problems that need to be solved is very simple, but it's also time-consuming. The process is to list your requirements down one side of the page, and list the candidate technology down the other. From there, create a short list of technology that's practical to test. Next, work with the vendors to bring the technology in and test its form and function. Does it do what it should do, and under the stresses that the SOA will place upon it?

Sure, it's tempting to call one of the big stack players and just leverage everything they have in their portfolio. Whereas some portions of such a technology suite will indeed be a good fit for your SOA, the rest of the stack will be the wrong match for your requirements. Accepting wrong technology for the convenience of a suite or single vendor relationship leads to SOA failure. Instead, in most cases the right technology solution for SOA will be a mix-and-match kind of exercise with many different vendors and technologies.

Make sure your architecture is technology-independent. It might sound easier to build architecture around a particular technology, as some vendors promote, but remember that technology is always in a state of change, whereas the core notion of the architecture needs to remain fairly consistent. You need to work the architecture out first, and then be concerned with the technology that supports it, understanding that the technology will change.

The bottom line for SOA success

SOA success is not rocket science; it's really a matter of good old IT disciplines that focus on all dimensions, including the people, processes, technology, and the business. Organizations that fail did not resist the temptation to chase the "quick and easy" technology-oriented solutions, and they did not consider the fundamentals.

The issue in most failed SOA development efforts is not having a clear understanding of the systemic change that SOA requires. It touches many parts of the organization, and therefore touches many political fiefdoms within enterprises, and turf issues come up quickly. Strong leadership is required to counter these issues, and most enterprises don't seem to have that leadership at the IT levels. Thus the project/movement founders and dies.

The core pattern of SOA success is around new, innovative thinking within the company among leaders who have the ability to drive enterprise-wide change based on that new thinking. Although some enterprises can step up to the requirements of SOA, most will need new leadership, management, and even staffers to pull SOA off. The ROI is real and significant, but it requires making the sacrifices at the very beginning.