If data integration hasn't crossed your horizon yet, then it's close. It's shaping up to be the next 'big thing'. And then again, you may already be in its grip, not formally perhaps, but if there's an ERP system or data warehousing project under way, then data integration is a part of that.
And it's set to accelerate as more and more demands are made on business intelligence for increasingly rapid analysis and more diverse reports of manufacturing, supply, sales and performance figures. Add to that the need to meet a growing array of regulation and compliance like Basel II in finance, Sarbanes-Oxley, the Corporate Law Economic Reform Program (Clerp 9), and others. Technology itself is adding fuel to the explosion as imaging comes into greater use in a growing number of fields - like patient records.
As a burgeoning segment in the technology mix, data integration has naturally caught the attention of both vendors and analysts and Gartner predicts that by 2009 structured data integration tools will have expanded their functionality to meet 50 percent of unstructured data integration needs.
John Brand, research director at Hydrasight, said most organizations are continually involved in data integration, though few realize it. It's tied to a specific project or implementation of a system, such as a new ERP, portal, or data warehousing project where it's treated as a one-off exercise.
"This is why we see so many high-profile failures in IT projects; it's this lack of recognition that the system is not as important as the data it is managing.
"Ironically though, we see organizations regularly establishing (and decommissioning) data quality and data cleansing projects in response to business issues driven by poor data quality or poor data integration.
"The underlying business causes of poor data integration or quality over time are rarely addressed. Instead organizations tend to choose an IT based 'solution' to the problem," Brand said.
Brand says DI has become one of the hottest areas in IT in recent times, highlighted by the continued interest in enterprise application integration, service oriented architectures, and portals as two common methods of 'implementing' data integration.
He said the fragmented nature of information assets in companies - where data is held in myriad enterprise applications, workgroups or small-scale collaborative style environments, e-mail, spreadsheets, and local custom databases - means they are continually struggling with creating the 'system of record' or a 'single version of the truth'.
"The irony though is that these issues rarely go away by implementing a centralized data store, a single, unified user interface, or a meta data management repository."
Phil Sargeant, research vice president at Gartner Australasia, said collating disparate information from different business units, like HR, finance, and sales, lets companies extract data that can help them perform better and make business decisions based on real-time data.
In a study into DI usage trends, Gartner found that, globally, companies expect to increase their expenditure by 60 percent in total (57 percent in Asia-Pacific compared to 69 percent in the US and 55 percent in Europe).
But as more companies move towards service oriented architecture, in which data integration plays a large role, it will find itself increasingly in the spotlight.
Brand said the assumption that data integration means centralizing data storage couldn't be further from the truth.
"There seems to be a commonly held belief that unless all of your data is integrated there's no value in any of the data being integrated, which is simply not the case.
"Leading organizations understand the value of the information itself and focus on that which will provide the greatest business benefit.
"Organizations that perform well with data integration projects understand how the data directly improves business efficiency - like analyzing supplier performance or optimizing asset yields," Brand said.
Mostly, organizations start with data warehousing or portal-based data integration projects.
However, Brand said, most business units have very little idea what they will do with it once they have it. "But they inherently feel that by integrating all the available information stores and centralizing data storage, business benefit will somehow be automagically achieved. In some cases though, starting by making a problem more visible (as in the case with many portal projects) can be a good place to start, depending on the culture of the organization."
As to which vendors are leading the race to tools to meet the challenge, it appears there is no outright winner at the stage with all the big contenders and a few niche players involved. What is apparent though, is that many more vendors are getting involved.
Gartner's research indicated that buyers tend towards database management system vendors, then business apps vendors with business intelligence tools and pure-play integration tools providers also in the mix.