How to overcome 3 hybrid cloud integration challenges
- 10 April, 2015 05:46
This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter's approach.
Companies adopting cloud technologies often end up with a public/private hybrid approach that balances business needs with infrastructure goals, and while the end result provides numerous benefits, hybrid clouds can result in integration challenges. But with the right planning and the strategies below, connecting cloud applications can be done easily and effectively:
1. Understand and work around individual SaaS API limitations
Surprisingly, not all APIs offer full create, read, update and delete (CRUD) operations on all of their entities, often exposing just a subset of the full data model. Always be sure to know what data entities are available and what operations you can perform on them before committing to any hybrid integration project, even when the API exposes the entity you are looking for and the operation you want.
Also be aware that SaaS applications typically exercise rate limiting, either through well-defined rules and policies or through poor performance exhibited under load, and plan accordingly.
2. Plan for large data migrations
You will likely need to come up with new strategies for moving on premise data to public/private cloud stores or SaaS applications. Three options include:
* Parallel processing, which allows you to break large data migrations into separate processes that can be run in parallel. Just make sure you understand how the entities you are migrating are related, how your data itself can be segregated and double-check that you won't run into a rate limiting issue.
* Incremental loading, which takes the "slow and steady" approach. Time available for a given data migration is typically based on how long you can have users off the system. By taking an incremental loading approach, data is moved based on change date from oldest to newest so the two systems eventually sync up to the point that down time for cut over is only based on the data change rate, not the size of your data.
* External key cross referencing. This last strategy is perhaps the strongest. It involves building a cross reference of keys between the systems on a fast local data store, which then means the integration never performs costly searches on the slow/rate limited SaaS application to relate entities.
3. Ensure data governance
Once you start integrating systems, you introduce the risk of losing data integrity if your data flow isn't designed to observe and maintain target applications' data governance rules. The two places we often see this become an issue are record ownership and auditing.
* Record Ownership. In the on-premise world, you could add additional users as part of your migrations/integrations, so the proper data was related to the proper principles on each side. SaaS applications often pose API issues around adding users, making maintaining these relationships difficult. Add SaaS users as needed, and maintain an external cross reference to map the users between the systems.
* Auditing. Integrations between systems based on periodically polling for changed data are notorious for causing anomalies in audit logs. If a record changes multiple times within a polling period, the details of the interim changes are lost, so if auditability is core to your integration, look for or build integration solutions triggered by data being changed instead of relying on polling.
Companies adopting a hybrid cloud approach have the chance to maximize their infrastructure's scalability, price, control and flexibility. Large companies, and those with deep technical expertise, can follow the steps above to successfully overcome these integration challenges, but the connected cloud isn't limited to just them.
Any business can achieve their hybrid integration goals by enlisting systems integrators and/or leveraging third party integration tools specializing in cloud integration. Regardless of your approach, be sure to plan for the cloud-specific challenges outlined here, and take the time to realistically assess your company resources and project timeline before diving in.