Open source software and open source inspired approaches to development and innovation are set to play an increasingly large role in the future of ASX-listed insurance group IAG.
IAG recently turned to the open source OpenStack cloud platform to help consolidate almost two dozen data warehouses, in the process saving millions of dollars.
The company has in the past been open about its mammoth project to consolidate its core policy and claims platforms, working to migrate from 32 systems to two.
But before the roll out of OpenStack, the multinational insurer also had 23 different data warehouses based on around 10 different technologies — a product of a string of acquisitions by IAG since it was founded in 2000, said the company’s data engineering and dataops chief Eddie Satterly.
“Some of the warehouses were built on IBM technology, some on Oracle, some on Informix, some on MS SQL,” Satterly said.
The situation was “super complex” and needed to be simplified to ensure the insurer was well-placed to leverage its massive data volumes. A decision was made to look towards open source, with an eye to taking advantage of community-driven innovation rather than trying to build everything with a commercial software vendor or purely in-house.
“Based on that, we started looking at what they had, what the assets were, validated what the workloads were, what the workload profiles were that we wanted to move to — including a bunch of open source tools such as Apache Kafka, Solr and Cassandra, Apache Storm and a few others,” Satterly said.
“Using a lot of the open source products is where we wanted to go for half of the platform for the data and analytics team,” he added.
“The other half was really around traditional data warehousing and using an existing incumbent Greenplum environment.”
IAG staged a performance bake-off between several different platforms to underpin the data warehouse consolidation including direct compute in SAN, VMware environments backed by SAN and vSAN, and two different versions of OpenStack: HP Enterprise’s Helion and RedHat’s OpenStack distribution.
“We really tested our workloads across all of those environments to see where we would fit the best and what would be most adept at handling our incredibly high IO workloads with all of our data movement,” Satterly said.
The new private cloud is based on the Red Hat OpenStack Platform, backed by Dell R730xd rack servers and Dell EMC ScaleIO. The setup has delivered “extreme performance” for IAG’s data workloads, Satterly said.
The move has paid off in a financial sense, too: IAG cut its hardware and software costs by $2.7 million last fiscal year by moving data workloads to OpenStack.
“We were able to give back a $1m+ worth of SAN and went to a hyperconverged node that was $20,000 a piece,” Satterly said.
“It started off as a two rack system — we’re now up to six racks and we’ve migrated the majority of our data workloads over to it, as well as moving over some of the core workloads that are critical to processing.”
The company has consolidated down to 11 data warehouses, with the eventual aim of shifting to just three. IAG has also slashed deployment times, with the company recently able to cut the time to deliver a new data service from three months to five days.
IAG initially began testing OpenStack, using Helion, in June 2016. “The Red Hat kit was brought in around October,” Satterly said. The insurer went live in production with OpenStack in January this year.
“It was about a six-month trial of different configurations, different workloads, different backend storage, really looking for the ultimate profile to allow us to be more agile and be able to build new things, test new things and fail fast, but delivering the kind of IO needs we have for moving around the 40 to 80 terabytes of data that we touch on a regular basis in the environment,” he said.
The OpenStack environment is being used as IAG prototypes and builds with a range of new technologies, including leveraging Solr and Cassandra to a deliver a single view of customer.
“It’s allowed us to build new applications, try new things, and it’s now enabled us to build some open source technologies inside of IAG that we’ll be sharing externally,” Satterly said.
Last week, IAG open sourced Data Pipeline: A Python based application for replicating data from source to target databases designed to help enable real-time analytics of data while having minimal impact on the original database housing it.
Some core insurance-related tooling is now running on OpenStack at IAG and the company is interested in the potential of shifting some of its core claims platform work to the platform. The insurer is taking a cautious approach however, Satterly said.
“There’s definitely an effort being made to start testing in that area to see if they could do that, at least from the pre-production side of things first and then production eventually,” he said.
The use of OpenStack is part of the broader effort to build an “open source culture,” Satterly added. There has been a push within IAG’s Customer Labs division, which is led by chief customer officer Julie Batch, to develop an “inner source” approach, rapidly prototyping based on emerging open source capabilities and then building on top of them, as well as contributing back to open source projects.