Open your data to the world

When Neil Fantom, a manager at the World Bank, sat down with the organisation's technology team in 2010 to talk about opening up the bank's data to the world at large, he encountered a bit of unfamiliar terminology. "At that time I didn't even know what 'API' meant," says Fantom.

As head of the bank's Open Data Initiative, announced in April 2010, Fantom was in charge of taking the group's vast trove of information, which previously had been available only by subscription, and making it available to anyone who wanted it. The method of doing that, he would learn, would be an application programming interface.

The API would place thousands of economic indicators, including rainfall amounts, education levels and birth rates - some metrics going back 50 years - at the disposal of developers to mix and match and present in any way that made sense to them. The hope was that this would advance the bank's mission of fighting poverty on a global scale by calling on the creativity of others. "There are many people outside the bank who can do things with the data set we never thought about," says Fantom.

"There are many people outside the bank who can do things with the data set we never thought about," says Neil Fantom, a manager at the World Bank.

One developer, for instance, created an app that married the bank's rainfall data to Google Maps to estimate how much rainwater could be collected on rooftops and subsequently used to water crops in different parts of the world. Another app provides facts about energy consumption and shows individuals what they can do to fight climate change.

Fantom and the World Bank aren't alone in this trajectory. A decade ago, open APIs were a novelty, but in the last few years they've been adopted at an accelerating rate. ProgrammableWeb, a website that tracks public APIs, listed more than 8,800 in early April. According to the site's data, it took from 2000 to 2008 for the number of APIs to reach 1,000, and then another 18 months for that to double. The jump from 7,000 to 8,000 took just three months.

The APIs cover a wide range of categories, including business, shopping, messaging, mapping, telephone, social, financial and government, according to the ProgrammableWeb website. They're becoming as necessary to an organization as a website. "In business today, an open API is more or less table stakes. It's something you have to have," says Stephen O'Grady, an analyst at RedMonk, an analysis firm that focuses on developers. "Increasingly, your traction is going to be driven by how open and how programmatically manipulable your product is."

When Best Buy first launched its API, BBYOpen, in 2009, it gave developers access only to the chain's products catalog, with descriptions and prices for all the items it had on sale, in the hopes that doing so would bring in more customers. That was part of a deliberate strategy to start slowly, says Steve Bendt, director of emerging platforms at Best Buy. "We had to prove these things over time," he says. "We started to prove out that this is a very vibrant and viable area to pursue."

Key takeaways

What you need to know about creating open APIs to your data:

Make it easy. Outside developers -- those at your customers' shops -- may have great ideas for how to use the data you make available, but the API itself needs to be understandable and easy to work with. Clear documentation and tools to help are must-haves.

Make sure your licensing terms are clear and fair. Successful APIs tend to have MIT-style open-source software licenses.

Use REST unless you absolutely need SOAP. About three quarters of all APIs are REST-based, according to ProgrammableWeb, with SOAP a distant second.

Be prepared for cultural resistance. Some of the data 'owners' may be reluctant at first to share the jewels. You might explain how the World Bank, Best Buy, Bloomberg and others have used the technique to reach customers in new ways and/or further their organisation's mission.

But external developers wanted more, so the company added the ability to access reviews and ratings for products, find a nearby store and check whether a product is available there, and purchase the item through the website or mobile app in question, perhaps with a single click if the user has linked a credit card to the app.

It's been a hit. The mobile apps ShopSavvy, RedLaser and Milo all use BBYOpen as part of their apps. The makers of the app get a commission on sales through Best Buy's affiliate program. Shoppers can search for an item, or scan a bar code, and get information on pricing from various sellers.

Of course, that might mean a customer using the app might wind up buying from a competitor instead, but Bendt says that since websites and mobile apps have changed how people shop, what's important for Best Buy is to be in the mix. "If we're not in the consideration set, that's a missed opportunity." And the fact that the API makes it possible for customers to find out if a product is available for pickup at a nearby store once they've purchased it helps provide a competitive edge over online-only retailers, he says. "Now you can search for, buy and pick up within a matter or 20 to 40 minutes."

Legacy data issues

The idea of an in-store pickup option actually came from external developers, Bendt says, and it took the chain some effort to adapt its legacy system to make inventory data available through the API; the data needed to be reformatted to be compatible. "The systems were built at a time before web services and APIs were in active use," he explains. "It wasn't built in a way to expose it externally to the developer."

The specifics of how they did that varied greatly depending on the data source, but generally the team would try to expose some "snapshot" of the data, updated as frequently as possible. If the data proved useful, they found ways to make it available in closer to real time.

Best Buy's strategy was to start slowly, says Steve Bendt,the retailer's director of emerging platforms. Over time, it's added more data for external developers to incorporate into apps..

Getting existing systems to work with the new API was also a challenge at the World Bank, says Malarvizhi Veerappan, open data systems lead. Her group originally struggled with latency issues because their 8,000 different economic indicators were not all directly linked to one another. It was important, she says, to create a structure that could incorporate all that historical data and grow as new information accumulated.

"We didn't want the API to be a separate application. We wanted it to be part of everything else we did with the data," she says. "We needed to connect it back to our data system. It did require our improving our internal data system."

As the API grew, the team added performance monitoring and instituted policies to ensure good traffic flow. The organization also increased server capacity and added server redundancy to assure availability of the API.

When financial information provider Bloomberg LP launched its Open Data Initiative in February 2012, the new open API -- BLPAPI -- was actually version 3 of the software development kit the company had already been using internally, says Shawn Edwards, Bloomberg's chief technology officer. In the old days, Bloomberg customers were given a dedicated terminal that connected them to the company's mainframe, which delivered market data, news and analysis.

Getting existing systems to work with the new API was also a challenge at the World Bank, says Malarvizhi Veerappan, open data systems lead.

(The name "Open Data Initiative" for both the World Bank and Bloomberg projects is just a coincidence; neither has any formal relationship with the Open Data Initiative that is about making use of publicly available data from various government sources.)

Bloomberg's project has since evolved into a software package that customers install on their own systems. Even before making it open, the company used the API to develop specific applications that allow customers to manipulate Bloomberg data on their own desktops.

With the launch of its open API, the company is now allowing customers to create their own apps, such as watch lists for selected securities or their own trading systems. They also allow outside developers to create apps that draw on other data sources as well as Bloomberg's. "We're not giving away market data. What this allows people to do is integrate with other services," Edwards says. "The API is a piece of software that connects to the Bloomberg cloud."

It just makes sense to let others do the app development, he explains. "We're not in the business of selling software," he says. "We're going to win their business by providing the best services and the best data."

When Bloomberg put out the open API, it decided to remove some of the old features that the previous versions supported. There was discussion as to whether the API should be backward compatible. "We said no," Edwards says. That meant some customers wound up with deprecated functions, but Edwards says it makes the API less cluttered with out-of-date functions.

Like most open APIs, the BLPAPI supports a variety of languages, so developers can choose the best one for their app. Someone running an overnight batch process might choose Perl, or the recently released Python version. An electronic trading system would probably run on C or C++. Quantitative analysts, or quants, generally use the data in Matlab. The API also supports Java, .Net, and C#, and Edwards says some developers are using an R wrapper as well.

One key to making an API successful lies in making it easy to use. Back in 2000, RedMonk's O'Grady says, APIs often used web services protocols, but they proved too complex. Now about three quarters of all APIs are REST-based, according to ProgrammableWeb, with SOAP a distant second. "Because developers overwhelmingly preferred this, it's now the dominant protocol for API systems," O'Grady says.

The importance of clarity

Another important requirement is having extensive, clear documentation, and tools to help developers do their jobs. Bloomberg's initial documentation was aimed more at the financial experts who are its customers, and had to be reworked to tell developers what they needed to know.

""The API is a piece of software that connects to the Bloomberg cloud," says Shawn Edwards, Bloomberg's chief technology officer. "We're not giving away market data. What this allows people to do is integrate with other services."

The BLPAPI also tried to make work easier for developers by providing a replay tool that allows them to perform trial runs of their apps, but that was not available when it first launched. Best Buy's BBYOpen also gives developers a set of tools, including a test console to run apps and an automatic widget generator. The World Bank offers a query builder that lets developers select options.

Tools and ideas don't all flow outward from the organizations; external developers often provide information and frameworks to help each other out. BBYOpen, for instance, offers libraries created by developers in Java, .NET, PHP and other languages. At the World Bank, there's a discussion forum where developers can ask questions, and others jump in with solutions.

"They don't wait for us to respond to questions in the forum," says Veerappan, who is working on giving the forum more features and converting it into a knowledge base. "It's kind of interesting to see the knowledge that other developers have gained in the API," she says.

Successful APIs tend to have MIT-style open-source software licenses; the World Bank, for example, uses an Open Source Attribution License. O'Grady says one key to success is being very clear about the terms of service, and not having an overly restrictive license that discourages use.

He says Stack Overflow, a collaboratively edited question-and-answer site for programmers, has a very nice API, for instance, but that the terms of using it are difficult to navigate. Twitter irritated some developers, he adds, by being too insistent about issues such as how the time stamp was formatted, or insisting that the word "tweet" must be capitalized. While developers are unlikely to shun Twitter for being difficult to work with, O'Grady says, "Certainly in some cases if your product isn't that popular people will abandon it."

Cultural resistance

Another non-technological challenge to creating an open API is getting other people in the organization, who are used to dealing in proprietary information and maintaining authority over their brand, to cede some control. "I had to do a lot of convincing," Bloomberg's Edwards says. "It's a different way of thinking, when you've been controlling your product." But he says it was important to distinguish between the market data Bloomberg sells and things like the symbology and software that the company doesn't need to control. "The time for all these proprietary interfaces is gone," he says. "It doesn't add value anymore."

Best Buy's Bendt also faced concerns. "It was tough when we first started talking about an API platform," he says. Colleagues wondered, "What are they going to build? What if they create a bad experience?" The company addressed that with rules about how developers could use the data; they must attribute it to Best Buy, for instance, and can't appropriate it for other purposes. There's no pre-approval of apps, but the company does regular audits to make sure the apps comply with the terms of service.

At the World Bank there was worry that giving away data would mean giving up the revenue that paid for curation of the data. Fantom says the bank decided that a free model would actually be better for the bank's main objective of fighting poverty. "By making these data available for free and using these tools, we've seen a massive increase in the use of our data," he says. "Once you start getting into this, it's pretty clear that this is the right thing to do."

All these organizations say their APIs continue to evolve, adding new functionality, responding to feedback from developers and customers and figuring out what data to make available. "You've got to release the right kind of data with the right documentation. Really, it comes down to what customer problems are you going to solve by doing what you do," Bendt says. "It's not a launch-it-and-leave-it type of capability. It's constant learning and constant improvement."

Neil Savage is a freelance science and technology writer in Lowell, Mass. He can be reached at neil@stefan.com.

Read more about applications in Computerworld's Applications Topic Center.