How OpenStack supports Verizon Media's publications

A $4.4bn deal in 2015 saw AOL-owned publications transferred to Verizon Media (at that time called Oath, the group was rebranded in January 2019). Despite the long legacy tail, the company now runs the majority of its publications on open source.

Speaking at the Open Infrastructure Summit in Denver this week, architecture director James Penick drew a direct line from the days of in-house development at Yahoo to the pivot to open source infrastructure which supports the titles today.

The Huffington Post, AOL, Yahoo and Techcrunch are, said Penick, "very popular sites", and consequently "we operate one of the largest infrastructure environments in the world".

Today there are "hundreds of thousands of physical machines with millions of cores," he said, most of which run on Openstack.

Penick started at the company several brand iterations ago when it was still Yahoo. Back then, almost all of the infrastructure had to be built on custom platforms because no one else was operating at Yahoo's scale, meaning off-the-shelf solutions just weren't available.

In the short term, Penick noted, building platforms internally as a closed shop appears to produce a higher return on investment.

"You have an idea and you build a prototype, you go to production...it was fast, easy," he said. "You can go from an idea to production very fast. But in the long run you will lose out to open infrastructure every time - because as I stand here, right now, someone in the world is working on a piece of software that's moving my infrastructure forward a little bit more.

"They don't work for my company, but somewhere in the world, somebody is improving my infrastructure."

He recalled that an ex boss had remarked that Penick was the only software architect he'd ever met who didn't insist on building everything from scratch himself.

"He asked why I chose to use a certain open source cloud platform [OpenStack] instead of design on my own, and I said, the reason is: because it exists," he said.

"It exists and it has an active and passionate global community who are constantly working to improve their product."

How did they do it?

Naturally, moving more than 20 years of infrastructure to open source tooling is not the simplest task.

"Yes, there are people around the world building tools for you, but it's not the tools that build the house, it's the carpenter," said Penick.

The team started out by running some experimental virtualisation clusters, however, in doing this they realised they were delaying the inevitable: focusing on the foundation of the infrastructure, the bare metal servers.

Although installing an operating system on a physical machine seemed like a basic problem on the surface, it turned out to be "very, very hard". But by getting their heads around deploying and operating bare metal, they eventually created a solid foundation to build on top of.

"VMs, containers, are all awesome," he said. "But each thing stands on the layer beneath: they all stand on top of the foundation - bare metal is not flashy, it's not glamorous, it's not the latest cool new toy that people are making breathless podcasts about. But you have got to deal with it, so we invested in them."

This process saw the business first adopting infrastructure-as-a-service on bare metal as an option, then a default, before finally making it mandatory.

"Are we done? No, we're managing the bulk of our infrastructure, hundreds of thousands of machines, millions of cores. We focused on the commodity 80 percent of the problem first, we took some of the special cases, then we made them into commodity cases," he added.

"And now we're taking the rest of the special cases, and we're focusing on that. It's very hard, and it takes a lot of the 'three Ps': passion, persistence, and persuasion. But it can be done."

Now the company is running Kubernetes containers on top of bare metal and on VMs. These containers and production workloads exist at "all levels" of the stack.

"One day, my dream is to push all of our workloads off bare metal and into higher order functions," he said, adding that even if that was successful, bare metal will still be supporting the foundations below.

"Our experiences moving over four million cores to technology like OpenStack Ironic, I hope can help encourage you and give you the confidence to do the same," he said.