Netflix has released Chaos Monkey, which it uses internally to test the resiliency of its Amazon Web Services cloud computing architecture, making available for free one of the tools the video streaming company uses to keep its massive cloud computing architecture running.
Chaos Monkey is a free download available from GitHub as of today. It works by randomly terminating instances of virtual machines in applications, simulating what would happen during a disaster event. "The best defense against major unexpected failures is to fail often," Netflix officials wrote in the blog post titled "Chaos Monkey released into the wild."
RELATED: Four tips to prepare for the next Amazon outage
YOU'VE GOT TO SEE THIS: Eye candy for IT: 25 award-winning designs
Just how secure public cloud computing offerings are from providers has come into focus this summer as Amazon Web Services suffered a significant outage that brought down Netflix, as well as other media companies Instagram and Pinterest. Salesforce.com, the major software-as-a-service (SaaS) provider, was hit with two outages in as many weeks earlier this summer as well.
Chaos Monkey can be configured to work on the Amazon Web Services offering or, with some tweaking, on other cloud computing offerings. It can be programmed to initiate a testing scheme with various frequencies and to be done during various times of the day, for example on average of once a week or once a day. In practice, a highly resilient cloud should automatically detect the outage and spin up new, identically configured virtual machines that keep the application running with no visible impact to the user.
Netflix says it has run Chaos Monkey internally to create 65,000 failed instances across its system. "Failures happen and they inevitably happen when least desired or expected," the blog reads, continuing later: "Even if you are confident that your architecture can tolerate an instance failure, are you sure it will still be able to next week? How about next month? Software is complex and dynamic and that 'simple fix' you put in place last week could have undesired consequences."
Jeremiah Peschka, managing director at IT consultancy Brent Ozar PLF, says clients he advises too often overlook installing disaster recovery plans, much less testing them. "This seems like a really sane and safe way to see if you're protected that doesn't cost a ton," he says. Netflix says in the blog the service could likely be run using Amazon SimpleDB, a relational database, and is small enough that it could be run within AWS's free usage tier, which covers up to 25 SimpleDB machine hours and 1GB of storage.
For users that may be nervous about intentionally failing their systems, Peschka recommends running Chaos Monkey in a scaled testing area that mimics a production environment. Other ways to test DR systems, he says, are traditionally from manually built processes of shutting down servers, but Netflix releasing Chaos Monkey is the first DR testing system he's seen released in an open source fashion.
"One of the biggest stumbling blocks of the cloud is that it can go down," he says. "People need to test this stuff and if you're really concerned about having 24/7 uptime, there's probably even more you can do." For example, highly reliant systems typically use elastic load balancers, which are available from Amazon Web Services, to reconfigure virtual machines away from failed instances. Additional testing to fail an entire availability zone and perhaps even an entire region are being tested by Netflix, the company has hinted at in what it calls "Chaos Gorilla." Netflix says it hopes to open source other tools it uses as well in the future.
Network World staff writer Brandon Butler covers cloud computing and social collaboration. He can be reached at BButler@nww.com and found on Twitter at @BButlerNWW.
Read more about data center in Network World's Data Center section.