Digital universe and its impact bigger than thought

A white paper released Tuesday from International Data (IDC) revised earlier estimates by the research firm to show that by 2011, the amount of electronic data created and stored will grow 10 times the 180 exabytes that existed in 2006, reflecting a compound annual growth rate of almost 60 per cent.

By 2011, there will be 1,800 exabytes of electronic data in existence or 1.8 zettabytes (an exabyte is equal to 1 billion gigabytes). In fact, the number of bits stored already exceeds the estimated number of stars in the universe, IDC stated. And, because data is growing by a factor of 10 every five years, by 2023 the number of stored bits will surpass Avogadro's number, which is the number of carbon atoms in 12 grams or 602,200,000,000,000,000,000,000 (6.022 x 1,023).

The study, "The Diverse and Exploding Digital Universe," also found that the rate at which electronic containers for that data -- files, images, packets, tag contents -- is growing 50 per cent faster than the data itself, and in the year 2011 data will be contained in more than 20 quadrillion -- 20 million billion -- of those containers, creating a tremendous management challenge for both businesses and consumers.

The study is an updated forecast to a white paper released last March.

Along with the new white paper, which was sponsored by EMC, IDC created a "Personal Digital Footprint Calculator," which resides on the same web page as EMC's worldwide digital growth tracking ticker that counts the amount of data created second by second.

"Obviously, none of us will relate to a number with 21 zeros at the end. The way that the ticker is practically blurred at the far end we hope will give people an idea of the pace - of how fast this is happening. In a way, this is the message of the study," John F. Gantz, IDC's chief research officer, said in a recorded interview.

IDC also admitted it underestimated earlier digital data figures for 2007, saying the actual amount of data -- 281 exabytes -- is 10 per cent greater than it had previously forecasted in a first "Digital Universe" study. IDC said the bigger numbers was the result of faster growth in digital cameras, televisions and data a better understanding of data replication.

The diversity of the digital data producing massive growth runs the gamut, from 6GB movies on DVD to 128-bit signals from RFID tags.

Less than half of the digital data being created by individuals can be accounted for by user activities, such as photos, phone calls, emails. The majority of the data is made up by what IDC calls digital "shadows" -- surveillance photos, Web search histories, financial transaction journals, mailing lists, etc.

Gantz said the impact of the tremendous data growth -- even if the data is not being created within the company's four walls -- will directly affect CIOs. A CIO in a data center that has been crunching numbers in IBM format for 40 years, "may be surprised what will happen" when all of the sudden all the voice calls in the enterprise are stored on disk and all the video cameras that used to be closed circuit TVs are now digital surveillance cameras forcing you to deal with video files, Gantz said.

Also, much of the data being created by consumers outside of an enterprise's four walls will still force that company to protect the data. "At some point in the life of every file, or bit or packet, 85 per cent of that information somewhere goes through a corporate computer, website, network or asset," Gantz said. "The corporation at that point in time has responsibility for that information. This is something that can be lost on the CIOs of the enterprises."

Page Break

"It's not until Viacom sues Google for a billion dollars for the information that its customers are putting on these basically rented electronic storage space[s], and all of a sudden Google has got to fight a lawsuit," Gantz said. "Whether they win or lose it's hundreds of millions of dollars in lawyer fees. So the enterprise is picking up responsibility for information that's not actually created at the hub of the enterprise."

To deal with this explosion of the digital universe in size and complexity, IT organizations will face three main imperatives:

Also adding to the deluge of digital data being created is the worldwide changeover in the next several years from analog TV systems digital. According to IDC, the number of digital TVs in the world doubled last year and should surpass 500 million by the end of 2011.

Digital growth's impact on the environment

IDC estimates that the amount of electronic waste being created alongside data is already accumulating at more than 1 billion units a year. Most of it comes from mobile phones, but personal digital electronics and PCs are also greatly impacting the environment.

As the world changes over to digital televisions, analog sets and obsolete set-top boxes and DVDs "will be heaped on the waste piles, which will double by 2011," IDC said. How much power will be consumed by the added data glut is more difficult to determine, but manufacturers developing power-saving chips and users installing power-saving systems, including new cooling and air conditioning and new management systems, should help. According to a 2006 study by IDC, server power and cooling costs were escalating rapidly as more powerful servers were being introduced. "Power consumption that was 1kW per server rack in 2000 is now closer to 10kW," the study found. "Customers building new datacenters are planning for 20kW per rack."

The main findings of the Digital Universe study:

• The digital universe is bigger than we thought: 281exabytes in 2007 - 10 per cent bigger than we said it would be last year.

• Digital cameras and security cameras out shipped our expectations.

• The digital universe will grow ten-fold in five years

• In 2007 the digital universe contained more bits (computer 1s and 0s) than there are stars in the physical universe

• While 70 per cent of the information in the digital universe is created by individuals, enterprises have some responsibility or liability for 85 per cent of it (think of Google getting sued by Viacom for US$1 billion because of the videos uploaded by consumers)

• There is now more information created in a year than there is capacity to store it responsibility for the information in the digital universe by industry does NOT match IT spend or GDP distribution by industry.

• A single email with a 1MB attachment sent to 4 colleagues can generate 50MB of information in the digital universe (based on an IDC-like architecture)