Storage industry tackles making sense of metadata

Here is a wager I bet I will win. How many of us took a photo, saved a video, or downloaded a music track and have no clue where it's hiding today? Maybe it's buried somewhere on your PC's hard drive or in one of dozen external drives.

The explosion of user-generated digital content and the increasing digitization of our day-to-day lives (music, photos, and video) is keeping that demand for evermore storage high. But it also means we're left with a messy morass of data that needs to be organized. For this reason, I find it encouraging that, in advance of the start of the Consumer Electronics Show, the discussion at the Storage Visions conference here in Las Vegas has turned to the issue of metadata.

I'm serious. And it's about time.

Today our storage is scattered among a hodgepodge of devices -our phones, our DVRs, our notebook PCs, our cameras, our set-top boxes, our favorite Web services. We have content scattered across all of these devices. As we ramp up the digital content we acquire, the need for indexing and tracking this content in a meaningful way has grown exponentially, and it's the elephant in the room every time we capture content, save content, purchase digital content online.

As the jumble of bits and bytes and mysteriously named DSC35493201111.jpg files grow on our hard drives, the situation is reaching critical mass. Unfortunately, the reality is this won't be solved overnight, and the consumer electronics industry as a whole is struggling with this issue. Every year at CES it feels as if we get closer to the home server-driven digital living room environment the industry likes to imagine for the mass-market future, but hang-ups like metadata will continue to hold this vision at bay.

Automatic metadata creation remains the Holy Grail, but the technology isn't ready yet to be affordable and mainstream. The way broadcast media handles metadata today doesn't work for this mass market scale: You can't have people dedicated to spending eight hours to index a one hour broadcast. And individuals can't do like Rovi, whose company has hundreds of people dedicated to metadata generation.

For metadata to work and work well, it has to be universal, and be understood in context by different devices, applications, and systems. The issues are complex, and involve interoperability across operating systems and devices. Over time, next-generation storage servers, for example, should do more than just store data-they should be smart, metadata aware devices that can help consumers access the vast content libraries they've amassed over time.

The consensus that easy and meaningful metadata generation is necessary may be there, but the biggest challenge of all may be the industry's tendency towards inertia. As one speaker on the panel noted, "For metadata to be useful, everyone needs to agree on it."

We're not there yet.