iPhone, Galaxy, iPad, Nexus: Here's how they really rate

The InfoWorld Test Center has reevaluated today's mobile devices in light of a changing technology landscape

Comments

Your smartphone or tablet is overrated. At least, that's the conclusion InfoWorld came to in assessing our scoring system for mobile devices. Much as changed since we developed our criteria and weightings four years ago. We saw an explosion of functionality for much of that period, so the average score climbed regularly -- until it was no longer average. And in the last year, there's been a real slowdown in mobile innovation, creating a plateau in our scores.

At the same time, there's been a major shift in the mobile ecosystem. Four years ago, security was a big differentiator, but it's less so today. The new mobile frontier for IT is management, where most platforms have little to offer. The major platforms have good browsers and a good basic set of applications available, so their differences have narrowed and their appropriate weight in our calculations has become too high.

[ Only at InfoWorld: Everything you need to know about securing, managing, and enabling mobile devices. | The mobile road warrior's essential toolkit. | Keep up on key mobile developments and insights with the Mobilize newsletter. ]

Then there's been the explosion of services in each platform -- Apple's iCloud and Siri, or Google's Google Now and Maps, for example -- as mobile devices have moved from being stand-alone endpoints to part of a computing fabric in which mobile devices are one endpoint among many. Our old scoring system didn't anticipate the rise of platform services.

Because only in Lake Woebegone are average children above average, because there's been a pause in mobile innovation, and because there's a big shift toward a fabric of computing that transcends mobile devices, we've tossed our old scoring system for mobile devices and developed a new one that both resets the average and creates headroom for those areas where we foresee innovation making the most strides.

Below, you can see the new scores for the major smartphones and tablets now on the market. In the following pages, I explain the new categories and provide a basic comparison of the major platforms for each.

Apps and WebOur previous scoring system had separate categories for Web and Internet support, business connectivity, and application support, totaling 55 percent of the score. That made sense when smartphones were new: At the time, apps and good browser experiences were the clear differentiators from the previous mobile generation best represented by the messaging-oriented BlackBerry. There was also a real question as to whether mobile devices could hope to be legitimate business devices, rather than merely personal ones.

Today, the distinction between the Web and apps is muddy, and all mobile platforms support the same basic capabilities. And there's no question that smartphones and tablets are real business aids. So, we've merged the Web and application support categories, as well as the portion of the business connectivity category related to apps (such as email, calendar, and contacts support), and reduced their combined weight to 20 percent of the score in what we now call the Apps and Web category.

Still, the same pattern remains in mobile devices' relative scores. iOS has the best selection of apps, in both numbers and functional richness, especially for business users; the Safari browser is very good too. Android's Chrome is arguably better than Safari -- its HTML5 support is certainly better -- but its app universe is both more limited and thinner in capabilities, though no longer at anemic levels.

Windows Phone, BlackBerry, and the emerging browser-based platforms such as Firefox OS and Ubuntu Touch are the least capable when it comes to apps -- not much better than when the original iPhone came on the scene in 2007 -- though their browser capabilities are typically decent. Both Microsoft and BlackBerry are plugging away, though slowly, in their app efforts, so we may see some breakthroughs here in future versions.

Platform servicesThis is a new category in our new mobile scoring system. It includes the services that the platforms provide across their devices, typically through cloud or other network connections. Previously, we'd been including them either in the App Support category or in the Business Connectivity category (if they related to communications, calendars, contacts, and the like). The Platform Services category accounts for 20 percent of the total score, up from perhaps 10 percent in the old scoring system's informal inclusion of such services.

The Platform Services category covers those services that work across devices to enrich and extend the user experience and the devices' functional capabilities:

For Apple, they include iCloud (including iCloud Keychain, data syncing, and shared storage), AirPlay, iTunes, notifications, data detectors, Apple Maps and related services like Find My Friends, and communications services such as iMessage and FaceTime.

For Google, they include Google Now and related search and profile services, Google Maps and related location services, notifications, and Google Drive and related Google accounts services.

For BlackBerry, they include BlackBerry Messenger (BBM) and BlackBerry Hub.

For Microsoft, they include OneDrive and related sync services, the Microsoft Account profile services, the Bing Maps (soon to be replaced with Nokia's Here Maps) and related location services, and communications services such as Skype as they get integrated into the OS.

iOS offers the richest set of such services, covering a wide range of capabilities. Android has a broad set of platform services as well, though Google tends to focus on services involving personal profiling that provide customized information and direct advertising to you; Google has few services that are only for your personal benefit. Microsoft has copied some of the core services from Apple and Google, but they're largely implemented as a haphazard collection. BlackBerry has the least developed notion of platform services, except for the Web-based platforms like Ubuntu Touch and Firefox OS, which are more mobile browsers than true platforms.

Security and managementIt wasn't until 2010 that Apple provided a security API to the iPhone; before then, you got a BlackBerry for security, and it's why BlackBerry owned the corporate market from the late 1990s until about 2012. (In the 2000s, Windows Mobile had some security capabilities through its use of the Exchange ActiveSync protocol, but Microsoft had all but abandoned Windows Mobile before the iPhone's debut, and it was no longer a factor in the market beyond some government segments.)

Apple kept improving its security APIs, and today the iPhone has displaced the BlackBerry as the corporate standard in all but the most security-sensitive organizations. Google copied Apple's approach, though with a smaller set of capabilities. At its release, Microsoft's Windows Phone didn't couldn't equal its Windows Mobile predecessor in terms of security capabilities. Even after its third version, Windows Phone has only reached parity with Windows Mobile, so it remains inadequate for most corporate enviroments.

Today, the security capabilities in BlackBerry and iOS are extensive, while Google chooses to be barely adequate and Microsoft chooses to pursue multiple, incompatible security approaches that keep it a mess. But it's clear that the security capabilities that make sense to include in a mobile platform are now pretty much available from the vendors that care about business users. Most future action will be in unifying mobile and desktop security management, we believe, not in adding more handcuffs and straitjackets to devices.

As a result, we expect security innovation to stagnate. Instead, we see the attention shifting to management, such as application distribution, application licensing, digital rights management, and content controls. The smarter mobile security vendors have been exploring various methods for digital rights management and content management, but it's still a Wild West environment. For application distribution and licensing, Apple's iOS 7 has made a huge step forward in creating a common API set for such management, and we believe it's an opportunity that some competitors will seek to replicate to make it easier for businesses to get actual value from their devices, not just protect them.

In our new scoring system, the weighting of security capabilities is unchanged at 20 percent, but management capabilities have a greater proportion of that weighting than in the previous version.

UsabilityPerhaps the most subjective category, Usability has jumped from 15 percent of the score in our old system to 20 percent in our new system. We made this change because mobile devices are both widely adopted and now a maturing technology. Usability matters more in both cases.

Our criteria, though, remain unchanged: Ease of use, ease of discovery, and consistent operation matter greatly, no matter what user interface approach a platform takes to deliver them. Excessive effort for users, needlessly complex interfaces, and inconsistent execution are all demerits in our scores.

Currently, Apple's devices do best in all three areas. Android devices as a class come in second, though the variations in Android user interfaces from vendor to vendor and even device to device mean that the Usability scores for Android devices can differ from one another. BlackBerry and Windows Phone fare worse than Android, for similar reasons: Both have very nice front ends (the tile interface in Windows Phone and Hub in BlackBerry OS), but as you move into specific apps and functions, you start seeing the seams underneath their façades.

HardwareWe've upped the weight of the Hardware category from 10 percent in the old scoring system to 20 percent in the new system, simply because the hardware has become more sophisticated than it was four years ago. Sophisticated cameras, quality audio, a growing array of sensors, increasing support for peripherals' connectivity, and a bevy of unique features such as Apple's Touch ID fingerprint sensor, Motorola's X8 motion coprocessor, and Samsung's stylus make today's mobile devices substantially different from their predecessors.

But we have reset our scores to lower the average, so there's room for growth in hardware capabiities. We now assume that a business-class device will have a responsive, crisp, high-density touchscreen; will support video-out and Bluetooth 4 connectivity; will be made of quality, durable materials; will have battery life will be sufficient for all-day use; and will provide enough storage and processing capability to perform like the computer it is.

A device scores at least a 6 if it does decently in these areas. To score higher than 6, a device has to bring in extra hardware capabilities that work well, or somehow take several standard capabilities and bring them to a useful new level. In our new scores, you'll see that nearly all devices score a 6 or 7 -- only the terribly underpowered ZTE Open gets the minimum score of 5, and none scores 8 or higher.

A note on scoringYou might have noticed the phrase "minimum score of 5" in the preceding section. InfoWorld has been testing products for nearly 35 years, and in that time, the scores between 1 and 4 have become increasingly uncommon as even bad products hit basic standards. Thus, today, the InfoWorld Test Center scores range from 5 (poor) to 10 (excellent). A score of 6 is fair, 7 is good, and 8 is very good. If these were grades in school, a score of 5 would correspond to a grade of D, 6 to C, 7 to C+, 8 to B, 9 to A, and 10 to A+.

What about F? These days, it's rare that a product is that bad. A product must lose data or be regularly unreliable, highly inadequate, or highly unsafe to score below 5. But if it's merited, we will score below 5.

Reviews recapIf you want to understand where our new scores came from, we invite you to read our reviews of the smartphones and tablets mentioned here. Although we have not revised our previously published reviews to incorporate the new scoring system, you'll learn where we found flaws, advantages, commonalities, and key differences among devices, and you should be able to see from those details how they affected the new scores. Those reviews: