Let’s Ship! A (Speculative) Lesson Learned from the iOS 8.0.1 Launch

iOS 8.0.1 was not a big success. It was released about a week after iOS 8.0 officially launched. The build’s goal was to fix a few bugs, but instead, 8.0.1 turned into a debacle that will cost Apple time and money.  Worse yet, it’s bound to lower people’s lofty perception of Apple’s commitment to quality.  iOS 8.0.1 had a handful of major blocking bugs, including the loss of cellular service and use of a device’s Touch ID. Apple quickly pulled the build, but of course the damage was done.

How did this happen? We’re left to speculate, but in my opinion those issues are unlikely ones that could be missed during manual testing. One of the simplest test cases from a smoke test case doc would be “User is able to use Touch ID to unlock device.” Another would have to be “User is able to make and receive calls.” Those would have to be some of the first things tested. Keep in mind test devices go to sleep all the time. Some, if not all, of the dozens of Apple testers who were testing this update would have noticed that they cannot unlock their device with this build. I’m doubtful that this failure stems from a testing issue.

But the process broke down somewhere. There aren’t many suspects left to consider. Knowing that a new build of an operating system is a prime candidate for automated testing, it is possible that automated testing was implemented incorrectly. But Touch ID testing is not something that can be easily automated. You need a human finger, and (according to my research) it is illegal to buy a human finger.  I’m willing to give Apple the benefit of the doubt on this point.

So you need people. If Apple used automation as a time saving tool for regression testing, Touch ID was likely reserved for manual regression testing. And if the time crunch was too extreme, manual testing would have been skipped.

Connecting the dots, it would appear the time crunch forced the project management team to choose on-time and on-budget over quality. If testing time is limited, bugs will leak through to production. There are hundreds if not thousands of acceptance test items for every build of something as robust as an operating system. They take time to test, even if they are automated. If time is chosen over quality, the result was a 4% drop in stock price (to the tune of $20 billion).  Adding insult to injury, their haste may lead to a loss in sales due to a perception of lack of quality.  A few days invested in testing doesn’t sound so bad after all, does it?