Quality Can’t Be Emulated, Imitated, or Simulated

Testers find bugs. Finding bugs is one of the main goals for someone in QA. We strive to achieve a high quality product by finding bugs and issues before launch. It’s not exactly how W. Edwards Deming described it, but it is the way that software development works for the most part in fast paced creative environments. One of the keys to being a good tester is to be able to communicate those bugs clearly so that a developer can reproduce the issue every time based on a bug report. Sounds simple – find a bug, describe the bug, fix a bug, check the fix, close the bug.

So I was very surprised recently when a bug which I believed to be very clearly written was returned to me with status “Cannot reproduce.” I had verified the bug on a few iPhones to make sure it was not specific to one device. The repro steps were very clear. I checked the bug on a few more devices to triple check.

Satisfied that the bug was still present, I walked over to the developer to attempt to troubleshoot together.  I asked him to show me what he was seeing on his test device. To my dismay, the developer took me over to his Mac and started up his iPhone emulator. He proceeded to show me that the bug never occurs, no matter which emulator he uses.

This brings me to the common knowledge among testers – which turns out to be not so common knowledge among others – that the best place to test your work is on real life physical devices and machines. In my experience, emulators and simulators, while close to the real thing, are not good enough for assuring quality.  After all, emulators run on completely different software and hardware than do the actual devices. The processing power of most desktop machines is greater than that of any iPhone, and desktop machines running Windows or OS X are not running iOS.   Because of this, results will vary between what an emulator shows and what a real device shows, especially for processor intensive functionality.

And so when we finally agreed to look at real devices for bug reproduction, the developer was able to easily replicate the bug and fix it. While emulators and simulators can work for initial testing among developers, QA should rarely use these tools. Testing on the real thing, when possible, is always best.