The phone we test on is usually the ideal phone: fully charged, fast internet, our own language, nobody calling. The player’s or the user’s phone is not that phone.
Most bugs live in that gap. The app behaves in normal conditions, and then someone opens it on the subway, at 5% battery, with the phone in Japanese.
The things that are hard to do by hand
Reproducing these states at your desk is a chore:
- Actually draining the battery. You either wait, or fiddle with battery saver settings.
- Breaking the network. Airplane mode is easy, but “a weak 3G signal” or “no service” on demand is not.
- Switching the phone’s language. Changing it in Settings and back again is separate work for every test.
- A call arriving mid-checkout. Timing that needs a second phone next to you.
- Step counter, location, sensors. Looking like you are walking, or leaving GPS without a fix, is not something you can do by hand.
So these scenarios drop off the test list. And then the bugs come from there.
In TestSafe the phone is a setting
In TestSafe the state of the device is part of the test. When you define an environment you say what state you want the phone in, and the test starts in that state. In our own testing we use these:
- Default device: Nothing customized. The nightly tests use this one.
- Turkish phone: The same device with the interface in Turkish. We want to see the app’s Turkish strings actually load.
- Weak signal, low battery: Battery pinned at 12%, cellular on 3G with a poor signal. The checkout and ride flows are tried here.
You prepare the environment once and reuse it in every test. You can also run the same scenario in two environments and put the results side by side.
It can change mid-test too
Some bugs don’t show up in a fixed state, but while the state changes. There are ready-made steps for that: drop the battery level, switch to airplane mode, change the network type, send an SMS, ring the phone, change the location, rotate the screen or switch the resolution to a tablet’s.
So a scenario can read: fill the cart, get to checkout, now cut the network, bring it back and check that the order was not created twice.
What we watch out for
Three habits help when writing this kind of test:
- Set the state close to where the bug lives. Dropping the battery after reaching checkout, rather than at the start, shows more.
- Write down what you expect. “The app must not crash” is thin. “Closing the warning must leave the cart as it was” gives the report something to check.
- Change one thing at a time. Break the language, the network and the battery together and it gets hard to tell which one produced the bug.
We collected what can be set on the device on the Device Conditions page. If you are wondering which state breaks your own app, the best starting point is usually what users complain about: “it happens on the bus”, “it happens when the battery is low”, “it happens when a call comes in”.