As a QA, if you are working on a mobile app project, you must consider setting up a mobile device lab to achieve comprehensive and efficient testing. This might give you a headache because there are many options, devices, prices, etc. But, don’t worry, with the right planning and approach you can set up a mobile device lab that will give you the best results.
Identify the Requirements
Before you buy any device, you must first identify the requirements of the app you are testing. Communicate with your team or other stakeholders about the devices and the OS versions the end-users are using or will use (if the app is still not released). You will get some information that could be a good starting point.
Now when you have this information, it is a good practice to think about the tests and the test scenarios.
First, let’s think about the tests. What kind of tests will you perform on the devices? Are you going to do functional testing, performance, security, etc?
Then, define the scenarios that you will test, like device synchronization, different screen sizes, different network speeds, sharing data between devices, working on different devices with different user roles, etc.
Select Your Devices
After defining all of these, you will get an idea of what kind of devices you need. And now it’s the time to get those devices. At this point, you have two options:
1. Cloud-Based Devices
If you are considering the cloud-based option, you can choose some of the most popular cloud providers like BrowserStack, Sauce Labs, or AWC Device Farm. There you can find various devices, and OS versions, that will probably suit your needs.
The benefit of this option is that you don’t have to pay a lot of money to provide all of those devices as physical devices. You can access those devices from everywhere without carrying them with you, without worrying about where to store them, etc. Also, the cloud providers are always updating their device farm with new devices and new OS versions, so you will always have access to the target device you need.
If you have automation tests, for example in Appium, you can connect your tests with the cloud device, and run your tests there, with any device that you specify. They can also be integrated into the CI/CD pipeline.
The drawback of this option is that you don’t have full control over the devices and their configurations and conditions, and the test that is running on a cloud device may give a different result than the one that is running on a physical device.
The second thing is that you are always dependent on the network. You need to have an internet connection to access these devices. If the internet connection is not very good at some moment, that can cause delays when we work with the devices.
The cloud providers will provide you with a lot of testing devices, but keep in mind that if you need a device that is not very common, you may not find it in their inventory.
Also, keep in mind that for some projects, the team and the stakeholders decide to not use the cloud devices because the app contains sensitive information that should not be shared. Sometimes, the clients or the company needs to provide approval from the higher management to use cloud devices.
2. Physical Devices
If you decide to use physical devices, you will be able to use the app like the end-users will do, on a real device, in a more realistic environment, and to perform real-world scenarios. You have full control over the devices, you are not dependent on some provider, or the network connection, and you can test a lot of stuff more realistically. For example, how the battery behaves, does the device overheating, etc.
If you have automation tests, you can also run them on a real physical device.
The drawback of this option is that you have a high upfront cost to buy the devices. When they arrive at your place, you need to think about where to securely store them. You will have to maintain them, manage the OS versions, etc, and you always have to carry them with you when you are going to work in a different place.
When you need a new device, you have to buy that device too, which will add additional cost.
So basically you are limited with the number of physical devices that you can provide and use for testing, in comparison to the cloud-based devices where you have a wide range of devices that you can use.
Based on your defined requirements from before, choose the devices that will suit your testing goal. Choose devices from the manufacturers that the end-users are using the most, for example, Apple, Samsung, Google, and Xiaomi.
Make sure that you have several devices where you can set up the newest OS version and the previous version so you can make comparisons. We have covered this topic in another post that you can read here.
And last, but not least, pick devices with different screen sizes so can test the UI properly.
Get the Devices
Now when you select your devices, you need to decide whether are you going to buy them all, whether are you going to use cloud-based devices, or a combination of both.
Now this depends on your project and your needs, but usually, the teams decide to use a combination of both options. For example, you can use cloud providers for running your automation tests on any targeted device or OS version, and to have that integrated into your CI/CD pipeline. On the other hand, to have a couple of physical devices, where you can perform real-world testing like the end-users would do.
In this way, you will have the most balanced approach and you can avoid the drawbacks of both of the options.
My recommendation is to pick a cloud provider that suits your needs based on the pricing of the devices and the possibilities they offer and to buy a couple of physical devices that are mostly used by the end-users.
Set Up Your Physical Lab
When the physical devices arrive at your place, make sure that you store them correctly. You can order or use some shelves or storage units next to your desk. Organize the charging stations so you can charge the devices on your desk or close to the desk. You could also buy a cable organizer bag to store all the charging cables and power bricks to not have a ton of cables on your desk.
It is a good practice to have a document where you will write what kind of devices you have, their model, screen size, OS version, and any other information that is important for your project. Update the document with every change that you made.
Make sure that they are secure when you are not there.
As we said before, with this kind of setup, you will have a group of physical devices with different OS versions that you can use for testing in a more realistic environment, very similar to what the end-users will have. On the other hand, on some cloud providers like BrowserStack, you will have various devices with different OS versions that you can use for running your automation tests or to test on some specific device that you don’t have in a physical form.
