Automated UI Testing with Visual Studio App Center
Steven Gerdes, Android Developer | 10/30/2018
The problem with UI testing
UI tests carry a reputation of being brittle, slow and hard to maintain, but this isn’t always the case. There are smart ways to implement UI testing that can help alleviate these problems. For example, the page object pattern can reduce the rigidity of UI tests and make them less brittle, and using a native UI testing framework can help increase test speeds. However, one very big concern with mobile UI testing is the vast and ever-growing number of Android and iOS devices that could be running your application.
Which devices should you test on? The cost of testing even a fraction of all the devices that exist on the market will quickly outweigh the benefit. The sample size needs to be small enough to get consistent and reliable results, yet large enough to accurately represent the devices that your app is running on. In addition, you need the flexibility to change devices in order to stay up-to-date with the preferences of your users. All of this makes building and maintaining a local device lab both time-consuming and very expensive. Cloud testing platforms can help mitigate this problem. At Tack Mobile, we have build servers that handle our CI/CD pipelines and a respectable number of test devices. It was only recently that we decided to explore the different cloud options for our automated UI testing needs.
UI testing on real devices in the cloud
A couple of months ago, we started working on a Xamarin, cross-platform project. The application had a robust layer of unit tests, but only a handful of UI tests, seemingly abandoned. The client wanted to grow their test portfolio by expanding the UI testing layer. We decided that our local test environment would not be the best option, and instead decided to run the tests on cloud devices. Here are a few reasons why we took this approach:
1. It puts the automated testing system directly in the customer’s hands. It’s autonomous and independent of the party that implemented it, and can be easily managed by the client if needed.
2. It can reduce time and device limitations. A cloud-based system offers 24-hour availability and some offer thousands of real devices running a variety of OS versions.
3. Cloud platforms usually have a simple UI for setting up, managing and analyzing tests. This can greatly reduce the time to competency for the customer and their in-house teams.
How we landed on App Center
We were using Xamarin.UITest for our automated testing framework because it has full IDE support, enabling us to run UI tests locally through Visual Studio. Also, Xamarin.UITest is written in C# allowing us to use a single language throughout the codebase. The downside to using Xamarin.UITest is that it is only supported by one cloud testing platform, Visual Studio App Center (formerly Xamarin Test Cloud). So we had to use App Center because it was our only option. That being said, App Center still stood out when we compared it to other cloud testing platforms. Here’s why:
1. Prior to its acquisition, Xamarin Test Cloud had a strong reputation within the Xamarin community.
2. Competitive pricing: App Center’s cheapest subscription is $99/month for 30 device hours, which is cheaper than or equal to many comparable cloud platform plans.
3. It offers hundreds of devices running a variety of major OS versions.
4. You can integrate App Center with other CI/CD platforms. Our project was using Bitrise.
How to start using App Center
The following steps outline the process for setting up and running automated UI tests on App Center. Hopefully, this will serve as a guide for developers and testers who are planning to do the same. Although, the steps are tailored for Xamarin, the process should be similar for native applications.
1. Create your Apps, Device Set, and Test Series
Start by adding two apps to your App Center account, one for iOS and one for Android. Make sure you select the appropriate platforms (ours was Xamarin for both apps). Once you have created your apps, create your device sets. Hover over the test icon on the left sidebar and click “Device sets.” Then click the “New device set” button in the top-right corner. You’ll get to choose from hundreds of devices running a variety of major OS versions. Next, create a test series. Hover over the test icon in the sidebar, click “Test runs,” then click the “Test series” button. Finally, click the “Create new series” button, name the series and click “Create.” If you have a good idea of the devices you’ll be testing on, the setup process can be completed in 5-10 minutes.
2. New Test Run
The next step is to create a new test run. First, click “Test runs” in the sidebar, followed by “New test run” in the top right-hand corner of the screen. Choose the device set you would like to run your tests on, then your test series, system language, and test framework.
The final step in the “New test run” flow is a page of instructions that describe how to upload the tests manually from your local terminal using the App Center Command Line Interface.
Final step in the App Center "New test run" flow
The instructions list the required versions of App Center CLI and Node.js, how to install appcenter-cli locally, CLI help commands and a generated CLI command for uploading the tests. The generated CLI command consists of six parameters, four of which are pre-filled with the options selected in the previous steps. The remaining two parameters should be replaced with your project-specific file paths.
3. Install App Center CLI and log in
From the terminal, install App Center CLI with npm install -g appcenter-cli. Once you have the CLI installed, log in using appcenter login. Follow the link and copy/paste the auth token from your browser to your terminal. This step is not described in the App Center instructions. Instead it can be found in the Microsoft App Center docs.
4. Manually upload tests to App Center and start a test run
Navigate to the directory that contains the Xamarin.UITest package (the packages directory in our project). This is where the upload command should be executed. Enter the command from the App Center instructions into your terminal. Remember to use your project-specific paths for --app-path and --test-dir. After a few seconds your tests should start uploading to App Center. You should see a progress indicator in your terminal.
5. Automate it
Using a Shell Script
If you’d like to write a script for uploading and starting tests on App Center you’ll need to add two more parameters to the App Center CLI command: --uitest-tools-dir and --token. Below is an example of the complete CLI command.
App Center CLI Command
The last two command options specify the location of the Xamarin.UITest package and your App Center auth token which you copied from your browser earlier. Run the command appcenter help test run uitest in your terminal to get a list of other App Center command options. Note: we use a wildcard in the --uitest-test-tools-dir parameter in case the version changes in the future.
Using Bitrise
If you’re using Bitrise for your CI/CD, like us, the process for uploading and starting tests on App Center is like writing a script, but with a convenient UI and predefined build steps provided by the Bitrise community. Find the App Center step and include it at the end of your test workflow. The step outlines all the command options required to upload and start UI tests on App Center. Then just fill in the command options with your project configurations and start a build.
Our App Center experience
While running our tests consistently for a of couple of weeks, we were able to make several observations about App Center. Some of our experiences may have been unique. We think they are worth sharing, nonetheless.
Buggy devices
After only a few test runs we noticed some issues with two of the devices that we were testing on. One device had location services turned off which caused nearly 50% of our tests to fail. We were told by App Center support that it was a bug, and the device had been removed from the lineup. We faced a similar issue when a device lost wifi for more than half of a test run. We contacted support and they resolved it by reimbursing us our monthly device hours, which was very generous of them. Fortunately, we didn’t experience any other device-specific issues after those two instances.
Waiting for…devices
The biggest problem we encountered was the surprisingly long time it took for our test runs to finish. Our original plan was to run our full UI test suite on Android and iOS three times per week, but we quickly realized that wasn’t going to be possible. Initially, we attributed the slow test runs to Xamarin.UITest, which is considerably slower than native UI testing frameworks. But after four test runs of 175-200 tests, we noticed that a large portion of the time was actually spent waiting for devices to become available.
Our Android tests ran on 2 devices: Samsung Galaxy S7 and S8 Plus, running Android 7.0 and 8.0.0, respectively. Our average test duration (time spent running the actual tests) was 176 minutes, or 2.9 hours. The average test runtime (time from uploading tests to receiving results) was 382 minutes, or 6.4 hours. Wait times had more than doubled our total testing time.
We knew we wouldn’t be able to run our tests three times per week. At 2.9 hours billed for each test run, Android alone would have put us over our 30 hours/month. But the wait times further complicated things for us. We had a time constraint. Our client’s dev server had nightly maintenance, which caused the service responses to be unpredictable after business hours. So, we were only able to run the tests on one platform per day. Even if we had more monthly device hours it still wouldn’t have been possible for us to run our tests three times per week. Essentially, we were left with two options. We could reduce the latency between test runs by upgrading our App Center plan to include device concurrency, or we could run our tests fewer times per week. Since we had already met our client’s budget for UI testing, we were stuck with the latter.
Waiting for…support
App Center support was very slow to respond to our inquiries. The chat window states that the team typically responds within a few hours. In our experience it took days. On the rare occasion, when we did get a response within a few hours, the subsequent text exchange inevitably dragged on for days. Nevertheless, all our issues did eventually get resolved. Despite the wait times, App Center support always provided us with good feedback and helpful information. They’ll solve all of your problems, just very slowly.
The test summaries are first-rate
App Center test results are neatly packaged and laid out in an intuitive way. The UI provides a simple way for accessing and searching the device logs, which was helpful for debugging tests initially. We added a liberal number of screenshots to our tests to get a clear visualization of problems that could occur before, during or after test failures. An advantage we gained from this was that it provided an easy way for us to QA new features as they were added; decreasing the need to run through the application flow manually on multiple devices. A possible disadvantage to this approach could be an increase in test times. Although, we found no documentation on the subject and could not verify that our test times were significantly impacted by the number of screenshots.
Should you test your application on App Center?
App Center has a lot of potential, but for us it was overshadowed by its deficiencies. We feel their biggest priority should be reducing wait times by increasing the number of test devices. The long wait times severely limited our number of test runs per week and it also eliminated the possibility for us to test on more than 5 devices (Android and iOS) without upgrading our subscription. We would also like to see them improve their testing docs and expand their support staff, which would make for a much faster and more enjoyable onboarding process.
If you are thinking of cloud testing, consider your situation. Our circumstances probably wouldn’t be deemed ordinary. We were working with a Xamarin application and executing cross-platform tests using Xamarin.UITest. We had an existing CI platform that we integrated with App Center. Also, our testing window was limited because our service was unusable outside of business hours. If your circumstances are similar to these, then you probably won’t find a reasonable alternative to App Center. Hopefully, Microsoft will improve cloud testing support for the Xamarin community in the future. Until then, take advantage of the App Center 30-day free trial and see what you think. For anyone else, not bound by these circumstances, we recommend exploring all of your options carefully.