Cloud-based Mobile App testing - Cloud-Based Platform for DevOps
Our experience with the Calabash and Xamarin Test Cloud pair
We wanted to:
- Our applications cover functional automated tests
- We can run the tests on physical devices,
- Have the ability to repeat the same thing on hundreds of physical devices through a cloud service.
This blog will talk about how we can achieve the above.
We are developing more and more mobile apps for the international market with a complex structure and reach up to hundreds of thousands of active users every month who use a wide variety of devices and operating systems.
At this magnitude, even a device that is not sustainable in the market, with usage at 0.1% can mean thousands of user sessions per month. This is no small headache if a bug slips through the manual test.
Although we have dozens of test devices that are the same in size and strength as the current statistics, we need to update this pool regularly and simply fail to provide 100% coverage – and it doesn’t make much sense.
But that’s exactly what cloud-based services do. W
Because we want to get this far, we start experimenting with BDD and look at the cloud-based test platforms available on the market to finally make an informed choice of the final runner, which is worth settling for in the longer term.
Clarification of the purpose
To do this, we wanted to choose a tool in which body scripts could be run both in-house on our build servers and in a cloud-based service without modification.
Our goal was to reduce the time spent on manual regression testing (close to zero) and to be able to run our developments on a much larger device pool, which also includes real devices, in addition to the physical devices available in-house before making another release.
But such a tool is also useful when you need to take over and develop a legacy app, which of course involves a certain level of refactoring – even in such cases, we start with tests related to the affected features.
A Behavior Driven Development (BDD) tool
As in the selection of the other tools used, in this case we thoroughly went around and compared the available solutions.
The following aspects were the most important for us:
- It should be a cross-platform solution (treat the two largest mobile platforms equally).
- The completed tests should be run locally and in cloud-based services that use physical devices.
- Try not to allocate separate Java, Swift, Objective-C, PHP, JS, Python, or C # developers for the task, test writing should be advanced, if possible, stay within the task in QA.
- A cross-platform approach should also be feasible in test writing: a test script should handle both platforms.
- The process should be well reportable, or better yet, if opportunities are outlined in the direction of collaborative testing, the implementation of which is our future goal.
- Reuse the code as much as possible.
So looking back, we had a bit high expectations of the new tool.
We also examined non-cross-platform solutions, but their application would have been a problem in the selection of device clouds, and we still did not consider it strategically fortunate to start on two platforms in two completely different directions.
By keeping the task within the QA division, we wanted to maintain the condition that no developer resources need to be allocated for functional testing of the developments – the QA manager is the third chance to find logical loops, resp. bugs as well as run development and testing simultaneously.
Which cloud-based service should we choose?
When selecting body clouds, we first considered the following aspects:
- Run tests on physically existing devices;
- Manage the framework (also) we use locally and on our build servers;
- The cloud device pool should largely fit the tools and OS versions measured in the analytics of our projects – e.g. it would have made no sense to use a device pool based on Indian conditions, which is only a few percents identical to the data measured in the EMEA region;
- Be it group and user management, our customers can also get insight into the testing processes (collaborative capabilities are an added advantage);
- Be affordable.
Emulator/simulator services have fallen out because we can implement them on our servers, we didn’t want to pay for them unnecessarily, and in our experience, testing on emulators for Android can’t return serious results anyway, and there can’t be a proper answer if testing in such a particular version of a fragmented market is the goal.
Based on the above considerations, solutions with technically unreliable, frivolous UX / UI or low pricing soon fell out at the beginning of the research. We were already looking for a test tool by monitoring the services that were built on it.
Anyway, it is worth constantly monitoring the market as a lot of changes have taken place in the last year and a half. New service providers have appeared, some have expanded functions, pricing is almost constantly changing everywhere, and there is no shortage of acquisitions (SeeTest, MonkeyTalk, TestDroid, Xamarin).
In the end, we filtered out the dozens of options pretty quickly, and the only question was which of the two most serious players to choose.
Appium vs. Calabash
Appium is an open-source test automation framework for use with native, hybrid, and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol.
Calabash enables you to write and execute automated acceptance tests of mobile apps. It’s cross-platform, supporting Android and iOS native apps. It’s also open-source and free, developed and maintained by Xamarin.
There are plenty of articles on the subject, but most are based on only superficial knowledge and provide misinformation that has made us a little unsure, but in the end we tried both and chose Calabash.
We also managed to dispel a few misconceptions
Myth 1: Calabash does not handle hybrid apps.
It handles and handles WebView. Web elements can be referenced with CSS, JS, and XPath. We also need it in some of our apps where third-party providers appear in WebView. But we find an article that raises the same issue with Appium.
Myth 2: Calabash is two devices, the Appium one.
It’s true that at Calabash we have to install a gem per platform, and there are some differences in the methods and query language capabilities that can be used between the two platforms, but these are not nearly as many differences that you can’t get used to in a short time.
Myth 3: To use Calabash, you need to create separate builds, not with Appium.
On Android, Calabash does this automatically, supplementing the APK file with an HTTP server before starting the test run.
You already need to embed a framework on iOS and build the app with a new target, but it’s nowhere near as devilish as it sounds at first. Especially since we have a separate job for this on our build server in GitLab CI, which automatically creates the augmented IPA in the test stage. For this reason, developers do not have a separate job.
Myth 4: There is no solution similar to Appium Inspector in Calabash.
There really isn’t a GUI, but instead, there is a text console in which we can display information about the same in a more structured manner, and we can also filter it quite efficiently using queries since this is practically an IRB Calabash library, there is also a way to test our snippets from a console before embedding them in the test script and want to run it all at once.
Anyone who wants to extract IDs in a graphical interface can still use Android Studio’s UI Automator Viewer and Xcode Accessibility Inspector.
Myth 5: Using Appium, we can write tests in multiple languages, but with Calabash we can only code in Ruby.
This used to be the most deterrent, but the reality is more complex. The fact is that Calabash is based on Ruby, but writing scripts don’t require nearly as much knowledge of the language as if we wanted to write a test in Appium in the same language. When writing Calabash tests, we must use the methods written in the Calabash API. If we have to keep going deeper than that, we’re doing something wrong. But it’s enough to just watch a script written in Ruby on Appium and Calabash. Which is more complex?
What decided the language of the balance towards Calabash is that it can be learned and used quickly even without any special developer experience, while Appium is a much more complex tool that requires basic developer knowledge.
The body cloud services that support Appium, which can be taken seriously in terms of operation and price, are:
Calabash is managed by the following two systems:
Unfortunately, the Sauce Labs solution only works with emulators and simulators, so it dropped out pretty quickly, and AWS Device Farm lagged behind the Xamarin Test Cloud on the Calabash platform.
Since the end of March 2016, TestObject can also be used with Appium, which is just starting for it, it’s worth running a lap with as well, but it’s still more expensive than the Xamarin TC, they have far fewer devices and aren’t functionally sure they can do the same to provide the level.
AWS Device Farm experience
For a very long time, we were with him that Amazon’s solution would be the run-in, on the one hand, due to its pricing and on the other hand – honestly – because of the weight of the brand, but in the end, we had to be disappointed:
- Running tests on a larger device pool produce false errors on an average of 15-20% of devices (e.g. timeout, gesture problem, failed installations). By repeating the tests on the affected devices, there is a good chance that the test set will run without error. This unstable function also occurs sometimes with Xamarin, but only in 1-2% of cases.
- In Calabash, as part of the test script, you can also configure parameters for the runtime environment, specify what happens between two scenarios, and set up hooks or formatting. This is completely ignored by Amazon which is only willing to run with its support directory.
- In a body run, it can spin the counter on a device for 50 to 60 minutes for an error that should immediately interrupt the process with a runtime error – and since you have to pay on a per-minute basis (too), it can pretty much throw away your monthly bill.
- The UI is not very mature, user and group management are also missing.
Surprisingly, but not good value for money (currently) for this service.
Calabash and Xamarin Test Cloud in action
So far, it’s mostly been about why we didn’t choose the other options, and now for a few sentences about why Calabash and Xamarin are.
Advantages for users
Calabash is a library of functions that can handle native and hybrid apps on Android and iOS, so a “robot hand” that can reach from the console to the mobile and control it. It can be used to perform any interaction that is also used for manual testing.
You can also request information about the running device, such as type, name, resolution, software environment, and much more.
These methods are called we need to include it in steps in Ruby, from which we will build the scripts and then the full set of tests.
These .feature files, which are also in plain text format and contain specific steps, are then based on these steps defined in Ruby as a separate layer. This is essentially a script written in basic English (more precisely in Gherkin ) to tell us what we want to do, in the background the step files process it and turn our instructions into real commands.
(The following entry describes the structure and use of Calabash feature files in more detail.)
And what brings together and executes our Calyash-based Ruby and Gherkin stocks is Cucumber. It allows you to run tests on physical devices, emulators, and simulators and then reports on the results of the tests.
The biggest advantage of using the three tools together is that the participants in the testing process ( customers, managers, testers, and developers ) can work together on the same platform. The text feature files and the many outputs, which are the input side of the tests, offer a wide range of export options and the development of collaboration tools. One of these most promising developments is Cucumber Pro.
Running a test without any magic can be run locally on a test device connected to the machine. The video below shows such a run, with real tests, on a real iPhone:
Life on the device
Once we have tests running on a local machine, all we have to do is upload them to the Xamarin Test Cloud from the command line and run the test set on the compiled device pool. We get clear, showy reports on the results of the tests. A screenshot is taken of each test step, we also see the CPU and memory usage.
We can divide our projects and bodies into teams and series (quasi-branches) and give individual rights to the actors involved in the process.
With Calabash, we had quite a bit of difficulty in the beginning, and the learning time wasn’t necessarily fast either, but we think it was worth it – paired with the Xamarin Test Cloud, we got an extremely versatile test tool and our testing processes got better.
We will write about the technical structure of the tests and their effective cross-platform application in a later article.