Mobile UI Testing with RSpec and Appium

When it comes to automated testing, it’s critical to devise a sensible testing strategy (i.e. what do we test and how we test it). That’s true for every kind of application and even more for mobile apps. To some extent, mobile development has lagged behind web for UI testing. Frameworks had limited capabilities and performance (slow tests), and as a consequence, testing was not as high a priority in mobile development as in other stacks.

Over the past few years, the situation has changed for the better. As both iOS and Android platforms matured, there has been more focus on application architecture (MVP, MVVM, Architecture Components) and testing. The latter has become a first-class citizen in mobile development. Still, setting up an efficient mobile testing is an arduous task.

First, the choice of testing frameworks can be overwhelming. Each platform comes with its own testing framework (JUnit for Android, XCTest and XCUITest for iOS). In addition to these “built-in” tools, there is a plethora of open source libraries designed to be better or faster. There are always new tools being released to the public, including many by tech giants such as Facebook or Google (EarlGrey). At the same time, other frameworks are getting discontinued such as Calaba.sh 😵

When it comes to testing, it’s critical to pick a framework that suits your workflow. What works for others might not for you as you don’t have the same requirements and needs. For us, one of the requirements is to standardise our tools across projects and stacks to minimise tech churns and leverage existing knowledge.

In addition, you should pick a framework that will get updated over time when new SDKs/OSs are released. In a well-tested codebase, tests are the only code without tests, so the framework should not get in the way of writing and maintaining tests.

Tools

After much time comparing the available options and evaluating our needs, we settled on using two tools: Appium and RSpec.

Appium is probably the most widely used automation tool. Its strength lies in the facts that it’s both cross-platform and testing framework agnostic. Appium can be used for not only native mobile application but also hybrid, web and desktop application. So pretty much any kind of applications out there. When it comes to writing tests, it‘s compatible with a long list of frameworks and languages. It’s the Swiss army knife of mobile UI testing 👷‍.

As for RSpec, it’s a suite of Ruby gems including everything to run tests: a test runner, an expectation framework and a mock framework. While RSpec is primarily used in the Ruby/Ruby on Rails community, it has been the source of inspiration for many testing frameworks in other languages such as Jasmine (JavaScript) and Goblin (Go). It has a neat API and provides powerful capabilities for writing tests. Being based on Ruby, it’s fully supported by Appium.

By picking Appium and RSpec, we are able to use the same tools to test both iOS and Android applications. At Nimble, as we often have to develop the same application for these two platforms at the same time, this is a win as test scenarios can be re-used between both projects. With frequent updates for Appium, there are also no issues in supporting new SDKs and OSs.

Setup

As mentioned previously, Appium is built to work with any testing framework. So the setup to make both tools work together is rather straightforward.

Environment Setup (your machine)

1) Install the latest version of the Appium either as a desktop applicationn or as an NPM package.

For iOS, open the Developer Settings Panel and add “ — native-instruments-lib” to Custom Server Flags

2) Check that you have all the required pre-requisites installed using appium-doctor. Install it using npm install -g appium-doctor, then run the appium-doctor command, supplying the --ios or --android flags to verify that all of the dependencies are set up correctly.

3) Make sure that you have a recent version of Ruby installed

➜  ~ ruby --version 
ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin17

In order to install Ruby, either follow the official documentation or use tools like RVM to install and manage multiple versions of Ruby.

4) Install the gem bundler

➜  ~ gem install bundler

This gem is required to install other gems defined in Gemfile files.

Project Setup (the application you are working on)

1) Add a Gemfile with the following content:

source "https://rubygems.org"

gem "appium_lib"
gem "rspec"

gem "fabrication"     # optional
gem "ffaker"          # optional
gem "rake"            # optional
gem "fastlane"        # optional

Only appium_lib and rspec are actually required. The other gems are what we use to make testing easier:

• fabrication and ffaker to write test fixtures

• rake and fastlane to create tasks to run tests

2) Add a directory spec at the root with the following structure:

└── spec
    ├── config.rb
    ├── spec_helper.rb
    └── support/

• The support directory host all test helpers required by the tests

• spec_helper.rb is used to load all the required files

  require ‘./spec/config.rb’ Dir[”./spec/support/*/.rb”].each {|file| require file }

• config.rb is platform-specific:

iOS:

require 'rubygems'
require 'appium_lib'
require 'fabrication'
require 'ffaker'
require 'yaml'

opts = {
  caps: {
    'deviceName' => ENV.fetch('DEVICE_NAME', 'iPhone SE'),
    'platformName' => 'iOS',
    'platformVersion' => ENV.fetch('DEVICE_VERSION', '11.2'),
    'app' => './derivedData/Build/Products/Debug-iphonesimulator/<REPLACE_WITH_APP_NAME>.app',
    'nativeWebTap' => true,
    'fullReset' => true,
    'automationName' => 'XCUITest',
    'autoAcceptAlerts' => true,
    'showXcodeLog' => false,
    'wdaLocalPort' => ENV.fetch('WDA_PORT', '8110').to_i
  },
  appium_lib: {
    wait_timeout: 30
  }
}

RSpec.configure do |config|
  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end

  config.before(:suite) do
    @implicit_wait_timeout = 15
    @driver = Appium::Driver.new(opts, true).start_driver
    @driver.manage.timeouts.implicit_wait = @implicit_wait_timeout
    Appium.promote_appium_methods Object
  end

  config.after(:all) do
    @driver.driver_quit
  end
end

Android:

require 'yaml'
require 'fabrication'
require 'faker'

RSpec.configure do |config|
  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end

  config.before(:suite) do
    puts 'Installing application...'
    adb -s $EMULATOR_ID install -r <REPLACE_WITH_PATH>/build/outputs/apk/developer_x86/debug/<REPLACE_WITH_APK_NAME>.apk > /dev/null
  end

  config.around(:each) do |example|
    puts 'Starting driver...'
    start_driver
    puts 'Waiting for splash screen...'
    wait_for(timeout: 15) { @driver.current_activity != @driver.caps[:appActivity] }
    example.run
    take_screenshot(example.metadata) if example.exception
    quit_driver
  end

  config.after(:suite) do
    adb -s $EMULATOR_ID uninstall com.<REPLACE_WITH_PACKAGE_NAME>
  end
end

Writing Tests

We won’t cover RSpec in details as the official documentation is comprehensive. But we will share our best practices and learnings 🤓

Isolate tests

Each test case should be isolated from each other (i.e. a test should not rely on another test). If this is respected, test cases can be executed in random order.

In practice, every time a test is executed, Appium uninstalls and re-installs the application for the next test. This is required to ensure that test cases are executed in a pristine application state and environment. But this process takes time (loads of it):

1. Start the simulator

2. Install the WebDriver

3. Install the Application

4. Navigate to the screen we want to test

5. Finally, execute a single scenario

6. Then repeat

So one could be tempted to run all their tests at once skipping steps 1. to 4.. But this exposes risks of having the state of other tests leaking to the current test 😓 Flaky tests are the worst nightmare when running tests suites. It makes the whole tests suite unstable and debugging almost impossible. In the end, it makes you hate your tests suite.

The isolation of UI tests does make the test suites much slower but at the same time, it makes the tests suite stable. It’s a worthy trade-off, but get ready to have long running tests suites 🐌 To give you an idea, unit tests take mere seconds to start and run, while a single UI test might take 20–40 seconds. As a result, it’s common for complete UI test suites to run for an hour or more.

Stable Test Environment

One could reduce the test environment to only the emulator in which the tests are executed. But it also includes the local network, the test drivers, the OS/library version, third-party applications installed on the emulator and so on. For instance, if you have a flaky internet connection on the machine running the tests, tests will fail for no apparent reason. So it’s critical to have an understanding of your test environment.

As for the emulators, we have found that the following actions help dramatically in making our test environment more stable:

• Always disable the software keyboard during tests. It also saves time when implementing scrolling.

• Use mock locations. Appium allows you to set up mocked geolocation for your test device by calling onSetLocation in the Appium Driver instance. This will help to reduce the time needed to request geolocation.

• Pre-install third-party applications in the emulator (e.g. email, phone or maps). Pre-installation is also useful when testing for permissions.

Debugging

RSpec provides useful debugging information when a test fails.

Screenshot

Always take a screenshot if the test fails. This is really useful when you are not getting what you expect.

# spec/config.rb
RSpec.configure do |config|

  ...

  config.after(:each) do |example|
    if example.exception
      meta = example.metadata
      filename = File.basename(meta[:file_path])
      line_number = meta[:line_number]
      screenshot_name = "#{Time.now.to_i}-#{filename}-#{line_number}.png"
      screenshot_path = "spec/screenshot/#{screenshot_name}"
      @driver.screenshot(screenshot_path)
    end
  end
end

Pry

Sometimes printing out the variables and statements is enough to debug failing specs (using puts). However, when the issue is a little bit more complex, use the ruby debugger pry. The debugger allows you to add breakpoints in your code so that you can check at runtime all the properties of elements, the driver state and/or try to find a better element selector.

Find View Elements Faster

When writing UI tests, we always need to access the view elements to either go through a scenario and/or verify the tests expectations.

There are several ways to access views elements (sorted by the performance from the fastest to the slowest):

1) Class name

UIDatePicker -> XCUIELementTypePickerWheel

While it’s the fastest selector, you need to make sure that the class names are unique. Otherwise, it could yield the wrong element.

2) Accessibility identifier (Recommended)

This is our go-to method to find view elements. However, it requires you to add string-based identifiers to each view element that will be used in UI tests. We recommend settling on nomenclature for consistency purposes e.g. sign_up_button , done_button , email_text_field . These strings will be used in the tests codebase so they need to be easy to understand.

@driver.find_element(:accessibility_id, 'sign_up_button')

3) XPath

XPath is the slowest as the driver needs to go through the whole tree of view elements to find elements with a given XPath. But sometimes it can’t be avoided. For instance, when using a third-party library or even the native components (e.g. UIAlertController), the accessibility identifiers might not be present at all. In these cases, it’s important to optimize the XPath by limiting the search scope.

Slow
//UITableViewCell[1]

Faster
//UITableView/UITableViewCell[1]

There are other methods which we don’t cover here as there are not used much: Link Text, Predicate and Class Chain.

Scroll like a Champ

The Appium scroll_to and scroll_to_exact helpers are not optimized especially for long scrolls. These helpers allow scrolling until an element is found which is a critical requirement in all mobile applications.

To deal with this issue, we customized the scrolling helpers to use swiping via:Appium::TouchAction.new.swipe. It works by performing some calculations to check if the current scrolled position is the end of the layout. While it’s a bit slower, it’s more accurate than the built-in helpers in both iOS and Android.

if upward
  Appium::TouchAction.new.swipe(start_x: x, start_y: y_start, offset_x: 0, offset_y: upward_offset_y, duration: 2000).perform

else
  Appium::TouchAction.new.swipe(start_x: x, start_y: y_end, offset_x: 0, offset_y: downward_offset_y, duration: 2000).perform
end

Speed Up Tests

As mentioned previously, UI test suites take loads of time. So we are always looking for ways to both make our tests run faster and get faster feedback:

• Avoid using sleep in your tests. The condition might be met before the timeout period but the test will wait for a fixed amount of time each time. A better way is to create a helper method to check for a condition e.g. waiting until a view element appears on the screen (RSpec has built-in support this).

  Note that this is different from other frameworks such as Espresso which requires changes in the source code to deal with idle application state.

• Group tests by criticality. Our workflow is to run only the critical tests on every commit but then run the complete tests suite before merging the branch.

context "Booking with invalid promo code" do

  it "shows an error message", focus: true do

    ...
  end

end

RSpec supports it out the box via the meta tag: focus .

• Optimize the test strategy in each environment. In the development environment (your local machine), use a “Fail Fast” strategy. When writing and executing tests, the suite will stop at the first error it encounters. This fast feedback allows you to know what is wrong without waiting for the whole test suite to complete. On the other hand, on the CI/CD server, you should let the whole test suite run until it’s fully completed. The test report would then show all the failing tests cases at once so that we can tackle all the issues at once.

• Run on multi-emulators in parallel if possible. Appium supports this out of the box.

Write Tests like Stories

One important aspect to makes tests great is that the code should be easy to read and understand. The Ruby language with its concise syntax does help a lot in that regard. But adequate naming and externalizing repeated code to helper methods also play a major role to write tests that read well.

A UI test is simply a series of steps that a user take to perform an action e.g. tap on the tab “Book”, select an option then enter the payment details. Some of these steps can be used in multiple tests cases, so it’s a good strategy to move this code to re-usable helper methods. At the same time, finding elements can generate some cryptic and repetitive code (XPath) that could be externalized to helper methods as well. As a result, writing a test is combining is adequately named helper methods together.

it 'shows the payment form'
  navigate_to_tab(:book_now)
  select_first_nearest_hotel
  navigate_from_hotel_list_to_date_selection_screen
  select_date(today, tomorrow) 
  navigate_from_date_selection_screen_to_payment_screen
end

The above test is easy to read and maintain. All the complexity to find elements or to perform a step has been externalized to small helper methods.

Conclusion

As we have seen, mobile UI testing is not simple. It requires lots of optimization and still runs slower than unit tests. But picking the tools that work for your workflow makes it easier. It did for us. It allows us not only to release mobile applications to production with confidence but also develop faster.

If you are not testing your mobile applications yet, you should! And why not giving a try to Appium and RSpec.

Thanks to Thuy and Byte for contributing with their best tips on writing mobile UI tests🤘