Testing

Hero image for Testing

Testing is an essential component of how the team develops and delivers robust software applications. Each time a new feature is added or a line of code is simply modified, developers write new tests or modify existing ones to validate the implementation.

These tests are then executed each time new code is pushed to the code repository. On top of that, product managers only accept a user story after going through a checklist of manual or automated end-to-end testing.

General Best Practices

  • Not over-engineering tests. In order to have maintainable test suites, tests need to be written as a series of simple procedures that can be read by a human. If a test fails, it should be fairly obvious why and fixed in a short amount of time. Many test frameworks come with their own complex domain syntax language (DSL) and convenience methods. But the team prefers focusing on the true essence of testing (arrange - act - assert) over complex and non-standard setup.

  • Each test must run in isolation. Flaky tests are often the offspring of tests over-sharing fixtures and setup. Instead, each test must set up its own fixtures and not inherit or rely on any previous states. While it might result in code repetition, it’s an acceptable trade-off for stable tests.

  • Continuously optimize tests suites:

    • Remove old or deprecated tests. As the software application evolves, some tests, while still passing, might not add much value to the overall suite as newer tests already perform the same validations. Don’t be afraid to remove tests.

    • Maintain or reduce the total time to run the whole test suite. Having long running test suites decreases productivity of developers and the effectiveness of tests. Having a short feedback loop is paramount.

Continuous Integration

Automate All The Things! – Anonymous meme character

  • Automate the execution of tests suites each time new code is pushed to the code repository. Developers get fast feedback on their current work thus can prepare the necessary changes if their branch breaks the application. In addition, code reviewers can only start reviewing and approve pull requests that pass the test suites.

  • Execute tests on a dedicated server i.e. developers should not have to run the tests suites on their machine. Running tests usually uses lots of computing resources and requires isolated environments. In addition, when working as a team, tests suites need to be broken down into several jobs and executed in parallel to have fast feedback.

For both Web and mobile applications, the team currently favors using Github Actions for Continuous Integration. For mobile applications, Bitrise and Jenkins CI are also often used. Read more details on the tools.

Testing Pyramid

As developers we need to ensure that all use cases of an application are working correctly. So, by writing a variety of tests we can verify the expected results. The image below illustrates how our projects should include different categories of tests, to maximize code stability:

Testing Pyramid

Its essential point is that we should have many more low-level tests than high-level tests running. As we work up the pyramid, from small tests to large tests, each test increases in fidelity, but also increases in execution time and effort to maintain and debug. However, to maintain the recommended balance between the different categories, we need to define the recommended proportion of tests for each category (which can vary depending on the platform). But in general, we should write more unit tests than integration tests, and more integration tests than UI tests:

  • Unit Tests: Small tests that validate individual, isolated components.

  • Integration Tests: Medium tests that validate interactions between multiple components.

  • UI Tests: Large tests that validate a user’s journey spanning over multiple components.

On a final note, it is important to know that if you get a failure in a high-level test, you also have a missing or incorrect low-level test. Thus it is advised that before fixing a bug exposed by a high-level test, you should replicate the bug with a low-level test as well.

Here’s a summary of our testing principles per platform: Android, iOS and Web