Testing

Testing is an essential component of how the team develops and delivers robust software applications. Each time a new feature is added or a line of code is simply modified, developers write new tests or modify existing ones to validate the implementation.

These tests are then executed each time new code is pushed to the code repository. On top of that, product managers only accept a user story after going through a checklist of manual or automated end-to-end testing.

General Best Practices

• Not over-engineering tests. To have maintainable test suites, tests need to be written as a series of simple procedures that a human can read. If a test fails, it should be pretty obvious why and fixed in a short amount of time. Many test frameworks come with complex domain syntax language (DSL) and convenience methods. But the team prefers focusing on the true essence of testing (arrange - act - assert) over complex and non-standard setups.

• Each test must run in isolation. Flaky tests are often the offspring of tests over-sharing fixtures and setup. Instead, each test must set up its own fixtures and not inherit or rely on any previous states. While it might result in code repetition, it’s an acceptable trade-off for stable tests.

• Continuously optimize tests suites:

  • Remove old or deprecated tests. Don’t be afraid to remove tests. As the software application evolves, some tests, while still passing, might add little value to the overall suite as newer tests already perform the same validations.

  • Maintain or reduce the total time to run the whole test suite. Having long-running test suites decreases the productivity of developers and the effectiveness of tests. Having a short feedback loop is paramount.

Continuous Integration

Automate All The Things! – Anonymous meme character

• Automate the execution of test suites each time new code is pushed to the code repository. Developers get fast feedback on their current work, and thus can prepare the necessary changes if their branch breaks the application. In addition, code reviewers can only start reviewing and approving pull requests that pass the test suites.

• Execute tests on a dedicated server i.e. developers should not have to run the tests suites on their machine. Running tests usually uses lots of computing resources and requires isolated environments. In addition, when working as a team, tests suites need to be broken down into several jobs and executed in parallel to have fast feedback.

For both Web and mobile applications, the team currently favors using GitHub Actions for Continuous Integration. For mobile applications, Bitrise and Jenkins CI are also often used. Read more details on the tools.

Testing Pyramid

Developers need to ensure that all use cases of an application are working correctly. So, by writing various tests, they can verify the expected results. The image below illustrates how our projects should include different categories of tests, to maximize code stability:

Its essential point is that developers should have many more low-level tests than high-level tests running. As developers work up the pyramid, from small tests to large tests, each test increases in fidelity and execution time, and effort to maintain and debug. However, to maintain the recommended balance between the different categories, developers need to define the recommended proportion of tests for each category (which can vary depending on the platform). But in general, the team should write more unit tests than integration tests, and more integration tests than UI tests:

• Unit Tests: Small tests to validate individual, isolated components.

• Integration Tests: Medium tests to validate interactions between multiple components.

• UI Tests: Large tests to validate a user’s journey spanning over multiple components.

On a final note, it is essential to know that getting a failure of a high-level test usually means that there is a missing or incorrect low-level test. Thus, it is advised that before fixing a bug exposed by a high-level test, developers should replicate the bug with a low-level test as well.

Here’s a summary of the testing principles per platform: Android, iOS, and Web