Mobile E2E tests are notoriously hard to run, especially if you run them with the app as a black box. Before testing, you need an app build, a real/virtual device and a driver that executes the tests. Nowadays, you have drivers that use LLMs to interact with the state of the screen and make it easy to run/maintain tests. But I think the problem is more fundamental - you still have to run the whole app on a device and interact with it.

// An example mobile end-to-end test with Appium 
test("View product detail screen", async ({ phone }) => {
	await phone.deviceUtils.startTheRNAppWithTokens();
	await phone.deviceUtils.followDeepLink(product.deeplink!);  // view PDP
	await addItemToCartFromPDPMobile({ phone });                // add to cart
}

Instead of trying to solve all the known problems with E2E tests, can we use a different type of tests which are cheaper and faster to run and still verify all the cases covered by an E2E test? I set to explore this over the weekend.

Small, medium and large tests #

A few months ago, while reading about testing at scale, I came across the great Software Engineering at Google book, especially the section Designing a test suite. Here's a short overview of what the book suggests:

There are majorly three types of tests: small, medium and large. These are categorised not by the scope of what they verify, but by how complex they are to run.

"A test’s size is determined not by its number of lines of code, but by how it runs, what it is allowed to do, and how many resources it consumes."

The book suggests to write more of these small tests which run using one process and do not use network or other I/O dependencies. And it is so true in practice! In the context of TypeScript applications, tests written with Jest/Vitest are the most stable and fast-running ones as compared to setups like Playwright, which run with more than one processes.

Looking at the typical E2Es in mobile, they would be categorised as large tests - they run with more than one process (the test driver, the emulator/device, the app itself, if real network is used then upstream dependencies) - it's like a cocktail of all the things that could go wrong while testing.

And the problem with large tests is that you cannot run them frequently on every change quickly as you'd have to build the app, set up the driver, execute the flow. It is hard to make all of this fast and reliable, which is why we often see E2E test suites going unmaintained and solutions like agentic testing coming through.

Agents need a quick way to verify that things work, and their changes don't break other features, a.k.a. a short feedback loop.

Large tests to small tests #

There's one more kind of test that potentially covers similar ground as E2Es, often called integration tests. These verify that changes in different parts of the system do not break some core features of the product while integrating these changes into the product.

The idea we need to double down on, in this age of LLM-written software, is to convert the scope of an end-to-end test to run as a small-sized test. How can you run tests that verify what you intend to verify with E2E and integration tests without having to run all the machinery needed for them? One way to do this is make the tests headless. Shopify recently shared how they are speeding up agentic development and verification by running headless, UI-less end-to-end tests by driving the logic directly that controls the UI. And this exactly fits the rule: make large tests smaller.

I took this up for Zalando's apps. We have a good number of E2E tests written to be run with Appium, a black-box testing framework. These ~100 tests take somewhere between 20-30 minutes to run across 10 parallel devices - good for release testing, but not very nice for quick feedback.

Looking at the example again:

test("View product detail screen", async ({ phone }) => {
  await phone.deviceUtils.startTheRNAppWithTokens();
  await phone.deviceUtils.followDeepLink(product.deeplink!);  // view PDP
  await addItemToCartFromPDPMobile({ phone });                // add to cart
}

Each step here either starts a real app, interacts with it through tools like ADB or navigates through real screens. To write a small test for this scenario, we'd use a shallow, headless testing approach that does not render the entire screen but rather only a virtual one. In this case @testing-library/react-native is perfect for implementing shallow E2Es, a.k.a. integration tests.

it("view product details screen", async () => {
  const traitStore = pdpTraitStore();
  await openPdp(traitStore).open("zalando://PDS?sku=ABC12A00B-K11");
  click(screen.getByLabelText("mobile.app.wardrobe.product.add_to_bag.content_description", { hidden: true }));
  expect(traitStore.cart.addToCart).not.toHaveBeenCalled();
  await waitFor(() => expect(traitStore.productSizePickerControl.openSizePicker).toHaveBeenCalled());
})

I tasked Copilot to port 10 of the E2E tests to @testing-library/react-native (RNTL) by simulating the app setup flows and providing mocked dependencies. Well, it took some time to perfect the test harness it created to simulate close-to-real setup while still limiting to just RNTL.

Test Suites: 10 passed, 10 total
Tests:       39 passed, 39 total
Snapshots:   8 passed, 8 total
Time:        12.427 s, estimated 13 s

How does it compare to the original E2E tests?

The idea is not to compare them to choose one, but rather to choose the right one for every point in the development lifecycle:

1. During active development, use tests that run as fast as possible while testing fairly widely. This is achieved by adding more of small-sized tests to cover various scenarios and running them often, i.e. per commit or step in the harness/agent loop.
2. When it is time to ship changes to a stage which expects stability (I intentionally avoid saying production because there may be multiple stages before where this is expected), run the wider, more expensive tests.

This is almost what the testing trophy describes, as coined by Kent C. Dodds in 2021 and I'm just sad/amazed that we (or, I) keep re-discovering this every year.

I would rephrase this model in terms of the concept of test size:

Have more of small-sized tests, make large tests small as much as possible, and keep a tiny set of important large tests.

Comparing coverage #

I know that the integration tests written in RNTL will not inherently provide coverage for what the larger E2E tests do. But can we measure the extent of how far RNTL tests take us? Can we figure out how much we are missing on by not running the heavier E2Es?

Yes. There is a way to know how good your tests are and it is called mutation testing. As the name suggests, in this technique, you tests your tests by mutating parts of the system-under-test and check if these mutants survive the tests. If they do, your tests miss coverage in parts of the source code which was mutated!

Mutating with stryker #

For the JS ecosystem, stryker is a mutation testing tool which provides a CLI and bunch of mutators that change your source code in different ways. A simple example is:

if(Date.now() > window.end) { drop() }
// mutated to:
if(Date.now() <= window.end) { drop() }

The hypothesis is that if your tests are good and cover all such scenarios, they should "kill" all the mutants produced by stryker. I selected 3 flows - a product catalog screen (catalog), a product detail screen (pdp) and an outfit detail screen (outfit) and ran a bunch of mutation rounds (it takes some time to run these as a set of files are modified) and the results were pretty bad overall for coverage:

I also kind of agree with these results as the tests are quite high level and may miss out on some granular changes in the code which may not reflect on the overall screen rendering. A "good" thing though is that the scores for both kinds of tests are almost the same, meaning the RNTL tests are already proving to be similar to E2Es in terms of coverage.

A problem with stryker though is that by default it makes changes randomly in your source code. There are ways to direct it to change specific parts, but I did not explore this yet. An easier way was to ask Copilot to detect areas in code that may be related to what the tests cover and try to mutate them.

Mutating with an agent #

In this round, Copilot identified 11 mutants and ran the same process manually without stryker (i.e. test -> record -> mutate -> test -> compare). Results:

Score: RNTL 9/11, E2E 3/11
RNTL tests did well in this round, probably because they were testing specific things on the screen and the E2Es tried to make the tests work by using alternative journeys. Some example mutants and how the tests reacted to them:

A surprising discovery for me was seeing the mutant Added-to-cart notification is never shown. In this case, the popup with a link saying "Go to cart" never appears, which should be a valid regression. But the E2E reacts to this missing popup by using the bottom tab navigation icons to go to the cart screen which passes the test. It's interesting because this is similar to human behaviour. But I strongly feel it was a fallback someone implemented for the test to "work" so that in cases where the popup is not clickable, the test uses a different way to reach the cart - thus ignoring the regression.

What these exercises highlight:

1. RNTL integration tests cover similar product features as E2E tests, and in case a regression was introduced they would successfully catch it. That said, they cannot detect an issue caused by an underlying dependency e.g. native or data API layers. This highlights a key fact: integration tests can replace E2E tests when the chance of a regression due to code change is high.
2. It is extremely cheap to extend the RNTL tests as compared to E2Es run with Appium. They detect issues faster - almost 5-10 times faster than E2Es.

New possibilities #

Being able to simulate the entire app creation and screen loading flow in a shallow format like RNTL invites new domains that do not need a full device for testing. A great example where we've often struggled is tracking events. In a distributed setup, ensuring that screens emit the right tracking data across hundreds of modules they load is important. We've had many incidents in the past where this contract broke and led to crucial data being missing which eventually led to impact on ad allocation and spending.

With RNTL integration tests, verifying tracking data is extremely easy. A new harness wraps runtime code and intercepts calls to APIs that emit tracking events and a test can then verify if an interaction leads to the right schema for tracking being used in the emitted data.

it("emits only schema-valid events", async () => {
	await openPdp().open(PDP);
	click(
	  within(screen.getByLabelText("Product image gallery")).getByLabelText(
		"mobile.app.add.to.wishlist",
		{ hidden: true },
	  ),
	);
	
	const events = trackedEvents.all().filter((event) => event.kind === "ga4");
	expect(events.length).toBeGreaterThan(0);
	for (const event of events) {
	  expect({
		event: event.name,
		violations: schemaViolations(event),
	  }).toEqual({
		event: event.name,
		violations: [],
	  });
	}
});

Next steps from here #

I wrote a testing strategy a month ago for my org at Zalando and one of key points there is use more of small tests and make large tests small and I am going to double down on this. This exercise is part of a larger idea to make testing mobile experiences fast, deterministic and easy to maintain with agents. As agentic engineering and the concept of software factories becomes more commonplace, testing and verifying software works is going to be the most challenging and interesting puzzle to solve!