About two weeks ago, Apple released the first beta of Xcode 7 which comes with shiny new features like built-in UI testing. Apart from Swift, this was easily the feature that excited me the most. I couldn’t wait to try it out. Not that UI testing is anything new; KIF has been around since 2011. It’s a great tool but it has its limitations: it’s slow, it’s not 100% reliable and it depends on private API. Therefore, an official solution seemed like a good idea.
This solution is integrated into XCTest, Apple’s unit testing framework. Using it is very similar to using KIF. However, it’s built on top of a new version of UIAutomation, which is more stable and reliable. Because it’s built right into Xcode, powerful features like UI recording and automatic screenshots make it even more useful and easier to use.
In the following sections I talk about the biggest differences between the two frameworks.
So I went ahead and began to rewrite the UI tests for ImagePickerSheetController, one of my open source projects. It’s essentially the custom action sheet in iMessage on iOS to select an image.
The first test that I wanted to rewrite was a very basic one. I simply test whether the controller is dismissed when tapping the cancel button.
While this looks fairly simple, it’s not possible to achieve using XCTest. The problem is that Xcode’s UI testing does not allow access to the actual app. The API it offers really only acts as a proxy. Trying to retrieve a view directly instead of using XCUIElement, which really only represents that view, does not work.
In the Developer Tools Labs at WWDC I was told that the primary goal of XCTest is to make apps more testable. That makes sense, I thought, and started rewriting the tests of one of my apps called Crimson.
After a couple of basic tests I started to work on the ‘accent view’. It shows the different accents of a certain letter. What I wanted to test is the following two things:
First I thought that it’s not possible to write a test for this. As before, you can’t just investigate the actual view to check the properties. I looked through the docs and after a couple of minutes I stumbled across this useful property of XCUIElement called value:
What it does is return the accessibilityValue of the underlying UI element. This makes it possible to give the test access to one property. So the only thing I need to do is to set the accessibilityValue in the accent view:
This works well until I wanted to check the color. Since the accessibilityValue is of type String and not Dictionary, it is not really possible to get access to a second property. Apart from that, accessibilityValue is, compared to accessibilityIdentifier, user facing. Setting the accessibilityValue to obscure values like a color is extremely bad practice and defeats the purpose entirely.
Writing this second test is therefore not possible either. While it’s easy to check the frame or label of an element, it’s nearly impossible to test anything else.
To benchmark the performance of the two frameworks I wrote an example project with 3 tests. In order to simulate a reasonable test suite, every test was performed 20 times, which makes 60 tests in total.
For the first measurement, I implemented the tests as show in the docs. For XCTest, this means that I would relaunch the app after every test. KIF does not support this, which makes it as fast for the first two benchmarks.
The second measurement resets the state of the target app by navigating back instead of relaunching.
For the third I additionally disabled all animations.
*Because Xcode 7 was constantly crashing, I had to decrease the number of iterations to 5 times and then multiplied it by 4.
As we can see in the first benchmark, relaunching the app for every spec makes the test suite extremely slow. It’s surprising that the UI test template suggests this way of resetting rather than by navigation since this is not necessary in most cases.
The second benchmark shows that in normal circumstances, XCTest is a little faster than KIF. Note that 10 seconds could be a huge time saver when working with a rather large test suite that needs to be evaluated multiple times a day.
The last benchmark shows where XCTest shines. Nevertheless, it’s not an option for every project to disable all the animations. Think of a project with many custom transitions. The test suite should verify that the transitions work as expected.
What workaround to this is to separate the tests into classes with different launch environments to control the app’s configuration.
The AppDelegate then sets the configuration according to the launch environment.
This can decrease the time you’re waiting for your tests to pass drastically.
XCTest makes it easy to write simple UI tests. More advanced tests are difficult or just not possible. KIF is a lot more flexible.
Compared to KIF, however, XCTest is slightly faster at finding the UI elements. With animations disabled, XCTest is really fast.
Although I looked forward to UI tests integrated into Xcode I gave up on rewriting them for my projects. I simply couldn’t find a way to write all the tests.
