Xcode UI Tests Review

After writing Xcode UI tests for the first time today, immediately some things formed my initial impression. I wanted to pass along my Xcode UI tests review here on cleanswifter.com in effort to help educate others. For at least the pass year, if not longer, I’ve been using KIF for writing my functional UI tests. I love KIF. KIF is a third party, open source tool. That brings both benefits and drawbacks. Right off the bat, due to changes in the iOS 10/Xcode 8 Accessibility Inspector, KIF was broken in the beta versions of our tools. It was quickly fixed, and the impact was low, but it does make you wonder about what’s in store for the future when there are less people available to maintain it? This wasn’t the only thing that drove me towards Xcode UI tests for this project, but it was certainly a concern. I still love KIF and will continue to use it on other projects for the foreseeable future.

Background

Xcode UI tests build on an older tool from Apple called UI Automation. UI Automation enabled the iOS developer to write UI tests for their apps, and it was officially supported by Apple. “Officially supported” is certainly a loaded term, especially with UI Automation. The documentation was pretty bad. While there was some light guides on developing UI Automation tests, the actual API documentation had very little information. Combine that with the fact that the tests were written in JavaScript and run with Instruments (outside of Xcode), I was never happy with the tool.

Then last year, in WWDC 2015, Apple announced a reimagined solution for writing UI tests, Xcode UI tests. Here’s my Xcode UI tests review.

Pros

Officially Supported by Apple

This shouldn’t be underplayed. I’d live through a lot of crappy documentation just to use an officially supported tool. Apple officially supports Xcode UI tests, and that means a lot to me. It means you get WWDC videos on the subject. You get subtle integration with Xcode that only Apple could have access to. It means that your tests are less likely to break in future versions of iOS and Xcode.

Tests Written In Swift

Unlike UI Automation tests, Xcode UI tests can be written in either Objective-C or Swift. This could be the single best improvement. This means that you get everything from code completion, to compile-time syntax checking, to full Xcode integration.

Integration with Xcode

It was really annoying with UI Automation that you needed to switch to Instruments just to run your UI tests. Furthermore, JavaScript support in Xcode is pretty bad too, so one was left searching for a good editor experience when writing tests. Xcode UI tests change this. Now, the tests are first-class citizens right in the same IDE that we are using to write our actual application code. There’s even a “New File template” for Xcode UI tests.

XCTestCase subclasses

Buliding on the integration with Xcode, Xcode UI tests subclass XCTestCase just like unit tests. This means the full API of XCTest is available. Methods like XCTestAssertEqual and XCTestAssertTrue can be leveraged for writing your tests. Additionally, things like Scheme integration into your Test action enable execution of your Xcode UI tests when running tests, right from Xcode. Also, all those others niceties of running unit tests and the wonderful keyboard shortcuts can also be used with your Xcode UI tests.

UI Recording

UI Recording for Xcode UI tests is pretty cool. You can click a “Record” button in Xcode, and your app will launch in the simulator. Then, as you tap and navigate through your app, actual code will be generated in a test reflecting the path through your app that you took. It’s not perfect though, and more importantly, it doesn’t actually generate any assertions. Also, to prevent your test from being really brittle, you do need to consider breaking your tests up into cohesive chunks, and not one giant test.

Cons

Xcode UI tests aren’t without their flaws.

Documentation Is Still Light

The documentation for Xcode UI tests is still light. It’s not as bad as UI Automation was, but it’s still not perfect. KIF has been around for years. There are tons of stackoverflow.com questions and answers for it, the GitHub repository is mature, and you’ll find plenty of tutorials. I still can’t find an official piece of web-hosted API documentation for Xcode UI tests.

See the end of the article for a list of resources that I’ve been using as reference for Xcode UI testing.

Slow

Obviously all UI tests are going to be slow, regardless of whatever platform and tools you are using. Something uniquely slow jumps out at me about Xcode UI tests though. As a developer, you are required to call XCUIApplication().launch() from your tests when you want the app to launch. Each test must call this. The common pattern from what I’ve seen is to place the call to XCUIApplication().launch() in setUp(). setUp() is an overridden method from the XCTestCase base class, and is executed by the XCTest framework before each test. The thing is, XCUIApplication().launch() is REALLY SLOW. It triggers a fresh launch of your app. Now I guess this is useful from the context of resetting state in your app, but if you aren’t careful about structuring your tests, you could inadvertently introduce a lot of overhead in the speed of your tests.

iOS9 and Up

One of the main advantages of a suite of automated tests, regardless of the platform or type of tests, is to be able to run those tests in all places that your code base is supported. If you are writing a web application, this might be different browsers at different screen resolutions and window sizes. In iOS development, you can exponential benefit from running the same suite of tests across different devices of different form factors and different iOS versions. Unfortunately, Xcode UI tests put a major hamstring in this, in that they will only run on devices with iOS 9 and greater. To me, this is a big deal. We don’t all have the luxury of requiring the minimum iOS version to be the latest on the market. At the time of this post, iOS 9 is the latest iOS version, so that means I can’t even run my automated UI test suite on one version back, iOS 8. There have been so many times in my testing career in which UI tests identify and/or verify a bug that is present on one iOS version and not another. I’m really disappointed that Xcode UI tests don’t work on anything older than iOS 9. I guess I just have to hope for my applications, we’ll soon be able to require iOS 9 and greater.

XCUIElement != UIView

One of the most awesome things about KIF is the overlap between the KIF APIs and UIKit. When writing KIF tests, there are many methods like:

func waitForViewWithAccessibilityLabel(label: String!) -> UIView!

It’s so useful to have a UIView returned. This can be cast to more specialized subclasses that actually represent the object returned. And once you do that, you have full access to all the methods available on that object as defined in the actual code. Code like this is possible:

let saveButton = tester().waitForViewWithAccessibilityLabel("Save") as! UIButton
XCTAssertEqual(UIColor.blue(), saveButton.currentTitleColor)

This code is really useful because once you have a handle on the UIView, you can access so many APIs to verify behavior.

Unfortunately, there is no such intersection in Xcode UI tests. In Xcode UI tests, the object returned from similar method calls is a XCUIElement, and there’s no way to translate this to UIView (that I’ve found). As such, you are limited to the limited API available through Xcode UI tests and XCUIElement – none of which translates to a UIKit equivalent.

Final Thoughts

Despite the drawbacks, I’m still excited to use Xcode UI tests and I hope this Xcode UI tests review conveys that. It’s always fun to learn something new, and I think there are plenty of positives to using the tool. Have you used Xcode UI tests? What’s your Xcode UI tests review?

Happy cleaning.

Useful Resources

• “UI Testing in Xcode” from WWDC 2015
• XCTest API Reference
• Xcode UI Testing Cheat Sheet and Examples
• UI Testing in Xcode 7, Part 1: UI Testing Gotchas