After writing Xcode UI tests for the first time today, immediately some things formed my initial impression. I wanted to pass along my Xcode UI tests review here on cleanswifter.com in effort to help educate others. For at least the pass year, if not longer, I’ve been using KIF for writing my functional UI tests. I love KIF. KIF is a third party, open source tool. That brings both benefits and drawbacks. Right off the bat, due to changes in the iOS 10/Xcode 8 Accessibility Inspector, KIF was broken in the beta versions of our tools. It was quickly fixed, and the impact was low, but it does make you wonder about what’s in store for the future when there are less people available to maintain it? This wasn’t the only thing that drove me towards Xcode UI tests for this project, but it was certainly a concern. I still love KIF and will continue to use it on other projects for the foreseeable future.
Then last year, in WWDC 2015, Apple announced a reimagined solution for writing UI tests, Xcode UI tests. Here’s my Xcode UI tests review.
Officially Supported by Apple
This shouldn’t be underplayed. I’d live through a lot of crappy documentation just to use an officially supported tool. Apple officially supports Xcode UI tests, and that means a lot to me. It means you get WWDC videos on the subject. You get subtle integration with Xcode that only Apple could have access to. It means that your tests are less likely to break in future versions of iOS and Xcode.
Tests Written In Swift
Unlike UI Automation tests, Xcode UI tests can be written in either Objective-C or Swift. This could be the single best improvement. This means that you get everything from code completion, to compile-time syntax checking, to full Xcode integration.
Integration with Xcode
Buliding on the integration with Xcode, Xcode UI tests subclass
XCTestCase just like unit tests. This means the full API of XCTest is available. Methods like
XCTestAssertTrue can be leveraged for writing your tests. Additionally, things like Scheme integration into your Test action enable execution of your Xcode UI tests when running tests, right from Xcode. Also, all those others niceties of running unit tests and the wonderful keyboard shortcuts can also be used with your Xcode UI tests.
UI Recording for Xcode UI tests is pretty cool. You can click a “Record” button in Xcode, and your app will launch in the simulator. Then, as you tap and navigate through your app, actual code will be generated in a test reflecting the path through your app that you took. It’s not perfect though, and more importantly, it doesn’t actually generate any assertions. Also, to prevent your test from being really brittle, you do need to consider breaking your tests up into cohesive chunks, and not one giant test.
Xcode UI tests aren’t without their flaws.
Documentation Is Still Light
The documentation for Xcode UI tests is still light. It’s not as bad as UI Automation was, but it’s still not perfect. KIF has been around for years. There are tons of stackoverflow.com questions and answers for it, the GitHub repository is mature, and you’ll find plenty of tutorials. I still can’t find an official piece of web-hosted API documentation for Xcode UI tests.
See the end of the article for a list of resources that I’ve been using as reference for Xcode UI testing.
Obviously all UI tests are going to be slow, regardless of whatever platform and tools you are using. Something uniquely slow jumps out at me about Xcode UI tests though. As a developer, you are required to call
XCUIApplication().launch() from your tests when you want the app to launch. Each test must call this. The common pattern from what I’ve seen is to place the call to
setUp() is an overridden method from the
XCTestCase base class, and is executed by the XCTest framework before each test. The thing is,
XCUIApplication().launch() is REALLY SLOW. It triggers a fresh launch of your app. Now I guess this is useful from the context of resetting state in your app, but if you aren’t careful about structuring your tests, you could inadvertently introduce a lot of overhead in the speed of your tests.
iOS9 and Up
One of the main advantages of a suite of automated tests, regardless of the platform or type of tests, is to be able to run those tests in all places that your code base is supported. If you are writing a web application, this might be different browsers at different screen resolutions and window sizes. In iOS development, you can exponential benefit from running the same suite of tests across different devices of different form factors and different iOS versions. Unfortunately, Xcode UI tests put a major hamstring in this, in that they will only run on devices with iOS 9 and greater. To me, this is a big deal. We don’t all have the luxury of requiring the minimum iOS version to be the latest on the market. At the time of this post, iOS 9 is the latest iOS version, so that means I can’t even run my automated UI test suite on one version back, iOS 8. There have been so many times in my testing career in which UI tests identify and/or verify a bug that is present on one iOS version and not another. I’m really disappointed that Xcode UI tests don’t work on anything older than iOS 9. I guess I just have to hope for my applications, we’ll soon be able to require iOS 9 and greater.
XCUIElement != UIView
One of the most awesome things about KIF is the overlap between the KIF APIs and UIKit. When writing KIF tests, there are many methods like:
func waitForViewWithAccessibilityLabel(label: String!) -> UIView!
It’s so useful to have a
UIView returned. This can be cast to more specialized subclasses that actually represent the object returned. And once you do that, you have full access to all the methods available on that object as defined in the actual code. Code like this is possible:
let saveButton = tester().waitForViewWithAccessibilityLabel("Save") as! UIButton
This code is really useful because once you have a handle on the
UIView, you can access so many APIs to verify behavior.
Unfortunately, there is no such intersection in Xcode UI tests. In Xcode UI tests, the object returned from similar method calls is a XCUIElement, and there’s no way to translate this to
UIView (that I’ve found). As such, you are limited to the limited API available through Xcode UI tests and
XCUIElement – none of which translates to a UIKit equivalent.
Despite the drawbacks, I’m still excited to use Xcode UI tests and I hope this Xcode UI tests review conveys that. It’s always fun to learn something new, and I think there are plenty of positives to using the tool. Have you used Xcode UI tests? What’s your Xcode UI tests review?