Asynchronous Xcode UI Testing

As I continue to delve into Xcode UI tests, I’m starting to discover some of the lesser advertised benefits and drawbacks of the tool. One specific nuance that has caught my eye is how to deal with asynchronous Xcode UI testing. Xcode UI tests are asynchronous in that they simulate end user interaction with your application. End user interaction triggers animation (and in some cases waiting on remote services), and animation doesn’t happen instantaneously. There is a disconnect between the speed at which the lines of code in your tests can be executed, and whether the simulator (where the app under test is located and running) can keep up or not. In most cases, the simulator can’t keep up. As a result, you need to explicitly instruct your tests to wait for the app to catch up in the simulator. There are two ways to do this that I’ve been using.

Smart Waiting

waitForExpectationsWithTimeout(_:handler:)

When performing asynchronous Xcode UI testing, it’s often really useful to be able to verify something has appeared on screen. This way you know that the application is a certain state, and can continue executing a test script. For example, imagine an application with a tab bar, selecting a different tab should show a different view controller. If you were writing a test that relied on features within the second tab of the tab bar, it’s important have some assurances that when you instruct your test to tap the second tab, the tab actually appears. Furthermore, the act of waiting for something on screen can also act as a test in itself. If tapping the second tab doesn’t actually show what you expect, something probably went wrong and the test should fail.

Inspired from Joe Masilotti’s Cheat Sheet for Xcode UI testing, here’s how you can leverage waitForExpectationsWithTimeout(_:handler:) to verify that something appeared on screen:

// 1
let goLabel = self.app.staticTexts["Go!"]
let exists = NSPredicate(format: "exists == true")
expectationForPredicate(exists, evaluatedWithObject: goLabel, handler: nil)

// 2 
app.buttons["Ready, set..."].tap()

// 3
waitForExpectationsWithTimeout(5, handler: nil)
XCTAssert(goLabel.exists)

Here’s an explanation for this code:

1. Define a NSPredicate to be used in a XCTestExpectation to verify that the Go! button exists in the app.
2. Tap the Ready, set… button.
3. Wait for the Go! button to appear.

waitForExpectationsWithTimeout(_:handler:) will repeatedly look for the NSPredicate to be true within the timeout provided, in this case 5 seconds.

This approach to verifying that something exists on the screen is incredibly useful. It’s very much like my old favorite KIF API waitForViewWithAccessibilityLabel. The thing is, it isn’t perfect.

Not Perfect

I’ve observed a number of cases when using this approach has led to false positives that an element “exists” on the screen. Essentially the XCTest UI framework observes that the element exists, but isn’t actually tappable, or in a state that a human-end-user would consider “on the screen.” If seen this most often happen when this technique is used in combination with animations happening in conjunction with the test action. For example, if you have an application that taps a button, which triggers a navigation controller to push a new view controller onto the stack, there’s a relatively slow animation that happens. With the predicate approach laid out above, asynchronous Xcode UI testing will actually identify incoming screen elements as “existing” before they are in their “final resting spot” at the end of the animation. So if you gate further steps of your test on the presence of something on the screen as determined by the XCTest UI framework, you’ll get false positives that the element is on the screen and ready for further interaction. Essentially, Xcode UI tests will tell you, “Yes, that button exists on the screen,” but in reality it isn’t yet “on the screen.” Don’t worry though, there’s a solution, and it’s incredibly dumb.

Dumb Waiting

sleep(1)

In those cases when you need to a little extra pause in your asynchronous Xcode UI testing in response to an asynchronous event like an animation, I’ve found that just using a sleep(1) goes a long way. Essentially this will block your tests from continuing for the specified duration of time, while the application under test is allowed to continue. To update the example code above, I add the sleep(1) in conjunction with the NSPredicate technique as follows:

let goLabel = self.app.staticTexts["Go!"]
let exists = NSPredicate(format: "exists == true")
expectationForPredicate(exists, evaluatedWithObject: goLabel, handler: nil)

app.buttons["Ready, set..."].tap()

waitForExpectationsWithTimeout(5, handler: nil)
XCTAssert(goLabel.exists)
sleep(1)

Notice that the sleep(1) happens right after the NSPredicate is determined to be true. This follows my discovery that Xcode UI tests identify things as “existing” before they are actually on screen. Using the NSPredicate technique with the sleep(1) let’s you verify the presence of specify UI elements, while also giving the app time to complete long running tasks like animations before continuing.

What’s funnier is that this sample code shown on screen at WWDC 2016 caught my eye:

Turns out, Apple uses the same technique!

Assertions in Closures

Another useful technique in asynchronous Xcode UI testing is to perform assertions on user-interface elements in response to closure execution. Often, indication of completion of long running calls like web service retrievals will be implemented with closures. I’ve written Xcode UI tests that explicitly call the web service separate from the application in order to cross-check data returned. Basically, I’ll ensure that the data the raw web service returns matches what is in the application. Whether this is your use case or not, you can use this pattern to make assertions in the closure on callback completion. The only requirement is that Xcode UI API calls need to happen on the main thread:

// 1
let expectation = expectationWithDescription("API Request Complete")
var response: Response?

// 2
RequestUpManager().get(id: "3688840") {
  (response: Response?, error: NSError?) -> () in
  
  if let response = response {
    response = response

    // 3
    dispatch_async(dispatch_get_main_queue(),{
      // Verify there is something on screen that matches what the API provided.
      let cell = self.app.cells.elementBoundByIndex(0)
      XCTAssertEqual(cell.staticTexts[response!.position].identifier, "Position")
      expectation.fulfill()
     })
  }
  else {
    // 4
    XCTFail("Unexpected object returned from API")
    expectation.fulfill()
  }
}
// 5
waitForExpectationsWithTimeout(WAIT_TIMEOUT, handler: nil)

1. Use expectations to keep the test running while the request loads.
2. Synchronously load data from the API.
3. Tell the main thread to verify what’s on screen. (This will crash if not done on the main thread).
4. Explicitly fail the test if unwrapping the optional data returned from the API fails.
5. Using XCTestExpectations, wait for the asynchronous code to complete.

Wrap Up

These are just a couple little tricks I’ve learned for asynchronous Xcode UI testing when taking my first dive into the API. It’s always nice to get in and try out a tool beyond what’s shown in documentation or videos, you get a much better handle for nuances of what’s available. Have you come up with any solutions for these challenges in asynchronous Xcode UI testing?

Happy cleaning.