commentary Archives - Page 2 of 3

Creating Effective iOS New User Experiences

I wrote a post for the AWeber Engineering Blog that I also wanted to pass on to you via the cleanswifter.com. The post is titled “Creating Effective iOS New User Experiences” and it’s all about what it took to build the first user experience in our latest app from AWeber, Curate. You can find the post here. I don’t want to spoil anything in the article, go take a look.

Behind The App

That post is also part of a series of blog posts we’re working on called “Behind the App.” As part of the mobile team at AWeber, we’re trying to tell some interesting parts of the story behind building Curate. You can find all the posts here.

Here’s an awesome video that tells the story behind Curate as well.

I have something really exciting coming next week, stay tuned.

An iOS Continuous Integration Machine Is Not A Pet

An iOS continuous integration machine should not be a pet. Do you know what I mean by a “pet?” Have you heard the analogy for determining whether a server is a pet or cattle? Continuous integration machines should be treated like cattle, not pets.

Without trying to sound cruel or inhumane, when cattle get sick, you can quickly replace it. When pets get sick, you spend a lot of time and money to try and help them get better. Now apply this to your continuous integration machine. If you have a dusty old Mac Mini sitting next to you that is acting as your continuous integration machine, eventually the hard drive is going to die or any other myriad of problems. Do you want to be in the business of resuscitating the machine when that happens? Not me, I’d much rather be focusing on what matters, writing awesome new features for my end users.

CI Should Not Get In Your Way

Imagine this scenario: you’ve been coding all night, and finally you get that hot new feature done, tests pass locally, and you’re ready to open a pull request. You push your branch, and for some reason Wolfie, your build server, isn’t powered on and won’t start up. Now your choice is to either spend time trying to fix Wolfie, or skip your whole iOS continuous integration workflow. What would you do?

Regardless of the choice that you made in that scenario, the point is, your CI server should be waiting to serve your needs, perfectly, every time. And if you are relying on some old piece of hardware lying around your home or office or home office, chances are, you’re going to spend more time caring and feeding for it, that you would if you had managed it as a disposable resource that can easily be replaced.

Don’t Waste Your Time

As tempting as it is to tryout Xcode Server on that old Mac you have lying in your garage collecting dust and rusting, I propose that you don’t waste your time keeping old Wolfie the build server around or even introducing him in the first place, and instead consider approaches where you aren’t in the business of maintaining your own hardware or an isolated build software stack. There’s two paths you can take towards this: all-in-one hosted CI solutions, or configuring an automated deployment of your CI environment to use something like Mac Mini colocation where you can spin up Mac Minis on demand to build your application. Here’s an overview how DayOne used macminicolo.net in conjunction with Jenkins for their builds. Want to run your tests on actual devices? Well then you could even write a script to automatically provision devices from Amazon Device Farm and run your tests on them. This approach grants you the most granularity and flexibility in your configuration, but takes a lot more effort to develop. You need to write the scripts that do the server provisioning, so that they are available and dependable in a repeatable fashion.

Sound like too much work? Alternatively, hosted iOS continuous integration solutions have been coming onto the scene lately and look very promising in that they advertise simplistic wizard-based setup of your CI jobs. Basically, login with your GitHub credentials (or Bitbucket), select your project, and click Build. Tomorrow, I’m going to give you an overview of buddybuild which is one of the hottest players on the market now in this area.

In the meantime, happy cleaning.

Continuous Integration Workflow For iOS

Cheaper Than A Dollar A Day

I came across this article today, called “Continuous Integration on a Dollar a Day” since Jon Reid of qualitycoding.org mentioned it on his Facebook page. I first want to recognize that this article was originally written back in 2006, so no need to repeat the obvious in case you discover this fact yourself. A lot has changed in 10 years, and a lot remains the same. For one, I absolutely recommend using continuous integration. The author also clarifies that the article is intended for people new to continuous integration, so some of the principles are introductory. While I recognize that, I think there are better introductory solutions for your continuous integration workflow For iOS.

To me, if you were to ask me on the street, off the top of my head, continuous integration ensures that: – Code that’s committed to your repository always compiles – Code always has tests passing

Having lived with it for so long, it’s not fathomable to consider anything else. To think that I could potentially check out what’s in the source code repository and it not compile is unthinkable. If clean code is your desired end state, then continuous integration can help you achieve it.

A Proposed Continuous Integration Workflow For iOS

I am in love with Github pull requests. A past version of me scoffed at thinking that my code must go through code review before it could be considered “done” or merged. I remember using Subversion back in 2010-ish and committing directly to the main trunk of code, with no one review it, and no automated builds. YIKES! Today though, I feel that a combination of test driven development, code review, and continuous integration truly remove the need for dedicated testers on the product team. That’s not to see we don’t manually test, I’m just saying that it removes the need for something dedicated in that role.

I don’t want to git into (pun intended) the pros and cons of other version control systems, I know there are plenty. I use GitHub for all my projects, and I love it. I think you’ll love it too.

To sum up pull requests, you work locally in a branch of code. When you’re ready to share it with others, you “push” the branch to the remote repository, and open a “pull request.” An open pull request is basically a proposed change set. Other team members can review the changes, chat about them, suggest improvements. Ultimately, the proposed change set can be accepted, and then is merged into the destination branch where you intended the code to end up.

If you want more detail, this is a great writeup directly from GitHub describing the process of using pull requests in your workflow to enable pre-merge code review.

So where does continuous integration fit into this? I propose that anytime code is about to change somewhere, continuous integration jobs should verify that the incoming change will compile AND pass tests. The thing is, as you move up the iOS testing pyramid, tests will run slower and slower. Your wait time can really grow if these gate parts of your development.

Setting Up Your Jobs

Continuous integration for iOS projects is a delicate balance between enough test coverage across OS versions and devices, running all your tests, and the putting the checks in just the right places. Here’s how I recommend striking this balance:

Run your full suite of tests on one device/OS combination each time code is pushed to the repo – I prefer to do this on the lowest iOS version that the app supports. I also usually do this on an iPhone for my Universal apps (no real reason, and I’ve gotten bit by iPad only bugs, so probably worth mentioning that some reasonable amount of thought is worth it here for your situation).
Run your full suite again on a different device/OS combination each time code merges on the result of the merge.
Each night, run the full suite of tests on every possible combination of devices and OS that you support that is possible within your means.

Let’s play this over the GitHub pull request model:

I push a branch to the repository and open a pull request.
CI server detects the push, runs all tests (unit, UI, and snapshot) on an iOS8 iPhone 5.
CI build either fails or passes.
CI machine puts a flag in the pull request based on the result of the build. If the build does not pass, it should not be merged.
If build is successful, once code merges, a new build is triggered on the destination branch using the result of the merge to run the full suite of tests. If this build fails, manual intervention is necessary to understand why, and be addressed.
Starting around 11pm, a sequence of dependent jobs kick off running the full suite of tests across every device that I own – iPhones, iPads, iOS7 (if supported), iOS8, iOS9, etc.
Results from “nightlies” are reviewed in the morning.

Anytime a build fails, it should be immediately addressed. Failing builds tend to build up over time, especially with the nightlies. If a nightly fails, make fixing it the first thing you do in the morning when you arrive. It usually represents a bug in your code.

Room For Improvement

There is definitely some manual intervention needed in this workflow, it’s not full proof. Ideally, continuous integration works along side you and prevents you from doing anything stupid. Removing any manual steps from your workflow is paramount, and something I’m still working on. I’d love to hear from you, have you tried a continuous integration workflow for iOS? How has it worked out? Or if not, why not?

Happy cleaning.

In Defense of Testing the iOS Service Layer

Following up from my presentation at CocoaHeads last week, I heard some feedback on my proposed iOS Testing Pyramid. Specifically, Andres Catalan challenged my removal of the service layer. He told me that he felt like I underplayed its importance. Removing it as something worth of dedicated testing meant that you’re missing out at a key way to add quality to your application. I think he made a good point, and I wanted to acknowledge it here.

The Service Layer’s API is Important

Tools like CocoaPods and Carthage (and soon the Swift Package Manager) make it so easy to include third party code with your apps. I’ve taken advantage of these to great result, they’ve certainly brought so much to my apps. Despite whether you use a dedicated package manager, or manually add the code to your project, the thing is, you are adding someone else’s code to your project. This code is subject to change, and certainly contains bugs (regardless of whether you’ve found them or not).

testing the iOS service layer

The question is, should you, as a consumer of the third party code, write tests for this code that you’re consuming? Often, this code that you’re consuming “as a service” connects with crucial parts of your application. Personally, I’ve gotten great use of SSKeychain as a wrapper on the keychain. The keychain stores secure data, so it’s hard to think of something more critical than that! Making sure that this third part service works as advertised is of fundamental importance to my app working, AND the security of my users. So the question presents itself, how should consumption of code like this be tested?

Testing the Service Layer

It’s no secret from my prior article on the iOS Testing Pyramid that I don’t actually advocate testing the Service Layer. If I did HAVE to test it, here’s what I would consider:

What should be tested?
When should it be tested?
Who should be testing it?

What Should Be tested?

If you, the consumer, do decide to test the service layer, there’s two things I advocate when considering “what should be tested?” Only the API that you are using should be tested, and only test from the outside-in, even if you have the source. For example, using SSKeychain, there are several different methods available for storing and retrieving data in the keychain. I only ended up using a subset of those in my application. If I were to write dedicated for my use of this “service layer” then I wouldn’t write tests for anything beyond the methods that I was actually using. In other words, you don’t need to write comprehensive tests for every publicly available API method for the third party library. And furthermore, when writing tests, consider your perspective as a consumer. Often, dependency tools like CocoaPods include dependencies from open source projects where the full source code is available. Despite having access to the full source code, DON’T TEST IT! Stick with the public APIs.

When Should It Be Tested?

I like to main strict manual control of the third party code that I integrate with my applications. As a result, I specify exact versions of CocoaPods and Carthage dependencies in my code, while also checking in the third party code into my repositories. Don’t forget how one programmer broke the internet by deleting a piece of code from his NPM package. I absolutely advocate maintaining a local copy of the third party code that you use, so that if it disappears from the public repository, you aren’t dead in the water looking for your third party dependency. When considering, “when this should be tested?” I put all of this into consideration. How frequently will the code change? How frequently will by use of the third party code change? To sum it up, if I ever upgrade versions of the third party code, or if I change my use of the third party API, specific consideration for testing must be made in order to maintain high quality of code.

Who Should Be Testing It?

Well, this is the million dollar question (not really). The long story short, in my opinion, is that the original author should be testing their code, and including those tests with their open source code repository. Assuming that the project is open source, it’s easy enough to see if the tests were included. This isn’t always the case. I can’t tell you the number of times that I’m browsing from open source project to project, and there are zero automated tests included. If that’s the case, you absolutely should consider writing these tests yourself through your own consumption of the API.

To Test Or Not To Test?

At the end of the day, if I do find myself writing tests for the service layer, more often than not those tests are going to be bundled with my other tests – the unit and UI tests. They won’t be dedicated tests for the third party code. And I’ll rely on design patterns like Adapter to encapsulate the third party code into a closed off portion of my own code base so it can be swapped out down the road if needed. I hope I gave you some useful things to consider in your efforts for deciding whether testing the iOS service layer in your iOS apps is worth it.

Happy cleaning.

Proposed iOS Testing Pyramid

The Original Testing Pyramid

Before I go into my proposed iOS testing pyramid, I know you’ve heard me mention Martin Fowler’s test pyramid before. In case you weren’t reading, or don’t remember, here it is:

Fowler advocates for three levels of testing: unit, service, and UI. If you’d like the full description, you can read about it here.

Proposed Revision

I disagree with this pyramid for iOS apps. I think we can do better. Here’s my **proposed revised iOS testing pyramid **:

The changes include:

Unit Tests Remain – If you don’t write any other automated tests, please write unit tests. In the least, unit tests open the door for things like test driven development and refactoring.
UI Tests Remain – With the vast variety of device sizes and OS versions, UI tests’ value can multiply with the more devices and OS’s you run them on.
No Service Layer – I think with most iOS apps, there’s no need for the Service layer. I think a comprehensive suite of unit tests will take care of anything that the service layer would have otherwise taken care of.
Add Snapshot Testing – Snapshot testing can really help “lock in” your user interface to ensure it looks pixel perfect and preventing any future code changes from introducing slop.
Add Manual Testing – There are simply bugs that no amount of automated testing will find. There was one bug we came across where a hidden `UIView` was covering a button in a very specific edge case such that the button was not tappable. This was found by a human during manual QA. It never would have been found by automated tests alone. TestFlight makes this so easy these days, don’t skimp on it.

Suggested Tools

Manual Testing – You and your friends’ own two fingers.
Snapshot Testing – As far as I can tell FBSnapshotTestCase is the only game in town.
UI Tests – Lots of options here: KIF, Xcode UI Tests, Frank, Calabash.
Unit Tests – XCTest FTW.

Marco Arment Does Not Unit Test

If you aren’t listening to the Accidental Tech Podcast, you should be. It’s by far my favorite podcast. On this week’s episode, episode 168, Marco Arment reaffirmed his stance that he does not unit test his code. The context of the conversation was that Marco was discussing how he had to put in an emergency fix for an upgrade to his sync algorithm for his iOS podcasting app, Overcast. The co-hosts on the show, John Siracusa, and Casey Liss immediately jumped all over him to point out that this was exactly the sort of problem that automated tests are intended to fix.

This isn’t new news that Marco doesn’t unit test his code, he’s talked about this in the past. I’ll be giving a talk at Philly CocoaHeads this week on automated testing, and one of the things that I’ve been wrestling with is: where do I begin the talk? How much background should I assume? Are we still at the point as a community where we need to debate whether automated testing is a good/worthwhile thing? If you’ve read other posts on this blog, my stance should be obvious- half the posts I’ve written so far have been in favor of automated testing! It’s been a while since I’ve worked in other platforms than iOS, but I hear communities like Ruby on Rails and .Net have incredibly deep adoption of automated testing and test driven development principles. And on iOS, there really are plenty of both open source options, and even Apple-endorsed options for adding automated tests to your code.

So why are we still debating whether automated testing is even a thing? To me, it’s not about critiquing Marco about his development practices. It’s about recognizing that 1 out 3 senior engineers on a popular podcast don’t write automated tests. Is that representative of our community?

For my talk this week, I want to address this, and I’m thinking this is where I’ll open it up for conversation at the end of the talk. The majority of the 30 minutes will be focused on some how-to techniques for adding automated tests into your apps.

I like this trend that I’ve started now of capturing “testing in the wild” – how last week I dropped the reference to Sam Soffes’ perspective on automated testing. In a lot of ways, these popular people in our industry have a lot of power. I can imagine if I were an aspiring iOS engineer hearing that Marco doesn’t unit test his software – it would certainly make me think twice about whether it’s a worthwhile endeavor.

Sam Soffes on Test Driven Development

It always feels good when your own perspective is validated by someone you respect. Listening to the Immutable podcast episode 36 today, Sam Soffes gave a great endorsement of test driven development that I agree with. Essentially the question was asking for Sam’s perspective on unit testing, specifically challenging whether they are worth the time or not. To summarize Sam’s response, he acknowledged that they might feel like a chore when you are first writing them, but it’s when you are going back to change the code later that the value really shines. This to me is one of the main reasons why I write unit tests. There have been many times when I’ve gone into an existing code base to make a change – either a bug fix or a feature enhancement, only to introduce another bug. It’s when you have an automated test catch this mistake that you’ll buy into the value as well.

Immutable Podcast

I also wanted to recognize this podcast as one that I’ve been enjoying lately. I think you’d like it too. Each episode focuses on five listener submitted questions to Sam Soffes, an iOS engineer, and Bryn Jackson, a designer. Bryan and Sam answer each question with a brief conversation, and the whole episode is 30 minutes or less.

Who is Sam Soffes

Sam Soffes first came onto my radar a couple years ago when I heard that he built and sold a todo app called Cheddar. Later on, I found some good use of his open source framework called SSKeychain. SSKeychain simplifies the storage and retrieval of data in the iOS keychain. It’s very popular on GitHub with over 3000 stars at the time of this writing.

A Role Model For Me

Sam is someone I recognize as a really good engineer in the industry, someone I aspire to be like. I feels really good when you have a role model reinforce one of your values. Take a listen to Sam Soffes on test driven development on Immutable, and let me know what you think.

Happy cleaning.

Code Coverage Is A Silver Bullet

Have you ever measured code coverage on any of your projects? How did it work out for you? What problems did it solve? Did it present any new problems? Code coverage is a silver bullet, a silver bullet for understanding how well your tests cover your code. Beyond that, it’s really what you make of it.

What Is Code Coverage

Code coverage is a metric that measures how much of your “production” codebase is being tested by your automated tests, usually unit tests. It’s usually measured at the granularity of a line of code, and sometimes measures methods and classes in aggregate. It’s nothing more than a black and white measurement, how much production code was executed when a given test suite was run. There’s no judgements made to the quality of the tests, simply, how much code did they execute.

Where Code Coverage Gets Tricky

Just to reiterate, code coverage as a metric makes no claims about the quality of tests. If you write a “test” (intentionally in quotes), that simply calls a method and does nothing with the result, the method it calls will have a high level of coverage. That being said, the “test” has done nothing to actually verify an outcome, or craftily provide edge case inducing input. It simply calls the method and discards the result. Voilah, high coverage, crappy test. For example:

class Adder {
  func add(x: Int, y: Int) -> Int {
    return x + y
  }
}

Here’s a corresponding test:

func testAdd() {
  let toTest = Adder()
  toTest.add(2, y: 3)   
}

Adder right now has 100% unit test code coverage! Yay!

Wait a minute, slow down hoss. There’s not a single assertion made in that test. While the test generates a high code coverage metric, it doesn’t validate squat.

TDD FTW

This is where following your test driven development cycle of “Red, Green, Refactor” you’ll never get into this state of test crappiness, and it can truly ensure that your code coverage is a silver bullet. The “red” step of that cycle is critically important, as it ensures that your test actually verifies something. Without knowing that your test can actually fail, you never know that your test actually does anything. And likewise, that your code coverage even means anything.

Be a Professional

The biggest counter argument you will hear about measuring code coverage, is that it can be cheated. Of course it can be cheated! Software “professionals” don’t cheat though. Craftsmen don’t take shortcuts. My life was changed when I read “The Clean Coder” by Bob Martin. He talks about what it takes to be a “professional software engineer.” (He also advocates for 100% code coverage despite all other costs, but I have other opinions on that). It’s a sense of taking pride in your work. With this post, I just want to put this call to action out there, be a professional. Code coverage measurement is just another tool in your toolbox. Code coverage is a silver bullet, but only one that returns to you what you put into it. Crap in, crap out. Use it appropriately, know it’s shortcomings, and know where it shines. When properly used in conjunction with TDD it can powerfully help you continually improve over time. I’d love for you to try it out, and let me know how it works for you.

I was inspired to write this article from listening to episode 67 of “This Agile Life” podcast, where they reference this article, “Is Code Coverage a Silver Bullet?“

Happy cleaning.

Three Reasons KIF is Awesome

As I introduced yesterday, KIF is a tool that enables functional testing of iOS applications. I wanted to go into a little more detail today on KIF’s capabilities by giving you three reasons KIF is awesome. Tomorrow, I’ll present an actual walkthrough of writing a KIF test case or two, so be sure to check back then.

What is KIF?

KIF stands for “keep it functional.” The name does it justice, it’s a test framework totally focused on providing a mechanism to write functional tests for your app. KIF tests hook into your app by interacting with the user interface. Similar to all automated tests of any sort, KIF enables you to write repeatable steps of verification to ensure that certain inputs determine certain outputs. Unlike a unit test where you might instantiate a model object, and then directly call a method, and verify the outcome, in KIF tests you instead write code to literally navigates through your app’s user interface and looks for things on the screen. You’ll never call viewDidLoad() explicitly from a KIF test, but you will instead simulate the tap of a button. In order to find things on the screen, KIF leverages accessibility labels and identifiers on your user interface elements.

Three Reasons KIF is Awesome

There are three main reasons why I like KIF:

Tests can be written in Swift or Objective-C. For anyone that has experience with the first iteration of UI Automation test cases, THIS IS HUGE. UI Automation was Apple’s original provided tool for developers to write functional tests, and it was lame because you had to write the tests in JavaScript with a poorly documented API.
Tests are executed as part of a normal old test target. Similarly replacing UI Automation JavaScript testing, THIS IS EVEN HUGER! Being able to execute your tests as a normal target in Xcode means first class citizenship with your build process, and thus more easily enabling repeatability and integration with continuous integration tools. With a single Command-U (the keyboard shortcut to run all tests), you can run both your unit tests and your KIF tests.
KIFTestCases, the class that you subclass to write your tests, are subclasses of XCTestCase. This means that the same familiar API from writing verification in unit tests applies to KIF tests, no need to learn a new API for making assertions in your KIF tests.

Can KIF Test Swift Code?

For sure! In fact, this is where I got my first introduction to writing Swift code as part of a “real” project. The first Swift code that we started adding into our repositories on my team was in KIF tests. If you’re looking for a way to start introducing Swift code into your Objective-C project, give it a try in your tests.

Wrap Up

I’ve really enjoyed using KIF for my functional tests, and I hope I helped you by explaining why. Tomorrow, I’m excited to walk you through creating your first KIF test.

Happy cleaning.

iOS Functional Testing Overview

What is Functional Testing

iOS functional testing is the process in which an app is tested as a “black box” and in an end to end fashion. Unlike unit testing, functional tests are written with no awareness of the inner working of the code of an app. Honestly, you could even hand your project to someone, and forbid them from looking at the main source, and they would still be able to write thorough functional tests just my observing how the app behaves on a device or in a simulator. As a result, in iOS functional testing, you’ll write a lot of code that simulates gestures (taps, swipes, shakes, rotations, etc.), and perform validation based what appears on the screen in response to varied input. Of course, knowing the ins and outs of how the code for the app is written helps a lot when writing iOS functional tests. You’ll be more able to know where the edge cases are, and be able to ensure there’s coverage for each nook and cranny.

I’m someone cloudy on how iOS functional testing might differ from “acceptance” testing or “integration” testing or “UI testing”, and a lot of the time I interchangeably use the terms. In more complex systems than iOS apps, there may be more nuances to where boundaries are drawn between these processes.

How to do Functional Testing on iOS

Unlike manual tests, iOS functional testing is performed via written tests that enable this outside-in approach in an automated and repeatable way. I’m currently using and loving KIF for my iOS functional testing. As a result, I’m going to make a couple other posts this week dedicated to writing great tests with KIF.

KIF isn’t without its limitations. There’s also a bunch of other tools available for iOS functional testing, including:

Down the road, I plan to cover each of these in-depth as well as KIF.

Can you do TDD with iOS Functional Testing?

Hell yes, you can do TDD with iOS Functional Testing. In fact, I believe in it, and advocate it, though there are caveats you should be aware of. Often, it will end up redundant with unit tests. And, if you are purely following the red, green, refactor cycle, you won’t often end up with compilation errors as a result of your iOS functional tests, and instead you’ll have to run your tests more frequently. And iOS functional testing is slow, so as a result, I end up usually spending my TDD cycles with unit testing, and then after the fact, perform my iOS functional testing with KIF.

Often, as bugs are found in the app, I might start with iOS functional testing in order to ensure that I have a functional test that actually leverages the broken code. That way, I have the confidence to know that the bug will never surface again.

Don’t Rely on iOS Functional Testing Alone

As much as I love iOS functional testing, I believe it should be used side by side with unit tests. There’s simply large pieces of code that will go untested if you only rely on iOS functional testing. Additionally, because these test are slow, the more you write, the more slow they get. And finally, depending on how many other “things” like persistence layers or backend services that your app uses, your iOS functional testing will rely on these to be “working” and reliable as well – a dependency that just isn’t there with unit tests, and thus they are more reliable. I liked this article on the Google Testing blog that goes into this in more detail.

Wrapping Up

iOS functional testing sit at the top of the testing pyramid. They are a powerful way to test your app in an end to end, blackbox fashion, simulating a user testing your app. Check back throughout the week for more detailed posts on my favorite functional testing tool, KIF.

Happy cleaning.