SFSpeechRecognizer Tips for iOS 10

Did you know that there’s a new iOS 10 API for transcribing audio in your iOS apps? I didn’t know that this was possible myself until I came across the Speech Framework introduced in iOS 10. In fact, I was actually watching one of Sam Davies’ screencasts on Ray Wenderlich’s new video site called Audio File Speech Transcription (both the screencast and the new videos site are fantastic by the way) where he covered how the API can be used to transcribe an audio file. The screencast itself is a fantastic short overview of how to use the framework to create an app that transcribes 90’s rap music files. I’m not going to try and recreate a tutorial on the basics of the API. Instead, when trying it out myself I discovered a couple nuances not mentioned in Sam’s screencast. Here’s a couple SFSpeechRecognizer tips.

Requires A Device

Use of SFSpeechRecognizer, the main class that makes speech transcription possible, will not actually transcribe anything unless you are running your app on a device (as of Xcode 8 beta 6). This was a surprise to me, especially considering that speech transcription of an audio file, rather than microphone input, has nothing to do with a physical device. I’m wondering if it has something to do with the underlying implementation of Siri and something that only exists on the actual device. Regardless, you are lucky enough to have access to a Bool on SFSpeechRecognizer called isAvailable. This Bool simply indicates whether speech recognition is available at the time of usage. I was actually banging my head trying to figure out how to get Sam’s sample project to actually work within the iOS Simulator. His screencast seemed to transcribe speech no problem in the app I was viewing on screen. Finally I looked closer and noticed that he was screensharing from an iOS device through Quicktime! Mystery solved! Either way, don’t make the same mistake as me and wonder why code like this didn’t work:

guard let recognizer = SFSpeechRecognizer() else {

if !recognizer.isAvailable {
  print("Speech recognition not available")


The other interesting discovery I made when playing around with SFSpeechRecognizer is that there is an undocumented limitation on how big of a file can be transcribed at once. I’m still playing around with the details as to where the limits are, but I have discovered, that longer running SFSpeechURLRecognitionRequest will timeout. I’m not even talking that long, like I had problems with a 5 minute video. For example, I tried transcribing my video Replace Temp With Query that is 4 minutes and 45 seconds long, and this was all the text that was returned before the timeout happens:

Hey what’s up everybody Danny from clean swiffer.com I’m here to show you another we factor in this week from our valors book we factoring in improving the design of existing time this week we’re gonna take a look at the re-factoring called replaced temp with Cory please temp with berries are factoring that’s a lot like expect nothing but there’s one difference it’s a little more specific with a place template query we’re going to specifically target temporary variables within our code and extract them into reusable piece of code so

Ya, not much (and not very accurate either). Either I should create a Radar for this, or Apple only intends for this API to be used for transcription of short audio clips. Time will tell.

Partial Transcription To The Rescue

Despite the undocumented timeout putting a crimp in my plans for using the Speech Framework in a couple longer running use cases, another Bool caught my eye: shouldReportPartialResults. It turns out my setting this flag to true, the Speech Framework will periodically provide transcribed results to you as they are discovered. Just set the value to true and you’ll see results continuously reported as they are determined:

let request = SFSpeechURLRecognitionRequest(url: url)
request.shouldReportPartialResults = true
recognizer.recognitionTask(with: request) {
  (result, error) in
  guard error == nil else { print("Error: \(error)"); return }
  guard let result = result else { print("No result!"); return }


Transcribing Realtime Playback

Despite these two shortcomings of short timeouts and requiring a device (which I hope Apple will fix at some point as the API matures), speech transcription is a really cool technology. Did you notice that voicemails in iOS 10 are automatically transcribed? It’s freakin awesome that you can glance at text for a voicemail rather than needing to listen?

Anyway another really cool real-world example of use of the Speech Framework is in the GitHub open source project Speech Recognition from zats. Apparently with some help from Apple, he came up with a way to transcribe a video on the fly. There’s some gnarly AVFoundation Objective-C code that made this possible. Be sure to take a look at his project and give it a go. And in fact, I’m wondering if I can use the techniques here to work around the timeout limitation I experienced with raw use of SFSpeechRecognizer. (Update: Turns out it did!)

Back Story

If you’ve read this far about SFSpeechRecognizer tips, I might as well bore you with some details on the back story as to why the SFSpeechRecognizer API caught my interest. With the videos I create for this site, it’s important to me to provide a transcription of the videos. I realize that watching a video isn’t always possible, so I like having a text alternative. It probably also helps with SEO as well. The thing is, transcribing the videos is a tedious process. For about a 5 minute video, it takes me about 30 minutes to transcribe it, and I type fast. Additionally, that’s only raw transcription. I’d like to do so much more. For example, I think the Realm videos are really well done. Specifically, two things I like are the links to jump to specific spots in the video from the transcription, and I also like the source code samples in the transcription. For me to do this it would take more time, so in attempt to look for a quick and easy fix to buy back some time, I figured I could use the new iOS 10 Speech Framework to code a speech transcription app to automatically transcribe my videos for me. I’m still working on it and definitely leveraging these SFSpeechRecognizer tips.

They say that necessity is the mother of all invention, right?

Wrap Up

How will you be putting these SFSpeechRecognizer tips to work? Have you had a chance to try out the new Speech Framework or any of the SFSpeechRecognizer APIs? Have you run into any other tricky spots or overcome any hurdles?

Happy cleaning.

UIRefreshControl in iOS 10

Did you know that adding a UIRefreshControl in iOS 10 to a table view, a collection view, or a scroll view has never been easier? UIRefreshControl is now a first class citizen in all of these scrollable views and officially supported by the iOS SDK. Having fought through hacked solutions to use a UIRefreshControl in anything but the officially supported UITableViewController in the past, I’m really pumped to see this. This change for iOS 10 first caught my eye in the WWDC 2016 session, What’s New in UICollectionView in iOS 10 (which was a really good session by the way).

How It Works

The key to easily adding a UIRefreshControl to a UITableView, a UICollectionView, and a UIScrollView is the UIRefreshControlHosting protocol. Just like UIRefreshControl in iOS 10, this is also new in iOS 10. It’s a protocol adopted by UIScrollView. And coincidentally, UITableView and UICollectionView inherit from UIScrollView so they easily get this behavior.

Adding a UIRefreshControl in iOS 10

Here’s how simple it is to add a UIRefreshControl in iOS10:

let tableView = UITableView()
let rc = UIRefreshControl()
tableView.refreshControl = rc

Yep, it’s that dirt simple. Want to add a UIRefreshControl to a UIScrollView? It’s just as easy!

let scrollView = UIScrollView()
let rc = UIRefreshControl()
scrollView.refreshControl = rc

And want to add it to a UICollectionView? Well, you probably get the idea:

let collectionView = UICollectionView()
let rc = UIRefreshControl()
scrollView.refreshControl = rc

Now don’t forget to add a target to the UIRefreshControl so it has some behavior associated with it. Luckily, nothing has changed with that API, so it’s just as you remember it from pre iOS 10 days. Here’s a full example with a UITableView:

override func viewDidLoad() {
  let tableView = UITableView()
  let rc = UIRefreshControl()
  rc.addTarget(self, action: #selector(TableViewController.refresh(refreshControl:)), for: UIControlEvents.valueChanged)
  tableView.refreshControl = rc

func refresh(refreshControl: UIRefreshControl) {

Refreshingly Simple

If you got this far, you probably aren’t rolling your eyes at how easy this is. If you’re questioning whether this is worth a blog post or not, fine. I believe it is. This is a subtle, yet awesome API change coming in iOS 10. I love to see that Apple continues to refine their API and adding niceties like this for developers. I look at this change and think, “Ya, this is how it always should have been.” Thank you, Apple. Has there been anything else on the edges that you’ve noticed and felt good about?

Happy cleaning.

iOS 10 UICollectionView Highlights

Did you get a chance to watch the WWDC 2016 session, “What’s New in UICollectionView in iOS10″ yet? I watched it today. There is some good stuff in the session, and I want to recap it for you in this post. The session is broken into three segments that capture everything that’s new for an iOS 10 UICollectionView :

Improvements to:

  • Smooth Scrolling
  • Self-Sizing Cells
  • Interactive Reordering

There is a special bonus fourth segment as well, but I’ll save that for later.

Smooth Scrolling Enhancements

In iOS 10, there are several enhancements that will improve the performance of your UICollectionViews – some of which you will manually need to leverage, and some you will get for free. Before dropping into the new features of iOS 10 UICollectionView, the presenter gives a nice overview of what it means to “drop frames” and what it’s bad.

Don’t Drop Frames

In order for your app to have “buttery smooth” performance, a hallmark of iOS apps, you must strive for app animation that performs at 60 frames per second. This means that a given frame of the user interface must be displayed in less than 16.67ms in order for the animation to appear “smooth.” Otherwise, when the frame rate drops lower than that, it’s apparent to the user in the form of a choppy animation. The easiest way to make the frame rate drop, is to do things like add blocking, long running method calls on the main thread in the middle of your animation. Here’s an in-depth article from Facebook on how they measure and ensure a highly performant news feed in their app.

Less Aggressive Loading and Unloading Cells

In a UICollectionView, the lifecycle of a cell is as follows:

  1. prepareForReuse
  2. cellForItemAtIndexPath – The heavy lifting of populating your cell’s data from your model happens here.
  3. willDisplayCell – Lightweight preparation for when you cell is about to go onscreen.
  4. Cell comes on screen, and as scrolling continues starts to move offscreen.
  5. didEndDisplayingCell

In general, this flow is unchanged between iOS 9 and iOS 10. The different is when these methods are called. Apple has optimized when willDisplayCell is called. In iOS 10, it’s now called at the very last minute before the cell goes on screen. This helps to balance out the CPU performance in drawing cells, but not executing that code too early. Additionally, another enhancement Apple has made in iOS 10 UICollectionView is that cells are not put on the reuse queue as aggressively as in the past. Instead, after a cell leaves the screen, it is kept around a little longer in case the user decides to swipe the cell back on screen.

Cell Pre-fetching

Apple also enhanced UICollectionView such that by default, cells are pre-fetched. This means that you can get even earlier awareness of when data for a cell is needed such that you can retrieve it. For example, if you are building a UICollectionView full of remote images. Leveraging the UICollectionViewDataSourcePrefetching, UIKit will call:


to allow for you to start downloading the images with Grand Central Dispatch of an NSOperationQueue such that when the cells containing the images comes on screen, the images will be downloaded and ready to go.

If you need to, you can opt out of this behavior by setting isPrefetchingEnabled to false, but why would you?

As part of pre-fetching, realizing that cellForItemAtIndexPath may be called for cells that never end up coming on screen – because the user stopped scrolling before they were shown. Also, it’s really important that you keep the work in willDisplayCell and didEndDisplayingCell really light. All the heavy lifting goes in cellForItemAtIndexPath. Apple described pre-fetching as an “adaptive technology” which I assume to mean that it’s level of “predictiveness” varies by use case for a given application.

And as bonus, this exact same pre-fetching API is also available on UITableView via UITableViewDataSourcePrefetching.

Self Sizing Enhancements

Prior to iOS 10, estimatedItemSize has existed on UICollectionViewFlowLayout in order for you to provide an estimate size for the items in your collection view. Sometimes it’s hard to predict the size of items in your UICollectionView. Realizing this, Apple has introduced automatic item sizing for your UICollectionViewFlowLayout. Simple set the item size to the constant UICollectionViewFlowLayoutAutomaticSize and UIKit will make smart guesses based on past measurements of your items in order to automatically predict item sizes for future items.

According to Apple:

It will keep a running tally of all the cells it’s already sized, and use that to influence its future sizing estimates…making the sizing much more accurate…leading to better performance and a more accurate layout.

Interactive Reordering Enhancements

Reordering a iOS 10 UICollectionView has also undergone some improvements as well. Prior to iOS 10, if you didn’t already know (and I only learned recently), it’s really easy to enable reordering on your UICollectionView – in your UICollectionViewDelegate, simply implement:

func collectionView(_ collectionView: UICollectionView, targetIndexPathForMoveFromItemAt originalIndexPath: IndexPath, toProposedIndexPath proposedIndexPath: IndexPath) -> IndexPath

There are some additional methods on UICollectionView that you should take a peak at too that enable you to add more advanced animation and update your data model if appropriate.

New in iOS 10 UICollectionView is the ability to reorder with paging!

collectionView.isPagingEnabled = true

That’s it! The presenter describes it as an interaction that feels just like moving icons between pages on your home screen.

The Bonus

Finally, the big reveal happens. It’s as if this is such exciting news worthy of a WWDC reveal, but there is no other session appropriate for it. The presenters reveal that pull to refresh will be supported on:

  • UIScrollView
  • UITableView
  • UICollectionView

If it wasn’t awesome enough that UIScrollView and UICollectionView got the control, but you are also no longer constrained to needing a UITableViewController if you want an out of the box pull to refresh control (which was a pretty annoying prior limitation in my opinion).

Final Thoughts

I have plans to do an in-depth example of how to use pre-fetching with Grand Central Dispatch in order load remote images in a UITableView sometime in the future. I recently ran into a problem in one of my apps that this exact thing would have solved. Essentially I had cells in a UITableView that were of varied height based on a remote UIImage being loaded. I ended up needing to set a static height on the cells to achieve a high frame rate and scaling the images, instead of properly sizing the cells according to the natural height of the image. It wasn’t the end of the world, but there was some extra white space in the cells that wasn’t needed. How do you plan on using these new changes to iOS 10 UICollectionView?

Happy cleaning.

iOS 10 UNUserNotificationCenterDelegate Overview

I just finished watching “Introduction to Notifications” from WWDC 2016 and there are some really cool new features for iOS 10 local notifications. Notifications are core to the mobile experience, regardless of what platform you’re using – Android, iOS, or something else. Apple really raised the bar with their iOS 10 local notifications through creation of a new framework, the User Notifications framework. The brand new iOS 10 UNUserNotificationCenterDelegate in the User Notifications Framework is one thing that jumped out at me.

Why Create A New Framework?

Push notifications have been available in iOS for years. Just like any framework, over time, as use cases and technology evolves, frameworks need to be reconsidered and improved. It is that time for push notifications on iOS. New in iOS 10, Apple created the User Notifications Framework. When implementing notifications, you have the choice to implement a local or remote notification. A local notification does not require a server to alert the user, where a remote notification does require a remote server to send information to the user. In past iOS versions, depending on the type of notification you are handling, there are two different methods in UIApplicationDelegate to implement to handle either notification type – remote or local. As such, this can lead to a lot of duplicate code since it’s likely that there is common handling of notifications regardless of whether they are local or remote.

One Notification Handler To Rule Them All

The User Notification Framework in iOS 10 fixes this as there is now a single method for handling both types of notifications. The new iOS 10 UNUserNotificationCenterDelegate now has a single set of methods for handling both remote and local notifications.


The old methods on UIApplicationDelegate are deprecated, including:

  • application(_:didReceive:) – Sent to the delegate when a running app receives a local notification.
  • application(_:didReceiveRemoteNotification:) – Called when your app has received a remote notification
  • application(_:handleActionWithIdentifier:forRemoteNotification:completionHandler:) – Tells the app delegate to perform the custom action specified by a remote notification.
  • application(_:handleActionWithIdentifier:for:withResponseInfo:completionHandler:) – Called when your app has been activated by the user selecting an action from a local notification.
  • application(_:handleActionWithIdentifier:forRemoteNotification:withResponseInfo:completionHandler:) – Called when your app has been activated by the user selecting an action from a remote notification.


Instead, these are all consolidated into a TWO METHODS in the new iOS 10 UNUserNotificationCenterDelegate protocol:

  • userNotificationCenter(_:didReceive:withCompletionHandler:) – Called to let your app know which action was selected by the user for a given notification.
  • userNotificationCenter(_:willPresent:withCompletionHandler:) – Delivers a notification to an app running in the foreground.

That’s right, two methods. And what’s better, is that now that they are moved into their own protocol, the iOS 10 UNUserNotificationCenterDelegate so this will help clean up your existing UIApplicationDelegate by being able to refactor all that old notification handling code into a shiny, new, cohesive protocol of it’s own.

Here’s an example:

extension NotificationManager: UNUserNotificationCenterDelegate {

    func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse, withCompletionHandler completionHandler: () -> Void) {

        switch response.actionIdentifier {

        // NotificationActions is a custom String enum I've defined
        case NotificationActions.HighFive.rawValue:
            print("High Five Delivered!")
        default: break

    func userNotificationCenter(_ center: UNUserNotificationCenter, willPresent notification: UNNotification, withCompletionHandler completionHandler: (UNNotificationPresentationOptions) -> Void) {

        // Delivers a notification to an app running in the foreground.

That’s literally it for handling notifications.

Bonus: Creating Local Notifications

Similar, creating local notifications has also been simplified. In order to verify the behavior of the new iOS 10 UNUserNotificationCenterDelegate protocol, I wrote some sample iOS 10 User Notifications Framework code to generate a local notification:

// 1
let highFiveAction = UNNotificationAction(identifier: NotificationActions.HighFive.rawValue, title: "High Five", options: [])
let category = UNNotificationCategory(identifier: "wassup", actions: [highFiveAction], minimalActions: [highFiveAction], intentIdentifiers: [], options: [.customDismissAction])

// 2
let highFiveContent = UNMutableNotificationContent()
highFiveContent.title = "Wassup?"
highFiveContent.body = "Can I get a high five?"

// 3
let trigger = UNTimeIntervalNotificationTrigger(timeInterval: 5, repeats: false)

// 4
let highFiveRequestIdentifier = "sampleRequest"
let highFiveRequest = UNNotificationRequest(identifier: highFiveRequestIdentifier, content: highFiveContent, trigger: trigger)
UNUserNotificationCenter.current().add(highFiveRequest) { (error) in
  // handle the error if needed

Here’s an overview of that code block:

  1. Create the custom “High Five” action for responding to the local notification, and set it as a category for the notification in addition to a system provided category for simply “dismissing” the notification.
  2. Create the content for the local notification.
  3. Create the trigger for the notification, for it to pop 5 seconds in the future.
  4. Add the notification to the notification system.

This code highlights two new features of local notifications in iOS 10, custom actions and time based triggers.

Custom actions now let you specify any number of actions for which the end user may respond to the notification.

iOS 10 UNUserNotificationCenterDelegate

Time based triggers allow you to delay the local notification by a time interval, designate it as repeating or not, and even schedule it based on an actual date.

I didn’t hear it confirmed in the WWDC presentation, but I’m pretty sure that the custom notification actions require 3D Touch. I could not figure out how to interact with them on my iPhone 5s device that is running iOS 10.

I’ve even bundled all this code up into an Xcode project for you on GitHub, here.

Notify Me

Notifications, remote and local, enable some of my favorite features on iOS so as an end user, I’m happy to see all these enhancements. As a developer, I can’t wait to try them out in my own app. This only scratches the surface of the new stuff Apple added. Do you use notifications in your app? What are you looking forward to most about the new iOS 10 User Notifications Framework? What about the iOS 10 UNUserNotificationCenterDelegate?

Happy cleaning.

UIPasteboard iOS 10 Example

I just finished watching What’s New in Cocoa Touch today from WWDC 2016, it was an awesome session. My coworker called it “the index for the rest of WWDC.” Olivier Gutknect basically goes on rapid fire for an hour citing change after change in UIKit for iOS 10. For each thing he mentions, he covers varied levels of detail, and usually references a different WWDC session where the framework or topic will be covered in more detail. The UIPasteboard iOS 10 API made an appearance in this session, just like the keynote. When I heard about the Universal Clipboard changes for iOS 10 and macOS Sierra, I was so excited to actually be able to use this new feature as an end user. It’s so cool, leveraging Handoff and continuity, iOS and macOS will intelligently share your clipboard across devices. This means I can copy something on my laptop, and then pickup my iPhone and have that most recent thing copied available for pasting.

As a developer, I was also curious to see what new changes were in store in the UIPasteboard iOS 10 API.

In order to take advantage of the new Universal Clipboard in your apps, you don’t need to do anything differently than you’ve been doing, just use the UIPasteboard API. That being said, in order to provide the best user experience, there’s two changes to the UIPasteboard iOS 10 API that are related to the Universal Clipboard:

  1. Methods to help understand what is in the clipboard.
  2. Methods to designate the lifetime of the clipboard item.

Understanding the Clipboard Contents

Some of the changes to the UIPasteboard iOS 10 API enable you to inspect the contents of the clipboard to understand what type of contents are in the clipboard. Is it a color? Is it an image? Is it a string? Is it a URL? As a developer, you can intelligently take advantage of this. For example, say your app contains a color picker. You could provide a standard looking color picker, while also observing that the clipboard contains a color and show it as well.

Here’s an example:

// Setup, let's put some stuff in the UIPasteboard

let pasteboard = UIPasteboard.general()
pasteboard.string = "andy"
pasteboard.url = URL(string: "http://cleanswifter.com")
pasteboard.image = UIImage()
pasteboard.color = UIColor.red()

// Understanding the UIPasteboard contents

if pasteboard.hasStrings {
    print("The pasteboard has Strings!")
if pasteboard.hasURLs {
    print("The pasteboard has URLs!")
if pasteboard.hasImages {
    print("The pasteboard has images!")
if pasteboard.hasColors {
    print("The pasteboard has colors!")

Designating The Lifetime of Clipboard Contents

With the changes to the UIPasteboard iOS 10 API that introduce Universal Clipboard, it also opens a slight security vulnerability in that an end user could copy a sensitive piece of data and inadvertently make it available across all their devices. The UIPasteboard iOS 10 API provides a way to lock this down. As a developer, you can either:

  • Flag a piece of data as “local only” in which it will not appear in the Universal Clipboard across devices, and
  • Set an expiration date on a piece of data such that it isn’t available after that date.

Here’s an example of flagging a pasteboard item as “local only”:

// Add a string
let aLocalOnlyStringKey = "Local only string key"
let aLocalOnlyStringValue = "Local only string value"

// Set the string in the LOCAL pasteboard
pasteboard.setItems([[aLocalOnlyStringKey: aLocalOnlyStringValue]], options: [UIPasteboardOption.localOnly : true])

In one line, you set the item in the UIPasteboard with an option localOnly as true.

Here’s an example of adding something to the pasteboard with an expiration date:

let aExpiringStringKey = "Local only string key"
let aExpiringStringValue = "Local only string value"
// Create a date 24 hours from now
let expirationDateOfTomorrow = Date().addingTimeInterval(60*60*24)

// Add the string and mark it to expire 24 hours from now
pasteboard.setItems([[aExpiringStringKey: aExpiringStringValue]], options: [UIPasteboardOption.expirationDate: expirationDateOfTomorrow])

Again, in one line you get to pass an expiration date for when the UIPasteboard item should expire. You can also use these together.

Gonna Needa Pasteboard

James Dempsey is going to need to update the lyrics to his song “Gonna Needa Pasteboard” to account for the Universal Clipboard and these new UIPasteboard iOS 10 API features. In the meantime, I absolutely can’t wait to use the Universal Clipboard, and all the while knowing that if developers properly take advantage of these new UIPasteboard iOS 10 API features, my data will be safe and my applications will be snappy.

The code from this post is available in GitHub as a playground, here.

Don’t forget, this UIPasteboard iOS 10 API is still in beta and could change between now and the public release.

Happy cleaning.