Neil Kandalgaonkar

hacker, maker of things

isign: resign iOS apps on Linux

Today I’m proud to announce that Sauce Labs has open sourced one of the coolest projects I’ve ever done: isign. Check out the source on Github!

isign can take an iOS app that was authorized to run only on one developer’s phone, and transform it so it can run on another developer’s phone.

This is not a hack around Apple’s security. We figured out how Apple’s code signing works and re-implemented it in Python. So now you can use our isign utility anywhere – even on Linux!

You can install an app signed for your phone, with your credentials But it won't work on another developer's phone

Why would you want to do this?

  • Maybe you want to integrate with some other Linux-based continuous integration process, so the app and the web API are released the same way.

  • Or, maybe you want to hold your company’s Apple developer credentials in some Linux-based system, rather than on Keychain in a random developer’s laptop.

  • Or maybe you just want to study how iOS’s security works.

You might ask why Sauce Labs – known for testing as a service on virtual machines – is messing around with the innards of iOS real devices? It’s kind of a strange story….

Apple’s code signing

Unlike every other computing environment since the beginning of time, you don’t have the right to run any old app on your own iPhone. Apple’s security goes deep into how the kernel loads executable code. The phone won’t even consider running an app unless it can prove who wrote this code, and that they were authorized by Apple to run this code, on this device. It does all this with cryptographic signatures embedded right into the executable format. Even non-executable content is verified with hashes, collected in another data file.

That’s what the Apple App Store does; after apps are approved, they are signed with one of Apple’s master keys. Then every iPhone in the world will allow the app to run.

So how do you develop an app, before Apple has approved it? You have to go through an elaborate procedure of registering your phone with Apple, and then “code signing” your app with your Apple ID. Then you’ll get an app which is authorized to run but only on your own phone.

That was a big problem for us at Sauce Labs, working on the Real Device Cloud project.

Sauce Labs is the leading provider of software testing as a service. Until the Real Device Cloud, our job had been to orchestrate virtual machines on demand. VMs to run the OS, VMs to relay test commands, and even VMs to manage secure tunnels opened back to the customer’s network.

Now, our customers were clamoring to run tests on real devices – that is, real Android phones, and real iPhones. No matter how good a virtual emulator is, there are always things that can only be done on a real device.

Sauce Labs Real Device Cloud

And so we had this tricky problem: apps in development only work on the developer’s phones. But our customers wanted us to test these exact same apps on our phones. Somehow, we would have to take apps signed for customer devices, and then “re-sign” the apps so they would work on our devices.


In 2008, pioneering iPhone researcher Jay Freeman solved this problem with ldid. ldid can turn off the iPhone’s code signing verification, but it only works for jailbroken devices. This was a possible solution, but we wanted to guarantee to our customers that the phones were stock, straight out of the magic “Designed by Apple in California” box.

It’s also possible to re-sign apps with Apple’s XCode development suite. But that would mean introducing Mac Minis into the mix somewhere, and who knows what else that would do to our performance and security guarantees. We preferred to be able to spin up all resources necessary to manage the job as VMs. Plus, Apple’s terms and conditions make it hard to run with the kind of device/VM density we’d already achieved with the Android Real Device Cloud. So the ideal solution would run entirely on Linux.

Into the matrix

So, Steve Hazel – one of the wizards who founded Sauce Labs – came up to Vancouver, Canada for a week to work with me on something a bit crazy: to see if we could duplicate Apple’s “code signing” procedure in Python, Sauce Labs’ favorite language.

This is perhaps the only time I’ve ever been paid for a pure research project, and it was a blast.

Steve and I made a pact: if it didn’t work by 5:00pm on Friday, we were giving up and going with the Mac Mini solution, even if it made the operations people groan.

I spent a lot of time staring at Apple’s C++ libraries. At the time, Apple just threw up snapshots occasionally at their site, and the Security section listed some libraries, but it wasn’t anything you could easily compile or re-use yourself. Still, I gleaned enough to understand how signing and hashing basically worked, as well as how non-executable files fit into all this.

Meanwhile, Steve forged ahead by analyzing the apps, pre- and post-signing. His knowledge of crypto enabled him to guess how a lot of it worked. At one point, he figured out the exact byte range of a particular hash, by simply hashing all the possibilities. It turns out, O(n2) is totally acceptable when you’re researching. I didn’t know you were allowed to do that.

Steve also found a great Python library, construct, which can create parsers and generators for complicated binary formats, from a single declarative configuration file. Comex’s cs got us most of the way towards parsing the Mach-O format and the code signature. These libraries still form the core of isign today.

Part of the Mach-O Construct

Even after all this progress, on Friday afternoon, after a week of staring at green-screen hexdumps, we were about to throw in the towel. I reflected that maybe the OS X Reverse Engineering IRC channel (#osxre) had been right to laugh at me for trying.

ideviceinstaller fail

We’d gotten pretty intimate with Apple engineer in-jokes in hexadecimal (e.g. 0xFEEDFACE), and we noticed some bytes were flipped around from their usual order. I pulled the repo on my laptop and tried to install our customer’s app one last time on the iPhone 6.

The colorful icon popped up on the iPhone. Nothing out of the ordinary – except, this app had never been authorized to run on this phone, and now it was.

ideviceinstaller win

All isign needs is developer credentials that work with a different set of phones, and it can re-sign the app, on Mac or (crucially for us) on Linux!

How code signing works

Right now, the best documentation for what isign does is code itself, but here’s a conceptual overview of what’s going on. Please don’t crucify us for skipping over details here – although we would welcome corrections! We don’t understand everything about Apple’s code signing, but we understand enough.

The fundamental thing to understand: the iOS device will not “phone home” to determine whether an app is good or not. The iOS device has to be able to verify the app, all by itself, without the network. So it relies on digital signatures, which will trace every single bit (literally) back to an Apple-authorized developer. Whether this is a good thing or not depends on your perspective. But here’s how it works, more or less:

Verify the developer

When you’re developing with XCode, most of this just happens by magic. Or, XCode tells you there’s some problem, and asks if it can “automatically fix” it. And then we all wonder why XCode even had to ask. But here’s what’s really going on:

On their computer, the developer generates their own private/public key pair. Then they submit the public key to Apple, with a certificate signing request, and obtain the resulting certificate from Apple. Lastly the developer requests that Apple generate a provisioning profile from their certificate and the universal device id of their iOS device. This provisioning profile now represents a relationship between Apple, that developer, and that phone. The developer then will embed the provisioning profile in their app.

When the phone gets that app, it will read and verify that provisioning profile, and conclude that Apple thinks the developer is an excellent person, who should totally be allowed to run an app on this device.

Verify the developer created all the data files

But now, the developer also has to show that they approved everything in this particular app. After all, someone could have intercepted that app or modified the contents. So the developer has to sign every single file in the app. (On iOS, apps are really just directories of files, with a well-known structure inside.)

It’s easy to sign all the data files. A cryptographic hash is performed on every file, which is then added to a special file listing all the hashes. Finally this file is signed with the user’s private key. Since the phone has the user’s public key from the provisioning profile, this signature this can be verified. And by extension, now every data file is now known to come from the original developer.

Verify the developer created the executables

Executables and dynamic libraries are more interesting. Here the signature is built right into the structure of the binary. iOS uses a format called Mach-O. These binaries have headers pointing to “load commands” which show the operating system how to load various parts of the application. The one we’re interested in is the code signature. This is located in the structure known as LC_CODE_SIGNATURE.

You can think of LC_CODE_SIGNATURE as an elaborately formatted set of assertions about the rest of the binary.

All the code is hashed, in 4096-byte segments. Once we sign these hashes, even a single bit of different code will cause the iPhone to get a different hash result when it verifies the calculation, and it can know that the code was tampered with.

However, that’s not always enough. The app could still be tricked into doing the wrong thing, perhaps by modifying things other than code. So a set of “special” hashes are also added, with negative offset numbers. These will include the application’s entitlements; these encode the privileges the app wants. It also hashes the app’s metadata, and includes the hash of all the other data files in the app that we calculated earlier.

In a “so meta” move, Apple also adds requirements: the app’s suggestions to the operating system about how to verify its contents. The requirements are like a tree of expressions – a little program within a program. Requirements used to be looser and more optional when it came to data files, but have become more standardized over the years, presumably to guard against tampering.

Finally, all the above is signed with the developer’s private key, and encoded (with some baroque wrapping) into the LC_CODE_SIGNATURE.

How Sauce Labs uses re-signing

The first day it was added to our Jenkins continuous integration, it raised a few eyebrows. At the time, it wasn’t called “isign”:

iresign in the build

Since signing takes time and computing resources, proportional to the size of the app, we knew we wanted to minimize that. So we built a layer of middleware, in between the customer’s uploads, and Sauce Labs’ cloud servers – a kind of re-signing server.

isign and test orchestration

As of early 2016, isign has been powering our iOS Real Device Cloud beta for over six months.

Best practices when you don’t know what you’re doing

Google used to tell its employees to have a “healthy disregard for the impossible”, and that worked out well for us here. Indeed, almost everything about the Real Device Cloud is about solving problems nobody else has solved before, but that’s another blog post (or how about we talk about it, if that excites you?)

But apart from motivational-poster bravado, I think there are a few concrete reasons why this project worked out well:

  • We had diverse skills on the team. Steve Hazel and I are opposites in terms of work style. I research a lot before starting, and try to find precedents. Steve dives into problems with a different toolbox – poking and prodding, experimenting, relying on his experience to guide him. Together we covered a lot of ground!
  • We scouted out alternatives. Before we started this project, we did a lot of research into other approaches, and gauged the risk for each. We even sketched out ways to solve this problem which didn’t involve app re-signing. Re-implementing Apple’s code signature format was one of the highest risk approaches, but also had the most reward.
  • We timeboxed. Since we were doing it in one of the high risk ways, we gave ourselves only a week. I think this helped us focus. We didn’t have to constantly re-evaluate whether this was worth it; now that was the clock’s job. It’s easier to climb a mountain if you aren’t constantly looking over your shoulder to see if maybe you should turn back.

Why open source?

Sauce Labs is a company with deep open source DNA. One of our cofounders invented Selenium, the open source web automation tool, and our flagship mobile automation framework, Appium, is also already open source. So from the start we wanted to add isign to that list.

We hope you find more uses for it, and that it sheds more light on how iOS works.