Basil Shkara

Automated OTA iOS App Distribution

comments [ ios appstore ota ] 27 October 2010

I’ve been spending some time setting up Hudson on a Mac Pro for continuous integration of Xcode projects. It’s worked out better than I expected. I started with a goal to allow over-the-air (OTA) distributions of iOS ad hoc builds and am pleased to say that this goal was accomplished.

OTA distribution is a new feature particular to iOS4, so users on versions prior to this will not be able to take advantage of the feature, however they are still catered for as I’ll share further down. The primary advantage of having your build server take care of distributing your builds to your beta testers is that you do not have to be concerned with creating correctly signed .ipa files, zipping them up together with the right provisioning profile and then emailing the result to your testers. It streamlines your development process by getting rid of one of the most annoying processes of iOS development: the beta release cycle.

For OTA distribution, you cannot simply create an ad hoc signed build, zip it up manually and give it the .ipa extension. If the user attempts to install this build and s/he doesn’t have the same provisioning profile you signed the build with, the installation will fail. Until recently, I thought the only way of creating a valid OTA build was via Xcode’s ‘Build and Archive’ feature which performs some trickery and embeds the provisioning profile somehow into the resulting .ipa. This allows users to install the app and during this installation process, the provisioning profile also gets installed onto the device automatically.

So what if you want this process automated? Turns out there is a command-line utility which allows you to do the exact same thing as the ‘Build and Archive’ tool. In fact, it’s what gets run when you hit ‘Build and Archive’.

The Setup

All our code is kept on a locally hosted Gitorious instance on a virtual machine somewhere. Hudson is installed on a Mac Pro running Snow Leopard. The rest is just configuration.

I created a parameterized build in Hudson which just means that you get to supply a lot of environment variables to your running script if you want. I hosted my build scripts in my VCS and I recommend anyone to do the same. Then it’s just a matter of having the following execute in Hudson: 

bash -c "$(curl -fsS http://github.com/baz/ios-build-scripts/raw/master/build_ipa.sh)"

The script in its entirety is below:

#!/bin/bash
# Below are required environment variables with some example content:
# SDK='iphoneos4.1'
# FORMATTED_TARGET_LIST='-alltargets'
# CONFIGURATION='Ad Hoc'
# DISTRIBUTION_CERTIFICATE='iPhone Distribution: Your Company Pty Ltd'
# PROVISIONING_PROFILE_PATH='/Users/tomcat/Library/MobileDevice/Provisioning Profiles/Your_Company_Ad_Hoc.mobileprovision'
# GIT_BINARY='/usr/local/git/bin/git'
# REMOTE_HOST='your.remote.host.com'
# REMOTE_PARENT_PATH='/www/docs/ios_builds'
# MANIFEST_SCRIPT_LOCATION='http://github.com/baz/ios-build-scripts/raw/master/generate_manifest.py'
# ROOT_DEPLOYMENT_ADDRESS='http://your.remote.host.com/ios_builds'
# ARCHIVE_FILENAME='beta_archive.zip'
# KEYCHAIN_LOCATION='/Users/tomcat/Library/Keychains/Your Company.keychain'
# KEYCHAIN_PASSWORD='Password'

# Build project
security default-keychain -s "$KEYCHAIN_LOCATION"
security unlock-keychain -p $KEYCHAIN_PASSWORD "$KEYCHAIN_LOCATION"
xcodebuild -sdk "$SDK" $FORMATTED_TARGET_LIST -configuration "$CONFIGURATION" clean build

GIT_HASH="$($GIT_BINARY log --pretty=format:'' | wc -l)-$($GIT_BINARY rev-parse --short HEAD)"
GIT_HASH=${GIT_HASH//[[:space:]]}
BUILD_DIRECTORY="$(pwd)/build/${CONFIGURATION}-iphoneos"
cd "$BUILD_DIRECTORY" || die "Build directory does not exist."
MANIFEST_SCRIPT=$(curl -fsS $MANIFEST_SCRIPT_LOCATION)
MANIFEST_OUTPUT_HTML_FILENAME='index.html'
MANIFEST_OUTPUT_MANIFEST_FILENAME='manifest.plist'
for APP_FILENAME in *.app; do
	APP_NAME=$(echo "$APP_FILENAME" | sed -e 's/.app//')
	IPA_FILENAME="$APP_NAME.ipa"
	DSYM_FILEPATH="$APP_FILENAME.dSYM"

	/usr/bin/xcrun -sdk iphoneos PackageApplication -v "$APP_FILENAME" -o "$BUILD_DIRECTORY/$IPA_FILENAME" --sign "$DISTRIBUTION_CERTIFICATE" --embed "$PROVISIONING_PROFILE_PATH"

	# Create legacy archive for pre iOS4.0 users
	cp "$PROVISIONING_PROFILE_PATH" .
	PROVISIONING_PROFILE_FILENAME=$(basename "$PROVISIONING_PROFILE_PATH")
	zip "$ARCHIVE_FILENAME" "$IPA_FILENAME" "$PROVISIONING_PROFILE_FILENAME"
	rm "$PROVISIONING_PROFILE_FILENAME"

	# Output of this is index.html and manifest.plist
	python -c "$MANIFEST_SCRIPT" -f "$APP_FILENAME" -d "$ROOT_DEPLOYMENT_ADDRESS/$APP_NAME/$GIT_HASH/$MANIFEST_OUTPUT_MANIFEST_FILENAME" -a "$ARCHIVE_FILENAME"

	# Create tarball with .ipa, dSYM directory, legacy build and generated manifest files and scp them all across
	PAYLOAD_FILENAME='payload.tar'
	tar -cf $PAYLOAD_FILENAME "$IPA_FILENAME" "$DSYM_FILEPATH" "$ARCHIVE_FILENAME" "$MANIFEST_OUTPUT_HTML_FILENAME" "$MANIFEST_OUTPUT_MANIFEST_FILENAME"

	QUOTE='"'
	ssh $REMOTE_HOST "cd $REMOTE_PARENT_PATH; rm -rf ${QUOTE}$APP_NAME${QUOTE}/$GIT_HASH; mkdir -p ${QUOTE}$APP_NAME${QUOTE}/$GIT_HASH;"
	scp "$PAYLOAD_FILENAME" "$REMOTE_HOST:$REMOTE_PARENT_PATH/${QUOTE}$APP_NAME${QUOTE}/$GIT_HASH"
	ssh $REMOTE_HOST "cd $REMOTE_PARENT_PATH/${QUOTE}$APP_NAME${QUOTE}/$GIT_HASH; tar -xf $PAYLOAD_FILENAME; rm $PAYLOAD_FILENAME"

	# Clean up
	rm "$IPA_FILENAME"
	rm "$ARCHIVE_FILENAME"
	rm "$MANIFEST_OUTPUT_HTML_FILENAME"
	rm "$MANIFEST_OUTPUT_MANIFEST_FILENAME"
	rm "$PAYLOAD_FILENAME"
done

Feel free to use and modify the above script. Both scripts I use are located here. The script above expects quite a lot of environment variables to be set so it knows what to do. The expectation is that this script is run in an automated fashion, so you just need to set it up the one time. The environment variables listed at the top of the file are all parameters set in Hudson. This way you can easily change any of the settings should you decide to switch provisioning profiles or server locations.

The most interesting line is possibly this one: 

/usr/bin/xcrun -sdk iphoneos PackageApplication -v "$APP_FILENAME" -o "$BUILD_DIRECTORY/$IPA_FILENAME" --sign "$DISTRIBUTION_CERTIFICATE" --embed "$PROVISIONING_PROFILE_PATH"

This will create the .ipa for you, codesign it correctly and embed the provisioning profile into the .ipa so that when your beta testers install the build OTA - the profile will be installed automatically as part of the installation. How did I discover this command? This lonely thread on the Apple developer forums. Turns out if you watch the Console whilst you run ‘Build and Archive’ from within Xcode, you’ll see the command that it runs.

There is also a Python script which gets executed: 

python -c "$MANIFEST_SCRIPT" -f "$APP_FILENAME" -d "$ROOT_DEPLOYMENT_ADDRESS/$APP_NAME/$GIT_HASH/$MANIFEST_OUTPUT_MANIFEST_FILENAME" -a "$ARCHIVE_FILENAME"

This generates the required manifest file (which is just a plist describing your app) and the web page presented to your users. The template for the web page was taken from the iOS Beta Builder project.

The end result is this:

  1. Hudson polls the VCS and ensures it has the latest copy of your codebase.
  2. If a change is detected in the codebase, it initiates a build using the above bash script.
  3. An .ipa is then created from the built app and the provisioning profile is embedded.
  4. A .zip file is created with the .ipa and the provisioning profile for legacy iOS3 users.
  5. The plist manifest file and the index.html web page is generated.
  6. All of these files plus the .dSYM directory are then copied across to your web server and versioned by the current Git hash.

App Sales Machine - pulls, parses, stores and emails App Store reports

comments [ appengine python appstore ] 05 September 2009

[UPDATE 20090910] Now parsing the Top 200 rankings.

[UPDATE 20090909] App Sales Machine now automatically generates graphs using Graphy and also sends HTML formatted emails.

I recently spent a couple days writing an App Engine application using the webapp framework. App Sales Machine does the following:

  1. Imports a tarball of your existing sales reports and stores them in the datastore.
  2. Logs into iTunes Connect every morning, downloads your latest sales report and stores it in the datastore.
  3. Parses the downloaded report and stores the parsed data in the datastore.
  4. Converts the reported income revenue of your sales to a configurable currency and stores that.
  5. Pulls the rankings for your app every 8 hours (configurable) and stores that in the datastore.
  6. Emails a report to a select group of recipients every morning with the latest cumulative figures, as well as the last downloaded report’s figures.

It utilises the task queues API to fetch your rankings asynchronously in a staggered approach. And if the daily sales report pull times out, it will add that task to the task queue to keep trying until it is downloaded.

All of this was made possible because of the following pieces of open software:

I made some modifications to itunes-connect-scraper to use the URL Fetch API so I could increase the connection timeout to the maximum currently allowed on App Engine (10 seconds).

The data parsing and currency converting code is borrowed and modified from iTunesConnectArchiver.

I also ported the app_rank.pl Perl script to Python. It was originally written by Ben Chatelain but it seems to have been taken down, however it still lives on in other forms albeit still in Perl (read the comments in his original blog post).

The variety of scripts available to download and parse your own report data is just an indication of the poor state of tools that Apple has provided iPhone developers with which to monitor their sales. In fact an entire ecosystem has sprung up to capitalise on this void (appFigures, heartbeat, TapMetrics, Appstatz, AppViz, My App Sales).

There are some free tools (as well as the above scripts) available such as MajicRank and AppSalesMobile but nothing really comparable to the kind of reporting you get from one of the above paid tools. App Sales Machine isn’t really comparable either at this stage and I don’t know if it ever will be.

There’s nothing wrong with any of the above tools, in fact I used appFigures for a while and was quite happy with it. App Sales Machine was inspired by their functionality but I just wanted to be in control of my own data.

In its current state App Sales Machine may offer all the functionality you require. It does for me currently but there is certainly room for extending it’s current feature set. There is currently a very minimal frontend interface which consists of just a file upload field to upload your existing sales reports. It wouldn’t be hard to integrate nice charts as email attachments (e.g. Graphy) or even as part of a frontend interface.

How to install

# SKU of the app when you uploaded it to the App Store
"<<SKU>>": {
	# Human-readable app name which corresponds to the above SKU
	"name": "<<app name>>",

	# App ID (can be found from the GET param of your iTunes URL i.e. ?id=xxxxxxxxx)
	"app_id": "<<app id>>",

	# Make sure your category name and corresponding ID exists in jobs/app_store_codes.py
	"category_name": "<<category name>>",

	# Is your app free?
	"paid": False,

	# List of dictionaries which represent the human-readable version number and the
	# date the version was released on the App Store
	"versions": [{'name': 'v1.0', 'date': datetime.date(2009, 6, 22)},
			{'name': 'v1.1', 'date': datetime.date(2009, 7, 18)}],

	# This must be the email address of a registered administrator for the application due to
	# mail API restrictions
	"from_address": "My Name <sender@example.com>",

	# List of email addresses you would like the daily reports sent to
	"to_addresses": [
		'Recipient Name <recipient@example.com>',
	]
}

cron:
- description: Daily report pull
  url: /jobs/pull_report
  schedule: every day 06:00
  timezone: Australia/Sydney

- description: Rankings pull
  url: /jobs/pull_rankings
  # Must be larger than 1 hour, see email_report.py
  schedule: every 8 hours
  timezone: Australia/Sydney

- description: Daily report email push
  url: /jobs/email_report
  schedule: every day 07:00
  timezone: Australia/Sydney

What you get

If all goes well, the scheduled email task should push a daily report to your inbox that looks something like this:

Hello,

Here is your daily report for <<app name>>
--

Yesterday's (2009-09-03) download figures:
       - xx

Yesterday's (2009-09-03) upgrade figures:
       - xx

Total number of downloads (2009-06-22 to 2009-09-03):
       - xxxxx

Total number of upgrades (2009-07-18 to 2009-09-03):
       - xxxxx

Upgrade rate (over base of xxxxx):
       - xx.xx%

Approximate total income revenue (AUD):
       - 0.0

Rankings (as of 2009-09-04 16:59:27.615497 UTC):

       Country                  Category                 Ranking
       -------                  --------                 -------
       New Zealand              Lifestyle                67
       Australia                Lifestyle                6

I’ve been using App Sales Machine for the past week and it’s been working beautifully. It lets me and the other stakeholders in the app know about its progress and how it’s tracking.

And because everything is stored in GAE’s datastore under your own administrator account, you can write new webapp handlers and templates to query the data and generate graphs, etc. if you wish.

The usual disclaimers apply here. Although I’ve been using it happily for a while now, I take no responsibility for incorrect reporting or loss of data that App Sales Machine causes. With that being said, I hope you find a use for it.

Using the LLVM/Clang Static Analyzer for iPhone apps

comments [ cocoa iphone ] 06 January 2009

This set of instructions will help you get started with finding potential bugs in your app. They are provided for developers who do not have their provisioning certificate yet.

Why would you want to use this static code analysis tool? Why wouldn’t you? It’s another tool in your belt to detect bugs early and is probably a good idea to do before submission to the App Store.

If you haven’t run this tool before, you may run into this output:

$ xcodebuild
=== BUILDING NATIVE TARGET Property Machine OF PROJECT Test Project WITH THE DEFAULT CONFIGURATION (Debug) ===

Checking Dependencies...
CodeSign error: Code Signing Identity 'iPhone Developer' does not match any code-signing certificate in your keychain.  Once added to the keychain, touch a file or clean the project to continue.
** BUILD FAILED **

In this particular case, the project was set up to build against the iPhone device and not the iPhone simulator. Building for the device requires you to use an iPhone provisioning certificate which can be had for USD 99 from the Apple iPhone Developer Program. If you wish to write and compile software without a provisioning certificate - you may still test your app in the simulator provided with the SDK however you will not be able to test your app on the device itself.

Xcode Project Properties

Clang/LLVM Output

Overview of Jekyll - a static site generator written in Ruby

comments [ github ruby ] 27 December 2008

GitHub Pages was announced a couple of weeks ago and seemed to be received quite well. Although I find this service extremely awesome, especially now that they’ve gone even further with it than their initial announcement, the bit of news that excited me more was released much earlier: Jekyll. In fact, I used Jekyll to generate this site for me.

The connection between GitHub Pages and Jekyll is that each Pages-bound repo is piped through Jekyll automatically, allowing you to create a site and host it at GitHub. The beauty of this system is that all your posts and site changes are stored in your Git repo allowing your site the benefits of a distributed VCS such as the ability to sync your site on any machine, pull or push changes to the GitHub repo and publishing posts remotely by pushing new posts in raw markdown form to GitHub.

Of course you can choose not to use Jekyll with GitHub and opt for another VCS or eschew VCS altogether. So far you may be thinking that these benefits don’t seem like much compared to a typical Wordpress install and you’d be right! After all Wordpress offers version controlled posts, the ability to submit new posts from any web browser and import/export tools for your posts. People who choose to use Jekyll will probably have these requirements:

  1. Limited hosting facilities such as lack of a DBMS.
  2. Command-line junkies.
  3. Need to merge changes and track revisions VCS style.

In fact, the latter point seems ideal for software projects with large groups of people contributing to the site documentation and is what looks to be addressed by the latest GitHub Pages generation feature for existing hosted projects.

As one redditor commented, the barrier to entry for Jekyll with a VCS is high. You must have knowledge of VCS’s as well as being very comfortable in the command line. If you are happy with that, the end gain and primary advantage of Jekyll + a VCS seems to be an efficient and transparent way of storing your version controlled posts, plus it lets you blog like a hacker.

Templating Troubles

The worst aspect in setting up this site was dealing with the templating engine Liquid. For example I could not find a way to loop over the site topics like so:

{% for topics in site.topic_list %}
	{% for post in site.topics.topic %}
		{{ post.date | date_to_long_string }} » {{ post.url }} {{ post.title }}
	{% endfor %}
{% endfor %}

(Just a note: there is no site.topic_list attribute. I forked the project and added it in /lib/site.rb. Also, there is no site.topics attribute in v0.3.0. If you wish to use it we’ll have to see if Tom accepts my pull request. Or you can apply my patch, or you can clone the forked repo.)

.topic_list was just a list of all topics belonging to a site. In this case, I was using Jekyll’s topics as tags and foregoing categories. The difference seems to me to be semantical and as I don’t require 2 levels of categorisation for posts, I chose the latter as it appears to me to be a cleaner way of organising posts in terms of directory structure, i.e.

/_posts/tag1/tag2/yyyy-mm-dd-post-title.markdown

as opposed to

/cat1/cat2/_posts/yyyy-mm-dd-post-title.markdown

So I wanted a method of tagging posts and the ability to display all tags with corresponding posts in a site. For this, I used the pre-existing topics functionality and added the .topics attribute to the Site Liquid object. This now allows me to display my posts (use the ‘topics’ link in this page’s header) organised by tag/topic.

The disadvantage of this comes back to Liquid. I could not figure out a way to use the enumerator inside the Liquid for loop tags as an attribute name (see above). So currently this means I must specify the topic name manually like so:

{% for post in site.topics.TOPICNAME %}
	{{ post.date | date_to_long_string }} » {{ post.url }} {{ post.title }}
{% endfor %}

Where TOPICNAME is ruby or github (i.e. the topics for this post, and all the topics currently in use for this site).

This was the only issue that came up and was not a dealbreaker. That is as complicated as the Liquid markup got.

How do I get started with Jekyll?

So from all this, you can see that Jekyll is still a young project but showing promise and has the ability right now to publish fully blown blogs. It will appeal to people who like to get their hands dirty. However once you have set up the infrastructure of your site, the process of creating and pushing your posts should be very simple.

Dr Nic has created a generator that creates a simple Jekyll site structure for you with a choice of 2 themes. It even includes the Disqus and Lighthouse code you need. Unfortunately he didn’t count on the agile skills of the GitHub team who released something even more polished a couple days later in the shape of their very own Pages generator which allows existing hosted repo owners to generate a project page from the Admin tab of their account.

The way in which you get started depends on the initial level of control you wish to have over the site.

  1. You can install the tool on your own server sudo gem install mojombo-jekyll -s http://gems.github.com/ and generate a Jekyll site structure with Dr Nic’s generator and then tweak it to your liking.
  2. If you already have a hosted project on GitHub, you can generate a project page from the Admin tab of your repo. (Note: this feature is currently unavailable. In place of the generate link is this text: ‘Page generation is disabled while we sort out a bug.’.)
  3. Or you can create a GitHub account if you don’t have one already, set up a new repo called your_username.github.com and push your Jekyll structure to it in the master branch.

Static-site generators

Having only used Jekyll, I found the whole process very satisfying. There is a minimalist simplicity in the ability to serve pre-generated static pages instead of relying on a DBMS to store content for you. As well, all your posts are in simple text files stored in a logical directory structure which is (hopefully) backed up to a VCS in the cloud, minimising the chance that you could ever lose your posts.

The Unix-like approach of storing everything in text seems to suit this particular application because in essence, there is little that needs to be done to produce content, able to be rendered by web browsers. As well, I am using Vim to author this post - it almost doesn’t feel like I’m writing text. It feels like I’m doing some sort of sysadmin task and that is probably part of the appeal and why I like it so much.