Concourse build system for iOS and Android (part 1)

Have you ever felt like waiting for your iOS app to be archived or a release APK to be built is just a waste of time? Wouldn't it be great if you could just push your commit to the repo without having to worry about anything else?

Imagine what you could do with all the time you could save so… Maybe you could spend it writing long, complex and repetitive configuration files in an unreadable, indentation sensitive language. Sounds interesting? Keep on reading!  

Continuous integration and mobile apps

Using a continuous integration solution in any development workflow comes with many benefits:

• It allows you to run tests automatically
• It can easily produce various build versions for different environments
• It can automate the deployment process to reduce a possibility of human error
• It makes it easier to keep track of version numbers and build artifacts
• It saves developers’ time (when it works reliably)  

Nowadays, there are many cloud based CI solutions available. They serve as great options to automate workflows for web applications and backend systems; however, mobile development poses its own challenges which often are hard to solve with cloud services.

IDEs for mobile platforms haven’t been designed keeping the automation and command line usage in mind. This is especially true for Xcode - it relies heavily on graphical configuration editors and runs only on macOS which is not the first OS choice for server environments. Android Studio and Xcode are quite handy when an individual developer works with them, but when it comes to a project involving many persons or a handful of persons working on a number of projects, their shortcomings become instantly apparent.

If you’d like to run certain UI tests on your application, you might need a specialized service allowing for the use of device simulators to run the tests. Some of them even offer paid farms of physical devices.. Alternatively, having an in-house CI solution, you can set up your own worker to perform tests or you can even hook up your own devices and run tests on them.

The complexity of code signing and provisioning procedures is another issue to consider when you try to automate the deployment process. The release version of the app needs to be signed and the signing assets should be kept in a secure place, ideally, separate from the source code. If you’re developing for iOS, provisioning profiles and entitlements might give you a serious headache. On top of everything, you still need a secured communication channel with a store to upload your final build and that requires another set of credentials.

Concourse CI

For the reasons mentioned above and just to experiment a bit, we decided to set up an in-house CI/CD environment capable of building and deploying Android, iOS and Flutter apps. When it comes to open source CI servers, the most obvious choice is Jenkins. However, having some less than thrilling previous experience with the old mustachioed waiter we were looking for other options. That’s how we found Concourse.

Concourse is a quite new CI system with a multitude of interesting features:

• It is distributed - it consists of many independent nodes
• It offers a user-friendly web interface to visualize build status and a convenient command line interface for management
• It is docker based - by default all the builds are executed in docker containers
• It uses YAML configuration files for describing pipelines which can be kept under version control
• It has a powerful concept of resources which can be used to implement custom integrations and new features easily

Let's take a closer look at these features to understand better how they translate into practice.  

The basic setup

Concourse is written in Go and is available as a standalone binary for Linux, Mac or Windows. A machine running a Concourse process constitutes a node. While the whole system can consist of many independent nodes, at least one web node must exist. The web node is responsible for providing the system with the web interface and scheduling tasks for worker nodes. It also requires a PostgreSQL database for storing configuration and build metadata; however, all other build artifacts will be kept elsewhere with the use of external resources.

In order to be effective, you will also need at least one worker node. The worker node is another machine running Concourse and a docker daemon. When a web node discovers a worker node, it can ask it to execute some tasks. Since everything happens inside docker containers, there is no need to manually set up the build environment. The worker node will automatically download necessary docker images, fetch codes from a git repo, execute build tasks inside a container and upload any artifacts to relevant resources. Unfortunately, this is not entirely true for iOS apps, but we will discuss this inconvenience later.

A fly command line tool has to be used to configure nodes. It allows you to check the status of worker nodes, update configurations and perform any other management tasks. More details about the basic Concourse setup can be found in the official docs.

Pipelines, resources, jobs, tasks

Concourse uses certain abstract terms to describe areas to be automated and how to do it. It might be useful to familiarize yourself with these terms before trying to configure your projects. We present a quick overview of the essential concepts below.

Pipelines

A pipeline is a description of a single continuous integration workflow. Most likely, you will want to have a pipeline for every project that you develop and maintain. It will be visualized on a web UI, which allows you to see the status of the project and view a build history. The pipeline contains a list of resources and jobs, and a declarative description of how to use them together to perform desired tasks. On the code level the pipeline is just a YAML file with certain mandatory sections and an established format. You can read more about pipelines here.

Resources

Resources are one the core concepts of Concourse. They represent various external artifacts which can be used as inputs or outputs for executed tasks. Resources are also versioned and can be automatically monitored for changes. When Concourse discovers that a new version of a resource is available, it can trigger a job execution using the new resource version as an input.

Here are some examples of resources:

• A git repository - with commits treated as subsequent versions
• S3 or similar storage bucket - a place where a finished build product can be saved
• Auto-incrementing version number
• Slack channel - can be used as an output to inform about build status

I think you’ve got the gist. What’s nice about the resources is that they are just docker images underneath, so it’s really easy to create your own resource if the already available ones don’t fit your particular needs. Sharing your new resource is a great way to contribute to the Concourse community.

Jobs

Jobs define the actual actions to be executed within a pipeline. Each job has a name, and will be visible as a node in a graph on the web UI. The most important part of the job is a list of steps to be executed. These can include among others:

• Fetching a new version of the resource
• Executing a task with the input provided by the resource
• Pushing a new version of a resource after performing some task on it

The resources that your job operates on will be visible on the web UI as graph edges entering and exiting the block representing the job.

Tasks

The task is the smallest unit of work that you can configure, usually it is a certain actual command to be executed. A basic task description consists of:

• A platform on which it can be executed (you will need a worker node with a matching platform)
• A docker image which will be used to execute the task
• A command to be executed within the container
• A list of inputs and outputs - matching resources available for the particular job

When you click on a job node in the web UI, you can see the list of tasks it contains. You can click on any task to see its details, for example, the console log.

Summary

I hope this introduction gives you a basic overview of how the Concourse CI works. In the following part we will describe in greater detail how to apply this knowledge to create working pipelines for building Android and iOS apps and how to solve certain problems you may encounter. Stay tuned!