Android and Bamboo – Testing Using the Android Emulator on a CI Server

Posted on Mon 01 June 2015 in Continuous Integration

In a previous post, I discussed how to set up Atlassian Bamboo to build unit tests as part of an Android project. In that post, I went through the steps needed to install the Android SDK on a Bamboo build server, and configure a Bamboo build project to build and test the codebase.

One of the assumptions that I made in the previous post was that all of the unit tests were written with a package called Robolectric (see my post on Android Unit Testing with Robolectric for details on how to set up and use it). Why use Robolectric? Well, it allows you to test your project without having to run the tests on an Android emulator. As you may already know, the emulator – while good – can be slow. However, there are many times when you need to test on an actual Android device. In this post, I will discuss how to run an emulator on your CI server, and configure Bamboo to run Android tests directly in the emulated device.


Assumptions

In this post, I’m going to make several assumptions:

  1. Your Bamboo server is running a Linux OS.
  2. You have already installed the SDK on the Bamboo server, and that it has the components you need to build and support your Android application. If you don’t already have this set up, see my previous post on how to do this.
  3. You are familiar enough with Bamboo that you know how to configure a project, and add several tasks to it.
  4. You have access to a shell on the build server so that you can define a few things that Bamboo needs.
  5. You want to run tests that actually require an emulator (or device) to run. Tests such as ActivityInstrumentationTestCase2 and ProviderTestCase2 are good examples of tests that need to be run on an emulator or device.

The Solution

Implementing this solution involves the following steps:

  1. Define an emulated device.
  2. Start a headless emulator in the background, and monitor it.
  3. In Bamboo, add a task to run the tests on the emulator.

Step 1: Define an Emulated Device

You need to shell into your build server now. Once there, you need to define an emulator with the configuration your application requires. In my case, I needed an emulator that supported API level 19 (KitKat). In the example below, I’m also using the x86 image for the emulator. I called my new emulator android_x86. Make sure you switch to the user that has access to the Android SDK that you installed.

android create avd --force -n android_x86 -t android-19 --abi x86

Side-note: why use the x86 image? If you are have either an Intel or AMD processor on Linux, the x86 image can be accelerated as long as you have KVM installed. See my post on Android Emulator VM Acceleration on AMD for details on how to install KVM.


Step 2: Starting and Monitoring a Headless Emulator

For this task, we will create a start/stop script for an Android emulator, and install and use the Linux program monit to spin up the emulator and make sure it continues to run. The monit program is a great monitoring program that will be able to see if an emulator is running, and if not, start one up in headless mode. Plus, it has a web interface that will let us start, stop, and monitor the status of the running Android emulator from within a browser.

An Emulator Start/Stop Script

The first thing to do is to create a script that will be responsible for starting and stopping the emulator in headless mode. What is headless mode? Usually a build server does not have a monitor attached to it, and thus anything that tries to open a window on the display will error out. Operating the emulator in headless mode will prevent that from happening, since it won’t try and display an emulator window. A few blogs out on the net have good instructions on how to do this (for example, Paul Estrada’s post on defining a headless emulator for Travis CI).

The script below acts as a simple start/stop script. I named the script android-emulator-control.sh and put it in /opt/bin for easy access. You will need to edit ANDROID_HOME, ANDROID_SDK_HOME, and the EMULATOR_PORT depending on where your Android environment is configured, and what port you want to run it on. You’ll also need to edit the AVD_NAME to match the name of the AVD you created in the previous step. The script will start up an emulator on the designated port (in this case port 5556). It will also create a process identification file at /tmp/android-emulator-5556.pid and a log file at /tmp/android-emulator-5556.log.

#!/bin/bash
export ANDROID_HOME=/opt/android/android-sdk-linux
export ANDROID_SDK_HOME=/opt/android/android-sdk-linux
EMULATOR_PORT=5556
PID_FILE="/tmp/android-emulator-$EMULATOR_PORT.pid"
LOG_FILE="/tmp/android-emulator-$EMULATOR_PORT.log"
EXEC_FILE="$ANDROID_HOME/tools/emulator"
AVD_NAME="android_x86"

case $1 in
    start)
        $EXEC_FILE -avd $AVD_NAME -port $EMULATOR_PORT -no-skin -no-audio -no-window \
            -qemu -m 512 -enable-kvm > $LOG_FILE 2>&1 & echo $! > $PID_FILE
        ;;
    stop)
        kill `cat $PID_FILE`
        ;;
    *)
        echo "usage: android-emulator-control.sh {start|stop}" ;;
esac
exit 0

Once you have created the script, you will need to make it executable with:

sudo chmod +x /opt/bin/android-emulator-control.sh

With the script executable, to start the emulator, just do a /opt/bin/android-emulator-control.sh start. To stop it is /opt/bin/android-emulator-control.sh stop.

Installing monit

Next is to install monit if it is not already installed. On a Ubuntu Server, this can be accomplished with:

sudo apt-get install monit

Configuring monit

The next step is to get the monit daemon configured. The general configuration file is located at /etc/monit/monitrc. To edit it, you will need to:

sudo vi /etc/monit/monitrc

You will need to enable or add the following lines in order to make it accessible on the web. Replace the IP address 10.0.0.44 with your server’s IP address, and the 10.0.0.0/24 with the subnet that you want to be able to access the monitor program (e.g. to give everyone in the 192.168.0 subnet access, make it 192.168.0.0/24):

 set httpd port 2812 and
    use address 10.0.0.44
    allow admin:monit
    allow 10.0.0.0/24

Save the file and exit. Next, you will need to create a configuration file for monit that will monitor the Android emulator. To do that, create a new file called /etc/monit/conf.d/android-emulator and add the following code:

check process emulator-5556 with pidfile /tmp/android-emulator-5556.pid
   start program = "/opt/bin/android-emulator-control.sh start"
      as uid bamboo and gid bamboo
   stop program = "/opt/bin/android-emulator-control.sh stop"
      as uid bamboo and gid bamboo
   if failed host localhost port 5556 then restart

Save the file and exit. This configuration file will tell monit to monitor the emulator PID, and restart the program if it finds it is no longer running. It also is configured to run the process as user bamboo with the group bamboo. This is because the bamboo user actually owns the emulator SDK files, and is the only user who is able to run the emulator and other Android commands.

Checking that monit is Running

Okay, now for the moment of truth. If you have configured monit correctly, it is time to start it up and see if it runs the emulator correctly. To start up monit:

sudo /etc/init.d/monit stop
sudo /etc/init.d/monit start

When it is restarted, aim your browser at the port 2812 on your CI server. You should be prompted for a username and password. Use the username and password that you defined in the monitrc file. You should be greeted with a screen that displays the status of your system, and the status of the emulator:

Monitor Main Screen

If you click on emulator-5556 you will be taken to a screen with more information about the process:

Monitor Options Screen

As you can see, there are options here to start and stop the emulator. This is really useful if Bamboo says that there was an error with the emulator. You can use this web screen to monitor its status when you are troubleshooting.


Step 3: Configuring Bamboo to Test the Project

Okay, the hard part is done. The final step is to configure the Bamboo project to execute the tests. Create a new Script task after code checkout has been completed and call it Run Tests on Emulator. You will need to pass the applicable ANDROID_HOME path. Simply create a script task, and add the following command to it:

./gradlew connectedAndroidTest

That’s all there is to it. Here is a screenshot of my Bamboo task that does this:

Bamboo Configuration

You will also need a JUnit Parser task in order to pick up the results of the tests. Add this as another task right after the test task. Call it Parse Test Results. You will need to specify the path to the generated XML reports. For Android projects, they are placed in the app/build directory under **/app/build/outputs/androidTest-results/connected/*.xml:

JUnit Parser Configuration


Run the Tests

With everything set up, you should now be able to run the tests. If everything works correctly, the results of your run should be available in Bamboo:

Successful Build

Here you can see my 3 tests ran correctly. These were tests that required instantiating an Activity and checking properties of various objects.


Troubleshooting

Bamboo Throws Errors

The most common error is that when the tests are run, the emulator isn’t up and running. If you look at the gradle output in Bamboo, you will see messages similar to the following:

:app:connectedAndroidTestDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:connectedAndroidTestDebug'.
> com.android.builder.testing.api.DeviceException: java.lang.RuntimeException: No connected devices!

The solution here is to make sure the emulator is actually up and running. If the above script is not working for you, you may need to take another approach to running the emulator.

The Emulator Won’t Run

There are many reasons why the emulator will not run from the simple start/stop script outlined above. For most of them, setting the path to the SDK is key, and making sure the AVD images are where the emulator command expects to see them.

Your best bet is to look at the log file that is generated when the start/stop script is called. In my example above, it will be at /tmp/android-emulator-5556.log. It will usually give you enough error information to help you figure out what went wrong. Also, be sure that you have created the emulator image with the android command with the user that has access to your Android SDK.

Multiple Build Agents

While the above works well if you have a single build agent, things get more complicated if you have multiple build agents running on the same machine. If more than one build agent tries to access the emulator at a time, you will probably get some unexpected results (crashes or failures). My suggestion would be to have several different remote agents. On each remote agent machine, you can have an emulator running using the process I described above.

Alternatively, define only a single agent with the capability to run Android tests. This will mean that on a busy server you will experience a backlog of jobs that need to be built, but at least you will not cause collisions with the emulator and different build agents. As a compromise, creating a single nightly run that executes these type of emulator tests on your development branch may be a good alternative to running them every time a branch is created or updated.


Wrapping Up

In this post, I discussed a solution for running Android tests on a Bamboo CI server. It involves running an emulator in the background so that the build agent can connect to it when it needs to run tests. Other CI servers have interesting plugins that help with the automated configuration and provisioning of emulators on the fly (for example, Jenkins has an Android Emulator Plugin that helps out with this task). The solution I describe above used a simple command called monit to run an emulator in the background.

Hopefully you have been able to implement the steps above, and have Bamboo building your latest Android project! Feel free to asks questions or leave comments in the comment section below.