Last time, I wrote about setting up Travis CI to effortlessly perform continuous integration and testing. My next step was to determine what was actually being tested in my Python Chip 8 emulator, and improve upon areas that had insufficient tests to cover them. While code coverage isn’t the best metric for code quality, it does at least provide visibility on what you are not actually testing.
A good tool that allows you to inspect your code is SonarQube (previously just called Sonar). Available as a standalone server, SonarQube gets you up and analyzing code in minutes. By default, the out-of-the-box configuration provides an H2 on-disk database that isn’t rated for production, but doesn't require any external dependencies (like PostgreSQL or MySQL). To get it up and running, simply unzip it and run it. On Linux this looks like:
unzip sonarqube-4.3.zip ./sonarqube-4.3/bin/linux-x86-64/sonar.sh start
This will start SonarQube on your local machine, and set the server listening on port
9000. To get to it, simply browse to
http://localhost:9000 in your web browser. When
you initially visit the link, there will be no projects loaded. To load your project
in to SonarQube, you will need one additional piece of software - the Sonar Runner.
The Sonar Runner
is responsible for running your test cases and posting the results to SonarQube.
In order for the Sonar Runner to be able to know what to do, each project needs
a file called
sonar-project.properties (unless you are using a Maven module -
but that's for a future post!). The properties file stores some pretty standard stuff:
sonar.projectKey=chip8:python sonar.projectName=Chip8 Python sonar.projectVersion=1.0 sonar.sources=chip8 sonar.tests=test sonar.language=py sonar.sourceEncoding=UTF-8 sonar.python.xunit.reportPath=nosetests.xml sonar.python.coverage.reportPath=coverage.xml sonar.python.coveragePlugin=cobertura
sonar.project* should be fairly self-explanatory. The
tells Sonar Runner where the source files for the project are located. The
sonar.tests property tells Sonar Runner where the unit tests are located.
The last three lines tell the Sonar Runner where to find the coverage reports,
and what format to find them in. Note that both the
property need to point to different sub-directories. If you keep them in the same
directory, you will get errors such as:
ERROR: Error during Sonar runner execution ERROR: Unable to execute Sonar ERROR: Caused by: File [relative=chip8/screen.py, abs=/export/disk1/emulators/python/chip8/chip8/screen.py] can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files
Okay, so with Sonar Runner configured, we need one more tool. Sonar Runner by itself
will not run the unit tests and gather coverage information. We need to use another
nosetests. Nose is an advanced test runner that can easily be
pip. We also need the
coverage tool. These can be installed with:
pip install nose pip install coverage
Once installed, you need to run
nosetests to run the unit tests for you, and
generate information relating to the source code. The following line runs the test
runner, generates coverage information, and generates an XML report that Sonar
Runner will use:
nosetests --with-coverage --cover-package=chip8 --cover-branches --cover-xml
--cover-package option. This restricts the coverage module to the
chip8 directory - without it, every single Python source file will be included
in the coverage report.
With SonarQube, Sonar Runner, and Nose, you are now ready to start inspecting your code. A typical session would be to make some changes to a source file, then run the following:
nosetests --with-coverage --cover-package=chip8 --cover-branches --cover-xml sed -i 's/filename="/filename=".\//g' coverage.xml sonar-runner
nosetests line we have seen before. But what’s with the
sed command? As
I discovered, it's a work-around to making the Sonar Runner properly identify
the file names in the
coverage.xml file. Without it, the Sonar Runner will
discard the coverage metrics for the files in the
chip8 package (see here
for more information and discussion). Finally, the
sonar-runner command will
execute the runner and post the results. Once again visit
to see changes to your project.
As I mentioned before, testing using coverage as the guiding principle isn't the best way to ensure you've tested everything. For a good run-down of why this is the case, check out Ned Batchelder’s talk at PyCon 2009 called Coverage testing, the good and the bad.
As a blunt tool, SonarQube can at least tell you what you're not testing. On the dashboard for the project, you can see metrics related to the unit test coverage and the unit test success:
Clicking on the unit tests coverage report will display the coverage breakdown per module. Clicking on a module will provide coverage details for that file. For example, checking out the CPU code for the Python emulator, we see that the code coverage is 69.6%:
The green lines at the left hand side of the code listing represent lines of code
that have been covered by running the unit test suite. Scrolling down a little in
the code, the functions
not tested by the unit test suite. SonarQube nicely highlights these areas in pink
so that we can quickly see what we're not testing:
Now it's up to me to go back and write tests to ensure that those functions are working as expected. Then, I can re-run the test suite, and perform another SonarQube analysis on the code.
SonarQube and the Sonar Runner provide a simple and effective way to inspect what your unit tests are actually testing with only a few extra packages. This only scratches the surface of what SonarQube can actually do. In a future post, I will examine some of the other SonarQube metrics, and how they can help improve code quality.