In this article I want to analyze how to get a full coverage test report of java code inside our SonarQube instance.

Lest start from the very beginning: what are Unit and Integration tests?

When you develop code you want to be sure that everything runs as expected, both inside your code and when your code is run together with other libraries.

Purpose of Unit Test is to check a small portion of the code (a “unit” as per name) , they are usually thin, quick test which are run every time the code under test is compiled and rely on the same set of library of the code itself

Integration Test on the other side check how the code behave when used together with other libraries, typically deployed on a target application server (If we are testing a web application).
These test are usually much more complicated to run since they need a full application stack to be assembled and take longer to execute since they involve several application layer.

But simply running successful test is not enough to really be sure that our application in acting fine, unless we start analyzing how good our tests are.
Purpose of testing phase is to check that no unexpected behavior arise on code execution, this means that all computational path must be simulated and their outcome compared with expected result.
Having an IF-clause and testing just one  alternative on the condition means that our code in just half way tested, since the other path is not checked at every build.
This means that the coverage of out test on such IF is just 50%.

Coverage is one of the most important metrics to take in account when writing test since it provide developers with immediate feedback on their testing suite.
Automate test coverage is one of the basic feature that SonarQube provide out-of-the box, granting users with a per-line coverage report which detail exactly which lines of code are covered an which one need further testing.

The main problem with default SonarQube analysis is that it provides only Unit Test coverage, while Integration Test even if present and running are ignored, while we would like to have a detail of the coverage of each phase together with overall final coverage.
Adding the Integration test coverage widget on SonarQube interface will simply report a 0% Coverage.

So let’s analyse how to obtain such achievement using Maven

First of all we have to separate our test in two different phases depending on their nature.
Unit test will be run during maven test phase suing Surefire plugin, while Integration will run during integration phase using its twin, the Failsafe plugin.

To identify and separate tests we will usa a naming convention on Class filenames (but you can saparate by package or whaterver you prefer): all integration test must end their name by IntegrationTest, while unit one will not, so

• MyClassTest.java : will be a Unit test for MyClass
• MyClassIntegrationTest.java: wil be its Integration test counterpart

to apply such separation between surefire an failsafe we will rely on inclusion/exclusion rules:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.12.1</version>
  <configuration>
  <excludes>
    <exclude>**/*IntegrationTest*</exclude>
  </excludes>
  </configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<includes>
<include>**/*IntegrationTest*</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>

Triggering your maven build with this configuration will let you  run all unit test at each mvn package build, to be sure all your code is acting as expected, no Integration test will slow you down in this phase.

When you will run a mvn install/deploy or just a mvn verify then also the integration phase will be invoked and all the test, both unit and integration will be performed on a longest but more complete build.

…but sadly still no integration test coverage on your nice SonarQube dashboard.

Here JaCoCo Agent comes into play!
JaCoCo is a free code coverage library for Java used by SonarQube, but the default configuration provides just one agent to be applied during the standard test phase, so we have to add another agent instance to be used during failsafe execution:

<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<!– Prepares a variable, jacoco.agent.ut.arg, that contains the info
to be passed to the JVM hosting the code being tested. –>
<execution>
<id>prepare-ut-agent</id>
<phase>process-test-classes</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.reportPath}</destFile>
<propertyName>jacoco.agent.ut.arg</propertyName>
<append>true</append>
</configuration>
</execution>
<!– Prepares a variable, jacoco.agent.it.arg, that contains the info
to be passed to the JVM hosting the code being tested. –>
<execution>
<id>prepare-it-agent</id>
<!– moved to package phase to be sure all pre-integration test have
it already set to bring up environments with jacoco agent as JVM params –>
<phase>package</phase>
<goals>
<goal>prepare-agent-integration</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.itReportPath}</destFile>
<propertyName>jacoco.agent.it.arg</propertyName>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>

and here a bunch of useful properties to be added to our project to have it act properly:

<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>

<jacoco.version>0.7.1.201405082137</jacoco.version>
<jacoco.outputDir>${project.build.directory}/jacoco</jacoco.outputDir>
<jacoco.out.ut.file>jacoco-ut.exec</jacoco.out.ut.file>
<jacoco.out.it.file>jacoco-it.exec</jacoco.out.it.file>
<sonar.jacoco.reportPath>${jacoco.outputDir}/${jacoco.out.ut.file}</sonar.jacoco.reportPath>
<sonar.jacoco.itReportPath>${jacoco.outputDir}/${jacoco.out.it.file}</sonar.jacoco.itReportPath>

let’s give them a closer look: first two properties instruct SonarQube about the fact that we are about to use JaCoCo agent, and that all already run test report (usually sonar phase is run as a second step of our build, when surefire and failsafe have already produce their report) must be reused instead of being evaluated again.
The second group of properties simply define name and path for both UT and IT report (jacoco ouput file are binaries).

As stated into the plugin configuration JaCoCo agent configuration string is created twice and stored in 2 different variables: ${jacoco.agent.ut.arg} for Unit Tests and ${jacoco.agent.it.arg} for Integration ones.

So all we still need to have our report in place is to pass such strings to our plugins (we add also some useful memory enhancement to show how to integrate parameters with other JVM args :

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.12.1</version>
  <configuration>
<argLine>-Xmx1G -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled ${jacoco.agent.ut.arg}</argLine>
  <excludes>
    <exclude>**/*IntegrationTest*</exclude>
  </excludes>
  </configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<argLine>-Xmx1G -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled ${jacoco.agent.it.arg}</argLine>
<includes>
<includes>
<include>**/*IntegrationTest*</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>

This time everything is in the right place, if we run a mvn clean verify you will notice that the two JaCoCo agent will be initialized, and the proper JVM argument string will be added to tests execution in order to inject and activate the agent during test execution.
Two different binary report files will be produced and SonarQuebe will correctly parse them during analyse phase.

And finally… open you project dashboard to get your final prize!
A wonderful, populated Integration Test Coverage widged with an additional Overall Coverage value as a bonus gift!

I think most of you would be happy up to this, but for the more Maven addicted users I add just a couple of additional info.

First of all, I expect you to work with a Hierarchical Maven structure so you can add all the configuration and plugins of this article into your parent pom and have them inherited automatically by all project.

But, if you want to manage properly SonarQube analysis over both multi-module and single-module you have to take care to add the following properties only to multi-module project, inside their parent pom

<jacoco.outputDir>../target/jacoco</jacoco.outputDir>

Doing so Jacoco will produce a single UT and IT report file for all the modules (did you notice the append = true configuration in plugins ? ) allowing SonarQube to properly import them at once.
Without such propery all sub-modules analysis would be skipped by SonarQube (even if ut an it file are present) and so would not show into the widget.

If, on the other hand you have some sub-module you want to skip during analysis (maybe since it’s containing code not related to test coverage), you can simply include their name into this property, comma separated:

<sonar.skippedModules>mySikppedModule1,mySkippedModule2</sonar.skippedModules>

ok, now it’ really all!
Enjoy your superb coverage report and start writing even better tests… you will be grateful to yourself one day! 😉