This short article explains how code coverage can be achieved using the JaCoCo plugin for Maven. The idea behind code coverage tools, such as JaCoCo is to find out which parts of the source code have been tested by keeping track of all the lines of code executed during a given test.
Introduction
JaCoCo (Java Code Coverage) offers both line and branch coverage. As opposed to other code coverage tools (like Cobertura for example), JaCoCo supports on the-fly-code instrumentation. This, it does by running a Java agent which instruments and monitors code execution. It can also be configured to store the collected data either locally in a file, or remotely through TCP. In addition, report data files from multiple runs can be easily merged.
The latest version (> 0.6.4.201312101107) of the JaCoCo plugin for Maven which is compatible with Java 7, provides two new goals, aimed at providing code coverage for integration tests as well. Using these Maven goals, one can easily generate two separate code coverage reports, one for unit testing, and one for integration testing.
Separating unit and integration test code coverage is clearly desirable, mainly since both tests tackle different aspects of system testing. Whereas plain and simple unit testing tackles the issue of code correctness, integration testing tackles the issue of whether or not a system test from start to end is successful. As such, integration testing does not exclusively focus on class implementation details, like say, making sure that all branching execution paths are covered, but rather focuses on the behavior of the system as a whole. Conversely, unit testing focuses only on the class implementation details, where usually asserts or mocks are employed to make sure that every branching execution path is covered.
Thus, it follows that if unit and integration test coverage reports are separated, untested code can be easily spotted, since now, one can exactly know what is being tested. More importantly, there is no risk of an integration test inadvertently affecting the general coverage result of a unit test, as holes in a unit test are easier to detect. If that is the case, the unit test in question can be beefed up accordingly, to cover those code sections which were just being targeted by integration tests.
Code coverage with Jenkins and Sonar
Jenkins is a Continuous Integration server which continually and repeatedly puts the source code through build cycles based on some criteria, such as for example, when it detects changes in the SCM after a commit has occurred. Sonar on the other hand, is a Continuous Inspection platform, usually set up in such a way that it constantly monitors code quality by processing results emitted from a Jenkins build to provide code metrics. This includes code coverage, violations and duplicated code instance reports. Commonly, Sonar is configured as a Jenkins post-build action, meaning that it is run only after Jenkins completes the build process. Yet if desired, it can also be configured to run as a Maven plugin, within the build process itself.
When configured as a Jenkins post-build action, code coverage reports can be fed to Sonar in two different ways:
Using its embedded engine to let Sonar launch the unit test code coverage execution directly.
Reusing existing reports, previously generated by external tools such as Maven and Ant.
This article opts to using the second method, and feed Sonar code coverage reports previously generated through a Maven-Jenkins build. The advantage of this method over the first is that it allows developers run code coverage tests manually, without invoking Sonar or requiring a Sonar installation locally. Instead, the JaCoCo Maven plugin is used to generate code coverage reports during a build. This considerably speeds up the testing process, as no interaction with Sonar is required. Moreover, code coverage reports generated by JaCoCo during a Maven build can be readily imported into Eclipse through the EclEmma plugin.
Unit tests vs. Integration tests
We already mentioned that the latest version of JaCoCo provides specialized goals to cover integration tests, separately from unit tests. Similarly, so does Maven distinguish between the two, and it provides two different plugins for running unit and integration tests through the Surefire and Failsafe plugins respectively.
The Surefire plugin is designed to run unit tests, and is by default bound to the test phase of the Maven build lifecycle. The Failsafe plugin on the other hand, is designed to run integration tests and should be used during the integration-test and verify phases of the Maven build lifecycle. Both of these plugins generate test reports, detailing which tests were successful, which failed, the time took to run a test suite, and the like.
"What makes these two seemingly similar plugins different is the following. If the Surefire plugin is used to run integration tests, and a test failure occurs, the build will stop at the integration-test phase, causing the build to fail entirely. This is clearly a disadvantage, since during an integration test, systems on which a given test depends might go out of service, causing the build to fail unnecessarily. The Failsafe plugin deals with this problem by not failing during the integration-test phase, enabling the post-integration-test phase to be executed.
With this in mind, we can easily decide when to use one and not the other. More detailed information on these two plugins can be found in the Maven Surefire Testing Framework project. In this article, a standard default plugin configuration is preferred over a custom one, as the resulting POM is smaller and less cluttered.
The means that for Surefire, tests matching these wildcard patterns will be automatically included:
**/Test*.java
Includes all of its sub-directories and all java file names that start with "Test".
**/*Test.java
Includes all of its sub-directories and all java file names that end with "Test".
**/*TestCase.java
Includes all of its sub-directories and all java file names that end with "TestCase".
For Failsafe, the following apply:
**/IT*.java
Includes all of its sub-directories and all java file names that start with "IT".
**/*IT.java
Includes all of its sub-directories and all java file names that end with "IT".
**/*ITCase.java
Includes all of its sub-directories and all java file names that end with "ITCase".
Surefire vs. JaCoCo
Where does JaCoCo come in, and how does it differ from Surefire? What needs to be understood is that these Maven plugins interact with test classes differently. While the job of Surefire (or Failsafe) is to run tests in a JVM, JaCoCo latches itself onto this JVM, and through its Java agent, is able to instrument and monitor.
This means that JaCoCo cannot function on its own, but needs to instrument code that is already being executed by a plugin such as Surefire. Also, since JaCoCo is instrumenting code on-the-fly, as mentioned earlier, it targets Java classes and the byte code contained within, where code coverage is calculated in terms of JVM byte code instructions, and not source lines. Only when the final JaCoCo report is generated are the actual Java source files consulted, so that proper line coverage statistics are calculated. On top of that, for the JaCoCo final report to include line numbers as well to show highlighted source code, compilation must be done with debugging symbols enabled, otherwise line information not available to the JaCoCo agent.
When running as a Maven plugin, the JaCoCo agent configuration is prepared by invoking the prepare-agent or prepare-agent-integration goals, before the actual tests are run. This sets a property named argLine which points to the JaCoCo agent, later passed as a JVM argument to the test runner. Once the test runner creates and starts the JVM with argLine, the JaCoCo agent configuration is already in place, and code instrumentation is performed transparently. As soon as the JVM process terminates, the JaCoCo agent writes the code coverage statistics to a file. Depending on which agent preparation goal was invoked, the file created is named jacoco.exec for unit tests, and jacoco-it.exec for integration tests by default. Both of these binary files are written in the target directory.
The JaCoCo code coverage data files can then be used to generate a final report, including all the execution statistics such as (byte code) instruction coverage, branch coverage and line coverage, together with the actual source code lines highlighted accordingly. These user-friendly reports can be generated using the report and report-integration goals, depending on whether unit test or integration test reports are required. Reports are available in HTML, XML and CSV files.
Preparing the POM for code coverage
Now that the basic Maven plugins have been introduced, we will see how a typical POM can be configured so that it runs unit and integration tests, and in doing so, generating separate code coverage reports for each. Firstly, the JaCoCo plugin needs to be configured.
Configuring JaCoCo
As explained above, for JaCoCo to be able to instrument code, the agent must be prepared prior to the execution of tests. The configuration prepares two agents, one for instrumenting unit tests, and the other for instrumenting integration tests. Integration test-related JaCoCo goals are only available in versions 0.6.4.201312101107 and higher. Apart from preparing these agents, we will also configure the JaCoCo plugin so that code coverage reports are produced automatically during the Maven build. These reports are by default created in the target/site/jacoco directory.
Configuring this plugin is done as follows:
where maven.jacoco.plugin.version is equal to 0.6.4.201312101107. The prepare-agent goal used for unit tests is bound by default to the initialize lifecycle phase, while the prepare-agent-integration goal is bound to the pre-integration-test lifecycle phase. Both report generation goals are bound to the verify lifecycle phase.
Configuring Surefire and Failsafe
Once the configuration of JaCoCo is in place, the test runners can be configured next as shown below:
where both maven.surefire.plugin.version and maven.failsafe.plugin.version are equal to 2.8.1. The test goal of the Surefire plugin is bound by default to the test lifecycle phase, while the integration-test and verify goals of the Failsafe plugins are bound to the integration-test and verify lifecycle phases respectively.
To summarize, we will list the relevant Maven lifecycle phases, and what actually occurs at each phase as it is executed.
initialize: The JaCoCo agent configuration for unit tests is set up, with the argLine property set to an agent configuration similar to:
test: The Surefire plugin launches unit tests in separate JVMs, according to the defined test inclusion patterns "**/Test*.java", "**/*Test.java" and "**/*TestCase.java". Surefire applies the argLine property to each JVM, which conveniently was set during up during the initialize lifecycle phase by JaCoCo. As soon as the JVM terminates, a corresponding JaCoCo jacoco.exec data file is written, one for each Maven module.
package: Packaging of the compiled source code and related resources occurs.
pre-integration-test: The JaCoCo agent configuration for integration tests is set up, with the argLine property set to an agent configuration similar to:
integration-test: The Failsafe plugin launches integration tests in separate JVMs, according to the defined test inclusion patterns "**/IT*.java", "**/*IT.java" and "**/*ITCase.java". Failsafe applies the argLine property to each JVM, which conveniently was set during up during the pre-integration-test lifecycle phase by JaCoCo. As soon as the JVM terminates, a corresponding JaCoCo jacoco-it.exec data file is written, one for each Maven module.
verify: The JaCoCo report and report-integration goals are run, processing the jacoco.exec and jacoco-it.exec data files respectively. The code coverage report for both is generated in the project's target directory.
Code coverage reports can be easily generated either by running a full Maven build (i.e. mvn clean install), or else by manually executing the JaCoCo report and report-integration goals, providing the jacoco.exec and jacoco-it.exec are available. If not, a full Maven build must be issued.
Before concluding this section, one very important thing must be noted. Recall that the Surefire and Failsafe plugins both use the argLine property as a means of allowing other plugins or even the user to start the testing JVM with extra arguments. The JaCoCo plugin uses this fact to its advantage, and silently sets this argLine property with its agent configuration. Only through this property is JaCoCo able to start. If somehow this property was overwritten, Surefire and Failsafe would commence executing tests without running the JaCoCo agent.
There might be however certain cases where extra command line arguments must be added to the JVM started up by Surefire or Failsafe. If this is the case, the Surefire/Failsafe argLine parameter must be used in conjunction with the ${argLine} property, otherwise the agent will not start. This is shown below, and applies to both Surefire and Failsafe. Note that at the time of writing, this scheme only works with versions less than 0.6.4.201312101107.
Finally, if tests are to be skipped, the Maven project should be run with -DskipUnitTests or -DskipIntegrationTests. To skip both, the -DskipTests flag can be used. As shown in the configuration of POM snippets above, we tend to use defaults provided by all plugins. This results in a leaner and cleaner POM, based on the standard behavior documented on each of the plugins' websites.
Configuring Sonar as a Jenkins post-build action
Earlier above, we saw how the JaCoCo report and report-integration goals are used in conjunction with the jacoco.exec, jacoco-it.exec and application source files to produce a legible report detailing code coverage statistics. JaCoCo reporting is but one tool which is able to process these data files. Sonar is another tool which is able to parse such files to generate a report of its own, providing it is configured correctly.
In this article, Sonar is configured on Jenkins as a post-build action. It is therefore assumed that both Jenkins and its Sonar plugin are correctly installed. It is also assumed that the Sonar Jenkins plugin is correctly configured with a database such as MySQL, and that an active Sonar server is refers to this database.
To set up the Sonar Jenkins plugin to process already generated JaCoCo data files, the following properties must be specified in the "Additional properties" text box, under the Sonar installation being configured:
sonar.dynamicAnalysis=reuseReports
Configures Sonar to rely on reports previously generated externally. In our case, these reports (jacoco.exec and jacoco-it.exec) are generated during the Maven build preceding the Sonar post-build action.
sonar.skipDesign=true
Skips the computation of design metrics and dependencies.
sonar.java.coveragePlugin=jacoco
Sets the code coverage engine being used to JaCoCo.
sonar.jacoco.reportPath=target/jacoco.exec
Sets the location of the unit test JaCoCo report file.
sonar.jacoco.itReportPath=target/jacoco-it.exec
Sets the location of the integration test JaCoCo report file.
Sets the location of the Surefire reports directory. Failsafe is not supported yet, as these are not pushed nor displayed in Sonar.
These properties can also be specified in an application POM, or a common super POM if desired. However, the reason these are specified in the Sonar Jenkins plugin configuration is twofold. Firstly, these would be presumably standardized and adopted throughout the whole of the organization. Secondly, removing such properties from a POM will make life easier for the developer, as these details are hidden away, not to mention that a typical developer should not be interested in whether Sonar is being used or not. Also, the POM would be decoupled from the reporting mechanism, since Sonar-related properties are configured elsewhere.
This does not mean that the developer cannot override these properties, but doing so will risk breaking the report generation. That said, Sonar properties can be manipulated, overridden and also added to the application's POM. Such properties should only be added only when really needed, or when they are specific to the application under question.
Two such properties are:
sonar.exclusions: Excludes one or more files or directories from the Sonar analysis. Multiple comma-separated entries are supported.
sonar.branch: Sets a named SCM branch. If named differently, two branches of the same project are still considered as being different Sonar projects.
These properties can either be specified in a Maven profile or simply included in the POM's list of project properties. For a complete list of a Sonar analysis properties (a.k.a. parameters), please refer to the Sonar Analysis Parameters wiki.
Finally, Sonar can also be invoked manually from the command line through its Maven plugin.
This will completely bypass the Jenkins Sonar plugin, and instead interact directly with the Sonar server and its database:
Once all the unit and integration test code coverage data is interpreted and stored in the Sonar database, project statistics can be made available by adding the "Integration Test Cover" widget to the its dashboard. As usual, drill-down is available from this widget. Also at file level, detailed code coverage information split by unit and integration test can be accessed from the "Coverage" tab, as shown below.
If we were to compare the statistics on the code coverage reports generated by the Sonar with those generated by the jacoco:report goal, we would immediately note that the percentages vary. This difference is due to the fact that despite both tools consume the same jacoco.exec/jacoco-it.exec file, coverage statistics are computed differently. The jacoco:report plugin bases its percentages on simple proportional calculations, while Sonar employs a subtle mix of line and branch coverage. On a closer look however, both JaCoCo and Sonar report the same figures in terms of line and branch coverage count (compare green and blue markings in figures below); it is only the percentage calculations that vary.
For a detailed description on the actual formulas used by Sonar to compute code coverage figures, refer to the Sonar Metric Definitions wiki. Recent versions of Sonar even combine the code coverage results of unit and integration tests by using the sonar.jacoco.reportPath and sonar.jacoco.itReportPath properties, in order to come up with an overall code coverage figure.
Conclusion
This article introduced the notion of code coverage and how it can be achieved in Maven using the Surefire and JaCoCo plugins. It then moved on to describing the differences between unit and integration tests, and proposed a simple scheme with which unit and integration test code coverage reports can be separated by JaCoCo when generated, and later by Sonar when consumed.
Stay up to date with the latest ecommerce, payment processing and online business tips from our industry experts.
For a detailed description on the actual formulas used by Sonar to compute code coverage figures, refer to the Sonar Metric Definitions wiki. Recent versions of Sonar even combine the code coverage results of unit and integration tests by using the sonar.jacoco.reportPath and sonar.jacoco.itReportPath properties, in order to come up with an overall code coverage figure.
For a detailed description on the actual formulas used by Sonar to compute code coverage figures, refer to the Sonar Metric Definitions wiki. Recent versions of Sonar even combine the code coverage results of unit and integration tests by using the sonar.jacoco.reportPath and sonar.jacoco.itReportPath properties, in order to come up with an overall code coverage figure.