This short article explains how code coverage can be achieved using the JaCoCo plugin for Maven. The idea behind code coverage tools, such as JaCoCo is to find out which parts of the source code have been tested by keeping track of all the lines of code executed during a given test.
IntroductionJaCoCo (Java Code Coverage) offers both line and branch coverage. As opposed to other code coverage tools (like Cobertura for example), JaCoCo supports on the-fly-code instrumentation. This, it does by running a Java agent which instruments and monitors code execution. It can also be configured to store the collected data either locally in a file, or remotely through TCP. In addition, report data files from multiple runs can be easily merged. The latest version (> 0.6.4.201312101107) of the JaCoCo plugin for Maven which is compatible with Java 7, provides two new goals, aimed at providing code coverage for integration tests as well. Using these Maven goals, one can easily generate two separate code coverage reports, one for unit testing, and one for integration testing. Separating unit and integration test code coverage is clearly desirable, mainly since both tests tackle different aspects of system testing. Whereas plain and simple unit testing tackles the issue of code correctness, integration testing tackles the issue of whether or not a system test from start to end is successful. As such, integration testing does not exclusively focus on class implementation details, like say, making sure that all branching execution paths are covered, but rather focuses on the behavior of the system as a whole. Conversely, unit testing focuses only on the class implementation details, where usually asserts or mocks are employed to make sure that every branching execution path is covered. Thus, it follows that if unit and integration test coverage reports are separated, untested code can be easily spotted, since now, one can exactly know what is being tested. More importantly, there is no risk of an integration test inadvertently affecting the general coverage result of a unit test, as holes in a unit test are easier to detect. If that is the case, the unit test in question can be beefed up accordingly, to cover those code sections which were just being targeted by integration tests.
Code coverage with Jenkins and SonarJenkins is a Continuous Integration server which continually and repeatedly puts the source code through build cycles based on some criteria, such as for example, when it detects changes in the SCM after a commit has occurred. Sonar on the other hand, is a Continuous Inspection platform, usually set up in such a way that it constantly monitors code quality by processing results emitted from a Jenkins build to provide code metrics. This includes code coverage, violations and duplicated code instance reports. Commonly, Sonar is configured as a Jenkins post-build action, meaning that it is run only after Jenkins completes the build process. Yet if desired, it can also be configured to run as a Maven plugin, within the build process itself. When configured as a Jenkins post-build action, code coverage reports can be fed to Sonar in two different ways:
- Using its embedded engine to let Sonar launch the unit test code coverage execution directly.
- Reusing existing reports, previously generated by external tools such as Maven and Ant.
Unit tests vs. Integration testsWe already mentioned that the latest version of JaCoCo provides specialized goals to cover integration tests, separately from unit tests. Similarly, so does Maven distinguish between the two, and it provides two different plugins for running unit and integration tests through the Surefire and Failsafe plugins respectively. The Surefire plugin is designed to run unit tests, and is by default bound to the test phase of the Maven build lifecycle. The Failsafe plugin on the other hand, is designed to run integration tests and should be used during the integration-test and verify phases of the Maven build lifecycle. Both of these plugins generate test reports, detailing which tests were successful, which failed, the time took to run a test suite, and the like.
"What makes these two seemingly similar plugins different is the following. If the Surefire plugin is used to run integration tests, and a test failure occurs, the build will stop at the integration-test phase, causing the build to fail entirely. This is clearly a disadvantage, since during an integration test, systems on which a given test depends might go out of service, causing the build to fail unnecessarily. The Failsafe plugin deals with this problem by not failing during the integration-test phase, enabling the post-integration-test phase to be executed.With this in mind, we can easily decide when to use one and not the other. More detailed information on these two plugins can be found in the Maven Surefire Testing Framework project. In this article, a standard default plugin configuration is preferred over a custom one, as the resulting POM is smaller and less cluttered. The means that for Surefire, tests matching these wildcard patterns will be automatically included:
Includes all of its sub-directories and all java file names that start with "Test".
Includes all of its sub-directories and all java file names that end with "Test".
Includes all of its sub-directories and all java file names that end with "TestCase". For Failsafe, the following apply:
Includes all of its sub-directories and all java file names that start with "IT".
Includes all of its sub-directories and all java file names that end with "IT".
Includes all of its sub-directories and all java file names that end with "ITCase".
Surefire vs. JaCoCoWhere does JaCoCo come in, and how does it differ from Surefire? What needs to be understood is that these Maven plugins interact with test classes differently. While the job of Surefire (or Failsafe) is to run tests in a JVM, JaCoCo latches itself onto this JVM, and through its Java agent, is able to instrument and monitor. This means that JaCoCo cannot function on its own, but needs to instrument code that is already being executed by a plugin such as Surefire. Also, since JaCoCo is instrumenting code on-the-fly, as mentioned earlier, it targets Java classes and the byte code contained within, where code coverage is calculated in terms of JVM byte code instructions, and not source lines. Only when the final JaCoCo report is generated are the actual Java source files consulted, so that proper line coverage statistics are calculated. On top of that, for the JaCoCo final report to include line numbers as well to show highlighted source code, compilation must be done with debugging symbols enabled, otherwise line information not available to the JaCoCo agent. When running as a Maven plugin, the JaCoCo agent configuration is prepared by invoking the prepare-agent or prepare-agent-integration goals, before the actual tests are run. This sets a property named argLine which points to the JaCoCo agent, later passed as a JVM argument to the test runner. Once the test runner creates and starts the JVM with argLine, the JaCoCo agent configuration is already in place, and code instrumentation is performed transparently. As soon as the JVM process terminates, the JaCoCo agent writes the code coverage statistics to a file. Depending on which agent preparation goal was invoked, the file created is named jacoco.exec for unit tests, and jacoco-it.exec for integration tests by default. Both of these binary files are written in the target directory. The JaCoCo code coverage data files can then be used to generate a final report, including all the execution statistics such as (byte code) instruction coverage, branch coverage and line coverage, together with the actual source code lines highlighted accordingly. These user-friendly reports can be generated using the report and report-integration goals, depending on whether unit test or integration test reports are required. Reports are available in HTML, XML and CSV files.
Preparing the POM for code coverageNow that the basic Maven plugins have been introduced, we will see how a typical POM can be configured so that it runs unit and integration tests, and in doing so, generating separate code coverage reports for each. Firstly, the JaCoCo plugin needs to be configured.
Configuring JaCoCoAs explained above, for JaCoCo to be able to instrument code, the agent must be prepared prior to the execution of tests. The configuration prepares two agents, one for instrumenting unit tests, and the other for instrumenting integration tests. Integration test-related JaCoCo goals are only available in versions 0.6.4.201312101107 and higher. Apart from preparing these agents, we will also configure the JaCoCo plugin so that code coverage reports are produced automatically during the Maven build. These reports are by default created in the target/site/jacoco directory. Configuring this plugin is done as follows: where maven.jacoco.plugin.version is equal to 0.6.4.201312101107. The prepare-agent goal used for unit tests is bound by default to the initialize lifecycle phase, while the prepare-agent-integration goal is bound to the pre-integration-test lifecycle phase. Both report generation goals are bound to the verify lifecycle phase.
Configuring Surefire and FailsafeOnce the configuration of JaCoCo is in place, the test runners can be configured next as shown below: where both maven.surefire.plugin.version and maven.failsafe.plugin.version are equal to 2.8.1. The test goal of the Surefire plugin is bound by default to the test lifecycle phase, while the integration-test and verify goals of the Failsafe plugins are bound to the integration-test and verify lifecycle phases respectively. To summarize, we will list the relevant Maven lifecycle phases, and what actually occurs at each phase as it is executed.
- initialize: The JaCoCo agent configuration for unit tests is set up, with the argLine property set to an agent configuration similar to:
- compile: Compilation of the source code occurs.
- test: The Surefire plugin launches unit tests in separate JVMs, according to the defined test inclusion patterns "**/Test*.java", "**/*Test.java" and "**/*TestCase.java". Surefire applies the argLine property to each JVM, which conveniently was set during up during the initialize lifecycle phase by JaCoCo. As soon as the JVM terminates, a corresponding JaCoCo jacoco.exec data file is written, one for each Maven module.
- package: Packaging of the compiled source code and related resources occurs.
- pre-integration-test: The JaCoCo agent configuration for integration tests is set up, with the argLine property set to an agent configuration similar to:
- integration-test: The Failsafe plugin launches integration tests in separate JVMs, according to the defined test inclusion patterns "**/IT*.java", "**/*IT.java" and "**/*ITCase.java". Failsafe applies the argLine property to each JVM, which conveniently was set during up during the pre-integration-test lifecycle phase by JaCoCo. As soon as the JVM terminates, a corresponding JaCoCo jacoco-it.exec data file is written, one for each Maven module.
- verify: The JaCoCo report and report-integration goals are run, processing the jacoco.exec and jacoco-it.exec data files respectively. The code coverage report for both is generated in the project's target directory.
Configuring Sonar as a Jenkins post-build actionEarlier above, we saw how the JaCoCo report and report-integration goals are used in conjunction with the jacoco.exec, jacoco-it.exec and application source files to produce a legible report detailing code coverage statistics. JaCoCo reporting is but one tool which is able to process these data files. Sonar is another tool which is able to parse such files to generate a report of its own, providing it is configured correctly. In this article, Sonar is configured on Jenkins as a post-build action. It is therefore assumed that both Jenkins and its Sonar plugin are correctly installed. It is also assumed that the Sonar Jenkins plugin is correctly configured with a database such as MySQL, and that an active Sonar server is refers to this database. To set up the Sonar Jenkins plugin to process already generated JaCoCo data files, the following properties must be specified in the "Additional properties" text box, under the Sonar installation being configured:
Configures Sonar to rely on reports previously generated externally. In our case, these reports (jacoco.exec and jacoco-it.exec) are generated during the Maven build preceding the Sonar post-build action.
Skips the computation of design metrics and dependencies.
Sets the code coverage engine being used to JaCoCo.
Sets the location of the unit test JaCoCo report file.
Sets the location of the integration test JaCoCo report file.
Sets the location of the Surefire reports directory. Failsafe is not supported yet, as these are not pushed nor displayed in Sonar.
- sonar.exclusions: Excludes one or more files or directories from the Sonar analysis. Multiple comma-separated entries are supported.
- sonar.branch: Sets a named SCM branch. If named differently, two branches of the same project are still considered as being different Sonar projects.
mvn clean install sonar:sonar -Plocalhost -Dsonar.host.url=http://localhost:9000 -Dsonar.jdbc.url=jdbc:mysql://localhost:3306/sonar -Dsonar.jdbc.username=root -Dsonar.jdbc.password=abcdefgh -Dsonar.dynamicAnalysis=reuseReports -Dsonar.skipDesign=true -Dsonar.java.coveragePlugin=jacoco -Dsonar.jacoco.reportPath=target/jacoco.exec -Dsonar.jacoco.itReportPath=target/jacoco-it.exec -Dsonar.surefire.reportsPath=target/surefire-reports