Integrating OpenCover Test Coverage Statistics with TeamCity
Originally published at https://www.fabienruffin.com on June 9, 2015.
A bit over a year ago I started revamping our build pipeline at work and moved from a 100% MSBuild process to mostly psake. We run everything from a single psake script, including cleanup, build, packaging, unit tests, unit test coverage and report generation. The main goal with this process was to produce a completely self-contained build process so that any developer in the team could just run the full build as it would run on the build server. This is great and it helps a lot when trying to troubleshoot a build issue. However, one of the drawback is that we can’t use any of our build server’s integrations to run our tests and test coverage and import the results. The main reason I really wanted this integration to work is that we could then fail builds when the test coverage drops. It would also make the coverage report more readily accessible for the team to look at when needed (they were previously only included as one of our build artifacts, meaning they would mostly get ignored)
We recently moved to TeamCity and importing our NUnit test results to the system was quite easy thanks to the XML report processing feature.
However, there still doesn’t seem to be an easy way to import our test coverage statistics from the OpenCover reports. OpenCover is great but the tooling support for it still leaves much to be improved. Well, that was until I found out almost by accident that it is possible to import these statistics into a build results by simple writing messages into the logs. This mechanism is called “service messages” and is available in the TeamCity online documentation.
They basically look like this:
##teamcity[buildStatisticValue key='{statisticsKey}' value='{statisticsValue}']The list of valid key you can use to report build statistics is available there: https://confluence.jetbrains.com/display/TCD9/Custom+Chart#CustomChart-DefaultStatisticsValuesProvidedbyTeamCity
For reporting test coverage statistics you just need the following:
I tend to prefer statement coverage to line coverage as one statement can be written over multiple lines for readability reasons. Therefore I decided to exclude line coverage from my build stats.
So what I was trying to achieve is to write something like this to the logs (this is a test project, so ignore the low coverage, this is normal):
##teamcity[buildStatisticValue key='CodeCoverageC' value='39.6']
##teamcity[buildStatisticValue key='CodeCoverageAbsCCovered' value='53']
##teamcity[buildStatisticValue key='CodeCoverageAbsCTotal' value='134'] ##teamcity[buildStatisticValue key='CodeCoverageM' value='37.5']
##teamcity[buildStatisticValue key='CodeCoverageAbsMCovered' value='151']
##teamcity[buildStatisticValue key='CodeCoverageAbsMTotal' value='403'] ##teamcity[buildStatisticValue key='CodeCoverageS' value='33.4']
##teamcity[buildStatisticValue key='CodeCoverageAbsSCovered' value='625']
##teamcity[buildStatisticValue key='CodeCoverageAbsSTotal' value='1872'] ##teamcity[buildStatisticValue key='CodeCoverageB' value='24.2']
##teamcity[buildStatisticValue key='CodeCoverageAbsBCovered' value='332']
##teamcity[buildStatisticValue key='CodeCoverageAbsBTotal' value='1372']
So now I need to edit my psake script to somehow parse the results. There are a few options here, and I initially tried to parse the OpenCover XML reports. This wasn’t really fun though as most of the values I needed were not readily available in the report and needed to be calculated by parsing the entire report and calculating them.
Then I realised that all I wanted was just there in front of me. Looking at my build logs, here is what I found:
....................N.N.N.N.N.N................
Tests run: 35, Errors: 0, Failures: 0, Inconclusive: 0, Time: 5.383586192 seconds
Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0 Committing...
Visited Classes 53 of 134 (39.55)
Visited Methods 151 of 403 (37.47)
Visited Points 625 of 1872 (33.39)
Visited Branches 332 of 1372 (24.20) ==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 54 of 161 (33.54)
Alternative Visited Methods 177 of 498 (35.54)
It turns out OpenCover is already writing out a summary to the logs, which contains all the information I needed. So I decided to take advantage of that and parse the output instead of parsing the coverage report. So back in my psake script, I am now storing the stdout from OpenCover in a variable (using | Out-String) instead of letting it write to the host directly:
Note that I am still writing out OpenCover’s stdout to the host so we still have human-readable output in the logs.
The last part was to write the “Export-TeamCityCoverage” function to parse the string and extract the coverage statistics:
There is a lot that can be improved with this script, but it does the job.
After running my psake script from TeamCity I can now verify that the code coverage statistics have been found, and indeed they have:
Another option would be to write a plugin for this, in the same fashion the “XML Report Processing” plugin parses my NUnit reports, but that would get much more complicated very quickly.
If you have another solution to publish your code coverage statistics to TeamCity, please share in the comments!
