Code coverage aggregation

[daviddoria]

Is there a way to aggregate the results of the coverage analysis? That is, it looks like CodeCoverage.cmake is a per-target (one per add_test()?) function. Say I have 10 subdirectories each with their own set of add_test() calls - how do I see the result as a coverage percentage for the whole project?

[bilke]

Yes, the coverage analysis is done per-target. What I do is to create a custom target which runs ctest:

add_custom_target(ctest COMMAND ${CMAKE_CTEST_COMMAND})

Then I set this up as a coverage target:

SETUP_TARGET_FOR_COVERAGE_COBERTURA(ctest_coverage_cobertura ctest "ctest_coverage_cobertura_results" "-j;${PROCESSOR_COUNT}")

This will produce an aggregated coverage report of everything which is part of the ctest-run.

Additionally I setup more coverage targets (e.g. GoogleTest runner) which produce each their own coverage report and at the end I aggregate them in Jenkins with the Cobertura Code Coverage Publisher.

[daviddoria]

Thanks, I'll try this today. I guess I'm surprised that the coverage analyzer is able to gather the output of multiple executables just because they are launched by the same executable.

Did I understand correctly that you use both CTest and GTest in the same project? How do you decide what to do in each testing system?

[bilke]

We use GTest for unit tests and CTest for end-to-end (executable) testing.

[daviddoria]

Hm, I must be doing something wrong. Here is my CMakeLists.txt:

Project(CoverageExampleProject)
include(CTest)

set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_MODULE_PATH})
INCLUDE(CodeCoverage)

SET(CMAKE_CXX_FLAGS="-g -O0 --coverage")
SET(CMAKE_C_FLAGS="-g -O0 --coverage")
SET(CMAKE_EXE_LINKER_FLAGS="--coverage")

add_executable(CoverageExample CoverageExample.cpp)
target_link_libraries(CoverageExample gcov)

add_test(CoverageExampleTest CoverageExample)

add_custom_target(ctestTarget COMMAND ${CMAKE_CTEST_COMMAND})

SETUP_TARGET_FOR_COVERAGE(coverageTarget ctestTarget "testResults")

and the output:

~/build/Examples/c++/CoverageExample2 $ make coverageTarget
-- Configuring done
-- Generating done
-- Build files have been written to: /home/doria/build/Examples/c++/CoverageExample2
[100%] Resetting code coverage counters to zero.
Processing code coverage counters and generating report.
Deleting all .da files in . and subdirectories
Done.
make[3]: ctestTarget: Command not found
make[3]: *** [CMakeFiles/coverageTarget] Error 127
make[2]: *** [CMakeFiles/coverageTarget.dir/all] Error 2
make[1]: *** [CMakeFiles/coverageTarget.dir/rule] Error 2
make: *** [coverageTarget] Error 2

Note that 'make ctestTarget' seems to work fine (it runs ctest). Any suggestions?

[bilke]

Ah thanks, this is a bug:

The script assumes that the second parameter is a valid target as well as a binary located in CMAKE_BINARY_DIR:

SETUP_TARGET_FOR_COVERAGE(coverageTarget ctestTarget "testResults")

Obviously ctestTarget is a target but not a binary. I have not experienced that because my target is simply called ctest which also is not a binary created from the project but a global binary from CMake and therefore it worked but not as intended. I will think about a solution tomorrow ...

[daviddoria]

Makes sense. Let me know if you'd like me to test a patch.

For the time being, I changed to :

add_custom_target(ctest COMMAND ${CMAKE_CTEST_COMMAND})
SETUP_TARGET_FOR_COVERAGE(coverageTarget ctest "testResults")

but I get:

~/build/Examples/c++/CoverageExample2 $ make coverageTarget
[100%] Resetting code coverage counters to zero.
Processing code coverage counters and generating report.
Deleting all .da files in . and subdirectories
Done.
Test project /home/doria/build/Examples/c++/CoverageExample2
    Start 1: CoverageExampleTest
1/1 Test #1: CoverageExampleTest ..............   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.00 sec
Capturing coverage data from .
Found gcov version: 4.8.4
Scanning . for .gcda files ...
geninfo: ERROR: no .gcda files found in .!
make[3]: *** [CMakeFiles/coverageTarget] Error 255
make[2]: *** [CMakeFiles/coverageTarget.dir/all] Error 2
make[1]: *** [CMakeFiles/coverageTarget.dir/rule] Error 2
make: *** [coverageTarget] Error 2

Any thoughts?

[bilke]

Maybe you are missing add_definitions(-fprofile-arcs -ftest-coverage)?

[bilke]

I have updated the coverage scripts in another project. I will test it there first.

• CodeCoverage.cmake
• Invocation of CodeCoverage

[daviddoria]

I tried it and noticed two things (I know you haven't tested yet, but I thought it may help):

1. I had to add
   include(CMakeParseArguments)
   or I got "Unknown CMake command "cmake_parse_arguments"

2. At this line:
   add_custom_command(TARGET ${Coverage_NAME} POST_BUILD ....

Coverage_NAME seems to be empty
--------- EDIT -------

This seems to be fixed by calling
SETUP_TARGET_FOR_COVERAGE(NAME coverageTarget EXECUTABLE ctestTarget "testResults")
rather than the old
SETUP_TARGET_FOR_COVERAGE(coverageTarget ctestTarget "testResults")

[daviddoria]

(Sorry we have parallel threads going)
Adding
add_definitions(-fprofile-arcs -ftest-coverage)
indeed fixed it. I had:
SET(CMAKE_CXX_FLAGS="-g -O0 -fprofile-arcs -ftest-coverage")
SET(CMAKE_C_FLAGS="-g -O0 -fprofile-arcs -ftest-coverage")
SET(CMAKE_EXE_LINKER_FLAGS="-fprofile-arcs -ftest-coverage")

which I thought was the same thing, but apparently it is not? haha

Also, FYI I learned that
add_definitions(--coverage)
is apparently the more modern alias for the same thing.

David

[dustingooding]

I'd like to have aggregate code coverage reporting as well, but I'm unable to use ctest. I'm using catkin to build code in a ROS ecosystem and switching to a different build system is prohibitive. Simply creating a custom target that depends on the multiple SETUP_TARGET_FOR_COVERAGE targets results in errors (cannot read various symbols), which I believe is due to the multiple calls to delete/zero in a shared build space. I think a reasonable approach would be to have three "steps": one that zeros, one that runs the binaries, and one that generates the aggregated report. The trick will be to get all these targets/commands to properly depend on each other such that you don't have a lot of fixturing in your projects' CMakeLists.txt.

Have you made any progress on this aggregation idea? If I get it to work, would you like a PR from me?

Edit: I've got something working based on your work. Here's the relevant parts:

Usage:

if (CATKIN_ENABLE_TESTING)
  SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread")
  catkin_add_gtest(ClassATest test/ClassA_Test.cpp)
  catkin_add_gtest(ClassBTest test/ClassB_Test.cpp)

  target_link_libraries(ClassATest ${PROJECT_NAME})
  target_link_libraries(ClassBTest ${PROJECT_NAME})

  include(CodeCoverage)
  coverage_add_exec(ClassATest)
  coverage_add_exec(ClassBTest)
endif()

The Goods:

# Set up coverage targets

set(COVERAGE_DIR ${CMAKE_BINARY_DIR}/coverage)

add_custom_target(${PROJECT_NAME}_coverage_dir
    COMMAND ${CMAKE_COMMAND} -E make_directory ${COVERAGE_DIR}
)

add_custom_target(${PROJECT_NAME}_coverage_prep
    COMMAND ${LCOV_PATH} --directory ${CMAKE_BINARY_DIR} --zerocounters

    WORKING_DIRECTORY ${COVERAGE_DIR}
    DEPENDS ${PROJECT_NAME}_coverage_dir
)

add_custom_target(${PROJECT_NAME}_coverage_exec)

add_custom_target(${PROJECT_NAME}_coverage
    COMMAND ${LCOV_PATH} --directory ${CMAKE_BINARY_DIR} --base-directory ${PROJECT_SOURCE_DIR} --capture --output-file ${PROJECT_NAME}.info
    COMMAND ${LCOV_PATH} --extract ${PROJECT_NAME}.info '${PROJECT_SOURCE_DIR}/*' --output-file ${PROJECT_NAME}.info.cleaned
    COMMAND ${GENHTML_PATH} -o ${COVERAGE_DIR} ${PROJECT_NAME}.info.cleaned
    COMMAND ${CMAKE_COMMAND} -E remove ${PROJECT_NAME}.info ${PROJECT_NAME}.info.cleaned

    WORKING_DIRECTORY ${COVERAGE_DIR}
    DEPENDS ${PROJECT_NAME}_coverage_exec
)

# Function for adding executables to the coverage analysis

function(coverage_add_exec exec)
    add_custom_target(${PROJECT_NAME}_coverage_exec_${exec}
        COMMAND ${exec}
        WORKING_DIRECTORY ${COVERAGE_DIR}
        DEPENDS ${PROJECT_NAME}_coverage_prep ${exec}
    )
    add_dependencies(${PROJECT_NAME}_coverage_exec ${PROJECT_NAME}_coverage_exec_${exec})
endfunction()

Command:

catkin build package_name -DCMAKE_BUILD_TYPE=Coverage --no-deps --make-args package_name_coverage

[CiaranWelsh]

Is there a complete example somewhere of how to do this using googletest? Thanks.

[rayment]

Hi, I've written a small tutorial in #88 with a basic template to get started with ctest and doctest, though doctest is entirely optional and you can use a different test framework should you desire.

Aggregation with ctest actually works well and I have had no issues so far.