Problems building dependent packages

[tmadlener]

This might again be just me not fully understanding the concept yet, but maybe it's also the way the packages are structured and depend on each other.

I tried to make a minimal non-trivial test by building podio and EDM4hep. EDM4hep depends on podio, as that provides the code generator and some libraries that it links against.

I did the following:

• create a new project via spack mpd-project -T $(pwd)/edm4hep-minimal --name edm4hep-minimal cxxstd=20 %[email protected]
• symlink the repositories into edm4hep-minimal/srcs (making sure to lowercase them to make match the names spack expects)
• spack mpd refresh
• spack env activate edm4hep-minimal/local
• spack mpd build

With this I run into the following issue:

CMake Error at edm4hep/CMakeLists.txt:12 (find_package):
  Could not find a package configuration file provided by "podio" (requested
  version 1.1) with any of the following names:

    podioConfig.cmake
    podio-config.cmake

  Add the installation prefix of "podio" to CMAKE_PREFIX_PATH or set
  "podio_DIR" to a directory containing one of the above files.  If "podio"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!

I am not sure I fully understand yet why this happens, as podioConfig.cmake and podioConfigVersion.cmake are present in the podio build folder:

$ ls edm4hep-minimal/build/podio/*.cmake
edm4hep-minimal/build/podio/podioConfig.cmake  
edm4hep-minimal/build/podio/podioConfigVersion.cmake

Is this a podio issue (not being completely fetch-content-able) or is this something that can be fixed in spack-mpd?

[knoepfel]

@tmadlener, one of my colleagues (@greenc-FNAL) noticed recently that I made an implicit assumption about MPD--namely, it really only works right now for developing dependent packages that are using the Cetmodules build system, which alters some of the CMake behavior to make inter-package development possible. I did not intend to limit MPD to only packages using Cetmodules, so I'd like to find a way to fix this. Stay tuned.

[tmadlener]

Just to add a bit more information to this after playing around a bit. I think in this case the problem is not only related to how spack-mpd does things or it's assumptions. It's simply not possible to include podio via a simple add_subdirectory(srcs/podio) with cmake at the moment. I still need to figure out whether it is possible to make podio work like that.

What does work is using the ExternalProject module, i.e. something like this:

include(ExternalProject)
ExternalProject_Add(podio
  SOURCE_DIR /path/to/podio
  CMAKE_ARGS <whatever-cmake-args>
)

ExternalProject_Add(EDM4HEP
  SOURCE_DIR /path/to/EDM4hep
  CMAKE_ARGS 
    <whatever-cmake-args> 
    -Dpodio_DIR=/podio-install-prefix/lib/cmake/podio   # This is rather brittle
 DEPENDS podio
)

NB: FetchContent almost works for podio. It seems like FetchContent and ExternalData need to be configured somewhat carefully together to work as expected (mainly related to where PROJECT_SOURCE_DIR and CMAKE_SOURCE_DIR point to).

ExternalProject is a bit more cumbersome to configure, since not all of the top-level cmake args (e.g. CMAKE_INSTALL_PREFIX, ...) are automatically passed down to the "external" projects.

As you can see dependency management probably get's complicated this way. This also has the following issues that would require additional work:

• changes to either project do not trigger re-builds via cmake --build build
• Tests are not collected nicely, such that they can be triggerd with a single ctest invocation. Instead the tests of the individual projects have to be run individually via the --test-dir argument and passing the appropriate prefix.

[knoepfel]

@tmadlener, thanks for the extra info. I was just about to update this issue. I've verified that MPD can be adjusted to instruct CMake where to find the (e.g.) podioConfig.cmake file without any changes required to the package. However, there are constructs within the file that would need to be handled:

CMake Error at /home/knoepfel/scratch/test-edm4hep/build/podio/podioConfig.cmake:29 (message):
  File or directory /home/knoepfel/scratch/include referenced by variable
  podio_INCLUDE_DIR does not exist !
Call Stack (most recent call first):
  /home/knoepfel/scratch/test-edm4hep/build/podio/podioConfig.cmake:45 (set_and_check)
  edm4hep/CMakeLists.txt:12 (find_package)

The error occurring simply because podio_INCLUDE_DIR refers to the installation location, which doesn't exist yet (let alone the files that would be installed in there). @greenc-FNAL has dealt with this type of problem a lot, but it required a lot of work. I'd like to talk with him and see what options there may be.