Redo app build caching #667

jmarrec · 2024-01-11T16:45:54Z

I noticed an issue with how we cached the entire build directory...

In most cases it doesn’t speed up anything
Github limits caching to 10GB. We have 4 runners that cache about 3 to 4 GB, so we constantly exceed it
We often pick up an outdated OS SDK installer

I’ve done some semi “clever” caching strategies now:

Cache the conan cache. The key for the cache is computed from the autodetected conan profile (so specific to an OS, architecture, compiler version) + the hash of the ConanInstall.cmake
Cache the build/Qt-install dir + OS SDK installer (but trash it later if it's not the exact one). This is dependent on the runner.os and the QT version we use
Enable CCACHE on all platforms. This is dependent on the hash of the conan profile
- I have a few caveats here: if the cache is hit, then it's not saved in post build. Fine for the conan cache, but probably want an update for the build stuff? ideally the cache would be updated constantly... I might revisit this. Note that cache is immutable, so that'd mean uploading a NEW cache entry

Build times improvements on a rebuild when the source hasn't changed:

Ubuntu: 41 min -> 10 min
macos-13 (where conan binaries needed to be built): 1h15 -> 27min (shaving 23 min due to conan cache only)
Windows: 1h15 -> 20 min (the compile step takes a penalty on initial ccache building, approx 1h versus 53 min without)

before:

after:

…sted (and see if caching works) Don't define CONAN_USER_HOME on self hosted

… github.workspace ``` runner.workspace=/home/runner/work/OpenStudioApplication GITHUB_WORKSPACE=/home/runner/work/OpenStudioApplication/OpenStudioApplication github.workspace=/home/runner/work/OpenStudioApplication/OpenStudioApplication ```

… stale cache from same profile but different ConanInstall.cmake

…ipes were built from source

…d OS SDK) This includes the Qt install, the .ccache (ccache) folder on Unix, the OpenStudio SDK tar.gz, and previous build artifacts # TODO: problem is that caching is limited to 10 GB. The build folder takes 3-4 GB per runner, and we have 4 of them that try to cache # Perhaps we should just cache the ccache. Anyways, for incremental builds triggered one after another, cache eviction hasn't happened yet and all of them do a cache hit # Build times improvements # * Ubuntu: 41 min -> 10 min # * macos-13 (where conan binaries needed to be built): 1h15 -> 27min (shaving 23 min due to conan cache only) # Another thing is that if the cache is hit, then it's not saved in post build. Fine for the conan cache, but probably want an update for the build stuff # Note that cache is immutable, so that'd mean uploading a NEW cache entry (so potentially 4GB again)

RULE_LAUNCH_COMPILE is now flagged as an internal implementation detail, and it makes the compile of the RC file fail on windows. See the note on https://cmake.org/cmake/help/latest/prop_gbl/RULE_LAUNCH_COMPILE.html > Note This property is intended for internal use by ctest(1). Projects and developers should use the <LANG>_COMPILER_LAUNCHER target properties or the associated CMAKE_<LANG>_COMPILER_LAUNCHER variables instead. ``` [ 96%][543/560] Building RC object src\openstudio_app\CMakeFiles\OpenStudioApp.dir\OpenStudioApp.rc.res FAILED: src/openstudio_app/CMakeFiles/OpenStudioApp.dir/OpenStudioApp.rc.res C:/bin/ccache.exe C:/PROGRA~1/CMake/bin/cmcldeps.exe RC C:\src\OpenStudioApplication\build\src\openstudio_app\OpenStudioApp.rc src\openstudio_app\CMakeFiles\OpenStudioApp.dir\OpenStudioApp.rc.res.d src\openstudio_app\CMakeFiles\OpenStudioApp.dir\OpenStudioApp.rc.res "Note: including file: " "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx64/x64/cl.exe" C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe -DBOOST_ALL_NO_LIB -DBOOST_STACKTRACE_USE_NOOP -DBOOST_STACKTRACE_USE_WINDBG -DBOOST_STACKTRACE_USE_WINDBG_CACHED -DBOOST_USE_WINAPI_VERSION=BOOST_WINAPI_VERSION_WIN7 -DBOOST_WINAPI_DEFINE_VERSION_MACROS -DLIBXML_STATIC -DNINJA=1 -DNOMINMAX -DQT_CORE5COMPAT_LIB -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_NO_DEBUG -DQT_OPENGL_LIB -DQT_POSITIONING_LIB -DQT_PRINTSUPPORT_LIB -DQT_QMLINTEGRATION_LIB -DQT_QMLMODELS_LIB -DQT_QML_LIB -DQT_QUICKWIDGETS_LIB -DQT_QUICK_LIB -DQT_SVG_LIB -DQT_WEBCHANNEL_LIB -DQT_WEBENGINECORE_LIB -DQT_WEBENGINEWIDGETS_LIB -DQT_WIDGETS_LIB -DQT_XML_LIB -DSHARED_OS_LIBS -DUNICODE -DWIN32 -DWIN32_LEAN_AND_MEAN -DWIN64 -D_AFXDLL -D_CRT_SECURE_NO_WARNINGS -D_ENABLE_EXTENDED_ALIGNED_STORAGE -D_NO_ASYNCRTIMP -D_NO_PPLXIMP -D_SCL_SECURE_NO_WARNINGS -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS -D_SILENCE_FPOS_SEEKPOS_DEPRECATION_WARNING -D_UNICODE -D_WIN64 -Dmodel_editor_EXPORTS -I C:\src\OpenStudioApplication -I C:\src\OpenStudioApplication\src -I C:\src\OpenStudioApplication\build -I C:\src\OpenStudioApplication\build\src -I C:\src\OpenStudioApplication\src\qtwinmigrate -I C:\src\OpenStudioApplication\build\src\openstudio_app -I C:\Qt\6.6.1\msvc2019_64\include\QtZlib -I C:\Qt\6.6.1\msvc2019_64\include\QtConcurrent -I C:\Qt\6.6.1\msvc2019_64\include\QtGui\6.6.1\QtGui -I C:\Qt\6.6.1\msvc2019_64\include\QtCore -I C:\Qt\6.6.1\msvc2019_64\include -I C:\Qt\6.6.1\msvc2019_64\include\QtWidgets -I C:\Qt\6.6.1\msvc2019_64\include\QtGui -I C:\Qt\6.6.1\msvc2019_64\include\QtXml -I C:\Qt\6.6.1\msvc2019_64\include\QtNetwork -I C:\Qt\6.6.1\msvc2019_64\include\QtPrintSupport -I C:\Qt\6.6.1\msvc2019_64\include\QtWebEngineCore -I C:\Qt\6.6.1\msvc2019_64\include\QtQuick -I C:\Qt\6.6.1\msvc2019_64\include\QtQml -I C:\Qt\6.6.1\msvc2019_64\include\QtQmlIntegration -I C:\Qt\6.6.1\msvc2019_64\include\QtQmlModels -I C:\Qt\6.6.1\msvc2019_64\include\QtOpenGL -I C:\Qt\6.6.1\msvc2019_64\include\QtWebChannel -I C:\Qt\6.6.1\msvc2019_64\include\QtPositioning -I C:\Qt\6.6.1\msvc2019_64\include\QtWebEngineWidgets -I C:\Qt\6.6.1\msvc2019_64\include\QtQuickWidgets -I C:\src\OpenStudioApplication\build\OpenStudio-3.7.0\OpenStudio-3.7.0+d5269793f1-Windows\include -I C:\src\OpenStudioApplication\build\OpenStudio-3.7.0\OpenStudio-3.7.0+d5269793f1-Windows\include\openstudio -I C:\.conan\117565\1\include -I C:\Users\julien\.conan\data\zlib\1.2.12\_\_\package\5a61a86bb3e07ce4262c80e1510f9c05e9b6d48b\include -I C:\Users\julien\.conan\data\bzip2\1.0.8\_\_\package\53b5fd22ba061620078deefdae9a437c5f693201\include -I C:\Users\julien\.conan\data\jsoncpp\1.9.5\_\_\package\5a61a86bb3e07ce4262c80e1510f9c05e9b6d48b\include -I C:\Users\julien\.conan\data\cpprestsdk\2.10.18\_\_\package\bc796986be5f5a06bb4dfcaa8b7ab497284d57e9\include -I C:\Users\julien\.conan\data\openssl\1.1.1o\_\_\package\5a61a86bb3e07ce4262c80e1510f9c05e9b6d48b\include -I C:\Users\julien\.conan\data\websocketpp\0.8.2\_\_\package\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9\include -I C:\Users\julien\.conan\data\pugixml\1.12.1\_\_\package\5a61a86bb3e07ce4262c80e1510f9c05e9b6d48b\include -I C:\Users\julien\.conan\data\libxml2\2.9.14\_\_\package\77c83943028aeda4b502557bd0b671409529a8bf\include -I C:\Users\julien\.conan\data\libxml2\2.9.14\_\_\package\77c83943028aeda4b502557bd0b671409529a8bf\include\libxml2 -I C:\Users\julien\.conan\data\libiconv\1.16\_\_\package\5a61a86bb3e07ce4262c80e1510f9c05e9b6d48b\include -I C:\Users\julien\.conan\data\fmt\9.1.0\_\_\package\8c0a53bc9b0a08b9311ca89254091ba7fb4f89b4\include -I C:\Qt\6.6.1\msvc2019_64\mkspecs\win32-msvc -I C:\Qt\6.6.1\msvc2019_64\include\QtCore5Compat -I C:\Qt\6.6.1\msvc2019_64\include\QtSvg -DWIN32 /wd4503 /wd4251 /wd4275 /wd4267 /WX /fo src\openstudio_app\CMakeFiles\OpenStudioApp.dir\OpenStudioApp.rc.res C:\src\OpenStudioApplication\build\src\openstudio_app\OpenStudioApp.rc cl : Command line warning D9002 : ignoring unknown option '/fo' c1: fatal error C1083: Cannot open source file: 'src\openstudio_app\CMakeFiles\OpenStudioApp.dir\OpenStudioApp.rc.res.dep.obj ': No such file or directory OpenStudioApp.rc [ 98%][552/560] Building CXX object src\openstudio_app\CMa...\OpenStudioApp_tests.dir\test\OpenStudioAppFixture.cpp.obj ninja: build stopped: subcommand failed. ```

… message) + Check if windows ccache works

``` ##[debug]::debug::Search path 'D:\a\OpenStudioApplication\OpenStudioApplication' ##[debug]D:\a\OpenStudioApplication\OpenStudioApplication\CMakeLists.txt ##[debug][Error: EACCES: permission denied, lstat 'D:\a\OpenStudioApplication\OpenStudioApplication\conan-cache\short\77a911\1\bin\msys64\dev\fd'] { ##[debug] errno: -4092, ##[debug] code: 'EACCES', ##[debug] syscall: 'lstat', ##[debug] path: 'D:\\a\\OpenStudioApplication\\OpenStudioApplication\\conan-cache\\short\\77a911\\1\\bin\\msys64\\dev\\fd' ##[debug]} ```

jmarrec · 2024-01-11T19:25:44Z

cd this workflow: without ccache hit (run 1) and with ccache hit (run 2): https://github.com/openstudiocoalition/OpenStudioApplication/actions/runs/7491867085/job/20399061453

Run1:

Run2 (ccache cache hit):

I have a small issue with the hashFiles('**/CMakeLists.txt') again on windows. I've had that some hours ago: it was picking up the conan cache which I’ve placed in the workplace, it was looking for hashFiles(‘**/CMakeLists.txt’) into that conan folder as well and failing to lstat a msys2 executable... https://github.com/openstudiocoalition/OpenStudioApplication/actions/runs/7491232254/job/20392818323#step:10:77

…n key cache not hit

jmarrec · 2024-01-11T20:27:27Z

That's 3.4x speedup, nice !

Ok all good, I even tested partial cache hit on https://github.com/openstudiocoalition/OpenStudioApplication/actions/runs/7494140697/job/20401388058 (the restore-key is indeed used when the key didn't match for the CCache and the Conan caches)

jmarrec added 18 commits January 11, 2024 17:40

Try to debug cpack

9e71d8b

Try to improve app_build: redo caching, add ccache

e587d65

Redo cache keys (nested templates)

3e059e5

Use latest conan 1.x instead of 1.59.0 + don't cache conan on self ho…

9ae7f26

…sted (and see if caching works) Don't define CONAN_USER_HOME on self hosted

adjsut conan user home: github.workspace in matrix is nil appparently

0cd62da

Debug why hashFiles fails now

ecd7333

tweak

429a37e

try quoting + backslash on windoze

7e6f18b

Try to set ccache configuration to place stuff inside the build dir

b0c485b

Shush apt update + use restore-keys for conan cache: allow retrieving…

50388e1

… stale cache from same profile but different ConanInstall.cmake

Clean conan build + source folders to reduce cache size when some rec…

c31adee

…ipes were built from source

See if ccache only manages to speed up build

e2b06e7

Install libxcb-cursor0 (Tests don't run on Ubuntu currently with this…

28ccc2a

… message) + Check if windows ccache works

Restore-keys is a prefix, so do it properly.

7dc7967

jmarrec added the Developer Issue 🤓 label Jan 11, 2024

jmarrec self-assigned this Jan 11, 2024

jmarrec added 2 commits January 11, 2024 20:52

More explicit hashFiles to prevent windoze error on post build

bc39a90

Dummy CMakeLists + ConanInstall to test restore-key partial match whe…

734f992

…n key cache not hit

jmarrec requested a review from macumber January 11, 2024 20:27

jmarrec merged commit 270dc56 into develop Jan 11, 2024
8 checks passed

jmarrec deleted the redo-app-build-caching branch January 11, 2024 21:19

github-actions bot locked and limited conversation to collaborators Jan 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redo app build caching #667

Redo app build caching #667

jmarrec commented Jan 11, 2024 •

edited

Loading

jmarrec commented Jan 11, 2024 •

edited

Loading

jmarrec commented Jan 11, 2024 •

edited

Loading

Redo app build caching #667

Redo app build caching #667

Conversation

jmarrec commented Jan 11, 2024 • edited Loading

jmarrec commented Jan 11, 2024 • edited Loading

jmarrec commented Jan 11, 2024 • edited Loading

jmarrec commented Jan 11, 2024 •

edited

Loading

jmarrec commented Jan 11, 2024 •

edited

Loading

jmarrec commented Jan 11, 2024 •

edited

Loading