-
-
Notifications
You must be signed in to change notification settings - Fork 429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Standardising Benchmarks, with support for nanobench as an option for its backend #6448
Merged
Merged
Changes from 23 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
5167e12
Add initial nanobench benchmark config
vrnimje 16e42ea
Pyperf integration for testing puporses
vrnimje 3d38818
Added interface for nanbench backend, and tested interface with minma…
vrnimje 35a3f1d
Added workflows to test out benchmarks
vrnimje 59363bd
Modified CI to build and test minmax benchmark only
vrnimje 506e240
Added separate CI for benchmark without nanobench
vrnimje cbee8b1
Changing new CI name
vrnimje 2979001
Correcting CI names
vrnimje 4683bea
Changing CI activations to PR only
vrnimje 1f2ae1c
Adding back namespace defs and add_library call for nanobench
vrnimje ee41a79
Clean up minmax test
vrnimje 4bb7d3d
Adding command line argument for detailed bench output
vrnimje 77fa197
Fixing CI format
vrnimje 2e6258a
Adding initial benchmark visualisation script
vrnimje 794de41
Added benchmark output specialisations for normal case as well
vrnimje af5726b
Standardised modules.datastructures.small_vector_benchmark, and made …
vrnimje 14374ed
Updated cmdline switching to local config, and refactored benchmarks …
vrnimje 21fc400
Removing definition of inline variable, and modified CIs to use the c…
vrnimje 1d18a5c
Applying cmake-format and clang-format
vrnimje 995ae0e
Adding explicit double to long double conversions for the default case
vrnimje cc25fb1
Fixed implicit conversion issue while using add_time function
vrnimje 33ea494
Standardised foreach_report, stream_report and future_overhead_report…
vrnimje 526e048
Fixing clang-format for performance.cpp and future_overhead_report
vrnimje c35e4e4
Removed perftests_plot.py, and made minor clean-up changes
vrnimje File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
# Copyright (c) 2024 Vedant Nimje | ||
# | ||
# SPDX-License-Identifier: BSL-1.0 | ||
# Distributed under the Boost Software License, Version 1.0. (See accompanying | ||
# file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) | ||
|
||
name: Linux CI (Debug) with Benchmark Test | ||
|
||
on: [pull_request] | ||
|
||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
container: stellargroup/build_env:14 | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Configure | ||
shell: bash | ||
run: | | ||
cmake \ | ||
. \ | ||
-Bbuild \ | ||
-GNinja \ | ||
-DCMAKE_BUILD_TYPE=Debug \ | ||
-DHPX_WITH_MALLOC=system \ | ||
-DHPX_WITH_FETCH_ASIO=ON \ | ||
-DHPX_WITH_EXAMPLES=ON \ | ||
-DHPX_WITH_TESTS=ON \ | ||
-DHPX_WITH_TESTS_MAX_THREADS_PER_LOCALITY=2 \ | ||
-DHPX_WITH_CHECK_MODULE_DEPENDENCIES=On | ||
- name: Build | ||
shell: bash | ||
run: | | ||
cmake --build build/ --target tests.performance.modules.segmented_algorithms.minmax_element_performance | ||
- name: Test | ||
shell: bash | ||
run: | | ||
cd build | ||
./bin/minmax_element_performance_test | ||
- name: Test with detailed output | ||
shell: bash | ||
run: | | ||
cd build | ||
./bin/minmax_element_performance_test --detailed_bench |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# Copyright (c) 2024 Vedant Nimje | ||
# | ||
# SPDX-License-Identifier: BSL-1.0 | ||
# Distributed under the Boost Software License, Version 1.0. (See accompanying | ||
# file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) | ||
|
||
name: Linux CI (Debug) with Nanobench Benchmark Test | ||
|
||
on: [pull_request] | ||
|
||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
container: stellargroup/build_env:14 | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Configure | ||
shell: bash | ||
run: | | ||
cmake \ | ||
. \ | ||
-Bbuild \ | ||
-GNinja \ | ||
-DCMAKE_BUILD_TYPE=Debug \ | ||
-DHPX_WITH_MALLOC=system \ | ||
-DHPX_WITH_FETCH_ASIO=ON \ | ||
-DHPX_WITH_EXAMPLES=ON \ | ||
-DHPX_WITH_TESTS=ON \ | ||
-DHPX_WITH_NANOBENCH=ON \ | ||
-DHPX_WITH_TESTS_MAX_THREADS_PER_LOCALITY=2 \ | ||
-DHPX_WITH_CHECK_MODULE_DEPENDENCIES=On | ||
- name: Build | ||
shell: bash | ||
run: | | ||
cmake --build build/ --target tests.performance.modules.segmented_algorithms.minmax_element_performance | ||
- name: Test | ||
shell: bash | ||
run: | | ||
cd build | ||
./bin/minmax_element_performance_test | ||
|
||
- name: Test with detailed output | ||
shell: bash | ||
run: | | ||
cd build | ||
./bin/minmax_element_performance_test --detailed_bench |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
include(FetchContent) | ||
|
||
fetchcontent_declare( | ||
nanobench | ||
GIT_REPOSITORY https://github.com/martinus/nanobench.git | ||
GIT_TAG v4.3.11 | ||
GIT_SHALLOW TRUE | ||
) | ||
|
||
if(NOT nanobench_POPULATED) | ||
fetchcontent_populate(nanobench) | ||
endif() | ||
set(Nanobench_ROOT ${nanobench_SOURCE_DIR}) | ||
|
||
add_library(nanobench INTERFACE) | ||
target_include_directories( | ||
nanobench SYSTEM INTERFACE $<BUILD_INTERFACE:${Nanobench_ROOT}/src/include/> | ||
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}> | ||
) | ||
|
||
install( | ||
TARGETS nanobench | ||
EXPORT HPXNanobenchTarget | ||
COMPONENT core | ||
) | ||
|
||
install( | ||
FILES ${NANOBENCH_ROOT}/include/nanobench.h | ||
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR} | ||
COMPONENT core | ||
) | ||
|
||
export( | ||
TARGETS nanobench | ||
NAMESPACE nanobench:: | ||
FILE "${CMAKE_CURRENT_BINARY_DIR}/lib/cmake/${HPX_PACKAGE_NAME}/HPXNanobenchTarget.cmake" | ||
) | ||
|
||
install( | ||
EXPORT HPXNanobenchTarget | ||
NAMESPACE nanobench:: | ||
FILE HPXNanobenchTarget.cmake | ||
DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/${HPX_PACKAGE_NAME} | ||
COMPONENT cmake | ||
) | ||
|
||
add_library(nanobench::nanobench ALIAS nanobench) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,6 +9,7 @@ | |
#include <hpx/datastructures/detail/small_vector.hpp> | ||
#include <hpx/functional.hpp> | ||
#include <hpx/init.hpp> | ||
#include <hpx/modules/testing.hpp> | ||
|
||
#include <cstddef> | ||
#include <cstdint> | ||
|
@@ -19,61 +20,85 @@ | |
|
||
/////////////////////////////////////////////////////////////////////////////// | ||
template <typename Container> | ||
std::uint64_t measure(std::size_t repeat, std::size_t size) | ||
void fill(std::size_t size) | ||
{ | ||
std::uint64_t start = hpx::chrono::high_resolution_clock::now(); | ||
for (std::size_t i = 0; i != repeat; ++i) | ||
Container cont; | ||
for (std::size_t i = 0; i != size; ++i) | ||
{ | ||
Container cont; | ||
for (std::size_t i = 0; i != size; ++i) | ||
{ | ||
cont.push_back(typename Container::value_type{}); | ||
} | ||
cont.push_back(typename Container::value_type{}); | ||
} | ||
return (hpx::chrono::high_resolution_clock::now() - start) / repeat; | ||
} | ||
|
||
template <typename T, std::size_t N> | ||
void compare(std::size_t repeat, std::size_t size) | ||
{ | ||
std::uint64_t time = measure<hpx::detail::small_vector<T, N>>(repeat, size); | ||
// template <typename T, std::size_t N> | ||
// void compare(std::size_t repeat, std::size_t size) | ||
// { | ||
// std::uint64_t time = measure<hpx::detail::small_vector<T, N>>(repeat, size); | ||
|
||
std::cout << "-----Average-(hpx::small_vector<" << typeid(T).name() << ", " | ||
<< N << ">)------ \n" | ||
<< std::left << "Average execution time : " << std::right | ||
<< std::setw(8) << time / 1e9 << "\n"; | ||
} | ||
// std::cout << "-----Average-(hpx::small_vector<" << typeid(T).name() << ", " | ||
// << N << ">)------ \n" | ||
// << std::left << "Average execution time : " << std::right | ||
// << std::setw(8) << time / 1e9 << "\n"; | ||
// } | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can be removed There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed in c35e4e4 |
||
|
||
int hpx_main(hpx::program_options::variables_map& vm) | ||
{ | ||
// pull values from cmd | ||
std::size_t repeat = vm["test_count"].as<std::size_t>(); | ||
std::size_t size = vm["vector_size"].as<std::size_t>(); | ||
|
||
std::cout << std::left | ||
<< "----------------Parameters---------------------\n" | ||
<< std::left | ||
<< "Vector size : " << std::right | ||
<< std::setw(8) << size << "\n" | ||
<< std::left | ||
<< "Number of tests : " << std::right | ||
<< std::setw(8) << repeat << "\n" | ||
<< std::left | ||
<< "Display time in : " << std::right | ||
<< std::setw(8) << "Seconds\n" | ||
<< std::flush; | ||
|
||
compare<int, 1>(repeat, size); | ||
compare<int, 2>(repeat, size); | ||
compare<int, 4>(repeat, size); | ||
compare<int, 8>(repeat, size); | ||
compare<int, 16>(repeat, size); | ||
|
||
compare<hpx::move_only_function<void()>, 1>(repeat, size); | ||
compare<hpx::move_only_function<void()>, 2>(repeat, size); | ||
compare<hpx::move_only_function<void()>, 4>(repeat, size); | ||
compare<hpx::move_only_function<void()>, 8>(repeat, size); | ||
compare<hpx::move_only_function<void()>, 16>(repeat, size); | ||
hpx::util::perftests_init(vm); | ||
|
||
// int | ||
|
||
hpx::util::perftests_report("hpx::small_vector", "<int, 1>", repeat, | ||
[&] { fill<hpx::detail::small_vector<int, 1>>(size); }); | ||
|
||
hpx::util::perftests_report("hpx::small_vector", "<int, 2>", repeat, | ||
[&] { fill<hpx::detail::small_vector<int, 2>>(size); }); | ||
|
||
hpx::util::perftests_report("hpx::small_vector", "<int, 4>", repeat, | ||
[&] { fill<hpx::detail::small_vector<int, 4>>(size); }); | ||
|
||
hpx::util::perftests_report("hpx::small_vector", "<int, 8>", repeat, | ||
[&] { fill<hpx::detail::small_vector<int, 8>>(size); }); | ||
|
||
hpx::util::perftests_report("hpx::small_vector", "<int, 16>", repeat, | ||
[&] { fill<hpx::detail::small_vector<int, 16>>(size); }); | ||
|
||
// hpx::move_only_function<void()> | ||
|
||
hpx::util::perftests_report( | ||
"hpx::small_vector", "<fxn<void()>, 1>", repeat, [&] { | ||
fill<hpx::detail::small_vector<hpx::move_only_function<void()>, 1>>( | ||
size); | ||
}); | ||
|
||
hpx::util::perftests_report( | ||
"hpx::small_vector", "<fxn<void()>, 2>", repeat, [&] { | ||
fill<hpx::detail::small_vector<hpx::move_only_function<void()>, 2>>( | ||
size); | ||
}); | ||
|
||
hpx::util::perftests_report( | ||
"hpx::small_vector", "<fxn<void()>, 4>", repeat, [&] { | ||
fill<hpx::detail::small_vector<hpx::move_only_function<void()>, 4>>( | ||
size); | ||
}); | ||
|
||
hpx::util::perftests_report( | ||
"hpx::small_vector", "<fxn<void()>, 8>", repeat, [&] { | ||
fill<hpx::detail::small_vector<hpx::move_only_function<void()>, 8>>( | ||
size); | ||
}); | ||
|
||
hpx::util::perftests_report( | ||
"hpx::small_vector", "<fxn<void()>, 16>", repeat, [&] { | ||
fill< | ||
hpx::detail::small_vector<hpx::move_only_function<void()>, 16>>( | ||
size); | ||
}); | ||
|
||
hpx::util::perftests_print_times(); | ||
|
||
return hpx::local::finalize(); | ||
} | ||
|
@@ -97,6 +122,8 @@ int main(int argc, char* argv[]) | |
; | ||
// clang-format on | ||
|
||
hpx::util::perftests_cfg(cmdline); | ||
|
||
hpx::local::init_params init_args; | ||
init_args.desc_cmdline = cmdline; | ||
init_args.cfg = cfg; | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in c35e4e4