From e8aa05bf361b85b36dce1fd42ecc09672055779c Mon Sep 17 00:00:00 2001 From: Michael Voss Date: Thu, 1 Aug 2024 13:59:33 -0500 Subject: [PATCH 01/34] Add rfcs directory --- rfcs/README.md | 66 +++++++++++++++++++++++++++++++++++ rfcs/archived/README.md | 12 +++++++ rfcs/experimental/README.md | 28 +++++++++++++++ rfcs/proposed/README.md | 10 ++++++ rfcs/supported/README.md | 11 ++++++ rfcs/template.md | 68 +++++++++++++++++++++++++++++++++++++ 6 files changed, 195 insertions(+) create mode 100644 rfcs/README.md create mode 100644 rfcs/archived/README.md create mode 100644 rfcs/experimental/README.md create mode 100644 rfcs/proposed/README.md create mode 100644 rfcs/supported/README.md create mode 100644 rfcs/template.md diff --git a/rfcs/README.md b/rfcs/README.md new file mode 100644 index 0000000000..10639129d9 --- /dev/null +++ b/rfcs/README.md @@ -0,0 +1,66 @@ +# oneTBB Design Documents / RFCs + +The purpose of the RFC process is to communicate the intent to make +library-wide changes, get feedback prior to the actual implementation, +increase the transparency on why and how decisions are made, and improve +the alignment between different teams involved in oneTBB development. + +This directory documents design documents (RFCs) that have been approved +for implementation in oneTBB. + +There are several possible states for an RFC: + +1. Initial RFC PR +2. proposed +3. experimental +4. supported +5. archived + +Most modifications or new features will naturally start as a part of a +Github issue or discussion. Small changes do not require a formal RFC. +However, if the issue or discussion leads to an idea for a modification +or new feature that significantly impacts the library's public API or +architecture, it will be suggested that a PR be opened to add a new rfc +to the `rfcs/proposed` directory. The RFC contains a more detail description +and design for the feature. + +## General Process + +A template for RFCs is available as [template.md](template.md). The modified +template should be placed in `rfcs/proposed` in subdirectory with a name +of the form `_`. For example, +a proposal for a new "my_op" flow graph node might be put into a directory +`rfcs/proposed/flow_graph_my_op_node`. The [template.md](template.md) should +be used to create the `README.md` file in that directory. The folder can +contain other files referenced by the `README.md` file, such as figures. + +When two maintainers approve the PR it will be merged to the `rfcs/proposed` +directory. As the RFC moves to different states, the RFC document should be +updated with additional information. + +A proposal that is subsequently implemented and released in oneTBB +as a preview feature will be moved to the `rfcs/experimental` folder. The +RFC for a preview feature in `rfcs/experimental` should include a description +of what is required to move from experimental to fully supported -- for example, feedback from users, demonstrated performance improvements, etc. + +A proposal that is implemented, added to the oneTBB specification, and +supported as a full feature appears in the `rfcs/supported` directory. An RFC for a fully support feature in the `rfcs/supported` directory should +have a link to the section in the oneTBB specification that has its +formal wording. + +A feature that is deprecated or removed or a proposal that is abandoned may be moved to the `rfcs/archived` folder. + +## Document Style + +The design documents are stored in the `rfcs` directory. + +- Each RFC is stored in a separate subdirectory + `rfcs/proposed/_` +- There must be a `README.md` file that contains the main RFC itself (or links to a file that contains it in the same directory). + - The body of the RFC should be based on [template.md](template.md) + - The directory can contain other supporting files, such as images, tex formulas, and sub-proposals / sub-RFCs. + - We will not insist on the use markdown or the exact template file for new RFCs. But we do strongly prefer a text-based file that can be rendered by Github to allow for easy collaboration using PR comments. Even so, files such as pdfs may be acceptable. +- If the RFC is written in markdown. The width of the text should be limited by + 80 symbols, unless there is a need to violate this rule, e.g. because of + long links or wide tables. +- It is also recommended to read through existing RFCs to better understand the general writing style and required elements. diff --git a/rfcs/archived/README.md b/rfcs/archived/README.md new file mode 100644 index 0000000000..05d1585c64 --- /dev/null +++ b/rfcs/archived/README.md @@ -0,0 +1,12 @@ +# Archived Design Documents + +Documents may appear in the `rfcs/archived` directory for one of +three reasons: + +1. The document describes a feature or extension that has been deprecated. +2. The document describes a feature or extension that has been deprecated and then removed. +3. The document describes a feature or extension that was proposed but did +not (ultimately) become a fully support feature. + +Design documents that appear in `rfcs/archived` folder should include a +section that describes why the document has been archived. diff --git a/rfcs/experimental/README.md b/rfcs/experimental/README.md new file mode 100644 index 0000000000..d3dcf95375 --- /dev/null +++ b/rfcs/experimental/README.md @@ -0,0 +1,28 @@ +# Design documents for experimental features + +Experimental proposals describe extensions that have been implemented and +have been released as preview features in the oneTBB library. A preview +feature is expected to have an implementation that is of comparable quality +to a fully supported feature. It should also have sufficient tests. + +An experimental feature does not yet appear as part of the oneTBB +specification and therefore its interface and design can still change. +There is no commitment to backwards compatibility for a preview +feature. + +Since documents in this directory describe preview features, they +should include a list the exit conditions that need to be met to move from +preview to fully supported. These conditions might include demonstrated +performance improvements, demonstrated interest from the community, +acceptance of the required oneTBB specification changes, etc. + +For features that require oneTBB specification changes, the document might +include wording for those changes or a link to any PRs that have been opened +against the specification. + +Proposals should not remain in the experimental directory forever. The +expectation is that the proposals will eventually move either to the +supported folder when they become fully supported or to the archived +folder if they are not fully accepted. It should be highly unusual for +a proposal to stay in the experimental folder for longer than a year or +two. diff --git a/rfcs/proposed/README.md b/rfcs/proposed/README.md new file mode 100644 index 0000000000..2a4d3e7110 --- /dev/null +++ b/rfcs/proposed/README.md @@ -0,0 +1,10 @@ +# Design documents for proposed features + +Proposed features have reached some level of consensus within the +community. A proposal appears in this directory because the +community believes it has merit and further work should be done. +However, the proposed changes have not yet been released as a +preview or fully supported feature of the library. + +RFCs in the `rfcs/proposed` directory, should describe the motivation, +design and open questions related to the proposed extension. diff --git a/rfcs/supported/README.md b/rfcs/supported/README.md new file mode 100644 index 0000000000..e349670506 --- /dev/null +++ b/rfcs/supported/README.md @@ -0,0 +1,11 @@ +# Design documents for supported features + +Supported proposals describe extensions that have been implemented and +released as fully supported features of the oneTBB library. A fully supported +feature has a high-quality implementation. If the proposal impacted the +public API of the library it should appear in the oneTBB specification and +have supporting documentation in the oneTBB Reference as needed. A fully +supported feature is regularly tested. + +Proposals that appear in `rfcs/supported` may be retained indefinitely to +provide insight into the design of existing features. diff --git a/rfcs/template.md b/rfcs/template.md new file mode 100644 index 0000000000..fe200f5d23 --- /dev/null +++ b/rfcs/template.md @@ -0,0 +1,68 @@ +# Descriptive name for the proposal + +## Introduction + +Short description of the idea proposed with explained motivation. + +The motivation could be: +- Improved users experience for API changes and extensions. Code snippets to + showcase the benefits would be nice here. +- Performance improvements with the data, if available. +- Improved engineering practices. + +Introduction may also include any additional information that sheds light on +the proposal, such as history of the matter, links to relevant issues and +discussions, etc. + +## Proposal + +A full and detailed description of the proposal, with highlighted consequences. + +Depending on the kind of the proposal, the description should cover: + +- New use cases that are supported by the extension. +- The expected performance benefit for a modification. +- The interface of extensions including class definitions or function +declarations. + +A proposal should include the alternatives that were considered with listed +pros and cons. The alternatives should be clearly separated to make possible +discussions clear. + +Pay close attention to the following aspects of the library: +- API and ABI backwards compatibility. The library follows semantic versioning + so if any of those interfaces are to be broken, we need to state that + explicitly. +- Performance implications, as performance is one of the main goals of the library. +- Changes to the build system. While the library's primary building system is + CMake, there are some frameworks that may build the library directly from the sources. +- Dependencies and support matrix: does the proposal brings any new + dependencies or affects the supported configurations. + +Some other common subsections here are: +- Discussion: some people like to list all the options first (as separate + subsections), and then have a dedicated section with the discussion/ +- Listing of the proposed API and examples of its usage. +- Testing aspects. +- Short explanation and links to the related sub-proposals, if any. Such + sub-proposals could be organized as separate standalone RFCs, but this is + not mandatory. If the changes is insignificant, or doesn't make any sense + without the original proposal it is fine to have it in the RFC. +- Execution plan if approved, aka next steps. + +## Process Specific Information + +Depending on the state of the proposal, additional information should be +included. + +For new proposals (i.e., those in the `rfcs/proposed` directory), list any +open questions. + +For proposals released as preview features that are in the `rfcs/experimental` +directory, list the exit conditions to move from preview to fully supported. +These conditions might include demonstrated performance improvements, +acceptance of specification changes, etc. + +For proposals in the `rfcs/supported` directory, provide a link to the +any section(s) in the oneTBB specification that related to the proposal. +For modifications that do not affect the public API, no link is needed. From 9e6b4ce0ef62b8374ebb5f92d3995fd23c5d6637 Mon Sep 17 00:00:00 2001 From: Michael Voss Date: Mon, 5 Aug 2024 07:52:09 -0500 Subject: [PATCH 02/34] Removed deprecated as cause for archive --- rfcs/archived/README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/rfcs/archived/README.md b/rfcs/archived/README.md index 05d1585c64..9dec2adca2 100644 --- a/rfcs/archived/README.md +++ b/rfcs/archived/README.md @@ -3,10 +3,11 @@ Documents may appear in the `rfcs/archived` directory for one of three reasons: -1. The document describes a feature or extension that has been deprecated. -2. The document describes a feature or extension that has been deprecated and then removed. -3. The document describes a feature or extension that was proposed but did +1. The document describes a feature or extension that has been deprecated and then removed. +2. The document describes a feature or extension that was proposed but did not (ultimately) become a fully support feature. -Design documents that appear in `rfcs/archived` folder should include a -section that describes why the document has been archived. +Design documents that appear in the `rfcs/archived` folder should include a +section that describes why the document has been archived. Documents may +remain in this folder indefinitely to serve as a source of information about +previous proposals and features. From 6b819a36bfbef9e6bda9ebf0242b047a6f6f0d12 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:36:49 -0500 Subject: [PATCH 03/34] Update rfcs/README.md Co-authored-by: Aleksei Fedotov --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 10639129d9..aa07436e04 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -21,7 +21,7 @@ Github issue or discussion. Small changes do not require a formal RFC. However, if the issue or discussion leads to an idea for a modification or new feature that significantly impacts the library's public API or architecture, it will be suggested that a PR be opened to add a new rfc -to the `rfcs/proposed` directory. The RFC contains a more detail description +to the `rfcs/proposed` directory. The RFC contains a more detailed description and design for the feature. ## General Process From 0bc4bba1f12ab3a3148d1db3b5efcf88726c3b82 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:37:20 -0500 Subject: [PATCH 04/34] Update rfcs/README.md Co-authored-by: Aleksei Fedotov --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index aa07436e04..0cfa560899 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -27,7 +27,7 @@ and design for the feature. ## General Process A template for RFCs is available as [template.md](template.md). The modified -template should be placed in `rfcs/proposed` in subdirectory with a name +template should be placed in subdirectory of the `rfcs/proposed` with a name of the form `_`. For example, a proposal for a new "my_op" flow graph node might be put into a directory `rfcs/proposed/flow_graph_my_op_node`. The [template.md](template.md) should From 7e0de370cf96d5870db2cb4093ad89d80db2fd75 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:37:56 -0500 Subject: [PATCH 05/34] Update rfcs/README.md Co-authored-by: Aleksei Fedotov --- rfcs/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 0cfa560899..7ef8ce169b 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -59,7 +59,11 @@ The design documents are stored in the `rfcs` directory. - There must be a `README.md` file that contains the main RFC itself (or links to a file that contains it in the same directory). - The body of the RFC should be based on [template.md](template.md) - The directory can contain other supporting files, such as images, tex formulas, and sub-proposals / sub-RFCs. - - We will not insist on the use markdown or the exact template file for new RFCs. But we do strongly prefer a text-based file that can be rendered by Github to allow for easy collaboration using PR comments. Even so, files such as pdfs may be acceptable. + - It is not required to use the markdown file format based on the exact + template file for writing RFCs. However, it is strongly recommended to use + text-based file format that can be rendered by GitHub to allow for easy + collaboration using PR comments. Even so, files such as pdfs may be + acceptable. - If the RFC is written in markdown. The width of the text should be limited by 80 symbols, unless there is a need to violate this rule, e.g. because of long links or wide tables. From 6eddaa5ec1101146fcf23d6f60d4ba887fad2c06 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:38:12 -0500 Subject: [PATCH 06/34] Update rfcs/template.md Co-authored-by: Aleksei Fedotov --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index fe200f5d23..d7655ded88 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -31,7 +31,7 @@ discussions clear. Pay close attention to the following aspects of the library: - API and ABI backwards compatibility. The library follows semantic versioning - so if any of those interfaces are to be broken, we need to state that + so if any of those interfaces are to be broken, the RFC needs to state that explicitly. - Performance implications, as performance is one of the main goals of the library. - Changes to the build system. While the library's primary building system is From f8d7709eb1d52308dd23a18b9e49350d901027dc Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:39:39 -0500 Subject: [PATCH 07/34] Update rfcs/experimental/README.md Co-authored-by: Aleksei Fedotov --- rfcs/experimental/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/experimental/README.md b/rfcs/experimental/README.md index d3dcf95375..036ddc325b 100644 --- a/rfcs/experimental/README.md +++ b/rfcs/experimental/README.md @@ -11,7 +11,7 @@ There is no commitment to backwards compatibility for a preview feature. Since documents in this directory describe preview features, they -should include a list the exit conditions that need to be met to move from +should include a list of the exit conditions that need to be met to move from preview to fully supported. These conditions might include demonstrated performance improvements, demonstrated interest from the community, acceptance of the required oneTBB specification changes, etc. From 86c7031e63ae9090b77714e028cf9186c0a36324 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:39:57 -0500 Subject: [PATCH 08/34] Update rfcs/template.md Co-authored-by: Aleksei Fedotov --- rfcs/template.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/template.md b/rfcs/template.md index d7655ded88..c011c7780a 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -36,8 +36,8 @@ Pay close attention to the following aspects of the library: - Performance implications, as performance is one of the main goals of the library. - Changes to the build system. While the library's primary building system is CMake, there are some frameworks that may build the library directly from the sources. -- Dependencies and support matrix: does the proposal brings any new - dependencies or affects the supported configurations. +- Dependencies and support matrix: does the proposal bring any new + dependencies or affect the supported configurations? Some other common subsections here are: - Discussion: some people like to list all the options first (as separate From 60e76234a7d7a4e3545df2878b0ce4ca66a2a98f Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Fri, 16 Aug 2024 07:40:24 -0500 Subject: [PATCH 09/34] Update rfcs/template.md Co-authored-by: Aleksei Fedotov --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index c011c7780a..44c0237476 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -41,7 +41,7 @@ Pay close attention to the following aspects of the library: Some other common subsections here are: - Discussion: some people like to list all the options first (as separate - subsections), and then have a dedicated section with the discussion/ + subsections), and then have a dedicated section with the discussion. - Listing of the proposed API and examples of its usage. - Testing aspects. - Short explanation and links to the related sub-proposals, if any. Such From d70a09a3c2fb79a1fc57abc2ed419f86b41c2f42 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:25:05 -0500 Subject: [PATCH 10/34] Update rfcs/README.md Co-authored-by: Aleksei Fedotov --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 7ef8ce169b..6beb1d6202 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -44,7 +44,7 @@ RFC for a preview feature in `rfcs/experimental` should include a description of what is required to move from experimental to fully supported -- for example, feedback from users, demonstrated performance improvements, etc. A proposal that is implemented, added to the oneTBB specification, and -supported as a full feature appears in the `rfcs/supported` directory. An RFC for a fully support feature in the `rfcs/supported` directory should +supported as a full feature appears in the `rfcs/supported` directory. An RFC for a fully supported feature in the `rfcs/supported` directory should have a link to the section in the oneTBB specification that has its formal wording. From 640ee5dbf91fb2263e2d02ff4e0805f7b6350df1 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:25:26 -0500 Subject: [PATCH 11/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 6beb1d6202..0d642ca931 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -1,4 +1,4 @@ -# oneTBB Design Documents / RFCs +# oneTBB Design Documents/RFCs The purpose of the RFC process is to communicate the intent to make library-wide changes, get feedback prior to the actual implementation, From ac8e08501432592c5d1a8a4cf45562a473ad824c Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:25:44 -0500 Subject: [PATCH 12/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 0d642ca931..af93f7bac3 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -5,7 +5,7 @@ library-wide changes, get feedback prior to the actual implementation, increase the transparency on why and how decisions are made, and improve the alignment between different teams involved in oneTBB development. -This directory documents design documents (RFCs) that have been approved +This directory contains design documents (RFCs) approved for implementation in oneTBB. There are several possible states for an RFC: From 20766705b7c545e04890c4feaf3ed1c5739537ad Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:25:54 -0500 Subject: [PATCH 13/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index af93f7bac3..6334e252eb 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -8,7 +8,7 @@ the alignment between different teams involved in oneTBB development. This directory contains design documents (RFCs) approved for implementation in oneTBB. -There are several possible states for an RFC: +The possible RFC states are: 1. Initial RFC PR 2. proposed From d329f70b5b344526b210d44a6f3bce35fde0aee3 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:26:01 -0500 Subject: [PATCH 14/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 6334e252eb..c34e99f698 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -11,7 +11,7 @@ for implementation in oneTBB. The possible RFC states are: 1. Initial RFC PR -2. proposed +2. Proposed 3. experimental 4. supported 5. archived From 06fd104d33892b52f8bed625fb81c47fee7146d0 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:26:13 -0500 Subject: [PATCH 15/34] Update rfcs/template.md Co-authored-by: Alexandra --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index 44c0237476..272f99a571 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -30,7 +30,7 @@ pros and cons. The alternatives should be clearly separated to make possible discussions clear. Pay close attention to the following aspects of the library: -- API and ABI backwards compatibility. The library follows semantic versioning +- API and ABI backward compatibility. The library follows semantic versioning so if any of those interfaces are to be broken, the RFC needs to state that explicitly. - Performance implications, as performance is one of the main goals of the library. From 758959a7f3123cbb49fad953224f48529f08d919 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:26:20 -0500 Subject: [PATCH 16/34] Update rfcs/template.md Co-authored-by: Alexandra --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index 272f99a571..d1e974bd35 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -42,7 +42,7 @@ Pay close attention to the following aspects of the library: Some other common subsections here are: - Discussion: some people like to list all the options first (as separate subsections), and then have a dedicated section with the discussion. -- Listing of the proposed API and examples of its usage. +- List of the proposed API and examples of its usage. - Testing aspects. - Short explanation and links to the related sub-proposals, if any. Such sub-proposals could be organized as separate standalone RFCs, but this is From 068c26ef679bd2c6586491a4bbf4ba83b90cfbc6 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:26:31 -0500 Subject: [PATCH 17/34] Update rfcs/template.md Co-authored-by: Alexandra --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index d1e974bd35..f439be40d7 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -46,7 +46,7 @@ Some other common subsections here are: - Testing aspects. - Short explanation and links to the related sub-proposals, if any. Such sub-proposals could be organized as separate standalone RFCs, but this is - not mandatory. If the changes is insignificant, or doesn't make any sense + not mandatory. If the change is insignificant or doesn't make any sense without the original proposal it is fine to have it in the RFC. - Execution plan if approved, aka next steps. From 4d67b5bc7b072d2dc39daa62b31683e69ee490f5 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:27:04 -0500 Subject: [PATCH 18/34] Update rfcs/template.md Co-authored-by: Alexandra --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index f439be40d7..c87366769a 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -47,7 +47,7 @@ Some other common subsections here are: - Short explanation and links to the related sub-proposals, if any. Such sub-proposals could be organized as separate standalone RFCs, but this is not mandatory. If the change is insignificant or doesn't make any sense - without the original proposal it is fine to have it in the RFC. + without the original proposal, you can have it in the RFC. - Execution plan if approved, aka next steps. ## Process Specific Information From 446927b6269adedcd494eeedf1ce20f263ed0d1b Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 11:27:20 -0500 Subject: [PATCH 19/34] Update rfcs/template.md Co-authored-by: Alexandra --- rfcs/template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/template.md b/rfcs/template.md index c87366769a..a841b82f2a 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -48,7 +48,7 @@ Some other common subsections here are: sub-proposals could be organized as separate standalone RFCs, but this is not mandatory. If the change is insignificant or doesn't make any sense without the original proposal, you can have it in the RFC. -- Execution plan if approved, aka next steps. +- Execution plan (next steps), if approved. ## Process Specific Information From e449916077f7f3a3a77eff901b52bab39de0991c Mon Sep 17 00:00:00 2001 From: Michael Voss Date: Tue, 3 Sep 2024 13:54:49 -0500 Subject: [PATCH 20/34] Made wording changes in response to review. --- rfcs/README.md | 30 +++++++++++++++++++----------- rfcs/archived/README.md | 5 +++-- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/rfcs/README.md b/rfcs/README.md index c34e99f698..32add37623 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -1,12 +1,14 @@ # oneTBB Design Documents/RFCs -The purpose of the RFC process is to communicate the intent to make -library-wide changes, get feedback prior to the actual implementation, -increase the transparency on why and how decisions are made, and improve -the alignment between different teams involved in oneTBB development. +The RFC process intends to: + +- Communicate library-wide changes +- Collect feedback before implementation +- Increase transparency in decision-making +- Align different teams involved in oneTBB development This directory contains design documents (RFCs) approved -for implementation in oneTBB. +or rejected for implementation in oneTBB. The possible RFC states are: @@ -41,14 +43,17 @@ updated with additional information. A proposal that is subsequently implemented and released in oneTBB as a preview feature will be moved to the `rfcs/experimental` folder. The RFC for a preview feature in `rfcs/experimental` should include a description -of what is required to move from experimental to fully supported -- for example, feedback from users, demonstrated performance improvements, etc. +of what is required to move from experimental to fully supported -- for +example, feedback from users, demonstrated performance improvements, etc. A proposal that is implemented, added to the oneTBB specification, and -supported as a full feature appears in the `rfcs/supported` directory. An RFC for a fully supported feature in the `rfcs/supported` directory should +supported as a full feature appears in the `rfcs/supported` directory. An RFC +for a fully supported feature in the `rfcs/supported` directory should have a link to the section in the oneTBB specification that has its formal wording. -A feature that is deprecated or removed or a proposal that is abandoned may be moved to the `rfcs/archived` folder. +A feature that is removed or a proposal that is abandoned or rejected will +be moved to the `rfcs/archived` folder. ## Document Style @@ -56,9 +61,11 @@ The design documents are stored in the `rfcs` directory. - Each RFC is stored in a separate subdirectory `rfcs/proposed/_` -- There must be a `README.md` file that contains the main RFC itself (or links to a file that contains it in the same directory). +- There must be a `README.md` file that contains the main RFC itself (or +links to a file that contains it in the same directory). - The body of the RFC should be based on [template.md](template.md) - - The directory can contain other supporting files, such as images, tex formulas, and sub-proposals / sub-RFCs. + - The directory can contain other supporting files, such as images, tex + formulas, and sub-proposals / sub-RFCs. - It is not required to use the markdown file format based on the exact template file for writing RFCs. However, it is strongly recommended to use text-based file format that can be rendered by GitHub to allow for easy @@ -67,4 +74,5 @@ The design documents are stored in the `rfcs` directory. - If the RFC is written in markdown. The width of the text should be limited by 80 symbols, unless there is a need to violate this rule, e.g. because of long links or wide tables. -- It is also recommended to read through existing RFCs to better understand the general writing style and required elements. +- It is also recommended to read through existing RFCs to better understand the +general writing style and required elements. diff --git a/rfcs/archived/README.md b/rfcs/archived/README.md index 9dec2adca2..dea690377c 100644 --- a/rfcs/archived/README.md +++ b/rfcs/archived/README.md @@ -1,9 +1,10 @@ # Archived Design Documents Documents may appear in the `rfcs/archived` directory for one of -three reasons: +two reasons: -1. The document describes a feature or extension that has been deprecated and then removed. +1. The document describes a feature or extension that has been deprecated and +then removed. 2. The document describes a feature or extension that was proposed but did not (ultimately) become a fully support feature. From ad8ee516b6f5f812262dc52f8394a2ec3885e343 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:57:02 -0500 Subject: [PATCH 21/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 32add37623..bc4c251c43 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -12,7 +12,7 @@ or rejected for implementation in oneTBB. The possible RFC states are: -1. Initial RFC PR +1. Initial 2. Proposed 3. experimental 4. supported From 543131089b1932c3daa48aa8545bbed853c42470 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:57:21 -0500 Subject: [PATCH 22/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index bc4c251c43..0cd45a56e7 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -14,7 +14,7 @@ The possible RFC states are: 1. Initial 2. Proposed -3. experimental +3. Experimental 4. supported 5. archived From 6f309e047839bcd94fa10ba302627fe5b4eb8a19 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:57:36 -0500 Subject: [PATCH 23/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 0cd45a56e7..517714e6a3 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -15,7 +15,7 @@ The possible RFC states are: 1. Initial 2. Proposed 3. Experimental -4. supported +4. Supported 5. archived Most modifications or new features will naturally start as a part of a From fd42065991a9377c21a5e38069ef419ed172e19b Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:57:49 -0500 Subject: [PATCH 24/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 517714e6a3..7b828f31c6 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -16,7 +16,7 @@ The possible RFC states are: 2. Proposed 3. Experimental 4. Supported -5. archived +5. Archived Most modifications or new features will naturally start as a part of a Github issue or discussion. Small changes do not require a formal RFC. From 344688e659affb044f6820a3848aa4dbb083cd2c Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:58:04 -0500 Subject: [PATCH 25/34] Update rfcs/README.md Co-authored-by: Aleksei Fedotov --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 7b828f31c6..bd5641ef13 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -19,7 +19,7 @@ The possible RFC states are: 5. Archived Most modifications or new features will naturally start as a part of a -Github issue or discussion. Small changes do not require a formal RFC. +GitHub issue or discussion. Small changes do not require a formal RFC. However, if the issue or discussion leads to an idea for a modification or new feature that significantly impacts the library's public API or architecture, it will be suggested that a PR be opened to add a new rfc From 074a1e2974364c656ffbb7ccc93bac4c46572aef Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:58:31 -0500 Subject: [PATCH 26/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index bd5641ef13..6280a775aa 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -20,7 +20,7 @@ The possible RFC states are: Most modifications or new features will naturally start as a part of a GitHub issue or discussion. Small changes do not require a formal RFC. -However, if the issue or discussion leads to an idea for a modification +However, if the issue or discussion results in an idea for a significant change or new feature that affects the library's public API or architecture, we recommend opening a PR to add a new RFC to the `rfcs/proposed` directory. The RFC should provide a detailed description and design of the proposed feature. or new feature that significantly impacts the library's public API or architecture, it will be suggested that a PR be opened to add a new rfc to the `rfcs/proposed` directory. The RFC contains a more detailed description From d407d0c759dfa73d0e94ba959667c98d91c82011 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:58:43 -0500 Subject: [PATCH 27/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 6280a775aa..5fa7bfe8cb 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -28,7 +28,7 @@ and design for the feature. ## General Process -A template for RFCs is available as [template.md](template.md). The modified +A template for RFCs is available as [template.md](template.md). Place the modified template should be placed in subdirectory of the `rfcs/proposed` with a name of the form `_`. For example, a proposal for a new "my_op" flow graph node might be put into a directory From 207f07d676f96100d9ea266dd855b324b5d26b0b Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 13:59:10 -0500 Subject: [PATCH 28/34] Update rfcs/README.md Co-authored-by: Alexandra --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index 5fa7bfe8cb..695cacccc6 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -29,7 +29,7 @@ and design for the feature. ## General Process A template for RFCs is available as [template.md](template.md). Place the modified -template should be placed in subdirectory of the `rfcs/proposed` with a name +template in the subdirectory of the `rfcs/proposed` with a name of the form `_`. For example, a proposal for a new "my_op" flow graph node might be put into a directory `rfcs/proposed/flow_graph_my_op_node`. The [template.md](template.md) should From 62d9188f03831d7585980bcde714f2c1a9c6a936 Mon Sep 17 00:00:00 2001 From: Mike Voss Date: Tue, 3 Sep 2024 14:04:31 -0500 Subject: [PATCH 29/34] Apply suggestions from code review Made suggested wording changes. Co-authored-by: Alexandra Co-authored-by: Aleksei Fedotov --- rfcs/README.md | 27 ++++++++++++--------------- rfcs/archived/README.md | 8 ++++---- rfcs/experimental/README.md | 20 ++++++++++---------- rfcs/proposed/README.md | 11 +++++------ rfcs/supported/README.md | 6 +++--- rfcs/template.md | 6 +++--- 6 files changed, 37 insertions(+), 41 deletions(-) diff --git a/rfcs/README.md b/rfcs/README.md index 695cacccc6..a204e34364 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -31,17 +31,16 @@ and design for the feature. A template for RFCs is available as [template.md](template.md). Place the modified template in the subdirectory of the `rfcs/proposed` with a name of the form `_`. For example, -a proposal for a new "my_op" flow graph node might be put into a directory -`rfcs/proposed/flow_graph_my_op_node`. The [template.md](template.md) should -be used to create the `README.md` file in that directory. The folder can +a proposal for a new ``my_op`` flow graph node should be put into the +`rfcs/proposed/flow_graph_my_op_node` directory. Use [template.md](template.md) +to create the `README.md` file in that directory. The folder can contain other files referenced by the `README.md` file, such as figures. -When two maintainers approve the PR it will be merged to the `rfcs/proposed` -directory. As the RFC moves to different states, the RFC document should be -updated with additional information. +Once two maintainers approve the PR, it is merged into the `rfcs/proposed` +directory. Update the RFC document with additional information as the RFC moves to different states. A proposal that is subsequently implemented and released in oneTBB -as a preview feature will be moved to the `rfcs/experimental` folder. The +as a preview feature is moved into the `rfcs/experimental` folder. The RFC for a preview feature in `rfcs/experimental` should include a description of what is required to move from experimental to fully supported -- for example, feedback from users, demonstrated performance improvements, etc. @@ -49,7 +48,7 @@ example, feedback from users, demonstrated performance improvements, etc. A proposal that is implemented, added to the oneTBB specification, and supported as a full feature appears in the `rfcs/supported` directory. An RFC for a fully supported feature in the `rfcs/supported` directory should -have a link to the section in the oneTBB specification that has its +have a link to the section in the oneTBB specification with its formal wording. A feature that is removed or a proposal that is abandoned or rejected will @@ -57,22 +56,20 @@ be moved to the `rfcs/archived` folder. ## Document Style -The design documents are stored in the `rfcs` directory. +The design documents are stored in the `rfcs` directory, and each RFC is placed in its subdirectory under `rfcs/proposed/_`. -- Each RFC is stored in a separate subdirectory - `rfcs/proposed/_` - There must be a `README.md` file that contains the main RFC itself (or links to a file that contains it in the same directory). - - The body of the RFC should be based on [template.md](template.md) + - The RFC should follow the [template.md](template.md) structure. - The directory can contain other supporting files, such as images, tex formulas, and sub-proposals / sub-RFCs. - - It is not required to use the markdown file format based on the exact + - We highly recommend using a text-based file format like markdown for easy collaboration on GitHub, but other formats like PDFs may also be acceptable. template file for writing RFCs. However, it is strongly recommended to use text-based file format that can be rendered by GitHub to allow for easy collaboration using PR comments. Even so, files such as pdfs may be acceptable. -- If the RFC is written in markdown. The width of the text should be limited by - 80 symbols, unless there is a need to violate this rule, e.g. because of +- For the markdown-written RFC, keep the text width within + 80 characters, unless there is a reason to violate this rule, e.g., long links or wide tables. - It is also recommended to read through existing RFCs to better understand the general writing style and required elements. diff --git a/rfcs/archived/README.md b/rfcs/archived/README.md index dea690377c..7c23aa3efe 100644 --- a/rfcs/archived/README.md +++ b/rfcs/archived/README.md @@ -5,10 +5,10 @@ two reasons: 1. The document describes a feature or extension that has been deprecated and then removed. -2. The document describes a feature or extension that was proposed but did -not (ultimately) become a fully support feature. +2. The document describes a proposed feature or extension that have +not (ultimately) become a fully supported feature. -Design documents that appear in the `rfcs/archived` folder should include a -section that describes why the document has been archived. Documents may +Design documents that appear in the `rfcs/archived` folder should describe a +reason for archiving. Documents may remain in this folder indefinitely to serve as a source of information about previous proposals and features. diff --git a/rfcs/experimental/README.md b/rfcs/experimental/README.md index 036ddc325b..3793e84f74 100644 --- a/rfcs/experimental/README.md +++ b/rfcs/experimental/README.md @@ -1,28 +1,28 @@ -# Design documents for experimental features +# Design Documents for Experimental Features -Experimental proposals describe extensions that have been implemented and -have been released as preview features in the oneTBB library. A preview +Experimental proposals describe extensions that are implemented and +released as preview features in the oneTBB library. A preview feature is expected to have an implementation that is of comparable quality -to a fully supported feature. It should also have sufficient tests. +to a fully supported feature. Sufficient tests are required. An experimental feature does not yet appear as part of the oneTBB -specification and therefore its interface and design can still change. -There is no commitment to backwards compatibility for a preview +specification. Therefore, the interface and design can change. +There is no commitment to backward compatibility for a preview feature. -Since documents in this directory describe preview features, they +The documents in this directory should include a list of the exit conditions that need to be met to move from preview to fully supported. These conditions might include demonstrated performance improvements, demonstrated interest from the community, acceptance of the required oneTBB specification changes, etc. For features that require oneTBB specification changes, the document might -include wording for those changes or a link to any PRs that have been opened +include wording for those changes or a link to any PRs that opened against the specification. Proposals should not remain in the experimental directory forever. The -expectation is that the proposals will eventually move either to the -supported folder when they become fully supported or to the archived +It should move either to the +supported folder when they become fully supported or the archived folder if they are not fully accepted. It should be highly unusual for a proposal to stay in the experimental folder for longer than a year or two. diff --git a/rfcs/proposed/README.md b/rfcs/proposed/README.md index 2a4d3e7110..af243fad5e 100644 --- a/rfcs/proposed/README.md +++ b/rfcs/proposed/README.md @@ -1,10 +1,9 @@ -# Design documents for proposed features +# Design Documents for Proposed Features -Proposed features have reached some level of consensus within the -community. A proposal appears in this directory because the -community believes it has merit and further work should be done. +Proposed features in this directory have reached some level of consensus within the +community, indicating that they have potential and deserve further development. However, the proposed changes have not yet been released as a preview or fully supported feature of the library. -RFCs in the `rfcs/proposed` directory, should describe the motivation, -design and open questions related to the proposed extension. +RFCs in the `rfcs/proposed` directory should explain the motivation, +design, and open questions related to the proposed extension. diff --git a/rfcs/supported/README.md b/rfcs/supported/README.md index e349670506..ebbb35150d 100644 --- a/rfcs/supported/README.md +++ b/rfcs/supported/README.md @@ -1,9 +1,9 @@ -# Design documents for supported features +# Design Documents for Supported Features -Supported proposals describe extensions that have been implemented and +Supported proposals describe extensions implemented and released as fully supported features of the oneTBB library. A fully supported feature has a high-quality implementation. If the proposal impacted the -public API of the library it should appear in the oneTBB specification and +public API of the library, it should appear in the oneTBB specification and have supporting documentation in the oneTBB Reference as needed. A fully supported feature is regularly tested. diff --git a/rfcs/template.md b/rfcs/template.md index a841b82f2a..654ed5920f 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -1,4 +1,4 @@ -# Descriptive name for the proposal +# Descriptive Name for the Proposal ## Introduction @@ -16,11 +16,11 @@ discussions, etc. ## Proposal -A full and detailed description of the proposal, with highlighted consequences. +A full and detailed description of the proposal with highlighted consequences. Depending on the kind of the proposal, the description should cover: -- New use cases that are supported by the extension. +- New use cases supported by the extension. - The expected performance benefit for a modification. - The interface of extensions including class definitions or function declarations. From 30c06cab2f021fa27d13596b8ffb64141fb940dc Mon Sep 17 00:00:00 2001 From: Michael Voss Date: Tue, 3 Sep 2024 14:08:44 -0500 Subject: [PATCH 30/34] Fixed line lengths and made suggested changes after review --- rfcs/README.md | 14 ++++++++++---- rfcs/template.md | 6 +++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/rfcs/README.md b/rfcs/README.md index a204e34364..b1ae271b50 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -20,7 +20,10 @@ The possible RFC states are: Most modifications or new features will naturally start as a part of a GitHub issue or discussion. Small changes do not require a formal RFC. -However, if the issue or discussion results in an idea for a significant change or new feature that affects the library's public API or architecture, we recommend opening a PR to add a new RFC to the `rfcs/proposed` directory. The RFC should provide a detailed description and design of the proposed feature. +However, if the issue or discussion results in an idea for a significant +change or new feature that affects the library's public API or architecture, +we recommend opening a PR to add a new RFC to the `rfcs/proposed` directory. +The RFC should provide a detailed description and design of the proposed feature. or new feature that significantly impacts the library's public API or architecture, it will be suggested that a PR be opened to add a new rfc to the `rfcs/proposed` directory. The RFC contains a more detailed description @@ -37,7 +40,8 @@ to create the `README.md` file in that directory. The folder can contain other files referenced by the `README.md` file, such as figures. Once two maintainers approve the PR, it is merged into the `rfcs/proposed` -directory. Update the RFC document with additional information as the RFC moves to different states. +directory. Update the RFC document with additional information as the RFC moves +to different states. A proposal that is subsequently implemented and released in oneTBB as a preview feature is moved into the `rfcs/experimental` folder. The @@ -56,14 +60,16 @@ be moved to the `rfcs/archived` folder. ## Document Style -The design documents are stored in the `rfcs` directory, and each RFC is placed in its subdirectory under `rfcs/proposed/_`. +The design documents are stored in the `rfcs` directory, and each RFC is placed +in its subdirectory under `rfcs/proposed/_`. - There must be a `README.md` file that contains the main RFC itself (or links to a file that contains it in the same directory). - The RFC should follow the [template.md](template.md) structure. - The directory can contain other supporting files, such as images, tex formulas, and sub-proposals / sub-RFCs. - - We highly recommend using a text-based file format like markdown for easy collaboration on GitHub, but other formats like PDFs may also be acceptable. + - We highly recommend using a text-based file format like markdown for easy + collaboration on GitHub, but other formats like PDFs may also be acceptable. template file for writing RFCs. However, it is strongly recommended to use text-based file format that can be rendered by GitHub to allow for easy collaboration using PR comments. Even so, files such as pdfs may be diff --git a/rfcs/template.md b/rfcs/template.md index 654ed5920f..cf407e78fb 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -25,9 +25,9 @@ Depending on the kind of the proposal, the description should cover: - The interface of extensions including class definitions or function declarations. -A proposal should include the alternatives that were considered with listed -pros and cons. The alternatives should be clearly separated to make possible -discussions clear. +A proposal should clearly outline the alternatives that were considered, +along with their pros and cons. Each alternative should be clearly separated +to make discussions easier to follow. Pay close attention to the following aspects of the library: - API and ABI backward compatibility. The library follows semantic versioning From 93a5a817522c6255b651320cddab798e9edfcf93 Mon Sep 17 00:00:00 2001 From: kboyarinov Date: Wed, 11 Sep 2024 05:04:57 -0500 Subject: [PATCH 31/34] Add RFC draft --- .../flow_graph_wait_for_one/README.md | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 rfcs/experimental/flow_graph_wait_for_one/README.md diff --git a/rfcs/experimental/flow_graph_wait_for_one/README.md b/rfcs/experimental/flow_graph_wait_for_one/README.md new file mode 100644 index 0000000000..9fc2e0d91f --- /dev/null +++ b/rfcs/experimental/flow_graph_wait_for_one/README.md @@ -0,0 +1,97 @@ +# Waiting for single messages in the Flow Graph + +Extending the oneTBB Flow Graph interface with the new ``try_put_and_wait(msg)`` API which allows to wait +for completion of the chain of tasks corresponding to the ``msg``. + +The feature should improve the Flow Graph performance in scenarios where multiple threads submitts the work into the +Flow Graph simultaneously and each of them should + +## Introduction + +Current API of oneTBB Flow Graph allows doing two basic actions after building the graph: + +- Submitting messages in some nodes using the ``node.try_put(msg)`` API. +- Waiting for completion of **all** messagges in ghe graph using the ``graph.wait_for_all()`` API. + +Since the only API currently available for waiting until the work would be completed waits for all tasks in the graph +to complete, there can be negative performance impact in use cases when the thread submitting the work should be notified as soon as possible +when the work is done. Having only ``graph.wait_for_all()`` forces the thread to wait until **all** of the tasks in the Flow Graph, no metter +corresponds to the waited message or not. + +Consider the following example: + +```cpp + +struct ComputeInput; +struct ComputeOutput; + +// High-level computation instance that performs some work +// using the oneTBB Flow Graph under the hood +// but the Flow Graph details are not expressed as part of public API +class ComputeTool +{ +private: + oneapi::tbb::flow::graph m_graph; + oneapi::tbb::flow::broadcast_node m_start_node; + // Other Flow Graph nodes +public: + ComputeTool() + { + // Builds the Flow Graph + } + + // Peforms the computation using the user-provided input + ComputeOutput compute(ComputeInput input) + { + m_start_node.try_put(input); // Submit work in the graph + m_graph.wait_for_all(); // Waiting for input to be processed + } +}; + +int main() +{ + ComputeTool shared_tool; + + oneapi::tbb::parallel_for(0, 10, + [&](int i) + { + // Preparing the input for index i + + ComputeOutput output = shared_tool.compute(input); + + // Post process output for index i + }); +} + +``` + +The ``ComputeTool`` is a user interface that performs some computations on top of oneTBB Flow Graph. The function ``compute(input)`` should +perform the computations for the provided input. Since the Flow Graph is used, it should submit the message into the graph and wait for its +completion. This function also can be called concurrently from several threads as it shown in the ``main``. + +Since the only available API in the Flow Graph is ``wait_for_all()``, each thread submitting the work to the graph would be required to wait +until **all** of the tasks would be done, no metter if these tasks corresponds to the input processed by this thread or not. If some +post-processing is required on each thread after receiving the computation result, it would be only possible to start it when the Flow Graph would be completed what can be inneficient if the post-processing of lightweight graph tasks would be blocked by processing the more mature input. + +To get rid of this negative performance effect, it would be useful to add some kind of new API to the Flow Graph that would wait for +completion of only one message (instead of the full completion of the graph): + +```cpp +ComputeOutput compute(ComputeInput input) +{ + m_start_node.try_put(input); // Submit work in the graph + g.wait_for_one(input); // Pseudo-code. Wait for completion of only messages for computations of input +} +``` + +## Proposal + +The idea of this proposal is to extend the existing Flow Graph API with the new member function of each receiver nodes - +``node.try_put_and_wait(msg)``. This function should submit the msg into the Flow Graph (similarly to ``try_put()``) and wait for its completion. The function should be exited only if all of the tasks corresponds to the ``msg`` and skip waiting for any other tasks to +complete. + + + +## Combined or separated + +## Process Specific Information From 8c29592f405cce5d0c4fe4c68cad3e85f401e05f Mon Sep 17 00:00:00 2001 From: "Boyarinov, Konstantin" Date: Fri, 13 Sep 2024 12:34:49 +0300 Subject: [PATCH 32/34] Add try_put_and_wait RFC --- .../flow_graph_wait_for_one/README.md | 251 +++++++++++++++++- 1 file changed, 250 insertions(+), 1 deletion(-) diff --git a/rfcs/experimental/flow_graph_wait_for_one/README.md b/rfcs/experimental/flow_graph_wait_for_one/README.md index 9fc2e0d91f..141f7590be 100644 --- a/rfcs/experimental/flow_graph_wait_for_one/README.md +++ b/rfcs/experimental/flow_graph_wait_for_one/README.md @@ -90,8 +90,257 @@ The idea of this proposal is to extend the existing Flow Graph API with the new ``node.try_put_and_wait(msg)``. This function should submit the msg into the Flow Graph (similarly to ``try_put()``) and wait for its completion. The function should be exited only if all of the tasks corresponds to the ``msg`` and skip waiting for any other tasks to complete. +Consider the following graph: +```cpp + +using namespace oneapi::tbb; + +flow::graph g; + +flow::broadcast_node start(g); + +flow::function_node f1(g, unlimited, f1_body); +flow::function_node f2(g, unlimited, f2_body); +flow::function_node f3(g, unlimited, f3_body); + +flow::join_node> join(g); + +flow::function_node pf(g, serial, pf_body); + +flow::make_edge(start, f1); +flow::make_edge(start, f2); +flow::make_edge(start, f3); + +flow::make_edge(f1, flow::input_port<0>(join)); +flow::make_edge(f2, flow::input_port<1>(join)); +flow::make_edge(f3, flow::input_port<2>(join)); + +flow::make_edge(join, pf); + +// Parallel submittion +oneapi::tbb::parallel_for (0, 100, [](int input) { + start.try_put(input); +}); + +start.try_put_and_wait(444); +// Post-processing 444 + +g.wait_for_all(); +``` + + + +Each message is broadcasted from ``start`` to three concurrent computational functions ``f1``, ``f2`` and ``f3``. The result is when joined into single tuple in ``join`` node and +post-processed in a serial ``pf`` function node. The task queue corresponding to each node in the graph is exposed under the node in the picture. The tasks that corresponds +to the parallel loop 0-100 are shown as blue tasks in the queue. Red tasks corresponds to the message submitted as an argument in ``try_put_and_wait``. +The ``try_put_and_wait`` is expected to exit when all of the red tasks and the necessary amount of blue tasks would be completed. Completion of all blue tasks as in ``wait_for_all`` +is not guaranteed. + +From the implementation perspective, the feature is implemented currently by creating an instance of special class ``message_metainfo`` with the input message in ``try_put_and_wait`` +and then broadcast it through the graph together with the message. The actual value of message can be changed during the computation but the stored metainformation should be preserved. + +When the message is buffered in one of the buffering nodes or one of the internal buffers (such as ``queueing`` ``function_node`` or ``join_node``), the corresponding metainformation +instance should be buffered as well. + +For reference counting on single messages, the dedicated ``wait_context`` is assigned to each message passed to ``try_put_and_wait``. It is possible to use ``wait_context`` itself +instead of ``message_metainfo``, but it can be useful to pass something with each message through the graph, not only for single message waiting. The initial implementation of +``message_metainfo`` just wrapping the ``wait_context``, but it can be extended to cover additional use-cases. Each task corresponding to the completion of the message +associated with the awaited message holds the reference counting on the corresponding ``wait_context``. In case of buffering the message somewhere in the graph, +the additional reference counter would be held and released when the item is removed from the buffer. + +From the implementation perspective, working with metainformation is exposed by adding the new internal virtual functions in the Flow Graph: + +| Base Template Class | Existing Function Signature | New Function Signature | Information | +|---------------------|-------------------------------|-----------------------------------------------------|--------------------------------------------------| +| receiver | bool try_put_task(const T& t) | bool try_put_task(const T& t) | Performs an action required by the node logic. | +| | | bool try_put_task(const T& t, | May buffer both ``t`` and ``metainfo``. | +| | | const message_metainfo& metainfo) | May broadcast the result and ``metainfo`` to | +| | | | successors of the node. | +| | | | The first function can reuse the second with | +| | | | the empty metainfo. | +|---------------------|-------------------------------|-----------------------------------------------------|--------------------------------------------------| +| sender | bool try_get(T& t) | bool try_get(T& t) | For buffers, gets the element from the buffer. | +| | | bool try_get(T& t, message_metainfo&) | The second function provides both placeholders | +| | | | for metainformation and the element. | +|---------------------|-------------------------------|-----------------------------------------------------|--------------------------------------------------| +| sender | bool try_reserve(T& t) | bool try_reserve(T& t) | For buffers, reserves the element in the buffer. | +| | | bool try_reserve(T& t, message_metainfo& metainfo) | The second function provides both placeholders | +| | | | for metainformantion and the element. | +|---------------------|-------------------------------|-----------------------------------------------------|--------------------------------------------------| + +The ``message_metainfo`` class is described in details in the [separate section](#details-about-metainformation-class). +The [Nodes behavior](#nodes-behavior) section describes the behavior of each particular node when the metainformation is received. + +## Nodes behavior + +This chapter describes detailed behavior of each Flow Graph node when the item and the metainformation is received. Similarly to the message itself, the metainformation +can be received from the predecessor node (explicit ``try_put_task`` call) or initially from ``try_put_and_wait``. + +### ``function_node`` + +If the concurrency of the ``function_node`` is ``unlimited``, the node creates a task for executting the body of the node. The created task should hold the metainfo +received by the function node and broadcast it to the node successors when the task is completed. + +Otherwise, similarly to the original ``function_node`` behavior, the node tries to occupy its concurrency. If the limit is not yet reached, creates a body task +similarly to the ``unlimited`` case. If the concurrency limit is reached, both input message and the associated metainformation would be stored in the internal queue, associated +with the node. When one of the other tasks, associated with the node would be completed, it will retrieve the postponed message together with the metainformation and spawn it as +a task. + +Since the ``function_node`` guarantees that all of the elements would be retrieved from the internal queue at some time, [buffering issues](#buffering-the-metainfo) cannot take place. + +### ``function_node`` + +For the ``unlimited`` use-case, behaves the same as ``queueing`` node. + +If the concurrency limit of the node is reached, both message and the associated metainfo would be rejected and it is a predecessor responsibility to buffer them. +If the predecessor is not the buffering node, both message and the metainfo would be lost. +When another task would be completed, it will try to get a buffered message together with the metainfo (by calling the ``try_get(msg, metainfo)`` method) from the predecessing node. + +Since the ``function_node`` guarantees that all of the elements would be retrieved from the internal queue at some time, [buffering issues](#buffering-the-metainfo) cannot take place +for buffering nodes, preceding the ``function_node``. + +### ``function_node`` + +In regard to the concurrency limit, the lightweight function node behaves as it described in the corresponding buffering policy section (``queueing`` or ``rejecting``). +The only difference is that for such nodes the tasks would not be spawned and the associated function will be executted by the calling thread. And since we don't have tasks, +the calling thread should broadcast the metainformation to the successors after completing the function. + +### ``continue_node`` + +The ``continue_node`` has one of the most specific semantics in regard to the metainformation. Since the node only executes the associated body (and broadcasts the signal +to the successors) if it receives ``N`` signals from it's predecessors (where ``N`` is the number of predecessors). It means that prior to executing the body, +the node can receive several metainformation instances from different predecessors. + +To handle this, the ``continue_node`` initially stores an empty metainfo instance and on each ``try_put_task(continue_msg, metainfo)`` call, it [merges](#details-about-metainformation-class) +the received metainformation with the stored instance. Under the hood the merged instance will contain the ``wait_context`` pointers from its previous state and all of the pointers from +the received ``metainfo``. + +When the ``continue_node`` receives ``N`` signals from the predecessors, it wraps stored metainformation into the task for completion of the associated body. Once the task is ready, the stored +metainformation instance switchs back to the empty state for further work. Once the function would be completed, the task is expected to broadcast the metainfo to the successors. + +The lightweight ``continue_node`` behaves the same as described above, but without spawning any tasks. Everything would be performed by the calling thread. + +### Multi-output functional nodes + +``multifunction_node`` and ``async_node`` classes are not currently supported by this feature because of issues described in [the separate section](#multi-output-nodes-support). + +Passing the metainformation to such a node by the predecessor would have no effect and no metainfo would be broadcasted to further successors. + +### Single-push buffering nodes + +This section describes the behavior for ``buffer_node``, ``queue_node``, ``priority_queue_node`` and ``sequencer_node`` classes. The only difference between them would be in +ordering of retrieving the elements from the buffer. + +Once the buffering node receives a message and the metainformation, both of them should be stored into the buffer. + +Since bufferring nodes are commonly used as part of the Flow Graph push-pull protocol, e.g. before the rejecting ``function_node`` or reserving ``join_node``, it means that +the waiting for the message should be prolonged once it stored in the buffer. In particular, once the metainformation is in the buffer, the buffer should call ``reserve(1)`` on each +associated ``wait_context`` to prolongue the wait and call ``release(1)`` once the element is retrieved from the buffer (while calling ``try_get`` or ``try_consume``). + +Once the element and the metainfo are stored in the buffer, the node will try to push them to the successor. If one of the successors accepts the message and the metainfo, +both of them are removed from the buffer. Otherwise, the push-pull protocol works and the successor should return for the item once it becomes available by calling +``try_get(msg, metainfo)`` or ``try_reserve(msg, metainfo)``. -## Combined or separated +Since placing the buffers before rejecting nodes is not the only use-case, there is a risk of issues relates to buffering. It is described in details in the [separate section](#buffering-the-metainfo). + +### Broadcast-push buffering nodes + +The issue with broadcast-push ``overwrite_node`` and ``write_once_node`` is these nodes stores the received item and even if this item is accepted by one successor, it would be broadcasted to others and +kept in the buffer. Since the metainformation is kept in the buffer together with the message itself, even if the computation is completed, the ``try_put_and_wait`` would stuck because of the reference +held by the buffer. + +Even the ``wait_for_all()`` call would be able to finish in this case since it counting only the tasks in progress and ``try_put_and_wait`` would still stuck. + +``try_put_and_wait`` feature for the graph containing these nodes should be used carefully because of this issue: + +* The ``overwrite_node`` should be explicitly resetted by calling ``node.reset()`` or the element with the stored metainfo should be overwritten with another element without metainfo. +* The ``write_once_node`` should be explicitly resetted by calling ``node.reset()`` since the item cannot be overwritten. + +### ``broadcast_node`` + +The behavior of ``broadcast_node` is preety obvious - the metainformation would just be broadcasted to each successor of the node. + +### ``limiter_node`` + +### ``join_node`` + +### ``join_node`` + +### ``join_node`` + +### ``split_node`` + +### ``indexer_node`` + +### ``composite_node`` + +## Additional implementation details + +### Details about metainformation class + +``message_metainfo`` class synopsis: + +```cpp +class message_metainfo +{ +public: + using waiters_type = std::forward_list; + + message_metainfo(); + + message_metainfo(const message_metainfo&); + message_metainfo(message_metainfo&&); + + message_metainfo& operator=(const message_metainfo&); + message_metainfo& operator=(message_metainfo&&); + + const waiters_type& waiters() const &; + waiters_type&& waiters() &&; + + bool empty() const; + + void merge(const message_metainfo&); +}; +``` + +The initial implementation of ``message_metainfo`` class wraps only the list of single message waiters. The class may be extended if necessary to cover additional use-cases. + +The metainfo is required to hold a list of message waiters instead of single waiter to cover the ``continue_node`` and ``join_node`` joining use-cases. Consider the example: + +```cpp +using namespace oneapi::tbb; + +flow::function_node start1(g, ...); +flow::function_node start2(g, ...); + +flow::join_node> join(g); + +flow::function_node, int> post_process(g, ...); + +flow::make_edge(start, flow::input_port<0>(join)); +flow::make_edge(start, flow::input_port<1>(join)); +flow::make_edge(join, post_process); + +std::thread t1([&]() { + start1.try_put_and_wait(1); +}); + +std::thread t2([&]() { + start2.try_put_and_wait(2); +}) +``` + + + +### Combined or separated wait + +### Buffering the metainfo + +### Multi-output nodes support + +### Optimization while retrieving from buffer to task ## Process Specific Information + +Open questions: From 9abea43fbf7fb49b5003059abd0029ecc1bd1391 Mon Sep 17 00:00:00 2001 From: "Boyarinov, Konstantin" Date: Mon, 6 Jan 2025 04:31:59 -0800 Subject: [PATCH 33/34] Remove whitespace changes --- rfcs/README.md | 58 ++++++++++++++++++------------------- rfcs/experimental/README.md | 12 ++++---- rfcs/template.md | 10 +++---- 3 files changed, 40 insertions(+), 40 deletions(-) diff --git a/rfcs/README.md b/rfcs/README.md index 831c60d00d..15995290bd 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -4,28 +4,28 @@ The RFC process intends to: - Communicate library-wide changes - Collect feedback before implementation -- Increase transparency in decision-making +- Increase transparency in decision-making - Align different teams involved in oneTBB development -This directory contains design documents (RFCs) approved +This directory contains design documents (RFCs) approved or rejected for implementation in oneTBB. The possible RFC states are: -1. Initial +1. Initial 2. Proposed 3. Experimental 4. Supported 5. Archived -Most modifications or new features will naturally start as a part of a -GitHub issue or discussion. Small changes do not require a formal RFC. -However, if the issue or discussion results in an idea for a significant -change or new feature that affects the library's public API or architecture, -we recommend opening a PR to add a new RFC to the `rfcs/proposed` directory. +Most modifications or new features will naturally start as a part of a +GitHub issue or discussion. Small changes do not require a formal RFC. +However, if the issue or discussion results in an idea for a significant +change or new feature that affects the library's public API or architecture, +we recommend opening a PR to add a new RFC to the `rfcs/proposed` directory. The RFC should provide a detailed description and design of the proposed feature. -or new feature that significantly impacts the library's public API or -architecture, it will be suggested that a PR be opened to add a new rfc +or new feature that significantly impacts the library's public API or +architecture, it will be suggested that a PR be opened to add a new rfc to the `rfcs/proposed` directory. The RFC contains a more detailed description and design for the feature. @@ -35,43 +35,43 @@ A template for RFCs is available as [template.md](template.md). Place the modifi template in the subdirectory of the `rfcs/proposed` with a name of the form `_`. For example, a proposal for a new ``my_op`` flow graph node should be put into the -`rfcs/proposed/flow_graph_my_op_node` directory. Use [template.md](template.md) -to create the `README.md` file in that directory. The folder can +`rfcs/proposed/flow_graph_my_op_node` directory. Use [template.md](template.md) +to create the `README.md` file in that directory. The folder can contain other files referenced by the `README.md` file, such as figures. Once two maintainers approve the PR, it is merged into the `rfcs/proposed` -directory. Update the RFC document with additional information as the RFC moves -to different states. +directory. Update the RFC document with additional information as the RFC moves +to different states. -A proposal that is subsequently implemented and released in oneTBB +A proposal that is subsequently implemented and released in oneTBB as a preview feature is moved into the `rfcs/experimental` folder. The RFC for a preview feature in `rfcs/experimental` should include a description -of what is required to move from experimental to fully supported -- for +of what is required to move from experimental to fully supported -- for example, feedback from users, demonstrated performance improvements, etc. -A proposal that is implemented, added to the oneTBB specification, and -supported as a full feature appears in the `rfcs/supported` directory. An RFC -for a fully supported feature in the `rfcs/supported` directory should -have a link to the section in the oneTBB specification with its +A proposal that is implemented, added to the oneTBB specification, and +supported as a full feature appears in the `rfcs/supported` directory. An RFC +for a fully supported feature in the `rfcs/supported` directory should +have a link to the section in the oneTBB specification with its formal wording. -A feature that is removed or a proposal that is abandoned or rejected will +A feature that is removed or a proposal that is abandoned or rejected will be moved to the `rfcs/archived` folder. ## Document Style -The design documents are stored in the `rfcs` directory, and each RFC is placed -in its subdirectory under `rfcs/proposed/_`. +The design documents are stored in the `rfcs` directory, and each RFC is placed +in its subdirectory under `rfcs/proposed/_`. -- There must be a `README.md` file that contains the main RFC itself (or +- There must be a `README.md` file that contains the main RFC itself (or links to a file that contains it in the same directory). - - The RFC should follow the [template.md](template.md) structure. - - The directory can contain other supporting files, such as images, tex + - The RFC should follow the [template.md](template.md) structure. + - The directory can contain other supporting files, such as images, tex formulas, and sub-proposals / sub-RFCs. - - We highly recommend using a text-based file format like markdown for easy + - We highly recommend using a text-based file format like markdown for easy collaboration on GitHub, but other formats like PDFs may also be acceptable. - For the markdown-written RFC, keep the text width within - 100 characters, unless there is a reason to violate this rule, e.g., + 100 characters, unless there is a reason to violate this rule, e.g., long links or wide tables. -- It is also recommended to read through existing RFCs to better understand the +- It is also recommended to read through existing RFCs to better understand the general writing style and required elements. diff --git a/rfcs/experimental/README.md b/rfcs/experimental/README.md index 9dcd233aee..612e69f032 100644 --- a/rfcs/experimental/README.md +++ b/rfcs/experimental/README.md @@ -5,16 +5,16 @@ released as preview features in the oneTBB library. A preview feature is expected to have an implementation that is of comparable quality to a fully supported feature. Sufficient tests are required. -An experimental feature does not yet appear as part of the oneTBB +An experimental feature does not yet appear as part of the oneTBB specification. Therefore, the interface and design can change. There is no commitment to backward compatibility for a preview feature. -The documents in this directory +The documents in this directory should include a list of the exit conditions that need to be met to move from preview to fully supported. These conditions might include demonstrated performance improvements, demonstrated interest from the community, -acceptance of the required oneTBB specification changes, etc. +acceptance of the required oneTBB specification changes, etc. For features that require oneTBB specification changes, the document might include wording for those changes or a link to any PRs that opened @@ -22,7 +22,7 @@ against the specification. Proposals should not remain in the experimental directory forever. It should move either to the -supported folder when they become fully supported or the archived -folder if they are not fully accepted. It should be highly unusual for -a proposal to stay in the experimental folder for longer than a year or +supported folder when they become fully supported or the archived +folder if they are not fully accepted. It should be highly unusual for +a proposal to stay in the experimental folder for longer than a year or two. diff --git a/rfcs/template.md b/rfcs/template.md index 9c589e60b9..834840a5ea 100644 --- a/rfcs/template.md +++ b/rfcs/template.md @@ -2,7 +2,7 @@ ## Introduction -Short description of the idea proposed with explained motivation. +Short description of the idea proposed with explained motivation. The motivation could be: - Improved users experience for API changes and extensions. Code snippets to @@ -21,12 +21,12 @@ A full and detailed description of the proposal with highlighted consequences. Depending on the kind of the proposal, the description should cover: - New use cases supported by the extension. -- The expected performance benefit for a modification. -- The interface of extensions including class definitions or function +- The expected performance benefit for a modification. +- The interface of extensions including class definitions or function declarations. -A proposal should clearly outline the alternatives that were considered, -along with their pros and cons. Each alternative should be clearly separated +A proposal should clearly outline the alternatives that were considered, +along with their pros and cons. Each alternative should be clearly separated to make discussions easier to follow. Pay close attention to the following aspects of the library: From ecd4e83cb2b286aa2669e20b11664c7134b4287c Mon Sep 17 00:00:00 2001 From: "Boyarinov, Konstantin" Date: Mon, 6 Jan 2025 06:29:52 -0800 Subject: [PATCH 34/34] Fix spelling, add information about multi-output nodes support --- .../flow_graph_wait_for_one/README.md | 327 ++++++++++++++++-- 1 file changed, 306 insertions(+), 21 deletions(-) diff --git a/rfcs/experimental/flow_graph_wait_for_one/README.md b/rfcs/experimental/flow_graph_wait_for_one/README.md index 141f7590be..da3d19ef91 100644 --- a/rfcs/experimental/flow_graph_wait_for_one/README.md +++ b/rfcs/experimental/flow_graph_wait_for_one/README.md @@ -3,7 +3,7 @@ Extending the oneTBB Flow Graph interface with the new ``try_put_and_wait(msg)`` API which allows to wait for completion of the chain of tasks corresponding to the ``msg``. -The feature should improve the Flow Graph performance in scenarios where multiple threads submitts the work into the +The feature should improve the Flow Graph performance in scenarios where multiple threads submits the work into the Flow Graph simultaneously and each of them should ## Introduction @@ -11,11 +11,11 @@ Flow Graph simultaneously and each of them should Current API of oneTBB Flow Graph allows doing two basic actions after building the graph: - Submitting messages in some nodes using the ``node.try_put(msg)`` API. -- Waiting for completion of **all** messagges in ghe graph using the ``graph.wait_for_all()`` API. +- Waiting for completion of **all** messages in ghe graph using the ``graph.wait_for_all()`` API. Since the only API currently available for waiting until the work would be completed waits for all tasks in the graph to complete, there can be negative performance impact in use cases when the thread submitting the work should be notified as soon as possible -when the work is done. Having only ``graph.wait_for_all()`` forces the thread to wait until **all** of the tasks in the Flow Graph, no metter +when the work is done. Having only ``graph.wait_for_all()`` forces the thread to wait until **all** of the tasks in the Flow Graph, no meter corresponds to the waited message or not. Consider the following example: @@ -40,7 +40,7 @@ public: // Builds the Flow Graph } - // Peforms the computation using the user-provided input + // Performs the computation using the user-provided input ComputeOutput compute(ComputeInput input) { m_start_node.try_put(input); // Submit work in the graph @@ -70,8 +70,8 @@ perform the computations for the provided input. Since the Flow Graph is used, i completion. This function also can be called concurrently from several threads as it shown in the ``main``. Since the only available API in the Flow Graph is ``wait_for_all()``, each thread submitting the work to the graph would be required to wait -until **all** of the tasks would be done, no metter if these tasks corresponds to the input processed by this thread or not. If some -post-processing is required on each thread after receiving the computation result, it would be only possible to start it when the Flow Graph would be completed what can be inneficient if the post-processing of lightweight graph tasks would be blocked by processing the more mature input. +until **all** of the tasks would be done, no meter if these tasks corresponds to the input processed by this thread or not. If some +post-processing is required on each thread after receiving the computation result, it would be only possible to start it when the Flow Graph would be completed what can be inefficient if the post-processing of lightweight graph tasks would be blocked by processing the more mature input. To get rid of this negative performance effect, it would be useful to add some kind of new API to the Flow Graph that would wait for completion of only one message (instead of the full completion of the graph): @@ -118,7 +118,7 @@ flow::make_edge(f3, flow::input_port<2>(join)); flow::make_edge(join, pf); -// Parallel submittion +// Parallel submission oneapi::tbb::parallel_for (0, 100, [](int input) { start.try_put(input); }); @@ -129,8 +129,6 @@ start.try_put_and_wait(444); g.wait_for_all(); ``` - - Each message is broadcasted from ``start`` to three concurrent computational functions ``f1``, ``f2`` and ``f3``. The result is when joined into single tuple in ``join`` node and post-processed in a serial ``pf`` function node. The task queue corresponding to each node in the graph is exposed under the node in the picture. The tasks that corresponds to the parallel loop 0-100 are shown as blue tasks in the queue. Red tasks corresponds to the message submitted as an argument in ``try_put_and_wait``. @@ -166,7 +164,7 @@ From the implementation perspective, working with metainformation is exposed by |---------------------|-------------------------------|-----------------------------------------------------|--------------------------------------------------| | sender | bool try_reserve(T& t) | bool try_reserve(T& t) | For buffers, reserves the element in the buffer. | | | | bool try_reserve(T& t, message_metainfo& metainfo) | The second function provides both placeholders | -| | | | for metainformantion and the element. | +| | | | for metainformation and the element. | |---------------------|-------------------------------|-----------------------------------------------------|--------------------------------------------------| The ``message_metainfo`` class is described in details in the [separate section](#details-about-metainformation-class). @@ -179,7 +177,7 @@ can be received from the predecessor node (explicit ``try_put_task`` call) or in ### ``function_node`` -If the concurrency of the ``function_node`` is ``unlimited``, the node creates a task for executting the body of the node. The created task should hold the metainfo +If the concurrency of the ``function_node`` is ``unlimited``, the node creates a task for executing the body of the node. The created task should hold the metainfo received by the function node and broadcast it to the node successors when the task is completed. Otherwise, similarly to the original ``function_node`` behavior, the node tries to occupy its concurrency. If the limit is not yet reached, creates a body task @@ -195,7 +193,7 @@ For the ``unlimited`` use-case, behaves the same as ``queueing`` node. If the concurrency limit of the node is reached, both message and the associated metainfo would be rejected and it is a predecessor responsibility to buffer them. If the predecessor is not the buffering node, both message and the metainfo would be lost. -When another task would be completed, it will try to get a buffered message together with the metainfo (by calling the ``try_get(msg, metainfo)`` method) from the predecessing node. +When another task would be completed, it will try to get a buffered message together with the metainfo (by calling the ``try_get(msg, metainfo)`` method) from the predecessor node. Since the ``function_node`` guarantees that all of the elements would be retrieved from the internal queue at some time, [buffering issues](#buffering-the-metainfo) cannot take place for buffering nodes, preceding the ``function_node``. @@ -203,7 +201,7 @@ for buffering nodes, preceding the ``function_node``. ### ``function_node`` In regard to the concurrency limit, the lightweight function node behaves as it described in the corresponding buffering policy section (``queueing`` or ``rejecting``). -The only difference is that for such nodes the tasks would not be spawned and the associated function will be executted by the calling thread. And since we don't have tasks, +The only difference is that for such nodes the tasks would not be spawned and the associated function will be executed by the calling thread. And since we don't have tasks, the calling thread should broadcast the metainformation to the successors after completing the function. ### ``continue_node`` @@ -217,7 +215,7 @@ the received metainformation with the stored instance. Under the hood the merged the received ``metainfo``. When the ``continue_node`` receives ``N`` signals from the predecessors, it wraps stored metainformation into the task for completion of the associated body. Once the task is ready, the stored -metainformation instance switchs back to the empty state for further work. Once the function would be completed, the task is expected to broadcast the metainfo to the successors. +metainformation instance switch back to the empty state for further work. Once the function would be completed, the task is expected to broadcast the metainfo to the successors. The lightweight ``continue_node`` behaves the same as described above, but without spawning any tasks. Everything would be performed by the calling thread. @@ -234,9 +232,9 @@ ordering of retrieving the elements from the buffer. Once the buffering node receives a message and the metainformation, both of them should be stored into the buffer. -Since bufferring nodes are commonly used as part of the Flow Graph push-pull protocol, e.g. before the rejecting ``function_node`` or reserving ``join_node``, it means that +Since buffering nodes are commonly used as part of the Flow Graph push-pull protocol, e.g. before the rejecting ``function_node`` or reserving ``join_node``, it means that the waiting for the message should be prolonged once it stored in the buffer. In particular, once the metainformation is in the buffer, the buffer should call ``reserve(1)`` on each -associated ``wait_context`` to prolongue the wait and call ``release(1)`` once the element is retrieved from the buffer (while calling ``try_get`` or ``try_consume``). +associated ``wait_context`` to prologue the wait and call ``release(1)`` once the element is retrieved from the buffer (while calling ``try_get`` or ``try_consume``). Once the element and the metainfo are stored in the buffer, the node will try to push them to the successor. If one of the successors accepts the message and the metainfo, both of them are removed from the buffer. Otherwise, the push-pull protocol works and the successor should return for the item once it becomes available by calling @@ -254,27 +252,51 @@ Even the ``wait_for_all()`` call would be able to finish in this case since it c ``try_put_and_wait`` feature for the graph containing these nodes should be used carefully because of this issue: -* The ``overwrite_node`` should be explicitly resetted by calling ``node.reset()`` or the element with the stored metainfo should be overwritten with another element without metainfo. -* The ``write_once_node`` should be explicitly resetted by calling ``node.reset()`` since the item cannot be overwritten. +* The ``overwrite_node`` should be explicitly reset by calling ``node.reset()`` or the element with the stored metainfo should be overwritten with another element without metainfo. +* The ``write_once_node`` should be explicitly reset by calling ``node.reset()`` since the item cannot be overwritten. ### ``broadcast_node`` -The behavior of ``broadcast_node` is preety obvious - the metainformation would just be broadcasted to each successor of the node. +The behavior of ``broadcast_node` is pretty obvious - the metainformation would just be broadcasted to each successor of the node. ### ``limiter_node`` +If the threshold was not reached, both value and the metainformation should be provided to the successors. Otherwise- both should be rejected and buffered by another node. + +Metainformation on the decrement port is ignored since this signal should not be considered part of working on the original message. + ### ``join_node`` +Each input port of the join_node should support the queue for both values and the associated metainformations. Once all of the input ports would contain the value, the values +should be combined into single tuple output and the metainformation objects should be combined into single metainfo using `metainfo1.merge(metainfo2)`, associated with the tuple +and submitted to successors. + ### ``join_node`` +Buffering node should be used before each input port for storing the values and the associated metainformations. Once all of the input ports would be triggered with the input value, +the values and the metainformations should be reserved from the buffering nodes, values should be combined into single tuple output and the metainformation objects should be +combined into single metainfo using `metainfo1.merge(metainfo2)`, associated with the tuple and submitted to successors. + ### ``join_node`` +Similar to other `join_node` implementations, except the values and the metainformation objects are stored in the hash map inside of the port. + ### ``split_node`` +The split node should take the tuple object and the corresponding metainformation, split the tuple and submit the single values from the tuple to the corresponding ports. +Metainformation object copy should be submitted together with each element into each output port. + +Metainfo should not be split since is is unclear what is the relation between elements in the tuple and the metainformation objects stored in the internal list so all of them should +be provided to the successors. + ### ``indexer_node`` +The behavior is pretty obvious - provide the tagged value to all of the successors together with the originally associated metainfo. + ### ``composite_node`` +``composite_node`` does not require any additional changes. `try_put_and_wait` feature and the explicit support for metainformation should be done by the nodes inside of the composite. + ## Additional implementation details ### Details about metainformation class @@ -331,16 +353,279 @@ std::thread t2([&]() { }) ``` +### Combined or separated wait +Current proposal describes only the case where submitting the work into the flow graph and waiting for it are combined in a single use API `node.try_put_and_wait`. +In theory, it can be useful to also have the ability to split these phases: -### Combined or separated wait +```cpp + +oneapi::tbb::flow::graph g; +oneapi::tbb::flow::function_node start_node(g, ...); + +auto desc = node.try_put(work); // other API name can be selected + +// Other work +// May submit more work into the Flow Graph +// May create other descriptors that would be waited later + +g.wait(desc); + +``` + +In that case it would be needed to extend the node with the new API returning some descriptor that can be used as a hit work waiting function `g.wait` that also should be added. +This descriptor can wrap the metainformation class for the current proposal, but the exact semantics and API should be defined since it makes the metainfo class public in some manner. ### Buffering the metainfo +As described in the [buffering nodes description section](#single-push-buffering-nodes), current proposal requires the nodes that stores the user values as part of buffers to store +also the metainformation associated with these values in the same buffer for future use and also to hold the additional reference counter to force the `try_put_and_wait` to wait +until the rejecting receiver take the item from the buffer and process it. + +Such a behavior may significantly affect the other common use-case for buffering nodes: when they are used to store the result of the computation at the end of the Flow Graph. +In such scenarios, the metainformation would be stored in the buffer and never taken from it since the buffering node is not used as part of push-pull protocol and hence there is +no rejecting successor. The `try_put_and_wait` function associated with such a metainformation will hang forever since the reference counter on the stored metainformation would +never be decreased. + +It is impossible to rely on the number of successors while making a decision to store the metainformation in the buffer since if the node is used as part of the push-pull protocol, +the number of successors is also equal to `0` since the edge is reversed. + +Current proposal considers this scenario is a `try_put_and_wait` feature limitation and does not add any support for that. + ### Multi-output nodes support -### Optimization while retrieving from buffer to task +Multi-output nodes (`multifunction_node` and `async_node`) creates an extra issue for the wait-for-one feature. Consider the following example (all of the examples shown +in this section are for `multifunction_node` but also affects `async_node` in the same manner): + +```cpp + +using mf_node_type = multifunction_node>; +using output_ports = typename mf_node_type::output_ports_type; + +mf_node_type mfn(g, unlimited, + [](int input, output_ports& ports) { + std::get<0>(ports).try_put(output); + std::get<1>(ports).try_put(output); + }) + +``` + +Unlike the `function_node`, the computed output is not propagated implicitly to the successors. It is done explicitly by the user by calling +`try_put(output)` on the corresponding output_port of the node. Since the metainformation associated with the particular input is hidden from the user +inside of the Flow Graph implementation, it cannot be propagated to the successors as part of the explicit user try_put. + +An other interesting use-case is when the `multifunction_node` is used as a reduction for multiple input values with only one output. In that case the metainformation +should not be automatically propagated at all and should be accumulated simultaneously with the outputs. + +The current proposal is to extend the body of the `multifunction_node` with the third optional parameter of some tag type that wraps the metainformation: + +```cpp + +using mf_node_type = multifunction_node>; +using output_ports = typename mf_node_type::output_ports_type; +using tag_type = typename mf_node_type::tag_type; + +mf_node_type mfn(g, unlimited, + [](int input, output_ports& ports, tag_type&& tag) { + std::get<0>(ports).try_put(output, tag); + std::get<1>(ports).try_put(output, std::move(tag)); + }) + +``` + +We still need to support the user body with just two parameter for backward compatibility. If such a body is provided, the associated metainformation would be ignored +and never broadcasted to any successors of the node. + +The tag can be saved on the user side and provided to some `try_put` call as part of other calls to the body. It will hold an extra reference counted on the associated metainfo +object to extend the wait until the corresponding item is processed. + +Current proposal describes several approaches to implement the `tag_type`: + +```cpp +class multi_tag { + message_metainfo my_metainfo; +public: + multi_tag() = default; + + // Should definitely be movable + multi_tag(multi_tag&&); + multi_tag& operator=(multi_tag&&); + + // Can be copyable + multi_tag(const multi_tag&) + multi_tag& operator=(const multi_tag&); + + ~multi_tag(); + + void reset(); // Decreases the ref counters in my_metainfo + void merge(const multi_tag&); // Useful for reduction use-cases, should be thread-safe +}; +``` + +For all of the implementation approaches, as stated above, once creating the `multi_tag` object, the extra reference counter would be added on the associated metainformation object. +The main question is when this reference counter should be decreased. + +The first option is to decrease it once the tag object is destroyed (similar to lifetime management in a smart pointer class). +The second - once it is move-consumed (passed to `try_put` as rvalue) by one of the output ports (similar to the raw pointer object lifetime management). + +Let's consider several `multifunction_node` use-cases: +* One-to-one - for each input we have one output on each output port of the node. +* One-to-zero - for some input we don't have any outputs on any of ports. +* Many-to-one - several inputs are accumulated somehow into a single one and submitted once (reduction). +* Many-to-zero - several inputs are accumulated but on some point the reduction is cancelled without output provided. + +### One-to-one use-case + +Raw-pointer-like implementation approach: + +```cpp +node_type node(g, unlimited, + // tag is created internally holding an extra ref counter + [](int input, ports_type& ports, tag_type&& tag) { + // copy-consume, no reference counters increased or decreased + std::get<0>(ports).try_put(input, tag); + + // 1: move-consume, decrease the ref-counter + std::get<1>(ports).try_put(input, std::move(tag)); + + // 2: explicit reset on the tag, decrease the ref-counter + tag.reset(); + }); +``` + +It is important to mention that in for such an implementation approach, if only the copy-consumes would be used by the user, the reference counter would never be decreased and +it will cause hangs in the corresponding `try_put_and_wait` function. It is really easy to make a mistake here. + +Smart-pointer-like implementation approach: + +```cpp +node_type node(g, unlimited, + // tag is created internally holding an extra ref counter + [](int input, ports_type& ports, tag_type&& tag) { + // No difference between copy and move-consume + // Reference counters are not touched in both use-cases + std::get<0>(ports).try_put(input, tag); + std::get<1>(ports).try_put(input, std::move(tag)); + }); +``` + +In that case, even if the tag was not move-consumed or explicitly reset, the corresponding ref counter would be decreased once internal tag type would be destroyed. + +### One-to-zero use-case + +Raw-pointer-like approach: + +```cpp +node_type node(g, unlimited, + // tag is created internally holding an extra ref counter + [](int input, ports_type& ports, tag_type&& tag) { + tag.reset(); + }); +``` + +Even if no outputs are generated for the specific input, the tag should be explicitly reset since the library should receive an external signal to decrease the ref counter. + +Smart-pointer-like approach: + +```cpp +node_type node(g, unlimited, + // tag is created internally holding an extra ref counter + [](int input, ports_type& ports, tag_type&& tag) {}); +``` + +Since the reference counter would be decreased once the internal tag object would be destroyed, no external signals are required and the core is clearly obvious +from the user's perspective. + +### Many-to-one use-case + +For this use-case, the user code is the same for both raw-pointer and smart-pointer-like approaches: + +```cpp +int accumulated_result = 0; +tag_type accumulated_tag; + +node_type node(g, unlimited, + // tag is created internally holding an extra ref counter + [](int input, ports_type& ports, tag_type&& tag) { + if (accumulate) { + accumulated_result += input; + accumulated_hint.merge(tag); // should be thread-safe + } else { + std::get<0>(ports).try_put(accumulated_result, accumulated_hint); // copy-consume + std::get<1>(ports).try_put(accumulated_result, std::move(accumulated_hint)); // move-consume + } + }); +``` + +Some amount of inputs are reduced into the single variable `accumulated_result` and provided as a single output once the accumulate condition is not met. +The idea is to accumulate the tag simultaneously in a `accumulated_tag` variable and submit it together with `accumulated_result`. + +For both use-cases, the associated reference counter would be decreased in a move-consume case (or an explicit `reset`), once the management of the `accumulated_hint` +would be transferred from the user side to the library. + +### Many-to-zero use-case + +For this use-case, the user code is the same for both raw-pointer and smart-pointer-like approaches: + +```cpp +int accumulated_result = 0; +tag_type accumulated_tag; + +node_type node(g, unlimited, + // tag is created internally holding an extra ref counter + [](int input, ports_type& ports, tag_type&& tag) { + if (accumulate) { + accumulated_result += input; + accumulated_hint.merge(tag); // should be thread-safe + } else if (cancel_accumulation) { + accumulated_result = 0; + accumulated_tag.reset(); + } + }); +``` + +In both use-cases, the tag is required to be explicitly reset to signal the library that the object is not necessary anymore. + +### Smart pointer: unique or shared + +For the smart-pointer like approach, there also can be two options +* unique_ptr-like semantics. The tag is non-copyable. The reference counter is increased once while creating the tag and decreased once while destroying the tag. +* shared_ptr-like semantics. The tag is copyable. The reference counter is increased when the tag is created or a copy of the tag is created. + The ref counter is decreased once the tag or it's copy is destroyed. + +The shared-ptr-like approach is more flexible from the user perspective because of copy semantics defined but at the same time more dangerous since all of the copies on the user +side are holding it's own reference counted on the metainfo and once the tag is consumed, the user would be expected to reset all of the copies for correct behavior. + +### Tag implementation approaches summary + +| Aspect | raw-pointer-like | shared_ptr-like | unique_ptr-like | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| All use-cases are covered | Yes | Yes | Yes | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| Tag is copyable | Yes | Yes | No | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| Copy semantics does not pressure the reference counter | Yes | No | Yes (no copy) | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| Tag is movable | Yes | Yes | Yes | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| Move does not increase ref counters | Yes | Yes | Yes | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| Explicit reset method is required | Yes | Yes | Yes | +|--------------------------------------------------------|------------------|-----------------|-----------------| +| Required less accuracy from the user | No | Yes | Yes | +|--------------------------------------------------------|------------------|-----------------|-----------------| + +Following the initial discussions on this proposal, the raw-pointer-like approach was considered the most flexible, but the most dangerous from the user perspective. +The shared_ptr-like approach was still considered flexible, but adds pressure on the reference counter and requires the user to manage all of the copies of the tag +to achieve the correct behavior. +The unique_ptr-like approach was considered a balance between flexibility and danger since it provides the required minimum of operations and requires minimal extra effort from the user. + +Currently, the proposal relies on the unique_ptr-like approach as the main one. If required, it would be easy to switch to the shared_ptr-like approach in the future. ## Process Specific Information Open questions: + +* Multi-output nodes support should be finalized +* More feedback from customers is required +* Move wide testing should be enabled for the proposed implementation