Merge pull request #17 from oreiche/stable-1.4

Release 1.4

.clang-tidy

@@ -1,8 +1,70 @@
+# Here is an explanation for why some of the checks are disabled:
+
+# bugprone-easily-swappable-parameters: This check would require significant 
+# refactoring effort.
+
+# bugprone-unchecked-optional-access: Too many false positives. For example, 
+# an explicit comparison with std::nullopt isn't considered a check.
+
+# cppcoreguidelines-avoid-const-or-ref-data-members: We believe, ref data 
+# members are a good way to express ownership, and const data members improve
+# readability.
+
+# cppcoreguidelines-rvalue-reference-param-not-moved: Too many false positives,
+# especially with partial moves from STL containers.
+
+# misc-const-correctness: Too many false positives, especially with STL 
+# containers.
+
+# misc-include-cleaner: There is no way for symbol mapping. For example, this
+# check requires to delete the "gsl/gsl" header, but at the same time asks to
+# include one for gsl::not_null.
+
+# misc-use-anonymous-namespace: This check would require significant refactoring
+# effort, but wouldn't improve readability equally.
+
+# misc-no-recursion: There are legitimate uses for us: we use recursion for
+# trees a lot.
+
+# modernize-return-braced-init-list: We think removing typenames and using only
+# braced-init can hurt readability.
+
+# performance-avoid-endl: There are too many legitimate uses of std::endl for
+# us.
+
+# readability-identifier-length: We would like to enable this check, but it
+# would require significant refactoring effort.
+
 FormatStyle: Google
-Checks: '*,-abseil-*,-altera-*,-android-*,-boost-*,-cert-*,-darwin-*,-fuchsia-*,-linuxkernel-*,-llvm-*,-llvmlibc-*,-mpi-*,-objc-*,-zircon-*'
-CheckOptions: [
-  {
-    key: misc-non-private-member-variables-in-classes.IgnoreClassesWithAllMemberVariablesBeingPublic,
-    value: '1'
-  }
-]
+Checks: >-
+  *,-abseil-*,-altera-*,-android-*,-boost-*,-cert-*,-darwin-*,-fuchsia-*,-linuxkernel-*,-llvm-*,-llvmlibc-*,-mpi-*,-objc-*,-zircon-*,
+  -bugprone-easily-swappable-parameters,-bugprone-unchecked-optional-access,
+  -clang-analyzer-cplusplus.NewDeleteLeaks,
+  -clang-diagnostic-unused-command-line-argument,
+  -concurrency-mt-unsafe,
+  -cppcoreguidelines-avoid-const-or-ref-data-members,-cppcoreguidelines-rvalue-reference-param-not-moved,
+  -misc-const-correctness,-misc-include-cleaner,-misc-use-anonymous-namespace,-misc-no-recursion,
+  -modernize-return-braced-init-list,
+  -performance-avoid-endl,
+  -readability-function-cognitive-complexity,-readability-identifier-length
+WarningsAsErrors: '*'
+CheckOptions:
+  - { key: misc-non-private-member-variables-in-classes.IgnoreClassesWithAllMemberVariablesBeingPublic, value: '1' }
+  - { key: readability-identifier-naming.ClassCase,                     value: CamelCase  }
+  - { key: readability-identifier-naming.EnumCase,                      value: CamelCase  }
+  - { key: readability-identifier-naming.StructCase,                    value: CamelCase  }
+  - { key: readability-identifier-naming.TypeTemplateParameterCase,     value: CamelCase  }
+  - { key: readability-identifier-naming.ConstexprVariableCase,         value: CamelCase  }
+  - { key: readability-identifier-naming.ConstexprVariablePrefix,       value: k          }
+  - { key: readability-identifier-naming.GlobalConstantCase,            value: CamelCase  }
+  - { key: readability-identifier-naming.GlobalConstantPrefix,          value: k          }
+  - { key: readability-identifier-naming.MemberConstantCase,            value: CamelCase  }
+  - { key: readability-identifier-naming.MemberConstantPrefix,          value: k          }
+  - { key: readability-identifier-naming.StaticConstantCase,            value: CamelCase  }
+  - { key: readability-identifier-naming.StaticConstantPrefix,          value: k          }
+  - { key: readability-identifier-naming.ValueTemplateParameterCase,    value: CamelCase  }
+  - { key: readability-identifier-naming.ValueTemplateParameterPrefix,  value: k          }
+  - { key: readability-identifier-naming.VariableCase,                  value: lower_case }
+  - { key: readability-identifier-naming.ClassMemberCase,               value: lower_case }
+  - { key: readability-identifier-naming.PrivateMemberSuffix,           value: _          }
+  - { key: readability-identifier-naming.ProtectedMemberSuffix,         value: _          }

CHANGELOG.md

@@ -1,9 +1,16 @@
-## Release `1.4.0` (UNRELEASED)
+## Release `1.4.0` (2024-11-04)
 
-A feature release on top of `1.3.0`, backwards compatible.
+A feature release on top of `1.3.0`, backwards compatible with
+respect to rule language, build description, repository description,
+and wire protocols. However, the internal representation in local
+build root has changed; it is therefore recommended to remove the
+local build root on upgrade.
 
 ### New features
 
+- `just serve` now also works together with a compatible remote-execution
+  endpoint. This uses an extended version of the serve protocol, so
+  both, `just-mr` and `just serve` need to be at the new version.
 - User-defined rules, as well as the built-in rule `"generic"` can
   now specify a subdirectory in which an action is to be executed.
 - `just-mr` now supports garbage collection for repository roots
@@ -20,15 +27,19 @@ A feature release on top of `1.3.0`, backwards compatible.
   will not lead to an inconsistent state; however, the directories
   at the old location will not be used anymore while still using
   disk space.
-- The expression language has been extended to contain quote and
-  quasi-quote expressions, as well as new built-in functions
+- The expression language has been extended to contain quote
+  and quasi-quote expressions, as well as new built-in functions
   `"from_subdir"`, `"nub_left"`.
 
 ### Fixes
 
 - The built-in rule `"generic"` now properly enforces that the
   obtained outputs form a well-formed artifact stage; a conflicting
   arrangement of artifacts was possilbe beforehand.
+- The built-in expression functions `"join"` and `"join_cmd"`
+  now properly enforce that the argument is a list of strings.
+  So far, they used to accept a single string, treating it as a
+  singleton list.
 - A bug was fixed that cased `just serve` to fail with an internal
   error when building against ignore-special roots.
 - `just` now accurately reports internal errors that occurred on
@@ -37,34 +48,41 @@ A feature release on top of `1.3.0`, backwards compatible.
   depended upon are also written to or found in cache; previously,
   it was assumed that all export targets not analysed locally
   were local cache hits, an assumption that no longer holds in
-  the presence of serve end points. This fixes a cache consistency
+  the presence of serve endpoints. This fixes a cache consistency
   problem if the same remote-execution endpoint is used both, with
   and without a serve endpoint.
-- A race condition in reconstructing executables from large CAS has
-  been removed that could lead to an open file descriptor being kept
-  alive for too long, resulting EBUSY failures of actions using this
-  binary.
+- A race condition in reconstructing executables from large CAS
+  has been removed that could lead to an open file descriptor being
+  kept alive for too long, resulting EBUSY failures of actions
+  using this binary.
 - Internal code clean up, reducing memory footprint, in particular
   for simultaneous upload of a large number of blobs.
 - Avoidence of duplicate requests and performance improvements when
   synchronizing artifacts with another CAS.
 - Dependencies have been updated to also build with gcc 14.
 - Portability improvements of the code by not relying on implementation
   details of the compiler.
-- Local execution no longer has the requirement that there exist no
-  more files with identical content than the hardlink limit of the
-  underlying file system.
-- The size of large object entries has been reduced. The cache and
-  CAS must be cleaned up since stable versions before `1.4.0` cannot use
-  the new format.
-- The way of storing intermediate keys of the action cache has been changed. 
-  The cache must be cleaned up since stable versions before `1.4.0` cannot 
-  use the new format.
+- Local execution no longer has the requirement that there exist
+  no more files with identical content than the hardlink limit of
+  the underlying file system.
+- Inside action descriptions, paths are always normalized; this improves
+  compatibility with existing remote-execution implementations.
+- The size of large object entries has been reduced. The cache
+  and CAS must be cleaned up since stable versions before `1.4.0`
+  cannot use the new format.
+- The way of storing intermediate keys of the action cache has
+  been changed. The cache must be cleaned up since stable versions
+  before `1.4.0` cannot use the new format.
 - Various improvements to the tests: dispatching of the summary
   action is now possible, tests are independent of a .just-mrrc
   file the user might have in their home directory
 - Various improvements of the documentation.
 
+## Release `1.4.0~beta1` (2024-10-30)
+
+First beta release for the upcoming `1.4.0` release; see release
+notes there.
+
 ## Release `1.3.0` (2024-05-08)
 
 A feature release on top of `1.2.0`, backwards compatible.

README.md

@@ -25,6 +25,7 @@ taken from user-defined rules described by functional expressions.
   - [Targets versus `FILE`, `GLOB`, and `TREE`](doc/tutorial/target-file-glob-tree.md)
   - [Ensuring reproducibility](doc/tutorial/rebuild.md)
   - [Using protobuf](doc/tutorial/proto.md)
+  - [Running linters](doc/tutorial/lint.md)
   - [How to create a single-node remote execution service](doc/tutorial/just-execute.org)
   - [Dependency management using Target-level Cache as a Service](doc/tutorial/just-serve.md)
   - [Cross compiling and testing cross-compiled targets](doc/tutorial/cross-compiling.md)

bin/bootstrap-traverser.py

@@ -16,6 +16,7 @@
 import hashlib
 import json
 import os
+import shutil
 import subprocess
 import sys
 from typing import Any, Dict, List, Optional, cast
@@ -79,12 +80,14 @@ def build_tree(desc: Json, *, config: Json, root: str, graph: Json) -> str:
     tree_dir = os.path.normpath(os.path.join(root, "TREE", tree_id))
     if os.path.isdir(tree_dir):
         return tree_dir
+    tree_dir_tmp = tree_dir + ".tmp"
     tree_desc = graph["trees"][tree_id]
     for location, desc in tree_desc.items():
         link(cast(str, build(desc, config=config, root=root, graph=graph)),
-             os.path.join(tree_dir, location))
+             os.path.join(tree_dir_tmp, location))
     # correctly handle the empty tree
-    os.makedirs(tree_dir, exist_ok=True)
+    os.makedirs(tree_dir_tmp, exist_ok=True)
+    shutil.copytree(tree_dir_tmp, tree_dir)
     return tree_dir
 
 

bin/parallel-bootstrap-traverser.py

@@ -17,6 +17,7 @@
 import json
 import multiprocessing
 import os
+import shutil
 import subprocess
 import sys
 import threading
@@ -254,14 +255,16 @@ def build_tree(desc: Json, *, config: Json, root: str, graph: Json,
         if state != AtomicListMap.Entry.INSERTED:  # tree ready, run callback
             callback(tree_dir)
         return
+    tree_dir_tmp = tree_dir + ".tmp"
     tree_desc: Json = graph["trees"][tree_id]
 
     num_entries = AtomicInt(len(tree_desc.items()))
 
     def run_callbacks() -> None:
         if num_entries.fetch_dec() <= 1:
             # correctly handle the empty tree
-            os.makedirs(tree_dir, exist_ok=True)
+            os.makedirs(tree_dir_tmp, exist_ok=True)
+            shutil.copytree(tree_dir_tmp, tree_dir)
             vals = g_CALLBACKS_PER_ID.fetch_clear(f"TREE/{tree_id}")
             if vals:
                 for cb in vals:  # mark ready
@@ -274,7 +277,7 @@ def run_callbacks() -> None:
 
         def create_link(location: str) -> Callable[..., None]:
             def do_link(path: str) -> None:
-                link(path, os.path.join(tree_dir, location))
+                link(path, os.path.join(tree_dir_tmp, location))
                 run_callbacks()
 
             return do_link

doc/concepts/expressions.md

@@ -248,7 +248,7 @@ those) argument(s) to obtain the final result.
    result is the input list, except that for all duplicate
    values, all but the rightmost occurrence is removed.
 
- - `"nub_list"` The argument has to be a list. It is an error
+ - `"nub_left"` The argument has to be a list. It is an error
    if that list contains (directly or indirectly) a name. The
    result is the input list, except that for all duplicate
    values, all but the leftmost occurrence is removed.

doc/tutorial/hello-world.md

@@ -30,7 +30,7 @@ Second, we also need to create the multi-repository configuration
     { "repository":
       { "type": "git"
       , "branch": "master"
-      , "commit": "0e9d13a761c878a647fde5a543946a02bf2c693d"
+      , "commit": "b8ae7e38c0c51467ead55361362a0fd0da3666d5"
       , "repository": "https://github.com/just-buildsystem/rules-cc.git"
       , "subdir": "rules"
       }
@@ -46,7 +46,7 @@ Second, we also need to create the multi-repository configuration
 In that configuration, two repositories are defined:
 
 1. The `"rules-cc"` repository located in the subdirectory `rules` of
-   [just-buildsystem/rules-cc:0e9d13a761c878a647fde5a543946a02bf2c693d](https://github.com/just-buildsystem/rules-cc/tree/0e9d13a761c878a647fde5a543946a02bf2c693d),
+   [just-buildsystem/rules-cc:b8ae7e38c0c51467ead55361362a0fd0da3666d5](https://github.com/just-buildsystem/rules-cc/tree/b8ae7e38c0c51467ead55361362a0fd0da3666d5),
    which contains the high-level concepts for building C/C++ binaries
    and libraries.
 
@@ -228,7 +228,7 @@ the following content:
     { "repository":
       { "type": "git"
       , "branch": "master"
-      , "commit": "0e9d13a761c878a647fde5a543946a02bf2c693d"
+      , "commit": "b8ae7e38c0c51467ead55361362a0fd0da3666d5"
       , "repository": "https://github.com/just-buildsystem/rules-cc.git"
       , "subdir": "rules"
       }

doc/tutorial/just-serve.md

@@ -293,7 +293,7 @@ Our `repos.json` at this stage reads:
     { "repository":
       { "type": "git"
       , "branch": "master"
-      , "commit": "0e9d13a761c878a647fde5a543946a02bf2c693d"
+      , "commit": "b8ae7e38c0c51467ead55361362a0fd0da3666d5"
       , "repository": "https://github.com/just-buildsystem/rules-cc.git"
       , "subdir": "rules"
       }
@@ -504,7 +504,7 @@ We are now ready to see how this setup works. At this point the `repos.json` is
     { "repository":
       { "type": "git"
       , "branch": "master"
-      , "commit": "0e9d13a761c878a647fde5a543946a02bf2c693d"
+      , "commit": "b8ae7e38c0c51467ead55361362a0fd0da3666d5"
       , "repository": "https://github.com/just-buildsystem/rules-cc.git"
       , "subdir": "rules"
       }
@@ -643,7 +643,7 @@ removing any `"absent"` pragma fields
     { "repository":
       { "type": "git"
       , "branch": "master"
-      , "commit": "0e9d13a761c878a647fde5a543946a02bf2c693d"
+      , "commit": "b8ae7e38c0c51467ead55361362a0fd0da3666d5"
       , "repository": "https://github.com/just-buildsystem/rules-cc.git"
       , "subdir": "rules"
       }