Merge branch 'just-buildsystem:master' into github-ci

CHANGELOG.md

@@ -38,6 +38,11 @@ A feature release on top of `1.2.0`, backwards compatible.
   enabled whatever the distro considers suitable.
 - The `gc` subcommand supports an option `--no-rotate` to carry out
   only local clean up.
+- The expression language has been extended and, in particular,
+  allows indexed access to an arry (basically using it as a tuple)
+  and a generic form of assertion (to report user errors).
+- The `analyse` subcommand supports a new flag `--dump-result` to dump
+  the analysis result to a file or stdout (if `-` is given).
 
 ### Fixes
 
@@ -71,6 +76,10 @@ A feature release on top of `1.2.0`, backwards compatible.
   `git` for fetching and the URL is passed to `git` unchanged.
 - Improved portability and update of the bundled dependencies.
 - Various minor improvements and typo fixes in the documentation.
+- Fixed a race condition in the task queue that could cause (with
+  probability roughly 1e-5) a premature termination of the queue
+  resulting in spurious analysis failures without explanation (despite
+  "failed to analyse target").
 - Fixed a race condition in an internal cache of `just execute`
   used for keeping track of running operations.
 - The built-in rule `"install"` now properly enforces that the
@@ -91,6 +100,11 @@ A feature release on top of `1.2.0`, backwards compatible.
   repeated multiple times to list all the properties, but only the
   last one was retained. This is fixed now.
 
+## Release `1.3.0~beta1` (2024-05-02)
+
+First beta release for the upcoming `1.3.0` release; see release
+notes there.
+
 ## Release `1.2.0` (2023-08-25)
 
 A feature release on top of `1.1.0`, backwards compatible.

INSTALL.md

@@ -149,3 +149,21 @@ ${BUILDDIR}/out/bin/just install \
   -c ${BUILDDIR}/build-conf.json \
   -o ${BUILDDIR}/out/ 'installed just-mr'
 ```
+
+# Installing `just-import-git`
+
+The file `bin/just-import-git.py` is a useful Python script that allows quick
+generation of a multi-repository build configuration file from a simpler
+template for projects with dependencies provided by Git repositories.
+
+It is recommended to make this script available in your `$PATH` as
+`just-import-git`. Running it requires, of course, a Python3 interpreter.
+
+# Installing `just-deduplicate-repos`
+
+The file `bin/just-deduplicate-repos.py` is a useful Python script that
+removes duplicates from a multi-repository configuration by merging
+indistinguishable repositories.
+
+It is recommended to make this script available in your `$PATH` as
+`just-deduplicate-repos`. Running it requires, of course, a Python3 interpreter.

README.md

@@ -25,6 +25,7 @@ taken from user-defined rules described by functional expressions.
   - [Ensuring reproducibility](doc/tutorial/rebuild.md)
   - [Using protobuf](doc/tutorial/proto.md)
   - [How to create a single-node remote execution service](doc/tutorial/just-execute.org)
+  - [Dependency management using Target-level Cache as a Service](doc/tutorial/just-serve.md)
   - [Cross compiling and testing cross-compiled targets](doc/tutorial/cross-compiling.md)
 
 ## Documentation

TARGETS

@@ -245,6 +245,7 @@
       , ["ENABLE_LIBXML2", false]
       , ["ENABLE_EXPAT", false]
       , ["ENABLE_PCREPOSIX", false]
+      , ["ENABLE_PCRE2POSIX", false]
       , ["ENABLE_LIBGCC", false]
       , ["ENABLE_CNG", false]
       , ["XATTR_PROVIDER", "gnu"]
@@ -260,6 +261,7 @@
         , "ENABLE_LIBXML2"
         , "ENABLE_EXPAT"
         , "ENABLE_PCREPOSIX"
+        , "ENABLE_PCRE2POSIX"
         , "ENABLE_LIBGCC"
         , "ENABLE_CNG"
         , "XATTR_PROVIDER"

bin/just-deduplicate-repos.py

@@ -16,7 +16,7 @@
 import json
 import sys
 
-from typing import Any, List, Optional
+from typing import Any, Dict, List, Optional, Union, cast
 
 # generic JSON type
 Json = Any
@@ -171,12 +171,49 @@ def choose_representative(c: List[str]) -> str:
         return sorted(candidates,
                       key=lambda s: (s.count("/"), len(s), s))[0]
 
+    def merge_pragma(rep: str, merged: List[str]) -> Json:
+        desc = cast(Union[str, Dict[str, Json]],
+                    repos["repositories"][rep]["repository"])
+        if not isinstance(desc, dict):
+            return desc
+        pragma = desc.get("pragma", {})
+        # Clear pragma absent unless all merged repos that are not references
+        # have the pragma
+        absent = pragma.get("absent", False)
+        for c in merged:
+            alt_desc = cast(Union[str, Dict[str, Json]],
+                            repos["repositories"][c]["repository"])
+            if (isinstance(alt_desc, dict)):
+                absent = \
+                    absent and alt_desc.get("pragma", {}).get("absent", False)
+        pragma = dict(pragma, **{"absent": absent})
+        if not absent:
+            del pragma["absent"]
+        # Add pragma to_git if at least one of the merged repos requires it
+        to_git = pragma.get("to_git", False)
+        for c in merged:
+            alt_desc = cast(Union[str, Dict[str, Json]],
+                            repos["repositories"][c]["repository"])
+            if (isinstance(alt_desc, dict)):
+                to_git = \
+                    to_git or alt_desc.get("pragma", {}).get("to_git", False)
+        pragma = dict(pragma, **{"to_git": to_git})
+        if not to_git:
+            del pragma["to_git"]
+        # Update the pragma
+        desc = dict(desc, **{"pragma": pragma})
+        if not pragma:
+            del desc["pragma"]
+        return desc
+
     bisim = bisimilar_repos(repos["repositories"])
     renaming = {}
+    updated_repos = {}
     for c in bisim:
         if len(c) == 1:
             continue
         rep = choose_representative(c)
+        updated_repos[rep] = merge_pragma(rep, c)
         for repo in c:
             if ((repo not in keep) and (repo != rep)):
                 renaming[repo] = rep
@@ -198,6 +235,8 @@ def final_root_reference(name: str) -> str:
     for name in repos["repositories"].keys():
         if name not in renaming:
             desc = repos["repositories"][name]
+            if name in updated_repos:
+                desc = dict(desc, **{"repository": updated_repos[name]})
             if "bindings" in desc:
                 bindings = desc["bindings"]
                 new_bindings = {}

bin/just-import-git.py

@@ -227,7 +227,7 @@ def find_name(name: str) -> str:
 
 
 def rewrite_repo(repo_spec: Json, *, remote: Dict[str, Any],
-                 assign: Json) -> Json:
+                 assign: Json, absent: bool) -> Json:
     new_spec: Json = {}
     repo = repo_spec.get("repository", {})
     if isinstance(repo, str):
@@ -242,6 +242,8 @@ def rewrite_repo(repo_spec: Json, *, remote: Dict[str, Any],
         existing_repos: List[str] = repo.get("repositories", [])
         new_repos = [assign[k] for k in existing_repos]
         repo = dict(repo, **{"repositories": new_repos})
+    if absent and isinstance(repo, dict):
+        repo["pragma"] = dict(repo.get("pragma", {}), **{"absent": True})
     new_spec["repository"] = repo
     for key in ["target_root", "rule_root", "expression_root"]:
         if key in repo_spec:
@@ -319,6 +321,7 @@ def handle_import(args: Namespace) -> Json:
             foreign_repos[repo],
             remote=remote,
             assign=total_assign,
+            absent=args.absent,
         )
     base_config["repositories"] = base_repos
     shutil.rmtree(to_cleanup)
@@ -351,6 +354,11 @@ def main():
         help="Pretend the remote repository description is the canonical" +
         " single-repository one",
     )
+    parser.add_argument(
+        "--absent",
+        action="store_true",
+        help="Import repository and all its dependencies as absent."
+    )
     parser.add_argument(
         "--as",
         dest="import_as",

doc/concepts/expressions.md

@@ -107,12 +107,12 @@ expression power. It is equivalent but lot shorter to multiple
 
 ###### Binary conditional: `"if"`
 
-First the key `"cond"` is evaluated. If it evaluates to a
-value that is logically true, then the key `"then"` is
-evaluated and its value is the result of the evaluation.
-Otherwise, the key `"else"` (if present, otherwise `[]` is
-taken as default) is evaluated and the obtained value is the
-result of the evaluation.
+First the key `"cond"` is evaluated. If it evaluates to a value that
+is logically true, then the key `"then"` (if present, otherwise `[]`
+is taken as default) is evaluated and its value is the result of
+the evaluation. Otherwise, the key `"else"` (if present, otherwise
+`[]` is taken as default) is evaluated and the obtained value is
+the result of the evaluation.
 
 ###### Sequential conditional: `"cond"`
 
@@ -213,6 +213,10 @@ those) argument(s) to obtain the final result.
 
 ##### Unary functions
 
+ - `"not"` Return the logical negation of the argument, i.e.,
+   if the argument is logically false, return `true`, and `false`
+   otherwise.
+
  - `"nub_right"` The argument has to be a list. It is an error
    if that list contains (directly or indirectly) a name. The
    result is the input list, except that for all duplicate
@@ -256,6 +260,9 @@ those) argument(s) to obtain the final result.
 - `"reverse"` The argument has to be a list. The result is a new list
   with the entries in reverse order.
 
+- `"length"` The argument has to be a list. The result is the length
+  of the list.
+
  - `"++"` The argument has to be a list of lists. The result is
    the concatenation of those lists.
 
@@ -264,7 +271,7 @@ those) argument(s) to obtain the final result.
    neutral element 0).
 
  - `"*"` The argument has to be a list of numbers. The result
-   is their product (where the producut of the empty list is, of
+   is their product (where the product of the empty list is, of
    course, the neutral element 1).
 
  - `"map_union"` The argument has to be a list of maps. The
@@ -300,7 +307,7 @@ those) argument(s) to obtain the final result.
    default `""`).
 
  - `"escape_chars"` Prefix every in the argument every
-   character occuring in `"chars"` (a string, default `""`) by
+   character occurring in `"chars"` (a string, default `""`) by
    `"escape_prefix"` (a strings, default `"\"`).
 
  - `"to_subdir"` The argument has to be a map (not necessarily
@@ -391,3 +398,12 @@ that evaluation included in the error message presented to the user.
    two (or more) maps contain the same key, but map it to different
    values. It is also an error if the argument is a name-containing
    value.
+
+ - `"assert"` Evaluate the argument (given by the parameter `"$1"`);
+   then evaluate the expression `"predicate"` with the variable given
+   at the key `"var"` (which has to be a string literal if given,
+   default value is `"_"`) bound to that value. If the predicate
+   evaluates to a true value, return the result of evaluating the
+   argument, otherwise fail; in evaluating the failure message
+   `"msg"`, also keep the variable specified by `"var"` bound to
+   the result of evaluating the argument.