Improve build documentation and scripts (#395)

Add virtualenv as a requirement and deal with more versions of it.
Add option to create a worktree instead of clone download.
No need to clone if there is already a local repository with the right commit.
Find Python3 outside of the virtualenv too.
Generate the compilation database by default for programming tools.
This allows for example clangd-based tools to understand the code.
Use subshell instead of changing directory, less robust.
Clarify the build recipe by referring to the CI files.
Take into account the submodules when cloning.

docs/Building.md

@@ -9,6 +9,7 @@ cmake 3.20.6
 ninja 1.8.2
 Xilinx Vitis 2022.2
 python 3.8.x and pip
+virtualenv
 pip3 install psutil rich pybind11 numpy
 clang/llvm 14+ from source https://github.com/llvm/llvm-project
 ```
@@ -32,48 +33,73 @@ In addition, the following optional packages may be useful:
 ```
 LibXAIE is a backend target used to execute designs in hardware: https://github.com/Xilinx/embeddedsw/tree/master/XilinxProcessorIPLib/drivers/aiengine
 ```
-Note that if you build one of the supported platforms like vck190_bare_prod, the generated sysroot 
-already contains the LibXAIE drivers so you do not need to download the embeddedsw repo or 
-define the LibXAIE_DIR cmake parameter.
+Note that if you build one of the supported platforms like `vck190_bare_prod`, the generated `sysroot`
+already contains the LibXAIE drivers so you do not need to download the `embeddedsw` repository or
+define the `LibXAIE_DIR` `cmake` parameter.
 
 Currently, the only supported target is the Xilinx VCK190 board, running Ubuntu-based Linux, however
 the tools are largely board and device independent and can be adapted to other environments.
 
 
 ## Building on X86
 
-1. Clone the mlir-aie repo.
+1. Clone the `mlir-aie` repository with its sub-modules:
     ```
-    git clone https://github.com/Xilinx/mlir-aie.git
+    git clone --recurse-submodules https://github.com/Xilinx/mlir-aie.git
     cd mlir-aie
     ```
 
-    __All subsequent steps should be run from inside the top-level directory of the mlir-aie repo cloned above.__
+    __All subsequent steps should be run from inside the top-level
+    directory of the `mlir-aie` repository cloned above.__
 
-2. Run utils/setup_python_packages.sh to setup the prerequisite python packages. This script creates and installs the python packages listed in utils/requirements.txt in a virtual python environment called 'sandbox'.
+2. Source `utils/setup_python_packages.sh` to setup the prerequisite python
+    packages. This script creates and installs the python packages
+    listed in `utils/requirements.txt` in a virtual python environment
+    called 'sandbox', then it enters the sandbox:
     ```
     source utils/setup_python_packages.sh
     ```
 
-3. Clone and compile LLVM, with the ability to target AArch64 as a cross-compiler, and with MLIR 
-enabled: in addition, we make some common build optimizations to use a linker ('lld' or 'gold') other 
-than 'ld' (which tends to be quite slow on large link jobs) and to link against libLLVM.so and libClang
-so. You may find that other options are also useful. Note that due to changing MLIR APIs, only a
-particular revision is expected to work.  
+    If you need to exit the sandbox later, type `deactivate`.  If you
+    have a recent Linux distribution, you might not need this, as you
+    are able to have all the required packages from the distribution.
 
-    To clone llvm, run utils/clone-llvm.sh (see utils/clone-llvm.sh for the correct llvm commithash).
+3. Clone and compile LLVM, with the ability to target AArch64 as a
+   cross-compiler, and with MLIR enabled: in addition, we make some
+   common build optimizations to use a linker (`lld` or `gold`) other
+   than `ld` (which tends to be quite slow on large link jobs) and to
+   link against `libLLVM.so` and `libClang.so`. You may find that other
+   options are also useful. Note that due to changing MLIR APIs, only
+   a particular revision is expected to work.
+
+    To clone `llvm`, run `utils/clone-llvm.sh` (see
+    `utils/clone-llvm.sh` for the correct `llvm` commit hash):
     ```
     ./utils/clone-llvm.sh
     ```
-    To build (compile and install) llvm, run utils/build-llvm-local.sh in the directory that llvm and 
-    cmakeModules are cloned in. See build-llvm-local.sh for additional shell script arguments. 
-    Note that build-llvm.sh is a variation of the llvm build script used for CI on github.
-    ```
-    ./utils/build-llvm-local.sh 
+
+    If you have already an LLVM repository, you can instead of cloning
+    just make a new worktree from it by using:
+    ````
+    ./utils/clone-llvm.sh --llvm-worktree <directory-of-existing-LLVM-repository>
+    ````
+
+    To build (compile and install) LLVM, run `utils/build-llvm-local.sh` in the directory that `llvm` is cloned in. See `utils/build-llvm-local.sh` for additional shell script arguments.
+    (Note that `build-llvm-local.sh` and `build-llvm.sh` are a
+    variation of the LLVM build script used for CI on GitHub and
+    looking at the continuous integration recipe
+    https://github.com/Xilinx/mlir-aie/blob/main/.github/workflows/buildAndTest.yml
+    and output https://github.com/Xilinx/mlir-aie/actions/ might help
+    in the case of compilation problem.)
+```
+    ./utils/build-llvm-local.sh
     ```
-    This will build llvm in llvm/build and install the llvm binaries under llvm/install.
+    This will build LLVM in `llvm/build` and install the LLVM binaries under `llvm/install`.
+
+4. Build the MLIR-AIE tools by calling `utils/build-mlir-aie.sh` with
+    the path to the `llvm/build` directory. The Vitis environment will
+    have to be set up for this to succeed.
 
-4. Build the mlir-aie tools by calling `utils/build-mlir-aie.sh` with the path to `llvm/build`. The Vitis enviroment will have to be set up for this to succeed. 
     ```
     source <Vitis Install Path>/settings64.sh
     ./utils/build-mlir-aie.sh <llvm dir>/<build dir>
@@ -82,8 +108,8 @@ particular revision is expected to work.
 
     The MLIR AIE tools will be able to generate binaries targetting a combination of AIEngine and ARM processors.
 
-5. In order to run all the tools, it is necessary to add some paths into your environment. This can be 
-done by calling the `utils/env_setup.sh` script with the paths to the install folders for mlir-aie
+5. In order to run all the tools, it is necessary to add some paths into your environment. This can be
+done by sourcing the `utils/env_setup.sh` script with the paths to the install folders for mlir-aie
 and llvm.
     ```
     source utils/env_setup.sh <mlir-aie>/install <llvm dir>/install

utils/build-llvm-local.sh

@@ -1,10 +1,10 @@
 #!/usr/bin/env bash
 ##===- utils/build-llvm-local.sh - Build LLVM on local machine --*- Script -*-===##
-# 
+#
 # This file licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-# 
+#
 ##===----------------------------------------------------------------------===##
 #
 # This script build LLVM with custom options intended to be called on your
@@ -24,14 +24,16 @@ INSTALL_DIR=${3:-"install"}
 
 mkdir -p $LLVM_DIR/$BUILD_DIR
 mkdir -p $LLVM_DIR/$INSTALL_DIR
+# Enter a sub-shell to avoid messing up with current directory in case of error
+(
 cd $LLVM_DIR/$BUILD_DIR
 set -o pipefail
 set -e
 cmake ../llvm \
   -GNinja \
   -DCMAKE_C_COMPILER=clang \
   -DCMAKE_CXX_COMPILER=clang++ \
-  -DPython3_FIND_VIRTUALENV=ONLY \
+  -DPython3_FIND_VIRTUALENV=FIRST \
   -DLLVM_LINK_LLVM_DYLIB=ON \
   -DCLANG_LINK_CLANG_DYLIB=ON \
   -DLLVM_BUILD_EXAMPLES=OFF \
@@ -43,8 +45,9 @@ cmake ../llvm \
   -DLLVM_ENABLE_PROJECTS="clang;lld;mlir" \
   -DLLVM_TARGETS_TO_BUILD:STRING="X86;ARM;AArch64;" \
   -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
   |& tee cmake.log
 
 ninja |& tee ninja.log
 ninja install |& tee ninja-install.log
-cd ../..
+)

utils/build-mlir-aie.sh

@@ -4,7 +4,6 @@
 # This file licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-# 
 ##===----------------------------------------------------------------------===##
 #
 # This script builds mlir-aie given <llvm dir>.
@@ -54,4 +53,3 @@ cmake -GNinja\
 ninja |& tee ninja.log
 ninja install |& tee ninja-install.log
 #ninja check-aie |& tee ninja-check-aie.log
-cd ..

utils/clone-llvm.sh

@@ -1,22 +1,41 @@
 #!/usr/bin/env bash
 ##===- utils/clone-llvm.sh - Build LLVM for github workflow --*- Script -*-===##
-# 
+#
 # This file licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-# 
+#
 ##===----------------------------------------------------------------------===##
 #
 # This script checks out LLVM.  We use this instead of a git submodule to avoid
 # excessive copies of the LLVM tree.
 #
 ##===----------------------------------------------------------------------===##
 
-export commithash=35ca64989a75c93ea7e935ef11c3d1883c21cccd
+# The LLVM commit to use.
+# TODO: create a branch or a tag instead, to avoid fetching main and
+# this commit later.
+commithash=35ca64989a75c93ea7e935ef11c3d1883c21cccd
 
-git clone --depth 1 https://github.com/llvm/llvm-project.git llvm
-pushd llvm
-git fetch --depth=1 origin $commithash
-git checkout $commithash
-popd
+here=$PWD
 
+# Use --worktree <directory-of-local-LLVM-repo> to reuse some existing
+# local LLVM git repository
+if [ x"$1" == x--llvm-worktree ]; then
+  git_central_llvm_repo_dir="$2"
+  (
+    cd $git_central_llvm_repo_dir
+    # Use force just in case there are various experimental iterations
+    # after you have removed the llvm directory
+    git worktree add --force "$here"/llvm $commithash
+  )
+else
+  # Fetch main first just to clone
+  git clone --depth 1 https://github.com/llvm/llvm-project.git llvm
+  (
+    cd llvm
+    # Then fetch the interesting part
+    git fetch --depth=1 origin $commithash
+    git checkout $commithash
+  )
+fi

utils/setup_python_packages.sh

@@ -14,6 +14,11 @@
 ##===----------------------------------------------------------------------===##
 
 python3 -m virtualenv sandbox
-source sandbox/bin/activate
+# The real path to source might depend on the virtualenv version
+if [ -r sandbox/local/bin/activate ]; then
+  source sandbox/local/bin/activate
+else
+  source sandbox/local/activate
+fi
 python3 -m pip install --upgrade pip
 python3 -m pip install -r utils/requirements.txt