Add remote modules support for OTB nixpkgs (#15)

* [ADD] Add support for multiple new OTB remote modules * Enabled optional integration of various OTB remote modules like MLUtils, Prefetch, and Phenology. * [ADD] Add otb-dev and otb-dev-docker packages * Add new `otb-dev` package build options for all the remote modules. * Also added `otb-dev-docker` for image based on all the remote modules. * [ADD] Add OTB_APPLICATION_PATH to environment variables * Add OTB_APPLICATION_PATH so that applications are importable via native python api of OTB * Additionally add a USER as `otbuser` and HOME directory for docker images * [FIX] Update the Makefile command for the updated docker package * [CI] Update CI build commands to include both the builds for otb modules * Builds OTB with all the remote modules and also build with minimal modules * [REFAC] Make all remote modules to be not activated by default * [DOC] Update Docker build instructions and handle remote modules * Add documentation for examples on remote modules, * Add some note on OTBTF related to TF library * [FIX] Fix typos in OTB default options * [FIX] Fix the wrong package name in CI build for OTB * [DOC] Correct typo in README.md * Add aarch64 runner (#16) * [CI] Add AArch64 GitHub Action Workflow * Add a new GitHub Actions workflow for AArch64 builds using ArchLinux example from https://github.com/marketplace/actions/arm-runner. This prepares an ArchLinux image, sets up the nix environment, and builds the project for AArch64. * [FIX] Fix Aarch64 workflow build system * Fix the matrix.machine.platform with aarch64-linux in Nix build commands. * [FIX] Fix build steps in aarch64 workflow * Move the 'Build system' task to correctly follow the cache setup. * [CI] Add permissions to CI workflows and update build steps * Updated the CI workflows to include `contents: read` and `id-token: write` permissions. Also adjusted build commands to focus on `otb-dev` target and included the use of `magic-nix-cache-action`. * [CI] Remove deprecated aarch64 GitHub Actions workflow * The aarch64 GitHub Actions workflow file (.github/workflows/aarch64.yml) was deleted. * This action is no more needed as the ci.yml works fine with Nix-Magic-Cache * Rather than building 2 derivation i.e. otb-dev (all modules) and otb (minimal modules) which takes a lot of time for ARM64 build and gets timed out so we build only one derivation i.e otb-dev which contains all the modules. * [REFAC] Move remote module to their own separate nix file for better readability * Move the remote module to its own separate file for better readability, also add information regarding remote module code LICENSE and copyright information * Updated related configurations in `flake.nix` and `docker.nix` to include licensing information and appropriate fetchGit instructions.
daspk04 · Aug 31, 2024 · 8c30e66 · 8c30e66
1 parent c7ed860
commit 8c30e66
Show file tree

Hide file tree

Showing 17 changed files with 436 additions and 26 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -9,6 +9,9 @@ on:
 jobs:
   build:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write
     strategy:
       fail-fast: false
       matrix:
@@ -28,6 +31,7 @@ jobs:
             http-connections = 128
             max-substitution-jobs = 128
             extra-platforms = aarch64-linux
+      - uses: DeterminateSystems/magic-nix-cache-action@main
       - name: Build system
         run: |
-          nix build --system ${{ matrix.machine.platform }} .
+          nix build --system ${{ matrix.machine.platform }} .#otb-dev
diff --git a/Makefile b/Makefile
@@ -1,5 +1,8 @@
-build_docker_x86_64:
-	nix run .\#otb-docker-x86_64.copyToDockerDaemon
+build_docker:
+	nix run .\#otb-docker.copyToDockerDaemon
+
+build_dev_docker:
+	nix run .\#otb-dev-docker.copyToDockerDaemon
 
 #build_docker_arch64:
 #	nix run .\#otb-docker-aarch64.copyToDockerDaemon
diff --git a/README.md b/README.md
@@ -70,11 +70,13 @@ compressor for `Zarr` dataset. [Related Issue](https://github.com/remicres/otbtf
 Nix pacakge solved those issues as `GDAL` on `Nixpkgs` already has those.
 - One can combine OTB with different geospatial python libraries such as `rasterio`, `geopandas` etc. 
 which as already available with [Nixpkgs](https://search.nixos.org/packages) no need to create a separate environment.
-- As of today July 2024, there isn't any conda package for OTB, although it is under plan; 
+- There isn't any conda package for OTB, although it is under plan; 
 this Nixpkgs should help people who want to use `OTB` with a different python version or packages 
 such as (gdal,rasterio, geopandas etc.), which should solve most of those use cases for OTB to be 
-imported in an custom python environment.
-
+imported in an custom python environment [As of today August 2024,].
+- One can also build multi-arch version of OTB package apart from `AMD64 (x86_64-linux)` i.e. 
+`ARM64 (aarch64-linux)` as currently OTB has no `aarch64-linux` version.
+- One can easily build docker image as well.
 
 ## Advanced Configuration
 
@@ -156,19 +158,26 @@ Here is an example of how to create an `flake.nix` with all the above python pac
 ## Docker
 - Docker Image for OTB: 
   - Linux AMD64
-  - Linux ARM64: To build OTB for `linux-aarach-64` there are 3 options:
+  - Linux ARM64: To build OTB for `linux-aarch-64` there are 3 options:
     - The easiest way would be to build natively with ARM64 (Still needs to be tested)
-    - Complie using an emulator (Tested with GitHub Action and it builds fine)
+    - Compile using an emulator (Tested with [GitHub Action](https://github.com/daspk04/otb-nix/actions/runs/10191427474) and it builds fine)
     - We can also via nix cross compiler (Still needs to be Tested)
 
-One can build a docker image for OTB with python support (default).
+1) One can build a docker image for OTB with python support and no remote modules (default).
 ```
 1) Clone this repo
-2) make build_docker_x86_64
+2) make build_docker
 2) docker run -it --rm otb
 ```
-Example: How build a docker image based on `OTB`, `pyotb`, `gdal` and `rasterio` 
-without cloning this repo. 
+2) One can build a docker image for OTB with python support and all the remote modules.
+```
+1) Clone this repo
+2) make build_dev_docker
+2) docker run -it --rm otb-dev
+```
+
+### Example 1: 
+##### Build a docker image based on `OTB`, `pyotb`, `gdal` and `rasterio` without cloning this repo. 
 
 1) Create a local directory
 2) Create a `flake.nix` as shown below which already has the required python packages and as 
@@ -184,10 +193,9 @@ docker.nix
 4) Run command as mentioned below:
 
 ```
-1) nix run .\#otb-docker-x86_64.copyToDockerDaemon
+1) nix run .\#otb-docker.copyToDockerDaemon
 2) docker run -it --rm otb
 ```
-
 Example of `flake.nix` with `OTB`, `Gdal`, `Pyotb` and `Rasterio`:
 ```nix
 {
@@ -245,7 +253,7 @@ Example of `flake.nix` with `OTB`, `Gdal`, `Pyotb` and `Rasterio`:
         };
       in rec {
         packages = {
-          otb-docker-x86_64 = pkgs.callPackage ./docker.nix {
+          otb-docker = pkgs.callPackage ./docker.nix {
             inherit pkgs otb nix2containerPkgs;
             img-name = "otb";
             img-tag = "latest";
@@ -272,11 +280,112 @@ Example of `flake.nix` with `OTB`, `Gdal`, `Pyotb` and `Rasterio`:
       }
     );
 }
+```
+
+### Example2: 
+Build a docker image based on `OTB`, `pyotb`, `gdal` and `rasterio` without cloning this repo and some specific remote modules
+
+1)  The steps and the command are same as `Example1` except we just change the `flake.nix` file 
+and the `otb` section for only remote modules.
+
+
+Example of `flake.nix` with only activated remote modules such as `prefetch` and `otbtf` [note on OTBTF](#note)
+```nix
+{
+  description = "A flake for Orfeo Toolbox";
+
+  inputs = {
+    nixpkgs.url = "github:nixos/nixpkgs/24.05";
+    nixpkgs-unstable.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
+    flake-utils.url = "github:numtide/flake-utils";
+    otbpkgs.url = "github:daspk04/otb-nix";
+    nix2container.url = "github:nlewo/nix2container";
+  };
+
+  outputs = {
+    self,
+    nixpkgs,
+    nixpkgs-unstable,
+    nix2container,
+    flake-utils,
+    otbpkgs,
+  }:
+    flake-utils.lib.eachDefaultSystem (
+      system: let
+        pkgs = import nixpkgs {inherit system;};
+        nix2containerPkgs = nix2container.packages.${system};
+        python = pkgs.python312;
+        pyPkgs = python.pkgs;
+        
+        ## we only enable Prefetch and Otbtf remote modules
+        otb = pkgs.callPackage ./pkgs/otb/. {
+            inherit system;
+            shark = packages.shark;
+            itk_4_13 = packages.itk_4_13;
+            gdal = gdal;
+            python3 = python; # build otb with fixed python version
+            enablePython = true;
+            enablePrefetch = true;
+            enableOtbtf = true;
+          };
+
+        otbPath = with pkgs; pkgs.lib.makeLibraryPath [otb];
+
+        pyotb = pyPkgs.buildPythonPackage rec {
+          pname = "pyotb";
+          version = "2.0.3.dev2";
+          format = "pyproject";
+          docheck = false;
+
+          src = builtins.fetchGit {
+            name = pname;
+            url = "https://github.com/orfeotoolbox/pyotb.git";
+            ref = "refs/tags/${version}";
+            rev = "de801eae7e2bd80706801df4a48b23998136a5cd";
+          };
+
+          nativeBuildInputs = with pyPkgs; [
+            setuptools
+          ];
+
+          propagatedBuildInputs = with pyPkgs; [
+            numpy
+          ];
+        };
+      in rec {
+        packages = {
+          otb-docker = pkgs.callPackage ./docker.nix {
+            inherit pkgs otb nix2containerPkgs;
+            img-name = "otb";
+            img-tag = "latest";
+            python3 = python;
+            extra-python-packages = with pyPkgs; [pyotb gdal rasterio];
+          };
+        };
+        devShells.default = pkgs.mkShell rec {
+          packages = with pkgs; [
+            bashInteractive
+            pyPkgs.gdal
+            pypkgs.rasterio
+            pyPkgs.python
+            pyPkgs.venvShellHook
+            pyotb
+            otb
+          ];
+          venvDir = "./.venv";
 
+          postShellHook = ''
+            export PYTHONPATH="$PYTHONPATH:${otbPath}/otb/python"
+          '';
+        };
+      }
+    );
+}
 ```
 
 
 
+
 ## Develop
 - In case one needs to develop or experiment then
 - Clone this repo and make changes and push to your repository 
@@ -289,7 +398,8 @@ otbpkgs.url = "github:{userName}/{repoName}?ref={branchName}";
 ### How to build OTB locally
 ```markdown
 1) clone this repo
-2) nix build #.otb
+2) nix build #.otb 
+or # nix build #.otb-all --> this will build OTB with all the remote modules
 3) ./result/bin/otbcli_BandMathX -help
 ```
 
@@ -312,7 +422,13 @@ Nix should be installed and flakes should be enabled.
 
 ## TODO
  - [X] Build a Nix Docker for the OTB package [Linux AMD64 and Linux ARM64]
- - [ ] Build OTB with [remote modules]((https://www.orfeo-toolbox.org/CookBook/RemoteModules.html)) (OTBTF, TimeSeriesGapFilling, etc)
+ - [X] Build OTB with [remote modules]((https://www.orfeo-toolbox.org/CookBook/RemoteModules.html)) (OTBTF, TimeSeriesGapFilling, etc)
+ - [ ] Build OTB with Tensorflow for OTBTF (OTBTF remote modules doesn't have Tensorflow)
+
+## NOTE:
+- The `OTBTF` build with `Nix` still doesn't have the `TF` applications such as `TensorflowModelServe` 
+`ImageClassifierFromDeepFeatures` `TensorflowModelTrain` `TrainClassifierFromDeepFeatures` `ImageClassifierFromDeepFeatures`
+as the `TF` building from source via Nix is yet to be done by me.
 
 
 #### **NOTE: This repo is currently experimental, please open an issue in case you encounter any.**
diff --git a/docker.nix b/docker.nix
@@ -1,3 +1,17 @@
+#   Copyright 2024 Pratyush Das
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+#   https://github.com/efabless/openlane2/blob/main/nix/docker.nix
 {
   nix2containerPkgs,
   python3,
@@ -9,6 +23,8 @@
   pkgs,
   ...
 }: let
+  user = "otbuser";
+  home-dir = "/home/${user}";
   py-env = (
     python3.withPackages (pp:
       with pp;
@@ -24,7 +40,7 @@ in
   nix2containerPkgs.nix2container.buildImage rec {
     name = img-name;
     tag = img-tag;
-
+    initializeNixDatabase = true;
     copyToRoot = pkgs.buildEnv {
       name = "root";
       paths = with pkgs.dockerTools;
@@ -50,6 +66,7 @@ in
           git
           neovim-unwrapped
           zsh
+          nix
 
           # Library
           otb
@@ -59,6 +76,12 @@ in
 
       pathsToLink = ["/bin" "/etc" "/var" "/run" "/tmp" "/lib"];
       postBuild = ''
+        if -e $HOME; then
+            echo "home directory exists"
+            exit 1
+        fi
+            mkdir -p $out/${home-dir}
+        mkdir -p ~/.config/nix/ && printf "experimental-features = nix-command flakes\n" > ~/.config/nix/nix.conf
         mkdir -p $out/tmp
         mkdir -p $out/etc
         cat <<HEREDOC > $out/etc/zshrc
@@ -79,8 +102,10 @@ in
         "EDITOR=nvim"
         "PYTHONPATH=${py-env-sitepackages}:${otbLibPath}/otb/python"
         "PATH=${py-env-bin}:${otbBinPath}:/bin"
+        "OTB_APPLICATION_PATH=${otbLibPath}/otb/applications"
         "TMPDIR=/tmp"
+        "USER=${user}"
+        "HOME=${home-dir}"
       ];
     };
   }
-
diff --git a/flake.nix b/flake.nix
@@ -1,3 +1,16 @@
+#   Copyright 2024 Pratyush Das
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
 {
   description = "A flake for Orfeo Toolbox";
 
@@ -25,11 +38,11 @@
         # The minimal version of gdal and doesn't have arrow support but helps in leaner build
         # we can mix as per requiremnt which will compile gdal from source
         # https://github.com/NixOS/nixpkgs/blob/8c50662509100d53229d4be607f1a3a31157fa12/pkgs/development/libraries/gdal/default.nix#L7
-#         gdal = pkgs.gdalMinimal.override {python3 = python;};
-#         gdal = pkgs.gdalMinimal.override {python3 = python;
-#                                            useArrow = true;
-#                                            useHDF = true;
-#                                            useNetCDF = true;};
+        #         gdal = pkgs.gdalMinimal.override {python3 = python;};
+        #         gdal = pkgs.gdalMinimal.override {python3 = python;
+        #                                            useArrow = true;
+        #                                            useHDF = true;
+        #                                            useNetCDF = true;};
         gdal = pkgs.gdal; # gdal full version
         pyPkgs = python.pkgs;
       in rec {
@@ -45,7 +58,27 @@
             enablePython = true;
           };
 
-          otb-docker-x86_64 = pkgs.callPackage ./docker.nix {
+          otb-dev = pkgs.callPackage ./pkgs/otb/. {
+            inherit system;
+            shark = packages.shark;
+            itk_4_13 = packages.itk_4_13;
+            gdal = gdal;
+            python3 = python; # build otb with fixed python version
+            enablePython = true;
+            enablePrefetch = true;
+            enableOtbtf = true;
+            enableMLUtils = true;
+            enablePhenology = true;
+            enableBioVars = true;
+            enableGRM = true;
+            enableLSGRM = true;
+            enableSimpleExtraction = true;
+            enableTemporalGapfilling = true;
+            enableTimeSeriesUtils = true;
+            enableTemporalSmoothing = true;
+          };
+
+          otb-docker = pkgs.callPackage ./docker.nix {
             inherit pkgs nix2containerPkgs;
             img-name = "otb";
             img-tag = "latest";
@@ -54,6 +87,15 @@
             extra-python-packages = with pyPkgs; [packages.otb.propagatedBuildInputs];
           };
 
+          otb-dev-docker = pkgs.callPackage ./docker.nix {
+            inherit pkgs nix2containerPkgs;
+            img-name = "otb-dev";
+            img-tag = "latest";
+            otb = packages.otb-dev;
+            python3 = python;
+            extra-python-packages = with pyPkgs; [packages.otb.propagatedBuildInputs];
+          };
+
           #          otb-aarch64 = pkgs.pkgsCross.aarch64-multiplatform.callPackage ./pkgs/otb/. {
           #            inherit system;
           #            shark = packages.shark;