This repo holds Nix flake, modules, packages, and reusable utility Nix language code for use across the OpenGen ecosystem. Either directly, or indirectly.
- Packages: comes packed with python packages used day-to-day by the team at OpenGen.
- Executables: can execute loom and other packages without complicated installation instructions.
- Docker/OCI image: builds and publishes common images to use in the cloud or for demos.
- Developer shell: run
nix developand start coding. - Shared cache: don't spend hours rebuilding the same packages.
- CUDA support.
Build and tested on Linux.
-
Because genjax is closed source, Nix needs to be configured to access the repository.
To do so, run:
$ gh auth login
$ mkdir -p ~/.config/nix
$ echo "access-tokens = github.com=$(gh auth token)" >> ~/.config/nix/nix.confTo start coding, run:
$ nix developThis creates a shell environment that includes Bayes3d, Jax, SciPy and Jupyter:
$ jupyter notebook notebooks/demo.ipynbOn top of the packages from this repository, you also have access to the ~8k packages from nixpkgs .
To add a dependency, edit the parts/devshell.nix file and add the package to
the "Add your dependencies here" part.
Then exit and re-launch the shell with nix develop.
If the package doesn't exist in nixpkgs, it's always possible to pip install
it as well. In which case it will be installed in the repo's .venv.
Some of the packages are also directly executable, allowing to use then without installing anything.
$ nix run github:numtide/OpenGen.nix#loom
Usage: loom COMMAND [ARG ARG ... KEY=VAL KEY=VAL ...]Or inside of the checked out repo:
$ nix run .#loom
Usage: loom COMMAND [ARG ARG ... KEY=VAL KEY=VAL ...]This project also proposes Docker / OCI images for common scenarios. This is useful to distribute the code to users for demos, or to the Cloud.
On Linux, build and load them with:
$ nix build github.com:numtide/OpenGen.nix#oci-gensql-loom
$ docker load -i ./resultThe resulting image can then be published to a Docker registry.
The code in this project can also be re-used in other repositories. The use-case is to develop experiments independently from the main repository.
For this you would add a bare bone flake.nix in your repository:
{
inputs = {
opengen.url = "github:numtide/OpenGen.nix";
nixpkgs.follows = "opengen/nixpkgs";
};
outputs = { self, nixpkgs, opengen }: let
eachSys = nixpkgs.lib.genAttrs ["x86_64-linux"];
in {
devShells = eachSys (system: {
default = nixpkgs.legacyPackages.${system}.mkShell {
packages = [
opengen.packages.${system}.python.withPackages (p: [
# Add your packages here
p.bayes3d
p.genjax
]);
];
};
};
};Then invoke nix develop to create your own developer environment.
A Jupyter environment with Bayes3D libraries available.
Example:
$ nix run github:numtide/OpenGen.nix#jupyter-bayes3d notebook ./notebooks/demo.ipynbThe ./oci folder defines all the images for the project.
A barebone image with common tools in it.
Docker image including the loom utility.
Docker image for gensql.query
Here are all the python packages this flake provides, on top of all the ones available in nixpkgs.
All the packages are compiled against Python 3.11.
Bayes3D is a 3D scene perception system based on probabilistic inverse graphics.
Implementation of CrossCat in Python. NOTE: this ONLY builds for x86_64 architectures and only runs on linux, because it depends on
platform-dependent distributions.
Your options are:
$ nix build '.#packages.x86_64-linux.loom' # same as `.#loom` if that is your OS/arch
$ nix build '.#packages.x86_64-darwin.oci-gensql-loom'If you are running on Mac silicon (aarch64-darwin), that OCI image will run but behavior is not defined or supported.
distinctipy is a lightweight python package providing functions to generate colours that are visually distinct from one another.
Native library for probability distributions in python used by Loom. NOTE: this ONLY builds for x86_64 architectures and only runs on linux.
Tree is a library for working with nested data structures. In a way, tree generalizes the builtin map function which only supports flat sequences, and allows to apply a function to each "leaf" preserving the overall structure.
GenJAX is an implementation of Gen on top of JAX - exposing the ability to programmatically construct and manipulate generative functions, as well as JIT compile + auto-batch inference computations using generative functions onto GPU devices.
Goftests is intended for unit testing random samplers that generate arbitrary plain-old-data, and focuses on robustness rather than statistical efficiency. In contrast to scipy.stats and statsmodels, goftests does not make assumptions on the distribution being tested, and requires only a simple (sample, prob) interface provided by MCMC samplers.
Open3D is an open-source library that supports rapid development of software that deals with 3D data.
Wrapper package for OpenCV python bindings.
Oryx is a library for probabilistic programming and deep learning built on top of Jax.
Parsable is a lightweight decorator-based command line parser library. Parsable was written to be simpler than argparse, optparse, and argh.
Multiple dispatch in Python.
PyMetis is a Python wrapper for the Metis graph partititioning software.
pyRANSAC-3D is an open source implementation of Random sample consensus (RANSAC) method. It fits primitive shapes such as planes, cuboids and cylinder in a point cloud to many aplications: 3D slam, 3D reconstruction, object tracking and many others.
Probabilistic programming system for fast and exact symbolic inference.
TensorFlow Probability is a library for probabilistic reasoning and statistical analysis in TensorFlow.
- Docker image for Bayes3D.
- Publish GCP images as well.
- Reduce Bayes3D closure size (20GB on my machine).
- Poetry2nix and jupyenv integrations.
- macOS and aarch64-linux compatibility.