3dem · do-jason · Jan 20, 2023 · Jan 20, 2023 · Jan 24, 2023 · Jan 24, 2023
diff --git a/AUTHORS b/AUTHORS
@@ -1,6 +1,7 @@
 This program is developed in the group of Sjors Scheres at the MRC Laboratory of Molecular Biology, with contributions from the following people (in alphabetical order):
 
 - Tom Burnley (from the CCP-EM team at STFC)
+- Alister Burt (from David Barford's group at the MRC-LMB)
 - Liyi Dong
 - Bjoern Forsberg (from the Lindahl group at SciLifeLabs)
 - Shaoda He
@@ -10,6 +11,7 @@ This program is developed in the group of Sjors Scheres at the MRC Laboratory of
 - Takanori Nakane
 - Joaquin Oton (from the Briggs group at MRC-LMB)
 - Colin Palmer (from the CCP-EM team at STFC)
+- Euan Pyle (from Giulia Zanetti's group at Birkbeck)
 - Sjors Scheres
 - Jasenko Zivanov
 

diff --git a/CMakeLists.txt b/CMakeLists.txt
diff --git a/README.md b/README.md
@@ -1,86 +1,23 @@
-RELION 4.0.1
-============
+RELION 5.0-beta
+===============
 
 RELION (for REgularised LIkelihood OptimisatioN) is a stand-alone computer
 program for Maximum A Posteriori refinement of (multiple) 3D reconstructions
 or 2D class averages in cryo-electron microscopy. It is developed in the
 research group of Sjors Scheres at the MRC Laboratory of Molecular Biology.
 
-The underlying theory of MAP refinement is given in a [scientific publication](https://www.ncbi.nlm.nih.gov/pubmed/22100448).
-If RELION is useful in your work, please cite this paper.
+If RELION is useful in your work, please cite our papers.
 
-The more comprehensive documentation of RELION is stored [here](https://relion.readthedocs.io/en/release-4.0/).
+Comprehensive documentation of RELION and tutorials are stored [here](https://relion.readthedocs.io/).
 
 ## Installation
 
-More extensive options and configurations are available [here](https://relion.readthedocs.io/en/release-4.0/Installation.html),
-but the outlines to clone and install relion for typical use are made easy through [cmake](https://en.wikipedia.org/wiki/CMake).
-
-On Debian or Ubuntu machines, installing cmake, the compiler, and additional dependencies (mpi, fftw) is as easy as:
-
-```
-sudo apt install cmake git build-essential mpi-default-bin mpi-default-dev libfftw3-dev libtiff-dev libpng-dev ghostscript libxft-dev
-```
-
-RedHat-like systems (CentOS, RHEL, Scientific Linux etc) use `yum` package manager:
-
-```
-sudo yum install cmake git gcc gcc-c++ openmpi-devel fftw-devel libtiff-devel libpng-devel ghostscript libXft-devel libX11-devel
-```
-
-Once git and cmake are installed, relion can be easily installed through:
-
-```
-git clone https://github.com/3dem/relion.git
-cd relion
-git checkout master # or ver4.0; see below
-mkdir build
-cd build
-cmake ..
-make
-```
-
-By performing `git checkout ver4.0` instead of `git checkout master`, you can access the latest
-(developmental) updates for RELION 4.0.x. The code there is not tested as throughfully as that in
-the master branch and not generally recommended.
-
-The binaries will be produced in the `build/bin` directory. If you want to copy binaries
-into somewhere else, run `cmake` with `-DCMAKE_INSTALL_PREFIX=/where/to/install/` and
-perform `make install` as the final step. Do not specify the build directory itself
-as `CMAKE_INSTALL_PREFIX`! This will not work.
-
-Also note that the MPI library used for compilation must be the one you intend to use RELION with.
-Compiling RELION with one version of MPI and running the resulting binary with mpirun from another
-version can cause crash. See our wiki below for details.
-
-In any case, you have to make sure your PATH environmental variable points to the directory
-containing relion binaries. Launching RELION as `/path/to/relion` is NOT a right way; this
-starts the right GUI, but the GUI might invoke other versions of RELION in the PATH.
-
-If FLTK related errors are reported, please add `-DFORCE_OWN_FLTK=ON` to
-`cmake`. For FFTW related errors, try `-DFORCE_OWN_FFTW=ON`.
-
-RELION also requires libtiff. Most Linux distributions have packages like `libtiff-dev` or `libtiff-devel`.
-Note that you need a developer package. You need version 4.0.x to read BigTIFF files. If you installed
-libtiff in a non-standard location, specify the location by
-`-DTIFF_INCLUDE_DIR=/path/to/include -DTIFF_LIBRARY=/path/to/libtiff.so.5`.
-
-## Updating
-
-RELION is intermittently updated, with both minor and major features.
-To update an existing installation, simply use the following commands
-
-```
-cd relion
-git pull
-cd build
-make
-make install # Only when you have specified CMAKE_INSTALL_PREFIX in the cmake step
-```
-
-If something went wrong, remove the `build` directory and try again from `cmake`.
+See our [installation instructions](https://relion.readthedocs.io/en/release-5.0/Installation.html).
 
+You will have to set up a Python environment to use Python modules (e.g. Blush, ModelAngelo and DynaMight).
+Thus, please read the above instructions carefully even if you are familiar with earlier versions.
 
 ## Class Ranker
+
 The default model for the class ranker has been trained and tested in Python 3.9.12 with Pytorch 1.10.0 and Numpy 1.20.0.
 If you wish to retrain the class ranker model with your own data, please refer to [this repo](https://github.com/3dem/relion-classranker).
diff --git a/README_sycl.md b/README_sycl.md
@@ -0,0 +1,98 @@
+# RELION SYCL/DPC++ version
+
+This is SYCL/DPC++ version for [RELION](https://github.com/3dem/relion)
+
+## Build & Running
+
++ Additional requirements for RELION SYCL/DPC++ version
+	+ Intel(R) oneAPI Base Toolkit and HPC Toolkit (Installing all components are recommended. - https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html)
+	+ Intel(R) software for general purpose GPU capabilities (https://dgpu-docs.intel.com)
+	+ Intel(R) CPU Runtime for OpenCL(TM) Applications (optional - https://www.intel.com/content/www/us/en/developer/articles/technical/intel-cpu-runtime-for-opencl-applications-with-sycl-support.html)
+	+ Codeplay(R) oneAPI for NVIDIA(R) GPU (optional - https://developer.codeplay.com/products/oneapi/nvidia/)
+	+ Codeplay(R) oneAPI for AMD GPU (optional - https://developer.codeplay.com/products/oneapi/amd/)
+
++ SYCL specific command line arguments
+	+ `--gpu`, `--sycl` or `--sycl-levelzero`: This is for Intel(R) GPU specific Level Zero backend (Recommended for Intel GPU)
+	+ `--sycl-opencl`: This is for OpenCL(TM) GPU backend
+	+ `--sycl-cpu`: This is for OpenCL(TM) CPU backend
+	+ `--sycl-cuda`: This is for CUDA(R) backend over SYCL (Not tested)
+	+ `--sycl-hip`: This is for HIP backend over SYCL (Not tested)
+	+ If you need finer control of SYCL devices, you can set `ONEAPI_DEVICE_SELECTOR` environment variable. For detailed description, please look at https://intel.github.io/llvm-docs/EnvironmentVariables.html#oneapi-device-selector
+
+
+```bash
+$ mkdir build_sycl; cd build_sycl
+$ {Load Intel oneAPI toolkit and SYCL/Level Zero/OpenCL runtime environment}
+$ sycl-ls                        # This will display available SYCL devices
+$ cmake \
+-DCMAKE_C_COMPILER=mpiicx \
+-DCMAKE_CXX_COMPILER=mpiicpx \
+-DMPI_C_COMPILER=mpiicx \
+-DMPI_CXX_COMPILER=mpiicpx \
+-DMKLFFT=ON \
+-DSYCL=ON \
+-DCUDA=OFF \
+-DCMAKE_BUILD_TYPE=Release \
+-DCMAKE_C_FLAGS="-O3" \
+-DCMAKE_CXX_FLAGS="-O3 -march=native" \
+..
+
+$ #### This is Intel GPU Level Zero backend specific #####
+$ export ZE_AFFINITY_MASK=0         # Use only the first available Level Zero device. This can be replaced by --gpu 0 syntax.
+$ export ZEX_NUMBER_OF_CCS=0:4,1:4  # Set this only if you are putting more than one MPI ranks per GPU. 0:4 means 4 MPI ranks running on card 0
+$ export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=2
+$ export SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=0
+$ export SYCL_PI_LEVEL_ZERO_USM_ALLOCATOR="1;4G;host:16M,512,64K;device:16M,1024,64K;shared:0,0,64K"
+$ #### End of Intel GPU Level Zero backend specific  #####
+$ # For finer control of SYCL devcices, please see the above descrpition on ONEAPI_DEVICE_SELECTOR
+$ 
+$ ulimit -n 512000 # this is necessary for multi-GPU jobs
+$ {Run 2D/3D/refinement application by replacing --gpu/--cpu with --gpu/--sycl/--sycl-opencl/--sycl-cpu/--sycl-cuda/--sycl-hip}
+```
+
+## Optional runtime environment variables
+
++ The below shell environment variables can be tested for more potential SYCL specific tuning. Setting it to "1" or "on" will enable these features.
+	+ `relionSyclUseCuda`: --sycl-cuda will be used even if --gpu/--sycl is specified in command lines
+	+ `relionSyclUseHip`: --sycl-hip will be used even if --gpu/--sycl is specified in command lines
+	+ `relionSyclUseInOrderQueue`: Use in-order SYCL queue. Without this, out-of-order SYCL queue is used by default. (experimental)
+	+ `relionSyclUseAsyncSubmission`: Remove wait() for each SYCL kernel submission. (experimental)
+	+ `relionSyclUseStream`: Create new in-order SYCL queue for each cudaStream. (experimental)
+	+ `relionSyclUseSubSubDevice`: Create separate SYCL queue for each CCS. (experimental)
+
+
+## Added macros
+
++ For CMake configuration
+	+ `SYCL`(=ON/OFF): Enable SYCL based acceleration build
+	+ `SyclForceOneDPL`(=ON/OFF): Use oneDPL(https://github.com/oneapi-src/oneDPL) if it can be used. This has the same effect as setting "-DUSE_ONEDPL" for CMAKE_CXX_FLAGS below. (experimental)
+	+ `SYCL_AOT_COMPILE`(=ON/OFF): Enable AOT(Ahead-Of-Time) compilation for SPIR64 target. Default target is pvc. (for future use)
+	+ `SYCL_AOT_TARGET`(=ON/OFF): Specify AOT(Ahead-Of-Time) SPIR64 target. Possible list can be checked using "ocloc compile --help" command. (for future use)
+	+ `SYCL_CUDA_COMPILE`(=ON/OFF): Enable SYCL compilation for CUDA target (Not tested)
+	+ `SYCL_CUDA_TARGET`: SYCL CUDA arch target (Not tested)
+	+ `SYCL_HIP_COMPILE`(=ON/OFF): Enable SYCL compilation for HIP target (Not tested)
+	+ `SYCL_HIP_TARGET`: SYCL HIP arch target (Not tested)
+	+ `SYCL_HOST_FLAGS`(=list of flags with space as separator): Additional flags for host compiler (for future use)
+	+ `SYCL_COMPILER_NAME`: SYCL compiler command name (for future use)
+	+ `SYCL_COMPILE_FLAGS`: Additional SYCL compile flags (for future use)
+	+ `SYCL_LINK_FLAGS`: SYCL link flags except "-lsycl -lOpenCL" if needed (for future use)
+
++ Others for testing purpose (just defining with -D* is needed in cmake -DCMAKE_CXX_FLAGS)
+	+ `USE_ONEDPL`: Use SYCL kernel for weight sorting routines. If this is set, oneDPL(https://github.com/oneapi-src/oneDPL) is used when it is beneficial. (experimental)
+		+ `USE_LESS_ONEDPL`: If this is set, oneDPL is used only when there is no other implementation.
+		+ `USE_MORE_ONEDPL`: If this is set, oneDPL is used everywhere when it is applicable. If this is set, you SHOULD NOT SET DisableIndirectAccess=1.
+	+ `SYCL_OFFLOAD_FFT`: Use SYCL kernel for the current FFTW routines. (Not implemented)
+	+ `INTEL_EXPLICIT_SIMD`: Use Explicit SIMD extension for SYCL kernels. (Not implemented)
+	+ `INTEL_SG_SIZE`: Used for Intel sub-group size in SYCL kernel. 32 is recommended for PVC and 16 is for ATS. (Not tested well)
+	+ `USE_IPP`: Use Intel IPP library's RadixSort for sortOnDevice instead of std::sort. Enabled by default if IPP library exists.
+	+ `USE_MPI_COLLECTIVE`: Use MPI collective whenever possible. Enabled by default for ALTCPU and SYCL.
+	+ `USE_EXISTING_SYCL_DEVICE`: This will copy and use created SYCL device pointer instead of creating new SYCL device for each thread. Not recommended.
+	+ `USE_SINCOS_TABLE`: Pre-calculate sine/cosine table before main loop in some kernels (Not implemented)
+
++ Macros defined in source code (No need to specify in CMake configuration and it is defined by -DSYCL=ON above)
+	+ `_SYCL_ENABLED`: source codes with SYCL which may be compiled with any SYCL compiler
+	+ `_DPCPP_ENABLED`: source codes with DPC++ which should be compiled with Intel oneAPI compiler only
+
++ Prefined macros by Intel(R) oneAPI compiler
+	+ `__INTEL_LLVM_COMPILER`: For icx and icpx
+
diff --git a/cmake/BuildTorch.cmake b/cmake/BuildTorch.cmake