Conda Manual For Mudd's Computing Environments (purves knuth hopper)

Conda manual for Mudd’s computing environments Written by Tom Fu

This manual is written because of the fact that the computing environments at Mudd were set up in a little bit of a tricky way. Firstly, your home directory only has a quota of around 300MB, not really enough for any big conda packages, such as pytorch. Secondly, different storage disks are also connected in a slightly tricky manner: while you are able to access the same /mnt/ folder via both purves and knuth, but not through hopper, while hopper is the only one that has a potentially usable gpu.

Motivated by this, I wrote this conda manual to save myself and future Bee Lab members the trouble of setting up the correct computing environment via conda. The main concern here is to install conda and conda packages into the correct disk with enough storage space, i.e. the space you get from your lab (such as ​​/mnt/biology/donaldson/tom/, will be later referred to as CORRECT_LOCATION), NOT your home directory (such as /home/tfu/).

STEP 1. Install conda to the correct disk location

• Find a link here: https://docs.conda.io/en/latest/miniconda.html. If you are using purves/knuth, you should go for the Linux option. Copy that link address (such as https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh), and in your CORRECT_LOCATION, do wget LINK_ADDRESS. This will give you a .sh file such as Miniconda3-latest-Linux-x86_64.sh.
• Do bash Miniconda3-latest-Linux-x86_64.sh.
  • When you bash this file, it will ask you a question about the location of the installation. When answering this question, make sure to write CORRECT_LOCATION/miniconda3, otherwise it will install to your home directory which is tiny!
• Follow the instructions and finish installing miniconda. After doing conda init, I will also recommend you to set in command line the path to miniconda by doing export PATH="CORRECT_LOCATION/miniconda3/bin:$PATH"
• After you finish the above steps, create a conda environment to make sure miniconda works. An example command is conda create --name trial_env python=3.7. If it works successfully, you should be able to activate it by doing conda activate trial_env. If you want to quit the environment, you can do conda activate inside an environment (a small memorization trick: double activate = deactivate).

STEP 2. Set the correct path for storing conda environments and packages

• To set correct storage path for conda envs, in command line, do ​​CONDA_ENVS_PATH=CORRECT_LOCATION/miniconda3/envs
• To set correct storage path for conda packages, in command line, do mkdir CORRECT_LOCATION/conda_pkgs, and then do ​​CONDA_PKGS_DIRS=CORRECT_LOCATION/conda_pkgs

STEP 3. After installing conda correctly

• Note that after STEP2, your conda envs will be automatically stored in CORRECT_LOCATION/miniconda3/envs, but you can still store it somewhere else if you really want to. For example, if you made a completely new conda configuration, but did not delete the conda envs from the old configuration, you can still activate the environment from your old conda by doing conda activate COMPLETE_PATH_TO_YOUR_OLD_CONDA_ENV. This complete path is usually within the envs/ subfolder of the miniconda3 folder.
• When you do conda install …, make sure to check whether you are inside a conda environment or not. If so, you are only installing this package to the current conda env; but if not, you are installing this conda package globally and every single conda environment should have this package after you install it.
• If you want to remove a conda environment, you can do conda env remove --name trial_env. If this return an error, you can also force remove the env by rm -rf COMPLETE_PATH_TO_YOUR_OLD_CONDA_ENV (caution is suggested on this step as it is unrecoverable).
• If you accidentally messed up your system path configurations, leading to problems such as you cannot do pwd or ls, then you can do: export PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin
• To check the names of all conda environments under your base conda installation (the one you installed in STEP1), you can do conda env list.

References: The conda cheatsheet: https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf https://askubuntu.com/questions/688318/after-editing-bashrc-commands-return-command-could-not-be-located-because-bi https://anaconda.zendesk.com/hc/en-us/articles/360024036713-CondaError-OSError-122-Disk-quota-exceeded-and-Changing-the-Default-Package-Directory