Using the system miniconda 3 install

I previously wrote about installing your own miniconda install here which I strongly suggest reading. However, after I received many questions about getting the conda set up properly, I decided to make the process of getting conda set up for you a bit easier.

I wrote a script that will:

1. Ensure you are not installing to your home directory.
2. Eliminate the need to download the miniconda installer.
3. Reduce the burden of getting the channels set up and the proper envs and pkgs settings in your .condarc

The script will also make sure you have write permissions to the particular directory specified and won’t make changes unless you approve. Lastly, you can say that you’d prefer to run the commands yourself, and then copy/paste the commands into your own terminal.

Running the script

I made a new miniconda3 install here:

/local/cluster/miniconda3_base

This miniconda install will have only ever have the base packages installed and expects you to install any new conda packages in your own environments.

To get this conda setup for you to use, you can run this script:

/local/cluster/miniconda3_base/init_miniconda3 <PATH TO DESIRED CONDA DIR>

e.g.

/local/cluster/miniconda3_base/init_miniconda3 /nfs4/core/home/davised/opt/conda

This will generate the specified dir for you, if necessary, and then set up your ~/.bashrc to access the new conda bin.

The script will confirm that you actually want to run the listed commands before running them. If you feel better about running the commands yourself, choose option 2 (No) from the list, and feel free to copy/paste them individually into your own terminal.

The script checks for your ~/.condarc and backs it up, if desired. It also lets you know if you have already cached environments in the ~/.conda dir.

If the script is unable to generate the specified conda dir for you, it will not make any changes and instead exit the script.

As an aside, if you’ve accidentally run the conda init command while in tcsh, you may have experienced problems accessing your programs. Just delete the ~/.tcshrc file that conda made and you should be good to go. As a reminder, we don’t support running conda in tcsh, only bash.

Regardless of which shell you run init_miniconda3 from, only your ~/.bashrc will be modified.

One reminder is to check out a node with qrsh when you want to build a new conda env, or your conda command will killed after 2 minutes.

To use a conda env over SGE_Batch, add this to your shell script:

1
2
3
4
5

eval "$(conda shell.bash hook)"
conda activate <ENV_NAME>
...
[Insert commands from conda env here]
...

It also helps to run the SGE_Batch command while in bash instead of tcsh if you happen to have tcsh as your default shell.

Final tips

Use mamba install and mamba create as a stand-in replacements for the conda install and conda create commands. Here is a link to the mamba docs. mamba is faster and will help resolve complex dependencies in a much shorter time than conda in some cases.

Also, it’s best practice to install all of your programs with a version number specified, e.g. mamba create -n python3.8 python=3.8, and to specify all of your programs when you create the environment so that all of the dependencies can be resolved at creation time. Sometimes you can run into issues when installing programs in different order after creation.

Please join the CQLS BUG Slack if you have additional questions regarding conda, and join the #conda channel.