Linux system setup

This page describes considerations and procedures for setting up a Linux environment for EM image analysis.

Storage
The Linux operating system and the EM packages should normally use no more than 100 GB of space. A good way to prepare for larger and larger EM software packages (mainly caused by multiple installations of python environments, these days) is to dedicate a 50 GB partition for the OS, and move the /usr/local directory to a larger partition(see below).

GPU
Most GPU computation software used for EM analysis depends on the Nvidia CUDA library. This mandates a Nvidia GPU with CUDA 3.0+ computation capability. Nvidia Geforce products after year 2012 (Kepler and newer architectures) should meet this requirement. On current market, the 1080TI and 2080TI cards are the best options below $2000. The 1050TI also offers very good performance for a low cost of $200.

Low cost system
A very minimalist system, for learning and testing, can also be set up on a low budget. This can be any desktop PC produced in the last 5 years, with >=8 GB memory, >=500 GB HDD, and an Nvidia 1050TI card. This system should not cost more than CAD 500 in total, including the peripherals. This setup should be able to process negative stain dataset of up to 100 images and generate 2D classes in less than a day.

If no GPU computation is needed, for example if only EMAN2 or cisTEM will be used, then the Nvidia 1050TI is not needed. However for learning purposes, it is best to have the GPU computation capability.

For a setup that is adequate for cryoEM single particle work, refer to Computer hardware setup.

Complications with CUDA versions
'''Update: CryoSPARC v2.5.0 now supports cuda 9 and 10. '''

CUDA-8.0 is commonly used by most GPU computing programs. Therefore if possible, CUDA-8.0 should be used for maximum compatibility.

However in recent tests on a system with 2080TI cards, we failed to run cryoSPARC2 with CUDA-8.0 or CUDA-9.2 ("sm_75" architecture not found). Installing CUDA-10.0 seems to have solved the problem.

In case certain program requires CUDA8.0: a workaround is to install CUDA8.0 as system-wide cuda (/usr/local/cuda -> /usr/local/cuda-8.0), while direct cryosparc directly to /usr/local/cuda-10.0 during installation.

Good practices

 * Install the OS on a dedicated partition of 50-80 GB. The /home directory stays on a separate partition.


 * All scientific programs and packages are installed in /usr/local, per convention.


 * Install each version of programs in a folder with version numbers, then use a symbolic link to represent the program. Therefore no editing of bash.bashrc is needed after each upgrade:


 * The /usr/local folder can also be placed outside the '/' partition. Then place a symbolic link in /usr directed to it (assuming /home is located on a separate, large partition, see above):




 * Besides allowing /usr/local to continuously grow, this also allows quick recovery from system disasters or clean upgrades.


 * All program-related environment variables should be placed in /etc/bash.bashrc file. This allows all users to share the computer's software.


 * Backup some important files from the /etc directory when changes to system are made. These include bash.bashrc, fstab, hosts, hostname, passwd ... or maybe the whole /etc directory.


 * Backup important data frequently. Save very important files on a cloud storage.

Setting up Linux system
Recommended Linux distribution: Ubuntu MATE, Version: 16.04 LTS.
 * The "MATE" version of Ubuntu uses traditional GNOME 2 interface, which provides better working efficiency for non apple users.

Important notice:


 * Ubuntu 18 uses GCC 6.0, which is incompatible with CUDA 8.0 installation. Therefore installing on Ubuntu 18 necessitates using CUDA 9.2 or higher. We found that Ubuntu-16.04-MATE provides a good environment for all current software.
 * Some experiences suggests that unless the GPU is not supported by CUDA8.0, using CUDA8.0 will ensure best compatibility with most GPU dependent software.
 * Nvidia 2080TI (sm_75) can only work under CUDA10.00

Step 1. Install Ubuntu MATE 16.04 LTS

 * 1. Download Ubuntu ISO:
 * https://ubuntu-mate.org/blog/ubuntu-mate-xenial-final-release/
 * 2. Make a bootable USB installation disk
 * In Windows: use RUFUS: https://rufus.ie/en_IE.html
 * In Linux: use GNU ddrescue or dd. Be very sure which block device is really the USB key.
 * 3. Install Ubuntu
 * Use manual partitioning if possible. See the example partitioning below.
 * It is better to give a password to your first user instead of leaving an empty one. An empty password won't allow you to login by SSH (therefore no sFTP either), or change the root user's password.
 * {| class="wikitable"


 * EFI
 * 500 MB
 * A partition reserved for EFI booting
 * Ext4
 * 50 GB
 * The root directory, storing the Linux OS
 * SWAP
 * 64 GB
 * A swap partition, not absolutely required. In absence of it, the system uses a file for caching.
 * /home
 * Ext4
 * All usable space
 * All users' home
 * }
 * A swap partition, not absolutely required. In absence of it, the system uses a file for caching.
 * /home
 * Ext4
 * All usable space
 * All users' home
 * }
 * }

Step 2. Initial preparation
After first boot of the new Ubuntu, install a better Vi editor:

Install evince, the PDF reader:

Give the system root user a password:

Step 3. Download and install CUDA 8.0

 * This is the most crucial step. If this fails there is no need to waste time on others. The tricky part is to install the Nvidia graphics driver.


 * 1. Download Nvidia graphics driver and CUDA 8.0 library.
 * Nvidia driver :https://www.nvidia.com/Download/index.aspx?lang=en-us


 * CUDA 8.0: https://developer.nvidia.com/cuda-80-ga2-download-archive


 * For an Intel CPU PC, download the x86_64 "runfile(local)". Do not use the Deb package. The Deb package will only leave some .deb packages in the /var directory, which won't function as a CUDA library unless you install some of them.


 * 2. Restart computer in textmode and install Nvidia driver and CUDA.


 * If a Nvidia driver was not previously installed, it can be installed when installing the CUDA library.
 * In order to install the Nvidia driver, we need to change the Linux to start in text mode first and restart:


 * After restarting in textmode, login with your username and password.


 * Then run the previously downloaded "cuda_8.0....run" file, hopefully the Nvidia driver will be installed too:


 * Answer questions and choose to also install the graphics driver. Leave the destination to the default /usr/local/cuda-8.0.


 * If the graphics driver installation fails, try installing the driver separately by running the "Nvidia...run" file.


 * The success of Nvidia driver installation can be tested by going into the Graphical interface ("X window", "X11") and run glxgears:
 * Open a terminal and run glxgears, rotating gears should appear.
 * Open a terminal and run glxgears, rotating gears should appear.


 * The difficulty for Nvidia driver installation comes from the presence of the open source "nouveau" driver. It needs to be "blacklisted" in order for the Nvidia driver to install. If it is already incorporated in the kernel, then a new kernel needs to be compiled to replace the current one and the system needs to be started with the new kernel. See this webpage:

Possible way to disable nouveau
Reference: https://askubuntu.com/questions/481414/install-nvidia-driver-instead-of-nouveau

1. Edit /etc/modprob.d/blacklist.conf, add this line:

2. Remove installed nvidia-* packages

3. Build a new kernel

Step 4. Compiling RELION
Once CUDA installation is successful, it is best to install RELION the next by following the RELION 2.0 compilation instructions: https://www2.mrc-lmb.cam.ac.uk/relion/index.php?title=Download_%26_install

A few development packages will be required for compiling RELION. Related information can be found at the bottom of this page.

The reason to compile RELION first is that this would force the system to check the existence of a functional CUDA library and a few frequently needed packages, making the installation of other packages less troublesome.

RELION compilation Procedure

 * 1) Install necessary packages:


 * 1) Download RELION-3.0 from GitHub:


 * 1) Configure and compile:

EM software installation with SBGrid
An alternative to the manual installation and maintenance of all pieces of software needed for EM, is to join the SBGrid consortium (https://sbgrid.org/) Once a laboratory becomes a member, it allows every lab member to deploy/install a collection of 270+ structural biology applications on Linux and OS X systems. Every user can deploy an SBGrid installer onto their computer and then select which pieces of software (and which versions of those) the user wants to have installed. SBGrid makes it easier to manage software installations, especially in situations when a single user is responsible for maintenance of multiple computers. SBGrid membership could also be used to fully setup computers for structural work other than EM due to the wide selection of applications available.

Steps to install SBGRid software:

1) Have the PI of your lab sign up for a membership with SBGrid

2) Verify whether your system meets the requirements for SBGrid deployment:

Linux: software is tested/built under Red Hat Enterprise Linux 6.x and 7.x (use Red Hat / CentOS / Scientific Linux / Fedora distributions) Other versions of Linux could in principle be used, but more issues might arise for users when using those other distributions.

Mac OSX: 10.11 - 10.13

Make sure you have admin/sudo user privileges for the account on your computer from which you would like to perform the installation.

3) To run the SBGrid installation client, set up an account by registering (https://sbgrid.org/registration/register/)

4) Install the software and activate it, use instructions from SBGRid Wiki (Mac OSX has a graphical installer, Linux uses the command-line based one (https://sbgrid.org/wiki/client_CLI_install))

Make sure you always launch the sbgrid installer from its correct location (where you donwloaded/unpacked it). You will do that every now and then to install/uninstall/update your software, so remember where you put it.

5) Consider your disc space. Originally, sbgrid gets installed to /opt/sbgrid folder (with a symlink automatically set up to /programs). This location can be changed however (in situations with a very small system disc it is required to move the sbgrid installation directory somewhere else). Follow the instructions from SBGrid website to do that https://sbgrid.org/wiki/client_CLI_install (under: Installation on an external drive or USB key). Note that any location can be chosen - you don't have to move it to an external disc. It can be even a different partition on the same disc. Since during this operation a symlink is created from your newly chosen location to /opt/sbgrid, all should work fine.

6) Install all the pieces of software you think you might need, following the Wiki instructions linked above.

7) Set up an alias for SBGRid environment: - use the following command to open your bashrc file

- add a line at the end of the file

- save the file, log in and log out for changes to take effect

8) Using SBGrid: - to initialize SBGrid environment, simply type this in a new terminal window:

- launch the software you need from that very terminal window, using the name under which it is displayed in the applications list

9) Other: cryoSPARC CryoSPARC is not part of the SBGrid library and needs to be installed separately. Follow the instructions here (https://cryosparc.com/docs/reference/install/) and for an installation on a single PC use Quick Install: Single workstation version of the manual. Pay attention to the PATH you specify during installation for CUDA drivers. Beginning with version 2.5, cryoSPARC can work with CUDA 9 and 10, but older version require CUDA 8. After the installation, simply navigate your web browser to http://localhost:39000, log into your account with cryoSPARC and enjoy.

Potential issues: - Lack of access to cryoSPARC through the web browser: Run the following command (it is not location-specific):

- Needs to change the PATH to CUDA drivers directory post-installation: Navigate to your cryoSPARC installation directory, find the file */cryosparc/cryosparc2_worker/config.sh and edit it:

Change the  to what you desire, e.g.:

Linux and Nvidia

 * Test if OpenGL is working, i.e., if the Nvidia driver is functional


 * Monitor GPU activity


 * Set Ubuntu to start in text mode, for installing the Nvidia driver


 * Set Ubuntu back to start in graphical mode


 * Restart Linux


 * Installing DKMS may reduce the frequency of Nvidia driver problems.

Compiling RELION

 * fix cmake "No CMAKE_CXX_COMPILER could be found."
 * RELION compilation
 * Also solves cryosparc2 pycuda installation error




 * fix cmake "No MPI_C lib"


 * fix cmake "no lib-X11 "


 * other libraries required to compile RELIOIN