Computer hardware setup

This page is created to provide some references for setting up computation hardware, mainly desktop systems, for small labs.

Observations and impressions

 * RELION generally requires large amount of GPU time for refinements. The RELION documents mention that two GPU cards are needed for 3D auto-refine: https://www2.mrc-lmb.cam.ac.uk/relion/index.php?title=Benchmarks_%26_computer_hardware
 * cryoSPARC is not very demanding on hardware. The minimum setup can be a single (good) GPU and 16 GB CPU memory. 32 GB CPU memory is recommended though. The cryoSPARC v1, now phased out, used to have a 32 GB memory limit for starting heterogeneous refinement jobs. Although in practice bypassing this limit on a 16 GB system, for a 300 KDal protein project with a small dataset, didn't show any negative effect.  Some cryoSPARC tasks can utilise multiple GPUs.
 * cisTEM uses CPU computation only. The time consumption is still in the acceptable range. It will try to use all physical cores of a CPU unless limited explicitly by the user.
 * EMAN2 does not use GPU except for particle picking.
 * Memory requirements vary among jobs. This is determined by the size of the particle images, the number of particles, and the software package.
 * Negative stain datasets are usually small (up to 10K particles) and the processing goals are usually limited to obtaining 2D classes. Therefore they do not really need high-end computation hardware. Even an obsolete 4th gen i5 computer with 8 GB memory and no GPU can handle such tasks.

A fast CPU
A flagship desktop CPU, such as Intel Core i7 or i9, or the AMD equivalent, should be adequate. The GPU-heavy tasks may not benefit much from more cores, as the CPU part of computations are often run on a single thread. However EMAN2 and cisTEM both use multicore CPUs, instead of GPUs, to do most computations.

Besides the speed and number of cores, the number of PCIe lanes usable for communicating with the graphics card is also a consideration, because it in theory put an upper limit to the data transfer rate between CPU and the GPU. Intel Core i7 and i9 CPUs always have 16 PCIe lanes for the graphics cards. Xeon CPUs have more PCIe lanes. But Xeon systems are usually much more expensive. In practise, it has been shown that PCIe lanes are rarely the bottle neck.

32 GB memory or more
Ideally 64 GB. Intel Core CPUs (i3, i5, i7, i9) can not use more than 64 GB memory per processor. Xeon CPUs can use more memory, but again, those systems cost much more than regular i7 or i9 systems.

99% of the time the OS and the EM programs will only use 8-16 GB of memory and we do not benefit from any of the extra memory sticks on the mother board, at all. But when the memory usage hits the max RAM size, HDDs or SSDs will be used for caching: some data from memory will be moved to caches on hard disk, then some will be read back for processing. Back and forth multiple times. This may happen during some complex 3D refinement with large particle sizes.

To give an idea: 64bit PC2133 DDR4 RAM moves data at 2133M*64/8=17GB/s, while the NVMe SSDs do that at ~1 GB/s and SATA SSDs at <500 MB/s.

Motherboard that has at least two full-length PCIe slots to accommodate 2 GPU cards.
RELION documents state that its auto-refine jobs require two GPUs to run fast. However 1 GPU is also capable of handling most tasks if not all. cryoSPARC jobs are fast in general, so it is quite likely a single GPU will be enough.

As stated above, for intel Core (i7, i9) CPUs, a total of 16 PCIe lanes are used for communication with the GPU. Therefore when there are two GPU cards in the system, both are communicating with the CPU at 8x PCIe speed. However, this is unlikely to produce much noticeable effect unless the data transfer becomes very frequent.

A side note: the graphics cards can actually run on the short PCIe slots, except that the data transfer speed would be limited to 1x or 4x, instead of 16x (1 card) or 8x (2 cards).

GPU cards
The Nvidia 1080TI or 2080TI video cards seem to give the biggest bang for the buck at the moment. 1080TI is disappearing from the market.

A better GPU card should have more GPU cores, more GPU memory, higher bandwidth (speed of moving data from/to CPU). The size of the GPU memory is most important, as a larger memory reduces the need to frequently transfer data between CPU and GPU.

Mixing different GPUs on a same machine should not be a problem. Some software even know which GPU is faster. Graphics cards of different producer but employing the same GPU chip, for example 1080TI cards from MSI and ASUS, will work largely the same. Therefore adding cards one at a time is not a problem. Before buying a new card, one should check that the computer does have enough power and a usable power cable for it. Each 1080TI card needs ~250W of power and will use one 12V (yellow-black) power cable bundle that carries two 8-pin (6+2) plugs. This cable should normally come with the power box (750W power box should have two such bundles). Some cables coming with some cards are not meant for being used as a power cable for the card. The SLI bridges coming with the cards only help gaming, not GPU computation (CUDA ignores SLI).

SSDs
1TB SATA SSD minimum. NVMe SSDs are roughly 2x the speed of SATA SSDs. However, the number of M.2 NVMe slots on a mother board is usually limited to two or three.

NVMe is a storage interface standard that allows an SSD to directly use the PCIe interface to communicate with the motherboard. Therefore it allows an SSD to transfer data at a high speed, instead of being limited by the slower SATA interface.

NVMe SSD drives usually come in the form of a slim M.2 card. Bear in mind though, that M.2 is only a physical form factor and that being M.2 does not necessarily imply NVMe interface. SATA M.2 SSDs exist in the market. Many slim laptops use M.2 SATA SSDs as storage, for example. The SATA M.2 SSDs are usually cheaper than the NVMe ones, but they are also only as fast as those SATA SSDs in 2.5 inch cases. Make sure to look for "NVMe" instead of "M.2" when trying to buy one.

On a desktop motherboard, an M.2 slot may be PCIe-only or SATA/PCIe (may be set through BIOS). Here PCIe indicates that it can accept an NVMe SSD. When checking a motherboard's manual, which kind of M.2 slots it provides is one thing that worth paying some attention to.

If a large SSD working directory is desirable, one can get multiple SSDs and make them into an LVM. An LVM is a logical volume that spans multiple physical hard drives but pretends to be made on a single drive. For example, if there are 3x 1TB SSDs in the computer and they are made into one LVM, we get a single 3TB, continuous logical volume, which can be mounted as a single directory in Linux.

Power
At full speed, one CPU consumes ~100W of power, one GPU consumes 250-280W. Motherboard, memories, drives and peripherals consume less than 100W in total. For a 2-GPU system, 750W power should be enough.