Informatics Support Group

High Performance Computing

Our Informatics support team provides support, architecture design and development of Sanger's traditional high performance compute (HPC) clusters and OpenStack private cloud (flexible compute infra-structure). Our traditional HPC environment currently provides:

• Approximately 38,000 cores of computational resource, using IBM Spectrum LSF DRM (https://www.ibm.com/us-en/marketplace/hpc-workload-management) and OpenStack (http://www.openstack.org)
• 23.5PB of lustre clustered file systems, (https://www.ddn.com/products/lustre-file-system-exascaler/, http://www.intel.com/content/www/us/en/lustre/intel-solutions-for-lustre-software.html)
• 30PB of iRODS managed storage repository (https://irods.org/about/). This is currently cross site replicated and provides management of all sequencing data created at Sanger since 2009.
• 160Gb/s network backbone. In total our datacenter network contains enough cabling to cover the distance from lands end to John’o’Groats !
• Our clusters are currently running the Ubuntu operating system.
• We provide support for both locally created code as well as for the for a range of centrally managed applications.
• We provide consultation support and assistance for all aspects of cluster use and run a cluster course here at Sanger for new starters.

Such traditional platforms have been the bedrock of computational processing within the bio-informatics field for the past 20 years. They are well known and have performed well to date. As Sanger moves towards the next 5 years, it does so with a clear vision towards bio-informatics research and developing healthcare applications (https://www.sanger.ac.uk/about/who-we-are/sanger-institute/our-vision). Supporting cutting edge research of this nature requires that we provide platforms that provide an increasingly cost effective, flexible and scalable platform with opportunities for resilience and self service.

Since Feb 2017, our team has been delivering an internal private OpenStack cloud to our scientific researcher community. This flexible compute platform will is based around RedHat OpenStack and CloudForms and now provides:

• ~ 12,000 cores (SuperMicro)
• > 100TB memory (SuperMicro)
• 5.5PB usable NVMe index accelerated Ceph storage (SuperMicro)
• 1 or 2TB SSD per compute node for high speed ephemeral storage
• 100GB/s point to point software defined networking (Arista)

We also supply a range of supporting applications and images to our research community as they migrate their workloads to a more distributed compute platform.