HPC and cloud distributed computing, as a journey

HPC and cloud distributed computing, as a journey

• 1.
  Flexible Compute Environment •Wehave successfully been using the Farms for a number of years. •These are a fantastic tool for our science. •But now - new cloud technologies are available. •We want to be able to take advantage of these industry standard technologies in our own private cloud – whilst still retaining the benefits of the farms.
• 2.
  HPC and Cloudcomputing are Complimentary Traditional HPC The highest possible performance for Sanger workloads • A mature and centrally managed compute platform. • High performance Lustre filesystems and system libraries optimised for heavy data centric workloads. • Efficient fair-share workload management across large scale infrastructure • The ability to add bespoke or disruptive technologies, such as FPGA or GPU accelerators. Flexible Compute A collaborative platform, with security and flexibility. • Full segregation between projects ensures data security throughout computation tasks. • Offers standard API’s and tools including Docker to our developers and community of collaborators. • Scale-out computing with the ability to over commit CPU allocation ensure efficient resource utilisation • Developers and collaborators are no longer tied to a single operating system or shared libraries. They are free to follow the latest technologies and trends
• 3.
  Cloud computing: aflexible platform for collaborative research •Cloud computing permits our customers to run their own images within a secured network environment. This means that: •We are no longer tied to the operating system and libraries that run on todays HPC platforms. •We can distribute our code in a pre-packed format that others can share. • By sharing these images we can send Sanger pipelines to collaborators without technical knowledge. This sends the workloads to where the data resides. (Not data to the work.) •Cloud computing provides common API’s and libraries to our research community. •We can support disruptive technologies, including Docker in a secure environment, accelerating development of new scientific pipelines and software providence. •We can overcommit CPU usage by at least 1.5 : 1. This allows for greater resource efficiency and cost management.
• 4.
  Cloud computing collaboration acrosssites Open Standard APIs Federated AAAI MRC Bioinformatics Centres Public clouds WTSI Flexible Compute Common Collaborative API’s, and familiar self service environments that are secure by design
• 5.
  Future HPC andcloud computing integration. OpenStack CASM HPC cluster Cloud environment Humgen Core … Software Defined Networking (Infrastructure as Code, IaaS) It is becoming possible to install and manage traditional HPC infrastructure using the same tools that are provided by our cloud environment This would allow us to: • Provide a consistent securable network infrastructure across all computation resources. • The provision of both HPC resources in a traditional manner and access to cloud resources on a per tenant basis. • Provide the best of both worlds for our customers. • Reduce management overheads • Expand to meet future demand • Accommodate disruptive technologies as they arise. Build from a menu requires no user knowledge….
• 6.
  Sanger flexible compute platformis coming soon •Our new flexible compute platform has been under acceptance and early testing for over a month. •We’ve had excellent feedback from our research customers. •On release, this platform will provide: • 9000 vCPUs • > 50Tb memory • 1Pb of storage, including externally accessible S3 object storage. Released on Wednesday the 22nd of February 2017 For more details, please contact Peter Clapham or Tim Cutts.
• 7.
  OpenStack day onCampus •On March the 10th we are hosting an OpenStack day •Guest speakers from the wider community. •We have Tim Bell from CERN as a key speaker. •Tim is responsible for CERN’S Openstack environment of > 200,000 cores. •Other attendees include representatives from:
• 8.
  Production openstack (I) •107 Compute nodes (Supermicro) each with: • 512GB of RAM, 2 * 25GB/s network interfaces, • 1 * 960GB local SSD, 2 * Intel E52690v4 ( 14 cores @ 2.6Ghz ) • 6 Control nodes (Supermicro) allow 2 openstack instances. • 256 GB RAM, 2 * 100 GB/s network interfaces, • 1 * 120 GB local SSD, 1 * Intel P3600 NVMe ( /var ) • 2 * Intel E52690v4 ( 14 cores @ 2.6Ghz ) • Total of 53 TB of RAM, 2996 cores, 5992 with hyperthreading. • Redhat Liberty deployed with Triple-O
• 9.
  Production openstack (II) •9 Storage nodes (Supermicro) each with: • 512GB of RAM, • 2 * 100GB/s network interfaces, • 60 * 6TB SAS discs, 2 system SSD. • 2 * Intel E52690v4 ( 14 cores @ 2.6Ghz ) • 4TB of Intel P3600 NVMe used for journal. • Ubuntu Xenial. • 3 PB of disc space, 1PB usable. • Single instance ( 1.3 GBytes/sec write, 200 MBytes/sec read ) • Ceph benchmarks imply 7 GBytes/second
• 10.
  Production openstack (III) •3 Racks of equipment, 24 KW load per rack. • 10 Arista 7060CX-32S switches . • 1U, 32 * 100Gb/s -> 128 * 25Gb/s . • Hardware VxLan support integrated with openstack *. • Layer two traffic limited to rack, VxLan used inter-rack. • Layer three between racks and interconnect to legacy systems. • All network switch software can be upgraded without disruption. • True linux systems. • 400 Gb/s from racks to spine, 160 Gb/s from spine to legacy systems. (* VxLan in ml2 plugin not used in first iteration)
• 11.
  But what arewe providing? CloudForms Service driven access OpenStack Horizon Granular control over instance Direct API Access Direct https access from anywhere Accessible only from within Sanger Ceph Object Storage (Used to provide volume and image storage) S3 Object Storage Layer
• 12.
  How Does thisFit with Existing Services? OpenStack “Bubble” Compute Ceph 100GB/s SDN network infrastructure Sanger internal systems Access to secured services i.e. iRODS Databases CIFS (Windows shares) S3 API access OpenStack API and GUI access 80GB/s connectivity No access to: NFS Lustre
• 13.
  CloudForms Interface
• 14.
  Horizon Interface
• 15.
  Efficient Resource Management OpenStack resourcesare managed at a tenant group level. • Each “tenant” group has an assigned quota for: • Disk • CPU • Memory Once limits are full, tenant members will either have to wait for resources to become available or shutdown or terminate a running instance. Initial quotas are agreed with the IC before creation
• 16.
  Quotas, they arenot all the same. Some groups have a requirement that they have an absolute number of spots available for essential services Other groups would like to burst to meet demand as required. These requirements do not fit well with each other.
• 17.
  The Proposed Workaround Forthose projects which require guaranteed access: • We create a dedicated tenant group that has specific access to a set quota allocation of vCPU, Disk and memory. • This is tied directly against reserved hardware This guarantees requested resource will be available when required, whilst providing security, operating system flexibility and instance management. BUT there is no ability to use more than the requested allocation
• 18.
  Dynamic Workflows Dynamic workflowscan expand to meet demand and collapse when not required. So a quota that matches the initial resource request will mean constantly under quota’ing the system For the initial release we will start by: • Overcommitting CPU by 1.5 : 1 (available total vCPU ~9000) • Overallocate quotas so that 115% of the overcommitted vCPU is available to tenants. So some initial ability to use more of the system than may be available.
• 19.
  For More Details,see https://docs.google.com/a/sanger.ac.uk/document/d/17z9urhh3bTLRhQo9b8Ccs ZW_3O7cxlGY9uiwpAS_GqQ/edit?usp=sharing Or http://tinyurl.com/zzurp5s We are adding monitoring and metrics gathering to the system. This will provide a feedback loop for quota and project management.
• 20.
  New Opportunities for ApplicationDevelopment Cloud application development aims to scale out compute and provide: • Auto scaling of key services • Making pipelines cost effective on commercial platform providers • Self-healing of service components that fail • Creating resilient services with reduced impact when service components fail • Not tied to any one specific environment • Enabling sharing of code, images and services with collaborators. This can dramatically reduce the need to copy large data sets around the world and permit running complex pipelines where the data resides.
• 21.
  How do wesee Migration ?
• 22.
  Initial Early Adopters. Wehave some early adopters ! 1. Mutational signatures 2. Imputation service 3. Blast service 4. Pan-prostate We look forward to hearing more from these groups soon !
• 23.
  Mostly Share aCommon Approach Web Interface Data upload Run analysis Update job status data base Present data Invoke Analysis Retain a copy
• 24.
  Adaption to Cloudbased tools Stage Current approach Cloud approach User details local databases, directory services, Oauth Oauth, directory services Data Downloads Globus or https S3, Globus Job status RDBM: MySQL, Oracle or PostgreSQL NoSQL: Mongodb, Cassandra or REDIS Invoke Job Analysis Hand crafted equest to LSF AMQP Run Analysis LSF job submission AMQP, Heat orchestration or API call to Openstack Present data Make available via sftp, Globus or https web upload S3 automatically generated URL's Keep data No consistent approach S3, archive as required Service failure Await systems Use IFTTT or add code to instance to raise or restart an instance as required Autoscale options Await systems Use IFTTT or add code to instance to raise or restart an instance as required Service discovery Manual Cloud init, heat templates, dynamic DNS