The Buyers group and its challenge
In the HNSciCloud PCP, CERN (CH) lead a buyers group of ten public research organisations from seven European countries: National Institute for Nuclear Physics INFN (IT), German Electron-Synchrotron DESY (DE), National Center for Scientific Research CNRS (FR), Karlsruhe Institute for Technology KIT (DE), SURFsara (NL), Science and Technology Facilities Council STFC (UK), European Molecular Biology Laboratory EMBL-EBI (DE), Institute for High Energy Physics IFAE (ES), European Synchrotron Radiation Facility (ESRF).
They joined forces in the PCP to get new solutions developed that meet the demands of large-scale scientific projects for big data storage and analysis tools. Upcoming projects like the High Luminosity LHC and in the domains of Genome Analysis, Astrophysics, Life Science and Photon Science have requirements for their resources to scale massively, being able to share the data and to some extent being available on demand. All of this being deployable in potential world-wide scientific collaborations including the ’long tail of science’.
The buyers group expected the following benefits from new hybrid cloud solutions:
• economy of scale – further reducing the costs for commercial cloud resources
• more elasticity – adaptable to changing demands for faster and more efficient scaling
• more concentration on science-specific demands not covered by commercial cloud providers
• standardization - make computational and storage infrastructure changeable and replaceable
The specific technical challenges included:
• Compute and Storage - support a range of virtual machine and container configurations including HPC working with datasets in the petabyte range accessible transparently
• Network Connectivity and Federated Identity Management - provide high-end network capacity via the publicly funded GEANT network for the whole platform with common identity and access management
• Service Payment Models - explore a range of purchasing options to determine those most appropriate for the scientific application workloads to be deploye
The HNSciCloud pre-commercial procurement
The PCP took place between January 2016 and December 2018. Initially 4 bidders were selected for phase 1 of the PCP to produce designs of which 3 were then selected in phase 2 to produce prototypes. In the pilot phase 3, 2 consortia were selected to deploy platforms offering each 10,000 CPU cores, 1 petabyte of storage and 40Gbps network connectivity. The buyers group has successfully deployed applications and datasets used by end-users from astronomy, high energy physics, life sciences and photon/neutron sciences on these innovative cloud services. More details on the solutions and the applications deployed.
The innovative cloud services of the 2 consortia - RHEA group (SixSq & Exoscale) and T-Systems (OTC) - that deployed their solutions in the last PCP phase, have already been integrated into the service catalogue for the European Open Science Cloud.
The PCP procurement process has led to the introduction of commercial cloud services to support the IT strategy of European research organisations. The PCP triggered the development of digital cloud services that are more adaptable to changing needs, provide a cloud-based system designed specifically for science and research, increase standardisation, benefit from economies of scale, and offer appropriate service payment models for research bodies using it. This has permitted the public research sector to make use of the latest, most energy efficient IT technologies hosted in recent data centres with excellent power usage effectiveness. The new cloud base systems offers certified services that ensure compliance with relevant standards and legislation notably in the domains of security, and data protection. The innovative solutions developed by the commercial contractors in the project are based on open source implementations that do not require licenses in order to be deployed on the in-house IT resources of research organizations connected to the hybrid platform. HNSciCloud put in place also an innovative voucher scheme that has encouraged the adoption of cloud services by individual researchers, frequently referred to as the ‘long tail of science.
Mid-term impact - Larger scale deployment
The results have the potential to deliver long-term impact for the SMEs involved in the PCP and also SMEs who may wish to bid to subsequent Scientific Cloud procurements. The project is very much in the vanguard of such Scientific Cloud procurement activities and its success may stimulate other companies, including other European SMEs to bid in the future.
The results accelerate and expand the research that can be performed in a number of scientific domains that contribute to address societal challenges including climate change and health. The innovative cloud services support the connection of the research infrastructures identified in the ESFRI Roadmap to the nascent European Open Science Cloud intended to create a single digital research space for Europe’s 1.8 million researchers.
In this context, the follow-up EU funded OCRE (Open Clouds for Research Environments) project will make the new innovative cloud services accessible to a broader set of public research and higher education bodies, supporting future cutting edge research. In the meantime, the buyers group has already grown to 40 research organisations that plan to launch together, under the lead of GEANT, a call for tender in November 2019 to buy the innovative cloud services.
The HNSciCloud pilot phase also identified a gap in the current market offering for long-term data preservation services that is being addressed in a new Pre-Commercial Procurement called ARCHIVER.
The Procura+ awards
More information about the Procura+ awards can be found here.