Infrastructure layer

What is the Infrastructure layer

This layer represents all the infrastructure elements required to set up an EBSI node including network, compute, and deployment capabilities.

The infrastructure layer is used to aggregate all the tools, methods, and services needed to develop, build, test and deploy, and maintain the EBSI node image. 

Anatomy of the layer

None of the services within the infrastructure layer are directly exposed to consumers. They are used to manage the EBSI node image and all the higher layers services.

• Host Image: The service that delivers the pre-build EBSI node images (which is a collection of 3 host images configured to play different roles).
• Containers: The service that builds the docker containers of the higher-layer services.
• Configuration Management: The service that produces the configuration policy for each service.
• Infrastructure as Code: The service producing the scripting for the definition of host images.
• QA testing: The automation approach for EBSI core infrastructure stack releases includes testing at every layer of the node stack, integrated within the agile development methodology and powered by multiple test automation tools.

Host Image

What is the Host Image service

In EBSI V2, the host image has been consolidated into one single host running all the EBSI services, packaged as containers. Also, the image is a shell, with no pre-configured containers. instead, the configuration to pull the containers is enabled when the EBSI V2 node and the Configuration manager exchange keys.

Why we need it

This service facilitates the deployment of nodes to the maximum extent possible. The idea is two-fold: from one side, to leverage the number of steps needed by technical teams operating the nodes. From another, to have the certainty that the Operative Environment, Software Packages and Configuration are delivered equally across all nodes, thus reducing the possibilities of diverse configuration combinations, making support much easier. That leaves the network, and to a certain extent, the hypervisor, as the only variables of change that can affect the run time of an EBSI node. It is the intention to keep reducing variability in order to achieve a decentralised but homogeneous network.

Anatomy of the service 

This service can be considered as a one-stop shop service for Member States to be able to implement a node. The Member States use it to have the current EBSI Image for their own computing environment. The image, which the EBSI DevOps team built to facilitate node deployment and reduce the need to plan resources and competency to create a deployment image, is aimed to be easy to operate and easy to upgrade from.

Each EBSI V2 node consists of one single node deployed using the EBSI host image. Distributed images are created with a single operator local account, with their password distributed in a separate email. Instructions are provided to execute one script in each host, which, based on the host-name, configures and starts the necessary services for each role. SSH keys for node operators can be sent to Support Office, who place them in a safe repository, and ensure that operators can access their node in a convenient and secure way.

Technical implementation

Host images are a self-contained operative environment with OS, tooling, and configuration necessary to bootstrap the EBSI node. The operating system of choice is CentOS 8, a derivative or RHEL 8, the Enterprise Edition of Red Hat Linux, but without any associated license cost. A number of components are installed in the images, packages, libraries, groups and users, in order to provide the necessary functionality to operate and configure each host.

The images are created in the following way.

A minimal Centos 8 installation is deployed in a standard hardware, VPC or cloud instance, where appropriate. A few changes are made, such as setting the disk sizes, partitions, file-systems and labels. Then a configuration management agent is installed and run against our infrastructure server, which acts as configuration management service. The build instance is furnished with all common configuration, plus all necessary components to start all EBSI services. The image is left ready, but empty, in order to be initialised by the Member State's operator.

At this point we have a ready to be initialised host. The next step is to run a scrip that de-voids the system from any identity, removes all logs, all keys and uniqueness, so it becomes an anonymous but fully loaded system. Then, a randomised password is set for the ebsi1-operator local account, and it is set to be changed on first login. Then, the host is shut-down.

From this shut-down system we can either create an image like in the case of cloud platforms, or package the resulting system disk as images from distribution as it happens in the case of hyper-visors. In the case of images for hyper-visors, such as ESX, KVM, and HyperV, an additional step is needed to shrink the image to a portable size, and in some cases, to make it viable.

For EBSI v2, we officially test and support AMI for AWS EC2 instances and OVA for Vmware ESX virtual machines. Only OVA virtual machines are distributed to member states.

The Host Image service is currently delivered for the following environments:

• ESXI v5, v6
  • An OVA image, providing settings and disks.
• Amazon Image
  
  • AMI For internal use in EBSI project.
  • It will be possible to share it with MS in the future.

Code repository

Host Images can be downloaded from the EBSI Infrastructure Server. The credentials are communicated to registered and authorised M.S., via CEF Support Team.

Containers

What is the Containers service

A service to build the docker containers of all the services within the EBSI image. The EBSI image is a collection of micro-services, bundled in multiple containers. The Containers service provides the tools to build, initialise, and maintain all these services. 

Why we need it

This service is to have a formal service distribution approach while building the new versions of services and to be able to deliver up-to-date services to an operating EBSI node if required.

Anatomy of the service

The Container Service is composed of three parts.

• Docker running in each EBSI host, alongside some auxiliary tooling.
• A directory structure with files providing instructions to start the containers in their Runtime Environment.
• A Container Registry, with Container Images produced by each development team.

Each Development team produces one or several Service Container images which are uploaded to a Container Registry. The instructions to start up and shut down services are placed in a code repository.

Technical implementation

The Host Image runs a Container Management Service, Docker in EBSI V2, and a series of vendor scripts, like Docker Compose and some EBSI developed ones. The entire setup is delivered in the image as per the Configuration Policy driven by the Configuration Management service, in EBSI V2 this is provided by Puppet CE, running in an infrastructure service.

For each Service, there is a directory structure with one or several files that describe their Runtime Environment. Often, some settings need to be set from the Operative Environment. These are placed by the Configuration Management when first running the Host initialisation script.

These are two different files.

• A Docker-Compose YAML formatted file that states dependencies, software, network, ports, and location of the container image to run.
• And an environment file that provides answers to place holder variables to setup the run time, such as the Host's IP address or credentials to start the Service.

The Container Image is found on the Container Registry, and an appropriate version of the Service is pulled by the Container Management Service based on a version tag.

This provides a way to initialise the Hosts with the correct Services at first bootstrap, and also gives the option to provide updates to operation EBSI V1 nodes.

Code repository

Each component uses its own code repository.

https://ec.europa.eu/digital-building-blocks/code/projects/EBSI

Configuration Management

What is the Configuration Management service

The Configuration Management service provides a formal approach to manage OS, middle-ware, and application configuration by defining the desired state that is applied to the host via an agent.

Why we need it

This service makes sure all images are created against the same configuration and that hosts are kept in a know state. It also ensures that configuration changes can be brought to operational existing nodes when new nodes join the network. Finally, it is instrumental in bringing runtime metrics of the EBSI network. 

Anatomy of the service

The Configuration Management uses Puppet which is composed of several software components working together as an ecosystem.

• Puppet Server: This provides Configuration Policy and Data, and a Classification method.
• Puppet Agent: This is running in the hosts that executes periodically and communicates with the server.
• Hiera: An extensible tool to obtain external data from different sources, defaults to hierarchical YAML.
• Facter: A small extensible tool and library that collects system facts, used to make decisions in policy, templates, and scripts.
• A custom ENC script: Assigns classes and acts as a gatekeeper. Uses the same external data structure as Hiera.
• PuppetDB: A content store that is a repository of host states, agent run reports, and collected facts. It has a nice API.
• A third party PuppetDB Dashboard: Used to easy access and check the collected data.

Technical implementation

Puppet Server runs from the infrastructure server and listens to agents' requests. The agent runs, gathers system and custom facts, and takes a snapshot of the system state and sends it to the Server. The server classifies the host against the manifest or an ENC script and assigns policy classes to it. The server then creates a catalogue of changes based on the policy and the gaps that need to be fulfilled to align with its policy. The changes are adapted to each system's native methods to carry out each change.

In order to make decisions – aside from policy – Puppet sends facts to the server which condition the contents of the catalogue that will be sent by the server. Part of the policy may be filled by variables that are provided by Hiera via a lookup function.

During a Puppet agent run, changes can be applied or not, but regardless, a report is generated and uploaded to PuppetDB. There, the node state history, facts, and reports are stored. The dashboard offers a visual way to navigate through all the data and quickly spot errors or nodes that may be out of sync or not reporting.

Changes to Puppet policy are done via version control, using master, int and dev branches. Each branch matches a Puppet Environment, that tracks it. Once a change is pushed to remote, Jenkins deploys the changes via a pull into the infrastructure server, for the appropriate branch. The same happens with data, which, for now, is in the same repository.

DevOps team writes policy, templates and custom facts to make the precise configuration reach every host.

Code repository

The EBSI code repository is not yet available.

The code is divided in three

https://ec.europa.eu/digital-building-blocks/code/projects/EBSI/repos/configuration-data/browse

https://ec.europa.eu/digital-building-blocks/code/projects/EBSI/repos/configuration-files/browse

https://ec.europa.eu/digital-building-blocks/code/projects/EBSI/repos/configuration-management/

Infrastructure as Code

What is the Infrastructure as Code service

A service producing the scripting for the definition of Cloud Infrastructure in different platforms such as AWS, Azure and others. In a more general way, it is a method of interaction in a programmatic way with any cloud platform that is interfaced by an API.

Why we need it

To have a formal approach to deliver Cloud Infrastructure following a modular approach that gives quality and repeatability to the structures needed to host the resulting images in those EBSI nodes that are hosted in supported Cloud Platforms.

Anatomy of the service

Terraform is used to create modular code to define AWS Cloud infrastructure for the EBSI nodes operated by the European Commission. It is possible that the Terraform plan files could be shared with external organisations once the code is tested and mature enough.

The structure is as follows:

• Terraform client in the infrastructure server. It validates code, creates plans, and applies the plans. It can also be used to inspect current infrastructure states.
• Terraform plugins are installed for each platform used.
• Modules that describe an EBSI node in AWS and other platforms.
• Node definitions, structured by Cloud Platform regions and EBSI environments, including all the data to bring the nodes on-line.

Technical implementation 

DevOps make changes or add features via version control, using repositories hosted in Bitbucket. Three branches, 'master', 'int' and 'dev' match the EBSI environments, as well as Puppet environments, so each change is aligned and stays in one 'tower'. Jenkins polls the changes in remote and deploys them to a local directory in the infrastructure server. The code then seats ready to be used by the operator, but is not automatically executed. In the future will be very possible to have parts of the IaC to deploy automatically as part of the Image and Services testing and Life Cycle.

The operator then can validate the code, create and name a plan, and apply or execute the plan. Terraform will then make the changes in the cloud objects that require it and produce a report on screen. Reports are currently not saved.

Code repository

The EBSI repository is not yet available. 

The old repository is available here: https://ec.europa.eu/digital-building-blocks/code/projects/EBSI/repos/6-infrastructure/browse

QA Testing

What is the Automation service

With Automation, we are referring to a series of processes and tooling used in Infrastructure to build, organise, test, plan, and deploy resources, objects and services to higher layers.

Why we need it

 This service ensures quality deliveries of code and facilitates the deployment of nodes to the maximum extent possible.

Anatomy of the service

The automation approach for EBSI core infrastructure stack releases includes testing at every layer of the node stack, integrated within the agile development methodology and powered by multiple test automation tools.

Technical implementation 

The technical implementation is based on a set of tools and standards that perform:

• Jenkins has been selected as central point of Continuous Integration Build orchestration tool, based on its popularity and extensive user-community. At the time of writing, a simple (master/no-nodes) server is being setup and running 24x7. This environment is more than enough to bundle the CI pipelines used by Developers, Testers, and DevOps team.
• The current Jenkins service is available at: https://infra.ebsi.xyz/jenkins/

The Jenkins dashboard will centralise the status of pipeline execution of the different steps involved, from software inception to final application or service deployment. Pipelines will also include Infrastructure-as-Code (IaC) Pipelines triggering the deployment of the underlying infrastructure where software will run.

• Unit testing considered part of the internal tooling available to different developers and no restriction or integration constrains are imposed from the QA/testing point of view at the time of writing.
  Developers have the final word regarding the quality and quantity of their unit-tests.

• SonarQube, static Javascript and TypeScript backend code analysis. Of interest to EBSI is its support for Javascript and TypeScript used for the development of front-end and back-end code.

  SonarQube can automatically detect security bugs, QA bugs, null pointers/undefined variables that in some cases compete with the skills of proficient developers and with low numbers of false positives. Note that it will always perform better with languages with support Type and metadata systems (vs. dynamic un-typed languages).

  SonarQube integration with Jenkins allows the user to visually display a report with statistics of each run (bugs, vulnerability, 'code smells', etc.).

  The current Sonar service is available at: https://infra.ebsi.xyz/sonar/

• Functional Testing of the backend API, will emulate a remote client triggering HTTP requests against the Rest API provided by each backend service. Node is used to create the happy/non-happy paths for each exposed public API and able to interpret the swagger API description. 

• Slither, static Solidity code analysis.  Slither was chosen for first static code analysis due to its simplicity when integrating with shell scripts, Jenkins, and JenkinsPipelines, with no restriction on the additional use of MythX if needed in the future.

The technical implementation follows the guidelines of the Testing Strategy.

Code repository 

The EBSI code repository is not yet available.