Discussions

Infrastructure for scientific web services and databases

  • Chris Morris profile
    Chris Morris
    29 February 2016 - updated 4 years ago
  • 2 comments
    Total votes: 1

Your experience

EUDAT does a lot to curate files produced by researchers. EGI provides processing resources. However, there is a major activity which has no infrastructural support at present: the development and curation of structured data resources, and the provision of services based on them. These are often relational databases with query interfaces.

Recently I have had a couple of inquiries from projects looking for a permanent home for such resources, after development funding ends. Doing this effectively and securely is hard. But it's not impossible. And having a long term home in mind from the beginning has some benefits.

Your ideas

Best practice for hosting a web service includes automatic end-to-end testing of functionality and availability, mod_security type filtering of requests, fuzz testing for security, and hardening of the server. Doing this properly is beyond the resources (and often the skill set) of most development projects. Providing a shared centre of competence makes this feasible.

These techniques are synergistic. If there are end-to-end tests, then it is possible to update the OS and know whether anything is broken. Fuzz testing goes hand in hand with training a custom request filter; and if this is done while there is still development resource, then the developers can make the service more robust.

A project to set up such a service should be done hand in hand with use cases, development projects in specific research communities.

Anyone interested in collaborating in such work is invited to add a comment.

Comments

Log in to post comments
Tiziana Ferrari's picture
Tiziana Ferrari,
01/03/2016 18:49

Dear Chris

the problem you raise has two aspects, one technical, the other of policy nature.
1- technical: I assume tests for hardening of the service end functionality tests can be automated if the use case is sufficiently general. EGI for example packages and distributes nagios probes for testing of public interfaces for many services including community ones, many tests are quite standard, test suites could be developed and distributed; a platform for hosting such web services could be provided (I am thinking of the INDIGO Datacloud project here)
2- policy: if the objective of the project is providing a service, not only a hosting environment is needed, but a business plan is required upfront in the development project: what are the assets/products to be maintained (software, data, technical support, operating costs etc.) and who will pay for that?

Both problems need to be addressed. (2) is as important as (1)!

Total votes: 2
Davide Salomoni's picture
Davide Salomoni,
02/03/2016 03:45

Dear Chris and Tiziana,

thanks for bringing this up. It goes at the core of some of the INDIGO-DataCloud objectives (INDIGO-DataCloud is the H2020 project I am coordinating).

First of all, I think this is an excellent example of the fact that we need a clear service portfolio in any European e-infrastructure that wants to offer services to scientists. The need here is I would say basic in its fundamentals: make sure that a certain scientific community is able to find a trusted place where to host its applications and databases in a highly available and reliable way.

So, on the one hand, Tiziana is clearly right when she says that policy decisions need to be addressed. So, yes, when you plan a project that goes beyond a development phase to eventually enter production mode, you should factor in a business plan. This is something that normally people won't disagree on (at least in theory). But Chris highlights an even more basic thing: *where* do I find such a service? What are its characteristics? And *how* is it run, maintained, customised to specific needs? And, last but not least, how much does it cost?

As I said, this should really be addressed in a general "Infrastructure for scientific web services and databases" (the title of this thread) through the definition of a clear service portfolio. If such a service portfolio (with associated costs and procedures) does not exist, we could hardly ask projects to factor in proper business plans.

On the other hand, let me now say something about the second point, the technical issues highlighted by Tiziana. I think that some thinking from the technical side is needed if one wants to include web services and databases in the service portfolio I alluded to above.

This is where INDIGO-DataCloud can help, I hope. We are about to deliver our first software release (July 2016), and that will include among several other things the ability to dynamically instantiate and monitor long-running services on a (public or private) Cloud infrastructure. From the point of view of a customer (a scientific community), this means that there will be a template describing the application and its needs; for example, the need for a database engine, or the desire to restart it on another resource should the one it is running on failed. So, no need to assume you need to package or run Nagios probes, to provision virtual machines somewhere to install and configure them or the application, to deal with OS updates, etc. Just focus on the (highly-available, trusted) service you want to offer, i.e. the application and its dependencies.

So, I am definitely interested in evaluating how we can help with that. In the INDIGO-DataCloud strategy, we have defined a number of pilot sites that will expose INDIGO services to scientific communities. This will happen as I say after July 2016 (the date of our first public software release). Chris, perhaps we could discuss if and how this can be useful to you and the communities you are in contact with. EGI.eu is part of the INDIGO collaboration and is involved in INDIGO exploitation activities so I think that EGI will be a key player in facilitating the potential uptake of these solutions. One last comment, which is outside the direct goals of INDIGO: I think that Chris is very right when he mentions that know-how and support to community must be an integral part of the service portfolio.

Hopefully these points will also be of help in the definition of a European open infrastructure for science.

Total votes: 0