Discussions

Open Public Reviews – an open consultation

John Magan

18 February 2016 - updated 4 years ago
10 comments
Total votes: 2

The potential benefits of Open Peer Reviews for scientific publications are well known: more openness and transparency; greater trust; credit for contributors; more visibility and reliability. As part of a general move towards applying Open Science principles more widely in the Commission's own operations, it has been suggested to apply the same principles to project reviews in H2020. For some more information on this idea, see the short background document.

We would like to hear from you on this topic. Specifically, we would like to know:

Do you think Open Public Reviews is a good idea for Horizon 2020 project reviews? Why?
Do you have experiences in open reviews?
What do you think are the main advantages and disadvantages? If you mention problems, can they be addressed? How?
Would you agree for your research results to be reviewed openly?

In your replies, please explain your position, and share any experiences you have had that could be useful or illustrative. Deadline for contributions is the 31st of March 2016.

Thank you for your time and responses; we hope it will help us to create a more open scientific community in Europe and beyond.

Short background document.

Update: November 17, 2016

Thanks to you all for your comments here; following on from this and taking account of your comments, especially on the confidentiality issues, we're preparing a small pilot action on open public review of projects in the area of eGovernment (which is by nature open and not commercially confidential). Based on the lessons learned in this pilot, we may roll it out more widely.

More details in due course!

John

Tags:

Comments

Andras Lorincz,

22/04/2016 19:15

I have seen comments on a Flagship project. They were contradictory, they were strongly influenced by personal feelings, e.g., about the control practices of the leadership of the consortium. I have also seen EU projects, where -- in my opinion -- the participants did not understand in full depth what they were up to. These are learning processes and this is OK. It is a good way for evaluation, but it will not help the project selection process, unless it is used somehow.

The key for the selection process -- again, this is my personal view -- is twofold:
1. the relevance of the calls and the person responsible for the big success, terrible failure, or the mediocre value of the outcome of the call
2. the credibility, or value of the reviewers
Both of these are of high relevance and I don't see any effort in these directions.

Open Review Process could be the solution if it is used for improving the selection process: the selection of the calls and the selection of the reviewers. You can make it clear in the call if the results, including the final report, will be subject to open reviewing or not. Still, the call itself can be evaluated, e.g., if it was timely, or not.

Greg Arrowsmith,

31/03/2016 15:25

Open public proposal reviews risk adding to the time needed to perform an evaluation. Time will be needed for the public scrutiny (which, because it is unremunerated, has to be longer than the time that paid evaluators get), also for the proposal coordinator or consortium to answer the comments, and for the evaluators to read and digest the exchanges.

Also, it will be impossible to put commercially sensitive information in a proposal that would be evaluated in this manner, so they are only suitable for programmes not seeking to attract participants from industry.

Some of the aims of 'open public reviews' could be met in other ways. For example, one aim is to help the paid evaluators evaluate better. EUREC proposed a mechanism for this in its position paper (see p5-6: http://www.eurec.be/en/News-Events/News/Feedback-on-two-years-of-Horizon...). The coordinator of a proposal would be given a chance to write, anonymously via the Participants' Portal, to the team that wrote the Evaluation Summary Report of his proposal to give his reaction to the ESR. This would give those evaluators pause for thought and an opportunity to consider whether their remarks were justified. They will do better next time.

John Magan,

19/04/2016 11:11

Thanks, Greg, for your comment. At this stage we're mooting the idea for project reviews, not proposal evaluations...
John

Richard Smits,

12/03/2016 21:44

H2020 aims "to drive economic growth and create jobs", and for this you need the participation of companies. This is quite difficult already, and openness is not making it more attractive because it is against the interests of companies who aim to make profit with the results.
Openness and transparency are beautiful aims, but also create more bureaucracy. I think the burden of having to prepare reports that can be made public will be bigger than the benefit of getting feedback for the project participants, so they will not volunteer to open up the review process.

09/03/2016 09:26

I mirrored this question in the Horizon2020 Group on LinkedIn: https://www.linkedin.com/groups/3731775
(question at at https://www.linkedin.com/groups/3731775/3731775-6108989479165456385?trk=... if the link works)

here's an anonymised summary of the comments there:

This is very good step in the right direction. But be prepared to accept the results and consequences. Even if there are projects that effectively deliver, overall the ROI is low. There are many reasons, political as well as ideological, but mainly because the methodological approach is not the one to really achieve innovation and economic growth. Innovation is done by daring individuals, not by consortia with 25 partners, not to speak of the delays. Most people have the same opinion on these matters, but as the system is as it is for years to come, to escape means standing on the side-lines. Economic competitiveness cannot be planned. It can only be fostered. The environment in which it has to happen is the dominant factor.
…
Thank you, XXX. I agree. We're trying to think of new ways of stimulating open innovation beyond the "procure a specific result" model we've been using for most FPs so far, and which does support the technological players but isn't responsive or agile enough...
…
XXX, your comment hits the center. It's all about getting the people on the side-line involved.
…
Good idea. Note that metrics like the ones provided by Google Scholar are already a sort of open public review. A rapid check of those metrics would tell a lot about the lack of scientific impact of many FPs (at least in the area I am familiar with, wireless communications). The current 'procure a specific result' approach supports well evolution of known ideas. What is missing is a mechanism that leverages on open innovation to explore new disruptive ideas. XXX, I see your point on the importance of individual contributions (the Silicon Valley model). However, the creation of Internet is an example of disruptive innovation where the role of groups of individuals have been essential.
…
My personal impression on this matter is that, although the guidelines on specific calls make references to words like innovation, transparency, open science and so on, decisions on publicly-funded research are still very much based on traditional parameters of impact indexes, citations and intuitional reputation. It is almost impossible for young researchers to get funding for pioneering ideas if those concepts haven’t yet been noticeably explored in order to be considered as “feasible”… what’s innovative about that? European commission needs to stop focusing on academic track records and make some room for those willing to embrace the risky task of creating new solutions for our social and technical problems instead of pursuing academic reputation and secure their faculty positions by following the traditional academic establishment policies.
…
Glad to see that many have a common view. Nevertheless, looking at the small number of comments the "impact" of the question asked seems low. I was expecting a flurry of comments. I think it is time for Europe to do some deep soul searching. The current approach is clearly not working well enough. A fundamental issue regarding innovation and economic growth in the EU is that the private sector (industry as well as investment community) seems to be withdrawing, e.g. from taking disruptive innovation risks. Risk reduction should not be risk avoidance but managed risk. The latter requires swiftness (the world doesn't wait). In the end, it is very much a cultural issue. The US seems to be better at it, but also many asian countries, even when centrally governed and technocratic. China for example works with 60 year plans. The result however is a very dynamic and boiling economy (yes, not always perfect). What can we learn from others? How can we transform our old world?
…
Good idea. The after the evaluation score adjustment to jolt the preferred winner should become public too.
…
I believe it would be an enormous enrichment of the H2020 selection process to use the general public (crowd) for ranking and preselecting the project ideas.
I already consulted several regions all over Europe to follow this approach. However, for that it would be necessary to present the project idea in a short and immediately understandable way.
For the regional funding schemes I consulted we solved that in following way.
The first step was not a long written application, but a well-structured pitching video. All the pitches were published on the same day and the crowd could vote for the best ideas/products.
The best ten were invited to an expert selection process.

I would love and recommend a similar approach for a big number of the H2020 calls.
…
I am not a researcher but an energy expert and working with H2020 projects mainly under "Secure, clean and efficient energy". However, to apply science and support first deployment of innovative solutions in the market is key for me. From my perspective, some answers.

I agree with much of that what already has been said. To make use of the "crowd" will give a broader perspective and support finding the really innovative ideas.

However, it might be that this process will exclude parts of the research community which for one reason or the other will not be part of an open review.

Another problem might be to ensure that the results of the open review are really the base for further decision making. Is there anything you can tell us about that bit?

In general, my impression is that one key challenge is to make use of research results and transfer them into effective instruments, products and services. Is there any possibility to include that in a new review process?
…
Interesting idea to improve the quality of the reviewing publicly funded project. You take as example the peer review process of publications which are anyhow meant to disclose information to the public at large. However, progress reports of publicly funded project contain (industrial) confidential information which are now protected by an NDA signed by the peer reviewer. To my opinion, industrial partners will be less eager to enter into H2020 collaborative projects once the progress reports will be distributed to unknown peer reviewers running the risk that company confidential information will be disclosed in an uncontrolled manner. Though I support the idea of peer review of publicly funded projects, it should not prohibit industrial partners to enter into collaborative projects as they ultimately generate economic growth and jobs badly needed in the European Union.
…
I find your idea no bad, but in practice no really working. 10 people and 100 meanings. How you will manage this 100 meanings of 10 people. You can realize this on an example. Write a paper about a topic with 10 important persons, and I will tell you, that you get many different meanings, about the importance, goals of your paper and how you will manage this?
…
I understand your very good approach, but we don't have in all topics of research a really comparable system of standardization and indicators for compness.
Therefore, I am not so excited about that
…
Horizon 2020 core aim is business orientated, taking innovation to market. However two thirds of evaluators are from the public sector. We need two thirds of evaluators from real businesses with hard won experience of what is involved in taking innovation to market.

MATILDE Machado,

25/02/2016 19:28

Hi, I am not familiar with the concept but from what I read it has to do with open up to the academic (or wider) communitty the refereeing of a proposal. I understand the transparency upside but there are obvious disadvantages as well. The proposal is supposed to contain ideas on the rough that are yet to be developed. When applying for a grant one implicitly trusts the funding entity to have chosen experts on the field who presumably enjoy certain professional ethics and reputation. When open up to the public anyone would have free access to those ideas I presume.
Matilde Machado

26/02/2016 09:21

Thank you, Matilde. In fact we were thinking at this stage more about open reviews of running projects than for open proposal evaluation, for the reasons you mention. And we'd expect it to be of work that's more open by nature, and for which the project participants agree to participate and see the benefit of it being opened up.
It's more really like the concept of "Public-Friendly Open Science" as described here:
http://www.science20.com/matteo_cantiello/publicfriendly_open_science-16...

For example, we could rethink how we require the projects to report:

1 to the EC, administratively;
2 to their peers, for more open innovation, and for reuse of results, code etc;
3 to the public, who have a right to know how public funds are spent.

if we design our requirements and procedures properly we could achieve this whilst saving time, money and effort. If not, we could end up just increasing the burden and frustration...

Hope this clarifies. Thanks for your comment.

Muki Haklay,

20/02/2016 12:55

I also join the comment below that the idea is good, as expertise and comments are not confined only to the people that are in a consortium or in the 3 reviewers which are selected to assess the project but exist elsewhere. It is especially valuable at early stages of a project, when there is a scope for adjusting and changing aspects of the project to address such comments.

However, there are significant problems:

First, deliverables are contractual and failure in them mean a direct loss of money to the consortium members (been there, and seen it!) and therefore highly negative comments which can stem from differences in worldview or disciplinary traditions can have an implication on the review outcome. The risk is too high to make these open without instructing reviewers to read the contractual documentation and the criteria that reviewers will use, and having a significant commitment of time from such a reviewer... and that's challenging.

Secondly, as Fermin below noted, such activity adds an additional burden - promoting the fact that the document is open to comments, addressing and responding to suggestions and revising the deliverable or output at the end. Sometimes, in order to get constructive feedback, you need to target people individually and solicit their comments and then add them to document. This was my experience when we developed a concept note for participatory mapping infrastructure as part of an FP7 project. It requires extra resources or delay the time in which a task can be accomplished.

Thirdly, the effort to carry out consultation requires extra time - for example, if an outputs for month 18 of a project is to go through such a review, it need to be ready by month 16, so it is updated in month 17, and finalised as a deliverable for month 18. This can disrupt the timelines and add burden on the project's team.

Thinking about the range of deliverables and outputs, many of them are unsuitable for open review because of the contractual constraints. I've now seen several times reviewers using two modes of thinking - one in which they compare the output to the project's contractual description and another one in the evening dinner they see the value of them generally and complain about the need to follow the criteria and documentation that was written 4 years ago strictly! But this is not true for all outputs, so in each project it is possible to identify several suitable outputs that can be open for comments.

Once certain outputs and deliverables that are suitable for such a review have been identified, they can be shared, as suggested. I don't think that ORCID is a good option - as this open the discussion only to other scholars. Putting information on an editable (but traceable) common editing system - e.g. a Google Docs links with commenting option - can be a start. There is a need - like here - for people to give comments and to pay attention to the information and suggest changes. There should be additional information about who is the output aimed at, what it is for etc. I do agree that anonymous comments should not be allowed, and in an open review the reviewer should say from what point of view they are commenting and what knowledge they have to comment.

Regarding 'A variant on this, regardless of open reviews, would be to encourage projects to crowdsource for
input and comments by posting material (reports and deliverables etc) online. This would also be
more transparent and serve as a source for innovation and collaboration' - I thought that this is already the case with many reports that are public? If not, of course that should be encouraged like this https://www.google.co.uk/search?q=everyaware%20deliverables - but you don't see much citations or serious use of these outcomes, or comments about them (which is a different issue). Even if you run a blog, or a book, there will be a huge majority of readers that you know nothing about, and very few comments...

In summary, there is a scope for such a process, and there is a need to consider what will be the incentives and mechanisms to engage people in commenting and reviewing, and how it is integrated into a project timeline and budget.

I'll add that personally I do keep my research outputs open - but also that I get very little feedback about it (e.g. discussions on my blog are infrequent)

24/02/2016 11:53

Thank you, Muki, for your very useful comments. There's certainly quite a lot for us to consider there.

Since posting the question it's been suggested to me (offline, unfortunately) that one way to proceed could be to separate the two functions of the review:
(i) assessment of project results as being on track, relevant, and useful in themselves - this would be of more general interest;
(ii) the more contractual, administrative obligation of checking that the project has actually properly done the work for which they're claiming costs.

The simplest way would be to just open up (i) to general comments.

This would require a rethink of how we oblige projects to report their results, as of course and as you point out, another restraint on us is that we should try to save time and money, and not increase the overall work, unless the increase in openness and transparency makes it worthwhile.

I hope to get other remarks on this...

Fermin Serrano Sanz,

19/02/2016 22:52

I think it is a good idea to open it even for the general public. The proposed pilot shouldn't be too hard to implement and to monitor. It should ensure proposers to consider extra budget and extra time within the work plan to provide such deliverables in appropriate ways. But the fact of saying publicly "this amount of € has been spent by this consortium on these tasks to achieve these outcomes because of this reason" is very nice.

A couple of personal experiences: When I heard in a project review the feedback "the impact is disproportionately large" I thought "what are you used to?" :) I have heard stakeholders (policy officials, researchers, people...) saying they have the impression that EC-funded H2020 are not having expected impacts so any effort trying to find potential solutions is pertinent.

Thanks for asking too.

I believe that public engagement is important in all the steps of the scientific work. Here it comes everything about citizen science, crowdfunding, crowdsourcing... and this kind of crowd-evaluating process you suggest. The mere fact of being publicly evaluated would encourage us all to work differently, with a different mindset from the early stages of each project.

some pros: coherence and responsibility; remove barriers; transparency-awareness for public budget; innovative open knowledge-governance from the EC which could promote similar practices afterwards; ...

some cons: populism- marketing campaigns; some disciplines are more accesible than others; language issues; it already takes a lot of effort to publish deliverables in a good way and it would be enormous to ensure clear-holistic understanding for the audience of the contract between EC-consortium; how to attract enough volunteer-evaluators (recognition, incentives...); how to balance the do-that-the-contract-says way of running a project versus being flexible,agile and responsive to challenges/opportunities...

these issues can be addressed. I guess you wonder how could they be covered by EC: better EC reference documents, guidance for proposers, central services to support the adoption of the pilot...

Fermin

Digital4Science

ARCHIVED PLATFORM

This website is archived since mid-May 2021 and it is not possible to login anymore.

But don't worry there is a new Futurium platform and all active communities have been migrated.

Discussions

Open Public Reviews – an open consultation

Comments

ARCHIVED PLATFORM

This website is archived since mid-May 2021 and it is not possible to login anymore.

But don't worry there is a new Futurium platform and all active communities have been migrated.

Share this:

Comments