Points of View:
Effective proposals: an evaluator's perspective
Funding for technological development and research is a key area of the EU's Research Framework Programme (FP7). The evaluation process that ensures SMEs receive the right funding is a challenging task. Dr Peter McGeehin, a leading technology evaluator, looks at the process and recommends what to include in a proposal.
Proposals submitted to SME Research cover a wide range of technologies and markets. It is therefore useful for evaluators to have broad interests in as many areas of science and technology as possible. Knowledge of management, economics and market trends is also useful as, typically, the evaluation process is organised into panels that take both a technological and economic standpoint when evaluating.
Potential evaluators nominate themselves by joining an EU-wide database of experts, which indicates professional experience and relevant technical interests. When selecting a particular evaluation, a balance of experience and knowledge is sought amongst academics, industrialists and consultants. Before an evaluation commences, evaluators are sent briefing material. If required, further background information is provided at meetings held in Brussels.
The written proposal document is evaluated against a fixed set of criteria. Each evaluator individually assigns scores to the various criteria, awarding '5' for the best and '0' for the worst.
Dr McGeehin said "there is direct correlation between the guidance given to those drafting proposals and the marking sheet that is used to score them. A proposal that addresses the required points explicitly, in the right order and clearly, in terms of structure, will inevitably be more straightforward to evaluate." Guidance on structure and maximum page counts is also included. It is wise to follow this guidance as a demanding reviewer might refuse to read beyond the allowance.
He said that "it can sometimes be quite intellectually challenging to come to a sound judgement, particularly in the case of a weakly structured document or one that does not address the criteria clearly. It is a pleasure to assess a really good proposal - but much more difficult to score proposals that are closer to the average."
Usually, the first stage of the evaluation process is carried out remotely at an individual reviewer's workplace. Typically in SME proposals, three individual readings are involved. Scores from each evaluator are collated to determine the proposal's final consensus score. This is either done remotely or in Brussels where evaluators attend consensus meetings.
"Meeting colleagues is good from a human perspective and enables evaluators to calibrate their individual judgements against those of colleagues" said Dr McGeehin. "Some of these meetings proceed easily, even with quite diverse scores. But every so often there are genuine differences and the debate can become quite heated. In such cases the independent chairperson (an EU member of staff) has to delicately steer the discussion to a conclusion. The temptation to settle with simply an average score must be avoided."
Projects are quite intense and generally last for two years. Dr McGeehin cautioned that a common fault with SME proposals is unrealistic over-ambition, either in respect to the technical idea or the timescale involved. "It can be refreshing to read a proposal that is led by SMEs with the RTD performers being subservient to the company's own market needs."
Evaluators sometimes get the chance to monitor the progress of projects, at the 12 month landmark and at their conclusion. Dr McGeehin said "it is interesting then to compare what has been planned with that which is being achieved".
"Evaluating proposals can be genuinely interesting and stimulating from an intellectual perspective, and quite rewarding in terms of the interesting colleagues that one gets a chance to meet from the different countries and cultures of Europe."
While reviewers receive a fee for the evaluation work they do, the real driving force is to serve the EU's community of SMEs and support the development of collaborative technological innovation.
Dr McGeehin concluded with advice for those drafting proposals: "Managing a consortium and addressing a problem in the form of a proposal is a challenging adventure. The key emphasis in any proposal has to be on the word 'convincing'. Each and every paragraph should be there to convince the evaluator of what you believe to be right, and why you should receive funding."
Dr Peter McGeehin
Technology Development Consultant
Tel. +44 560 255 7085