enEnglish
CROS

2011 11 17-18 ICT PIloting Kick-off meeting, Copenhague

Kick –off meeting in Copenhagen – Minutes

 

 

Date and venue

17-18. November 2011, Statistics Denmark

 

Agenda

1.               Welcome and presentation of meeting structure (STATDK)

2.               Eurostat – expectations , purpose, etc. (Eurostat and DG INFSO)

3.               Project content (STATDK)

4.               Project management (STATDK)

5.               Testing of ICT HH 2010 - Presentation re. experiences (Germany)

6.               WP1 + WP2 (Testing of existing questions) – Discussion

7.               WP3 + WP4 (Designing and testing of new questions) – Discussion

8.               Conclusions and summing up (STATDK)

 

Minutes including action points

               

  1. Welcome and presentation of meeting structure (STATDK)

Project leader Peter Bøegh Nielsen opened the meeting by welcoming the group. The project leader underlined the importance of the project. Rethinking and redesigning questions is a major issue in the framework of ESS projects.  The increasing need to consider positive and negative priorities in the European Statistical System is reflected in the purpose of the current project.

 

Both the project itself and the kick-off meeting has an ambitious agenda. Peter Bøegh Nielsen identified subjects related to WP1 and WP2 as well as decisions relevant for all work packages  to be main priorities during the kick-off meeting.

 

 

  1. Eurostat – expectations , purpose, etc. (Eurostat)

The project’s purpose and importance – seen from Eurostat’s point of view -  was presented by Konstantinos Giannakouris. 

 

The project’s overall objectives are to satisfy user requirements, to monitor the rapid developments in ICT usage and to enforce questionnaire consistency and harmonization of Community statistics re. ICT usage. More specifically, the project is meant to facilitate the process of annual development of model questionnaires and produce a toll-box of thoroughly tested questions and definitions. Project outcomes will also serve as input to the upcoming revision of the Digital Agenda.

 

The Evaluation Committee expressed a number of requirements re. the projects:

  • Clear description of involvement of each co-partner                     
  • Sufficient number of test and well-documented results
  • Inclusion of latest changes and developments
  • Testing as close as possible to the model questionnaire
  • Consideration of dependencies and internal consistency
  • Consideration of possible future developments
  • Inclusion of possible corrections and alternatives
  • Availability of draft results for at least one subject under WP3 and WP4 by September 2012

 

Konstantinos Giannakouris  informed the group about Eurostat’s priorities re. the various subjects in the four work packages. For more details see  Eurostat’s presentation in Annex 1.

 

The subject ‘e-skills’ has the highest priority among the subjects covered in WP3 and WP4 and draft results for a new module on e-skills will have to be produced latest by September 2012. Results for all subjects covered by WP1 and WP2 will have to be available by July 2012.

 

Stefano Abruzzini from DG INFSO added that the Commission is paying great attention to the project. Piloting is not an easy exercise and there are several risk elements in pre-testing. It is therefore important to learn from earlier experiencies.

 

DG INFSO outlined policy priorities from the coming years and referred to the upcoming revision of the Digital Agenda. Policy priorities are not yet finalized but the list of focus areas will certainly include e-commerce, e-skills and economic and social impact of ICT’s.

 

The project website will be established on ESSNet portal.  Eurostat  and the Commission shall be consulted re. major decisions and project results.

 

The following persons shall receive copies of communication re. the project’s work and progress:

 

Albrecht Wirthmann - Head of project - Central contact point for WP1,2,3,4, and especially for WP5

Heidi Seybert (HH/IND)  shall be informed about communication re. WP1 and WP3

Konstantinos Giannakouris (ENT)  shall be informed about communication re. WP2 and WP4

 

Stefano Abruzzini  is main contact to DG INFSO users and experts.

 

 

Action 1

Eurostat to forward relevant Commission communications re. future policy priorities

Ongoing

Action 2

DG INFSO to forward final report from previous pre-testing excercise

Done

Action 3

Eurostat should provide further information to STATDK concerning the ESSNet website to be used by the project

January 2012

 

 

  1. Project content (STATDK)

Agnes Tassy presented a short summary of the project’s structure and content as described in Annex I and Annex II  to the grant agreement  (Technical specifications and Application). For more details please refer to Annex 2 – STATDK Presentation project content.

 

Work under the project is organized in five work packages:

 

WP1: Test of existing questions – HH survey - Sweden, Holland, Denmark

WP2: Test of existing questions ENT survey - Slovenia, Holland, Sweden, Finland

WP3. Development and test of new questions – HH survey - Denmark, Poland, Finland

WP4: Development and test of new questions – ENT survey - Denmark, Poland, Finland

WP5: Overall project management and coordination incl. preparation and transmission of project deliverables - Denmark

 

 

 

 

 

  1. Project management (STATDK)

Peter Bøegh Nielsen described roles and responsibilities connected to the participation in the project.

 

The coordinator will be the first and main contact with Eurostat. STATDK will prepare quarterly monitoring reports, and other relevant documents in order to secure that deadlines are kept and deliverables are delivered in time. STATDK will also produce the final report on the results and deliverables of the project.

 

The activities to be carried out by STATDK are:

•                 Management of the project including the follow up of budgetary matters.

•                 Reporting to Eurostat in regular intervals (every quarter) on the state of the project, including resources used by each participating partner

•                 Elaboration of draft reports from WP3 and WP4 in time to be available for input to Task Force and WG meetings.

•                 Coordination of the different actions of the project and communication within the consortium and with stakeholders, especially Eurostat.

•                 Design, production and distribution of standard reporting templates for all participants

•                 Verification and quality assurance of deliverables

•                 Elaboration of final reports from WP’s and for the overall project.

 

Each WP will have at least 3 partners and all tests will be carried out in at least three languages. One participating country will act as Lead Country  in each of the four WP’s WP1-WP4.

 

Lead Countries’ activities include:

•                 Development and drafting of the documents on the statistical concepts including consultation of users and requesting input from the other partners of the project.

•                 Development of questions for the respective modules in coordination with the partners and Eurostat, organising and conducting the expert review and drafting the questionnaires.

•                 Organising meetings for their WP.

•                 Conducting and coordinating cognitive testing according to an agreed and common procedure.

•                 Compiling the results of the testing and revising the respective questionnaires.

•                 Updating the conceptual document and drafting the notes for the Methodological Manual.

 

All  partners  carry out the following activities:

•                 Contribution to the ongoing work.

•                 Translation of the questionnaires into national languages

•                 Organising and conducting the testing according to an agreed and common procedure.

•                 Documentation of test results incl. suggestion for improved questions by using the project’s standard reporting template

•                 Providing results of the testing to the leading NSIs for the specific subjects.

•                 Providing administrative information related to the execution of the project to the project coordinator.

 

The project leader has informed the group about the fact that the project’s official starting date is the 17th November 2011.

 

 

Action 4

STATDK to forward electronic copies of the duly signed grant agreement to project group members.

22. November 2011

 

 

  1. Testing of ICT HH 2010 - Presentation re. experiences (Germany)

Karen Blanke gave an interesting presentation concerning pre-testing of the 2010 model questionnaire for households and individuals. Her presentation  described test methodology, main findings and challenges. For more details please see Annex 3 – DESTATIS presentation.

 

  1. WP1 + WP2 (Testing of existing questions) – Discussion

With the presentation of the German pre-testing project the project group started to discuss questions related to performing pre-tests under WP1 and WP2.

 

Hanne-Pernille Stax introduced the discussion by presenting  main  issues relating to testing of questions. See annex 4 – presentation _ test if existing ICT questions for more details.

 

As the first step existing data sets (2010, 2011 and the e-commerce module from ICT HH model questionnaire 2009) shall be analysed. Even though all questions shall be tested special attention should be payed to ‘problematic’ questions and definitions identified by this analysis.

In order to facilitate harmonised methodology and documentation of  data analysis and expert review to identify problematic question the group decided to prepare a template for reporting data analysis results.

During  the meeting’s second day Sweden and Slovenia (lead countries for WP1 and WP2) presented a draft template for reporting analysis results. This template was discussed by the project group. Based on input from the project group a refined version of the template will be prepared by Sweden and Slovenia.

 

 

Action 5

Sweden and Slovenia circulates revised template for reporting data analysis and expert review results to project group members.

25. November 2011

Action 6

Members of WP1 and WP2 perform analysis of existing data sets and expert reviews before the Christmas holidays

  1. January 2012

 

Benchmarking  indicators shall also be treated with special attention during pre-testing. On the 2012 ICT ENT model questionnaire benchmarking indicators are marked with an asterix (*). This marking is not applied on ICT HH model questionnaire.

 

 

 

Action 7

Eurostat is asked to provide the group with a list of existing questions to be tested  from the 2009, 2010, 2011 and 2012 model questionnaires, where benchmarking questions are indicated. The list should be a word document making it possible to copy text.

As soon as possible

Action 8

STATDK to request input from all WG members re. problematic questions and issues related to ICT HH and ICT ENT model questionnaires.

  1. December 2011

Action 9

Eurostat will send feedback from the Quality Reports of both surveys to STATDK in order to identify problematic questions

As soon as possible

 

The project group exchanged views on possible harmonization of pre-test method e.g. by paper questionnaire, CATI interview or web-based questionnaire. Test-mode has an influence on test-outcome. A mix of different modes would be thus most preferred. On the other hand it is not feasible to harmonise test mode as ICT usage statistics are based on an output regulation. Countries may choose their preferred method of data collection.  Participating countries expect to use the same data collection method for pre-testing as they apply for their large scale data collection.

 

An overview of existing large scale data collection methods shows that pre-test are expected to be carried out in a mix of methods that cover most alternatives:

 

Project member

HH tests

ENT tests

Sweden

CATI

Web

Holland

CATI

Web

Slovenia

CATI + CAPI mix

Paper

Finland

CATI

Web

Poland

Face-to-face

Paper

Denmark

CATI + web mix

Web

 

The group continued the discussion by reflecting on the issue of number of tests.

 

Aarno Airaksinen informed the group about Finland’s experiences on this question. In Finland normally 6 cognitive interviews are performed in the Question Laboratory. All interviews are carefully documented by video recording. Experience confirms that raising the number of interviews higher than 6 will not add considerable extra information and thus value to test results and all possible problems will be identified during the first 6 tests. This observation was confirmed by other group members.

 

When designing interviews it is important to ensure that the selected test respondents represent different groups e.g. both service sector and manufacturing, both young and old citizens.

 

Based on the discussions the group concluded that minimum 6 test pr. country will have to be conducted when testing the questions. Most countries expect to perform more tests than the minimum amount:

 

 

 

WP1

WP2

WP3

WP4

Sweden

6

6

n/a

n/a

Holland

10

10

n/a

n/a

Denmark

10

n/a

10

6

Slovenia

n/a

10

n/a

n/a

Finland

n/a

6

15-20

6

Poland

n/a

n/a

10

10

 

 

 

 

 

The discussion continued with the question of how to document tests. The following issues are considered important when documenting cognitive tests:

 

1. Wording, comprehension, terms

2. Access to relevant information

3. Processing of information

4. Response options and format

5. Context of questions

6. Anticipated issues (prepared probes)

7. Suggestions.

 

For the current project it is essential to record suggestions for alternative questions or  revised formulations. The end product of the test should be a new module questionnaire and new modules on the selected issues.

 

To facilitate standardized documentation of test results STATDK has prepared draft reporting templates. See annex 5 Template for test results – micro level and annex 6 Template for test results – macro level. There reporting templates will be further discussed during an upcoming meeting in Stockholm in January 2012.

 

 

Action 10

Project members to forward comments and suggestions re. annex 5 and annex 6 Template for test results micro and macro level.

  1. December 2011

Action 11

STATDK to prepare final versions of test results reporting templates based on input received from other project members.

  1. January 2012

Action 12

Test in WP1 and WP2 to be conducted latest in March 2012

  1. April 2012

 



 

 

  1. WP3 + WP4 (Designing and testing of new questions) – Discussion

The above mentioned reporting templates will be used to document test results in WP3 and WP4 (development and test of ‘new’ questions covering the subjects e-skills, green ICT and cloud computing.)

 

Work in WP3 and WP4 will have to start with conceptualization of the selected subjects including hearing of important users and other interested parties. All participating members will conduct hearing in their respective countries. Hearing of international users, etc. should be coordinated, e.g. only one WP member or Eurostat performs international hearing for a given subject. The hearings will only cover the three subjects in WP3 and WP4 and shall not include other ICT-related issues.

 

 

Action 13

Eurostat provides members of WP3 and WP4 with a list of international fora /organisations to be included in the hearing.

End of January 2012

Action 14

Eurostat will contact and establish a link with OECD for the development of new questions/modules

 

Done

 

The group discussed timing issues related to WP3 and WP4. According to the project application, work for WP3 and WP4 would start in the spring of 2012. According to the Evaluations Committees request, draft results for e-skills will have to be available latest in September 2012. This means that tests will have to be conducted during the summer 2012.

 

The group has therefore concluded that it would be wise to push the starting date forward, e.g. to January 2012.

 

 

Action 15

The lead country for WP3 and WP4 prepares updated time-planning for these two WP’s.

End of January 2012

 

Based on discussions the project’s meeting schedule was revised.  The group found it useful to combine the two first meetings for WP1 and WP2. Similarly, first meetings for WP3 and WP4 could be combined as well. This can be possible by switching meeting venue for WP4 meeting 1 and 2, so the first meeting can be held in Copenhagen together with meeting 1 for WP3.

 

The revised draft meeting schedule will be further refined during the first months of 2012.

 

 

 

 

 

 

 

Meetings planned (revised)

 

 

WP5 – Kick off

November 2011

Copenhagen (DK)

WP1 – meeting 1

Mid January 2012

Stockholm (SE) (back-to-back with WP2 meeting 1)

WP1 – meeting 2

2012

Heerlen or Den Haag (NL)

WP2 – meeting 1

Mid January 2012

Stockholm (SE) (back-to-back with WP1 meeting 1)

WP2 – meeting 2

2012

Ljubljana (SI)

WP3 – meeting 1

Late March 2012

Copenhagen (DK) (back-to-back with WP4 meeting 1 ?)

WP3 meeting 2

2012

Stettin (PL)

WP4 – meeting 1

Late March- Early April 2012

Copenhagen ?(DK) (back-to-back with WP3 meeting 1 ?)

WP4 – meeting 2

2012

Helsinki  ?(FI)

WP5 – Round up

February- March 2013

Copenhagen (DK)

 

 

  1. Conclusions and summing up (STATDK)

Casper Larsen drew the group’s attention to practical matters re. the recording of costs, etc. he informed the group  about pre-financing. Details can be found in the grant agreement.

Peter Bøegh Nielsen closed the meeting and thanked group members  for their active and  constructive participation.



 

 

Summary of actions

 

 

Id. No.

Description

Deadline

Responsible

Action 1

Eurostat to forward relevant Commission communications re. future policy priorities

Ongoing

Eurostat

Action 2

DG INFSO to forward final report from previous pre-testing excercise

Done

DG INFSO

Action 3

Eurostat should provide further information to STATDK concerning the ESSNet website

January 2012

Eurostat

Action 4

STATDK to forward electronic copies of the duly signed grant agreement

22. November 2011

Denmark

Action 5

Sweden and Slovenia circulates revised template for reporting data analysis and expert review results to project group members.

25. November 2011

Sweden and Slovenia

Action 6

Members of WP1 and WP2 perform analysis of existing data sets and expert reviews before the Christmas holidays

1. January 2012

Sweden, Slovenia, Holland, Denmark, Finland

Action 7

Eurostat is asked to provide the group with a list of existing questions to be tested  from the 2009, 2010, 2011 and 2012 model questionnaires, where benchmarking questions are indicated. The list should be a word document.

As soon as possible

Eurostat

Action 8

STATDK to request input from all WG members re. problematic questions and issues related to ICT HH and ICT ENT model questionnaires.

December 2011

Denmark

Action 9

Eurostat will send feedback from the Quality Reports of both surveys to STATDK in order to identify problematic questions

As soon as possible

Eurostat

Action 10

Project members to forward comments and suggestions re. annex 5 and annex 6 Template for test results micro and macro level.

1. December 2011

Sweden, Slovenia, Holland, Denmark, Finland, Poland

Action 11

STATDK to prepare final versions of test results reporting templates based on input received from other project members.

28. January 2012

Denmark

Action 12

Test in WP1 and WP2 to be conducted latest in March 2012

1. April 2012

Sweden, Slovenia, Holland, Denmark, Finland

Action 13

Eurostat provides members of WP3 and WP4 with a list of international fora /organisations to be included in the hearing.

End of January 2012

Eurostat

Action 14

Eurostat will contact and establish a link with OECD for the development of new questions/modules

 

Done

Eurostat

Action 15

The lead country for WP3 and WP4 prepares updated time-planning for these two WP’s.

End of January 2012

Denmark

 

List of participants

 

STATFI

Rauli Kohvakka

STATFI

Aarno Airaksinen

STATFI

Perttu Melkas

STATPOL

Justyna Berezowska

CBS

Rachel Vis - Visschers

CBS

Ger Linden

STATSLO

Gregor Zupan

STATSLO

Eva Belak

SCB

Jennica Wallenborg

SCB

Ingrid Persson

DESTATIS

Karen Blanke

Eurostat

Konstantinos Giannakouris

DG INFSO

Stefano Abruzzini

STATDK

Hanne-Pernille Stax

STATDK

Martin Lundø

STATDK

Peter Bøegh Nielsen

STATDK

Casper Larsen

STATDK

Agnes Tassy