Navigation path

Staging Manager

The Staging Manager tool http://doteu.staging.cc.cec.eu.int:8888/ offers DGs all the functions they need to manage static sites on EUROPA:

  • FTP: transfer/delete their sites' files/directories
  • checking
  • preparation
  • release of updates
  • monitoring of update transfer to the production environment
  • synchronisation of staging and production sites. 

It is intended for all EUROPA site managers, including external contractors (contractors should see External accessRestricted area: This link points to internal pages and may not work if you are browsing as an external user..) 

 

Description

Built by Datacentre, the programme uses only the HTTP protocol to connect to the server and works only with the standard Commission browser (MS Internet Explorer 7 and higher) - it does not work with other browsers.

The Staging Manager does not enable you to load more than one local file to the server staging without individual selection: this tool is therefore best used for small numbers of files. To transfer larger numbers of files, Staging Manager enables you to use ZIP files, with automatic unzipping.

Traditional FTP tools are still the suggested alternative for large-scale transfers of local files to the staging environment.

The Staging Manager also compares files and indexes between the staging and production environments, enabling each manager to synchronise and tidy up their contents.

Read and write authorisations are identical to the ftp access allocated to webmasters in their individual staging environment: depending on the specific rights they have, they can write to the file request.update or request.delete and launch a production run (mise en production).

Please follow the procedure Access for transfers to static sitesRestricted area: This link points to internal pages and may not work if you are browsing as an external user. for new access authorisations or changes to existing access authorisations.

Production runs are processed in parallel (up to 20 at a time), to speed up the process and limit queuing. 

Sites are put online in 3 steps:

  • new files are copied from the local or development site to the staging server;
  • new files are verified on the staging server;
  • once verified, the files are copied to the production server.

The transfer to the production server applies to the files/directories on the staging server that are indicated in the .request.update or .request.delete folders placed at the root of the site in order to be copied or deleted on the production server.

It is possible in an emergency to immediately copy files/directories to a production server by using the Staging Manager's save and execute function.

 
 

Procedure description

Four special files are being used by the staging to production transfer procedure (also know as MEP). For each 'subsite' these files have to be located in the root directory of that 'subsite' on the staging server.
All file names listed inside these special files are relative to the root directory of the subsite they apply to (metacharacters "* or ." are not allowed). This should be clarified with an example below.
The special files used by the MEP (Mise En Production) update procedure and at any time are likely to exist within each subsite document root directory of the staging server are the following:

  • .request.delete lists all files and/or directories that are to be deleted from the production server.
  • .request.update lists files and/or directories that have to be added or updated from the staging server to the production server.
    • Files are copied only if a newer version exists on the staging server (based on the last modification date).
    • Directories include recursively all subdirectories. All files found in the production directories that do not have a counterpart on the staging site any longer will be deleted from the production site!
    • "<ALL>" can be specified as the only entry in order to copy the whole hierarchy. Even if this is the easiest way to proceed, this must be avoided as much as possible, because it implies a much longer processing time (<ALL> really does mean ALL).
    • The files are first copied to a temporary environment where checks are made. Missing files or files containing errors (absolute links, etc.) might stop the procedure and none of the files will be copied to the production server at all.
  • .request.lock is created at the start of the automatic procedure meaning that no one should modify any of the files from the staging directory until this file has been removed by the update procedure.
  • .request.output contains the results of the procedure. All the files which have been deleted and copied are listed. Error and warning messages can be found.

Important remarks:

  • The '.request.update' and '.request.delete' files should be created only after all modifications on the staging site have been completed, because the presence of these files means that the listed data are ready to be copied.
  • The presence of '.request.lock' file means that the procedure has started and has not completed yet. No one should modify any of the transferred data on the staging server until this file has been removed by the procedure.
  • '.request.delete', '.request.update' and '.request.lock' are always deleted at the end of the procedure.
  • '.request.output' contains only the result of the latest execution of the procedure. This file is overwritten at each execution, so the preceding results will be lost. If you want to keep a history of your modifications, then you will need to save this file under another name. The contents of the .request.output can be sent to automatically to the relevant webmasters by email.
 
 

Scheduled or immediate production

Files and directories are automatically copied to the production server everyday between 13.00 and 22.00.
In certain circumstances, the webmaster can ask for this timeframe to be changed by emailing COMM EUROPA MANAGEMENT.

The '.request', etc. files must be ready and correct prior to the scheduled move to the production server!

It is possible in an emergency to immediately copy files/directories to a production server by using the Staging Manager's save and execute function.

Transfers to production servers use significant amounts of resources and reduce server performance. 

N.B. Because of the reverse proxy's caching mechanism, it may take up to 15 minutes before newly copied html pages are displayed, slowing down the operation! If necessary, you can force a refresh of the proxy with Firefox pressing ctrl + shift - reload.

As a reminder, the reverse proxy makes possible faster access to EUROPA by memorising all the static pages (with the exception of PDFs). The refresh algorithm works like this:

  • < 15 min. for html pages
  • = 8 hours for static files (images, css, js, etc.)
 
 

Responsibility for transfers to production servers

 Managers authorised to carry out transfers to a production server must agree to:

  • Check the pages on the staging server before they are copied to the production server. The IPG checklist details what needs to be checked.
  • Check that the files have been as soon as they see the production report and, if necessary, correct any errors so that the transfer can take place correctly.
  • Limit immediate transfers to emergency situations and for a limited number of files. If these limits are not respected, the Data Centre will inform the EUROPA team.
  • Properly synchronize the content of the staging and production sites using the Staging Manager tool.
 
 

File names, extensions and invalid characters

Some file names and extensions cannot be copied to the production server. In addition, certain characters may not be used, because they are reserved for the operating systems of the Data Center servers. The use of these characters would prevent the server management and maintenance programs from functioning correctly. See the list of directory/file names excluded from the transfer mechanism.

Every time a transfer is carried out, a '.request.output' file is created and placed at the root of the site. This report is emailed to recipients chosen by the DG's webmaster.

It is important to read each report to be sure that the operation was successfully completed. The last lines of the report indicate the status of the transfer:

  • "Errors": the transfer was a complete failure
  • "Warnings": check the details and make any necessary corrections
  • nnn bytes: size of the files transferred.

Warnings are typically displayed when certain files are not copied. It is important to verify that the problem files do not impact the overall coherency of the site.

Here are 4 examples:

  1. Correct transfer: no errors, no warnings and nnn octets copied.
    Total bytes copied: 541955
    no error
    no warning
  2. Correct transfer: a file was not copied; check whether it is in fact necessary on the production server.
    Warning File not copied: north_korea/nav/_vti_cnf/north_korea_nav.htm

    [...]
    Total bytes copied: 242601
    no error
    1 warning(s) during update
  3. Transfer failed: If even 1 error occurs, no file will be copied or deleted.
    Error: can not read news_corner/speech12_en.htm, file not found
    Too many errors (1). No file copied or deleted.
    1 error(s) during update
    no warning
  4. Transfer rejected because the date of the files on the staging server is older than those on the production server. No error detected but 0 octet transfered.
    Total bytes copied: 0
    no error
    no warning
 
 

Assistance

The Data Centre's EUROPA Webdesk team is available to provide assistance during transfers. Please submit a detailed description of the problem via the local helpdesk.