Raw data management and backups

The most important rule

While you are strongly advised to read all this page, in case you do not please remember this fundamental rule

If the data is important to you, have a backup (or more!) that is under your control

Raw data management

A few suggestions on how to do raw data management:

  1. Make sure metadata is always attached to the data (e.g. a Excel Spreadsheet or a text file). Make really sure you have all the metadata you need
  2. If the data is important to you, make sure you have a copy under your control. Do not trust only exiting backup mechanisms
  3. We do not offer backups of carnation (core server) data, so do not expect data to be safe on the core server

If you are on Zac's group make sure your data is put on /data/raw of haldane

If you are on Jeff's group make sure your data is put on /data/raw of musculus. Data put on other servers (lepus and phodopus) will be overwritten

If you are on John's group make sure your data is put on /data/raw of xebec. Data put on other servers (xylemsap and xu) will be overwritten

Final Data management

TBD. Ask Tiago if you have doubts

Backups

Carnation is not backed up. If you have important data there, make sure you have your own backups

Zac's server is backed up offsite. The same is valid for John and Jeff's servers, but these have an extra procedure:

There is a main server (musculus for Jeff and xebec for John), lets call the other servers "satellites".

  1. The raw data (/data/raw) is mirrored from the main server to the satellites. If you need to put new raw data, use the main server. Any stuff put on /data of the satellite servers will probaby be overwritten
  2. Home directories are cross-backed up among machines. For example you can find the backup of xylemsap /home on /data/backups/xylemsap/homes of xu
  3. Some older machines do not have enough disk space to participate on the cross-backups. Namely xylemsap and phodopus

What is backed up offsite:

  • /homes
  • /data/raw
  • /data/final
  • and eventually scratches, but this is offered on a best attempt basis

Auditing

If you are on Jeff and John's servers you are responsible for auditing the backup procedure for the data and homes that concerns you. This is what you have to do:

  1. Go to all the servers and make sure you home directory is properly backed up. Check /data/backups/SERVER/homes/YOUR_USER for that
  2. Make sure /data/raw on all machines has your data

A procedure to audit off-site backups will be offered soon