Raw data management and backups
The most important rule
While you are strongly advised to read all this page, in case you do not please remember this fundamental rule
If the data is important to you, have a backup (or more!) that is under your control
Raw data management
A few suggestions on how to do raw data management:
- Make sure metadata is always attached to the data (e.g. a Excel Spreadsheet or a text file). Make really sure you have all the metadata you need
- If the data is important to you, make sure you have a copy under your control. Do not trust only exiting backup mechanisms
- We do not offer backups of carnation (core server) data, so do not expect data to be safe on the core server
If you are on Zac's group make sure your data is put on /data/raw of haldane
If you are on Jeff's group make sure your data is put on /data/raw of musculus. Data put on other servers (lepus and phodopus) will be overwritten
If you are on John's group make sure your data is put on /data/raw of xebec. Data put on other servers (xylemsap and xu) will be overwritten
Final Data management
Backups
Carnation is not backed up. If you have important data there, make sure you have your own backups
Zac's server is backed up offsite. The same is valid for John and Jeff's servers, but these have an extra procedure:
There is a main server (musculus for Jeff and xebec for John), lets call the other servers "satellites".
- The raw data (/data/raw) is mirrored from the main server to the satellites. If you need to put new raw data, use the main server. Any stuff put on /data of the satellite servers will probaby be overwritten
- Home directories are cross-backed up among machines. For example you can find the backup of xylemsap /home on /data/backups/xylemsap/homes of xu
- Some older machines do not have enough disk space to participate on the cross-backups. Namely xylemsap and phodopus
What is backed up offsite:
- /homes
- /data/raw
- /data/final
- and eventually scratches, but this is offered on a best attempt basis
Auditing
If you are on Jeff and John's servers you are responsible for auditing the backup procedure for the data and homes that concerns you. This is what you have to do:
- Go to all the servers and make sure you home directory is properly backed up. Check /data/backups/SERVER/homes/YOUR_USER for that
- Make sure /data/raw on all machines has your data
A procedure to audit off-site backups will be offered soon