Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
/
Data Migration HowTo
Updated Jan. 14

Data Migration HowTo

You need to migrate your data before January 22.

 

We expect best performance for the data transfers on the genoa system. You need to migrate relevant data from your user, project, and share folders in the old WORK file system:

  • /old-scratch/usr/username

  • /old-scratch/projects/projectname

  • /old-scratch/share/sharename

  • (/old-scratch/tmp/username)

We recommend migrating your data by

  • running a rsync on genoa-login node

  • use tmux to keep your transfer running without being logged in to the system

  • example rsync call:

    ssh username@genoa-login.nhr.zib.de $ rsync -av /old-scratch/usr/username/folder_i_want_to_keep /scratch/usr/username/

 

Moving large directories with parallel rsyncs

For directories with a lot of data and many files in different subdirectories, the transfer might be sped up by running several rsync processes in parallel. For this we first need a list of files/folders up to a certain depth.

We can generate a directory and file list e.g. with find with the -maxdepth flag:

blogin9:~ $ find /old-scratch/usr/myusername/testfolder -maxdepth 1 /old-scratch/usr/myusername/testfolder /old-scratch/usr/myusername/testfolder/debian-10.2.0-amd64-netinst.iso /old-scratch/usr/myusername/testfolder/firmware-10.2.0-amd64-netinst.iso /old-scratch/usr/myusername/testfolder/folder1 blogin9:~ $ find /old-scratch/usr/myusername/testfolder -maxdepth 2 /old-scratch/usr/myusername/testfolder /old-scratch/usr/myusername/testfolder/debian-10.2.0-amd64-netinst.iso /old-scratch/usr/myusername/testfolder/firmware-10.2.0-amd64-netinst.iso /old-scratch/usr/myusername/testfolder/folder1 /old-scratch/usr/myusername/testfolder/folder1/file2 /old-scratch/usr/myusername/testfolder/folder1/folder11

An then run one rsync for every line in that output with the following script (adjust TARGET and DIRFILE):

#!/bin/bash set -e PARALLEL=10 RSYNC_OPTS="-aH --itemize-changes --stats --delete --numeric-ids --one-file-system --no-v" TARGET="/scratch/myusername/old_files" DIRFILE="/home/myusername/list_of_files_and_folders.log" #dirfile with full paths - take care of trailing slashes restart_rsync() { RC=30 count=0 while [ "$RC" = "30" ]; do let count=$count+1 /usr/bin/rsync $RSYNC_OPTS --timeout=120 $1 $TARGET RC=$? if [ "$count" -gt "5" ]; then break fi done } export -f restart_rsync export RSYNC_OPTS export TARGET echo "Using up to $PARALLEL processes for transfer..." echo "Using directories from file $DIRFILE ..." cat $DIRFILE | parallel -q -j $PARALLEL -t restart_rsync {}

 

User Manual

User Manual
Results will update as you type.
  • Application Guide
  • Status of System
  • Usage Guide
    • SSH Login
    • Data Transfer
    • File Systems
      • File system replacement
        • Data Migration HowTo
      • System Quota
      • Metadata Usage on WORK
      • Special Filesystems
      • Sharing data
    • MPI, OpenMP build and run workflows
    • Slurm usage
  • Compute partitions
  • Software
  • FAQ
  • NHR Community
  • Contact

    You‘re viewing this with anonymous access, so some content might be blocked.
    {"serverDuration": 10, "requestCorrelationId": "33754eefb2724ae8ba8c9c63fe268ca3"}