Data Migration HowTo

You need do migrate your data before January 15.

We expect best performance for the data transfers on the genoa system. You need to migrate relevant data from your user, project and share folders in the old WORK file system:

/old-scratch/usr/username
/old-scratch/projects/projectname
/old-scratch/share/sharename
(/old-scratch/tmp/username)

We recommend migrating your data by running a rsync on genoa-login nodes (use tmux to keep your transfer running without being logged into the system):

ssh username@genoa-login.nhr.zib.de
$ rsync -av /old-scratch/usr/username/folder_i_want_to_keep /scratch/usr/username/

Moving large directories with parallel rsyncs

For directories with a lot of data and many files in different subdirectories, the transfer might be sped up by running many rsync processes in parallel. For this we first need a list of files/folders up to a certain depth.

We can generate a directory and file list e.g. with find with the -maxdepth flag:

blogin9:~ $ find /home/bzadmwat/testfolder -maxdepth 1
/home/bzadmwat/testfolder
/home/bzadmwat/testfolder/debian-10.2.0-amd64-netinst.iso
/home/bzadmwat/testfolder/firmware-10.2.0-amd64-netinst.iso
/home/bzadmwat/testfolder/folder1
blogin9:~ $ find /home/bzadmwat/testfolder -maxdepth 2
/home/bzadmwat/testfolder
/home/bzadmwat/testfolder/debian-10.2.0-amd64-netinst.iso
/home/bzadmwat/testfolder/firmware-10.2.0-amd64-netinst.iso
/home/bzadmwat/testfolder/folder1
/home/bzadmwat/testfolder/folder1/file2
/home/bzadmwat/testfolder/folder1/folder11

An then run one rsync for every line in that output with the following script (adjust TARGET and DIRFILE):

#!/bin/bash
set -e

PARALLEL=10
RSYNC_OPTS="-aH --itemize-changes --stats --delete --numeric-ids --one-file-system --no-v"
TARGET="/scratch/usr/myusername/old_files"
DIRFILE="/home/myusername/list_of_files_and_folders.log"
#dirfile with full paths - take care of trailing slashes

restart_rsync() {
    RC=30
    count=0
    while [ "$RC" = "30" ]; do
        let count=$count+1
            /usr/bin/rsync $RSYNC_OPTS --timeout=120 $1 $TARGET
        RC=$?
        if [ "$count" -gt "5" ]; then
            break
        fi  
    done
}

export -f restart_rsync
export RSYNC_OPTS
export TARGET

echo "Using up to $PARALLEL processes for transfer..."
echo "Using directories from file $DIRFILE ..."
cat $DIRFILE | parallel -q -j $PARALLEL -t restart_rsync {}