You need do to migrate your data before January 1522.
We expect best performance for the data transfers on the genoa system. You need to migrate relevant data from your user, project, and share folders in the old work/scratch WORK file system:
/old_-scratch/usr/username
/old_-scratch/projects/projectname
/old_-scratch/share/sharename
(/old_-scratch/tmp/username)
Migrating data could be as simple as We recommend migrating your data by
running a rsync on
...
genoa-login
...
node
use tmux to keep your transfer running without being logged in to the system
example rsync call:
Codeblock ssh username@genoa-login.nhr.zib.de $ rsync -av /old
...
-scratch/usr/username/folder_i_want_to_keep /scratch/usr/username/
...
Moving large directories with parallel rsyncs
For directories with a lot of data and many files in different subdirectories, the transfer might be sped up by running many several rsync processes in parallel. For this we first need a list of files/folders up to a certain depth.
...
| Codeblock |
|---|
blogin9:~ $ find /home/bzadmwatold-scratch/usr/myusername/testfolder -maxdepth 1 /home/bzadmwatold-scratch/usr/myusername/testfolder /home/bzadmwatold-scratch/usr/myusername/testfolder/debian-10.2.0-amd64-netinst.iso /old-scratch/homeusr/bzadmwatmyusername/testfolder/firmware-10.2.0-amd64-netinst.iso /home/bzadmwatold-scratch/usr/myusername/testfolder/folder1 blogin9:~ $ find /home/bzadmwatold-scratch/usr/myusername/testfolder -maxdepth 2 /old-scratch/homeusr/bzadmwatmyusername/testfolder /old-scratch/homeusr/bzadmwatmyusername/testfolder/debian-10.2.0-amd64-netinst.iso /old-scratch/homeusr/bzadmwatmyusername/testfolder/firmware-10.2.0-amd64-netinst.iso /home/bzadmwatold-scratch/usr/myusername/testfolder/folder1 /home/bzadmwatold-scratch/usr/myusername/testfolder/folder1/file2 /home/bzadmwatold-scratch/usr/myusername/testfolder/folder1/folder11 |
...
| Codeblock |
|---|
#!/bin/bash
set -e
PARALLEL=10
RSYNC_OPTS="-aH --itemize-changes --stats --delete --numeric-ids --one-file-system --no-v"
TARGET="/scratch/usr/myusername/old_files"
DIRFILE="/home/myusername/list_of_files_and_folders.log"
#dirfile with full paths - take care of trailing slashes
restart_rsync() {
RC=30
count=0
while [ "$RC" = "30" ]; do
let count=$count+1
/usr/bin/rsync $RSYNC_OPTS --timeout=120 $1 $TARGET
RC=$?
if [ "$count" -gt "5" ]; then
break
fi
done
}
export -f restart_rsync
export RSYNC_OPTS
export TARGET
echo "Using up to $PARALLEL processes for transfer..."
echo "Using directories from file $DIRFILE ..."
cat $DIRFILE | parallel -q -j $PARALLEL -t restart_rsync {}
|
...