-
Notifications
You must be signed in to change notification settings - Fork 2
/
rsync_command_list.Rmd
79 lines (60 loc) · 2.38 KB
/
rsync_command_list.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
---
title: "Rsync nanopore data onto a HPC"
subtitle: ""
author: "Alexis Lucattini"
date: "2017/08/17"
output:
xaringan::moon_reader:
lib_dir: libs
nature:
highlightStyle: github
highlightLines: true
countIncrementalSlides: false
---
# Sending data onto a server
The most common way to send data from a MinION laptop to a cluster is through the rsync command.
Rsync is installed by default on Ubuntu and MacOS systems.
Unforunately for Windows users, life is less straight forward.
You can either use:
1. Cygwin
2. A docker container such as andthensome/docker-node-rsync
+ This will require you to mount the appropriate directory and re-perform the ssh key setup in the previous step from within the docker container.
---
# Setting up a ssh-key
Make sure you can ssh into the rsync server prior to running the command.
See [tutorial 1](./basic_shell_logging.html) for assistance.
---
# The actual rsync command
This is the command to use. Do not be alarmed. We will go through it in next slides
```{bash, eval=FALSE}
# Setup variables
local_read_dir=/path/to/reads
username=<server_username>
servername=<name_of_server
remote_read_dir=/path/to/server
# Rsync command
rsync --recursive --times --checksum \
--prune-empty-dirs --remove-source-files --stats \
--include '*/' --include '*.fast5.tar.gz' --include '*.tsv' \
--exclude '*' \
${local_read_dir} ${username}@${servername}:${remote_read_dir}
```
---
# Rsync options
* --recusive: Copy files within folder
* --times: Preserve time stamps
* --prune-empty-dirs: This ensures that any directories with no matching files are not placed on the server
* --remove-source-files: Deletes files from the source folder.
* --stats: Print verbose output of the transfer
* --include / --exclude: Select which files to send across.
+ Order of commands important.
+ When using this option, we must first use --include '*/' to search recursively.
+ Then we include all files ending with '\*.fast5.tar.gz' and '\*.tsv'
+ We then exclude all other files.
* Local path
* Remote path
---
# Debugging
Prior to running the command, add the --dry-run parameter.
This will print a summary of the files that would have been moved/modified without actually doing the moving.
Alternatively, you can remove the final parameter `${username}@${servername}:${remote_read_dir}` first. This will just print the list of files that will be transferred.