File Transfer

Updated Mar 21, 2021 ·

SCP

SCP (Secure Copy) is a command-line utility that allows secure file transfers between local and remote systems using the SSH protocol. It provides encryption during data transfer and authentication, making it suitable for secure file copying over networks.

Pre-requisites

• rsa keys should be generate (id_rsa and id_rsa.pub)
• contents of id_rsa.pub of source-svr should be added to the authorized_keys of dest. server

To copy a file from source-svr to dest-svr:

    scp -i ~/.ssh/id_rsa-rhel ~/1-transfers/hello.txt eden@1.1.1.1:/home/eden

Note that it consists of the following:
- "-i <path-of-private-key>"    --- this is the generated rsa key 
- "<path/file.txt>              --- this is the file to be copied across
- "username@<ip>:<path-on-dest> --- this is the path on the destination svr

Note that the specific folder on the destination server should be specified.
If folder exists, file will be copied there.
If not, folder will be created and the file will be copied there.

In my local VM, I created two local files and then transferred it to the remote EC2 machine, specifically to the /tmp.

[root@localhost ~]# touch /tmp/edenlocalfile-1
[root@localhost ~]# touch /tmp/edenlocalfile-2
[root@localhost ~]# 
[root@localhost ~]# ll /tmp/edenlocalfile-*
-rw-r--r--. 1 root root 0 Jan  3 23:16 /tmp/edenlocalfile-1
-rw-r--r--. 1 root root 0 Jan  3 23:16 /tmp/edenlocalfile-2
[root@localhost ~]# 
[root@localhost ~]# scp /tmp/edenlocalfile-* root@1.2.3.4:/tmp
Enter passphrase for key '/root/.ssh/id_rsa': 
edenlocalfile-1                                                 100%    0     0.0KB/s   00:00    
edenlocalfile-2                                                 100%    0     0.0KB/s   00:00   

Checking on my remote machine:

$ ll /tmp/edenlocalfile-*
-rw-r--r--. 1 root root 0 Jan  3 15:17 /tmp/edenlocalfile-1
-rw-r--r--. 1 root root 0 Jan  3 15:17 /tmp/edenlocalfile-2

SFTP

SFTP (SSH File Transfer Protocol) is a secure file transfer protocol that provides file access, transfer, and management functionalities over SSH sessions. SFTP operates similarly to FTP but uses the SSH protocol to encrypt both authentication credentials and data, ensuring confidentiality and integrity during file transfers.

SFTP is enabled by default. You can find it in /etc/ssh/sshd_config. Search for "Sftp"

137 # override default of no subsystems
138 Subsystem sftp  /usr/libexec/openssh/sftp-server

You first need to connect via sftp (from source) to the destserver.

$ sftp -i .ssh/id_rsa-rhel eden@1.1.1.1
Enter passphrase for key '.ssh/id_rsa-rhel':
Connected to eden@1.1.1.1
sftp>
sftp>

Once connected, source-svr has access to the files inside dest-svr. To grab the file and copy:

sftp> get howdy.txt
Fetching /home/eden/howdy.txt to howdy.txt
/home/eden/howdy.txt 

Files can also be renamed:

sftp> get howdy.txt howdy-yall.txt
Fetching /home/eden/howdy.txt to howdy-yall.txt
/home/eden/howdy.txt                                                     
sftp>

To upload a file:

sftp> put howdy.txt

To close and ftp session:

exit  

Sync files using rsync

Rsync is a robust file copying tool that operates over SSH or directly over the network. It efficiently synchronizes files and directories between two locations while minimizing data transfer by only copying parts of files that have changed.

• If source and target files already exists, rsync will only sync their differences.
• rsysc can be used with many options.

To test it out, I created a few files on my remote EC2 machine.

$ touch /tmp/edenremotefile{1..10}
$ ls /tmp/edenremotefile*
/tmp/edenremotefile1   /tmp/edenremotefile2  /tmp/edenremotefile4  /tmp/edenremotefile6  /tmp/edenremotefile8
/tmp/edenremotefile10  /tmp/edenremotefile3  /tmp/edenremotefile5  /tmp/edenremotefile7  /tmp/edenremotefile9

On my local machine, I ran the rsync to sync the remote's tmp files to my local /tmp/data/.

[root@localhost ~]# mkdir /tmp/data
[root@localhost ~]# ll /tmp/data/
total 0
[root@localhost ~]# rsync -ar root@1.2.3.4:/tmp/eden* /tmp/data/
Enter passphrase for key '/root/.ssh/id_rsa': 
[root@localhost ~]# 
[root@localhost ~]# ls /tmp/data/
edenremotefile1   edenremotefile2  edenremotefile4  edenremotefile6  edenremotefile8
edenremotefile10  edenremotefile3  edenremotefile5  edenremotefile7  edenremotefile9

Let's say I delete some of the files in my remote EC2.

$ ll /tmp/edenremotefile*
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile1
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile10
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile2
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile3
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile4
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile5
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile6
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile7
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile8
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile9

$ rm -f /tmp/edenremotefile{5..8}

$ ll /tmp/edenremotefile*
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile1
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile10
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile2
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile3
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile4
-rw-r--r--. 1 root root 0 Jan  3 15:31 /tmp/edenremotefile9

Now I want to sync these files to another folder in my local machine.

[root@localhost ~]# rsync -ar root@1.2.3.4:/tmp/eden* /tmp/data/
Enter passphrase for key '/root/.ssh/id_rsa': 
[root@localhost ~]# 
[root@localhost ~]# ll /tmp/data2/
total 0
-rw-r--r--. 1 root root 0 Jan  3 23:31 edenremotefile1
-rw-r--r--. 1 root root 0 Jan  3 23:31 edenremotefile10
-rw-r--r--. 1 root root 0 Jan  3 23:31 edenremotefile2
-rw-r--r--. 1 root root 0 Jan  3 23:31 edenremotefile3
-rw-r--r--. 1 root root 0 Jan  3 23:31 edenremotefile4
-rw-r--r--. 1 root root 0 Jan  3 23:31 edenremotefile9