We recently had to create a cron job that downloads some files over SFTP, which is different from the similarly named FTPS.
SFTP (Secure File Transfer Protocol) uses SSH to ensure the security of file transfers, unlike FTPS, which uses SSL.
The problem with downloading files over SFTP using a cron job, however, is that password-based authentication, which is what we were using to log into the SFTP server, normally only works in an interactive shell, where the user can physically type the password into the terminal. But shell scripts run by a cron job are non-interactive.
One way to get around this is to try passwordless authentication using SSH keys.
But let’s assume we really, really want to use password-based authentication.
We’re in luck: There are some ways to emulate an interactive shell using a script.
Below, we’ll present two approaches, both in Python. Each of these scripts can be run in a cron job.
Using the pexpect module
More info about pexpect is at [http://www.noah.org/wiki/pexpect].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
#!/usr/bin/env python2.5 """ Usage: %prog [options] file1 [file2] [extra_files] destination Downloads a file (or files) via SFTP. Example: sftp_download.py --host=sftp.example.com --path=/path/to/file/ --username=monetate --password=PaSsWoRd download_me.xml download_me_too.xml /destination/ """ from optparse import OptionParser import os import sys import pexpect ######################### parameters & settings ######################## parser = OptionParser(usage=__doc__.strip()) parser.add_option('--host', dest='host', action='store', type='string', help='SFTP host') parser.add_option('--path', dest='path', action='store', type='string', help='SFTP path') parser.add_option('--username', dest='username', action='store', type='string', help='SFTP login username') parser.add_option('--password', dest='password', action='store', type='string', help='SFTP login password') ############################### functions ############################## def download_files(files, destination, host, path, username, password): """ Log in to the SFTP server using the username and password provided and download the specified files. """ sftp_opts = ['-o', 'PasswordAuthentication=yes', '%s@%s' % (username, host)] p = pexpect.spawn('sftp', sftp_opts) p.logfile = sys.stdout try: p.expect('(?i)password:') x = p.sendline(password) x = p.expect(['Permission denied','sftp>']) if x == 0: print 'Permission denied for password:' print password p.kill(0) else: x = p.sendline('cd ' + path) for file in files: x = p.expect('sftp>') x = p.sendline('get ' + file + ' ' + destination) x = p.expect('sftp>') x = p.isalive() x = p.close() retval = p.exitstatus except pexpect.EOF: print str(p) return 'SFTP file transfer failed due to premature end of file.' except pexpect.TIMEOUT: print str(p) return 'SFTP file transfer failed due to timeout.' ############################## main block ############################## if __name__ == '__main__': options, args = parser.parse_args() if not args: parser.print_help() sys.exit(1) destination = args[-1] files = args[:-1] status = download_files(files, destination, options.host, options.path, options.username, options.password) sys.exit(status) |
Using the paramiko module
More info about paramiko is at [http://www.lag.net/paramiko/].
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
#!/usr/bin/env python2.5 """ Usage: %prog [options] file1 [file2] [extra_files] destination Downloads a file (or files) via SFTP. Example: sftp_download.py --host=sftp.example.com --path=/path/to/file/ --username=monetate --password=PaSsWoRd download_me.xml download_me_too.xml /destination/ """ from optparse import OptionParser import os import sys import paramiko ######################### parameters & settings ######################## parser = OptionParser(usage=__doc__.strip()) parser.add_option('--host', dest='host', action='store', type='string', help='SFTP host') parser.add_option('--path', dest='path', action='store', type='string', help='SFTP path') parser.add_option('--username', dest='username', action='store', type='string', help='SFTP login username') parser.add_option('--password', dest='password', action='store', type='string', help='SFTP login password') ############################### functions ############################## def download_files(files, destination, host, path, username, password): """ Log in to the SFTP server using the username and password provided and download the specified files. """ transport=paramiko.Transport(host) transport.connect(username=username, password=password) sftp=paramiko.SFTPClient.from_transport(transport) for file in files: sftp.get(path + file, destination + file) ############################## main block ############################## if __name__ == '__main__': options, args = parser.parse_args() if not args: parser.print_help() sys.exit(1) destination = args[-1] files = args[:-1] status = download_files(files, destination, options.host, options.path, options.username, options.password) sys.exit(status) |
Clearly, using the paramiko module is the simpler approach, although if you are already using the pexpect module for other things and don’t want to go through the small trouble of installing paramiko, you may wish to use the first script.
Copyright © 2013 Monetate Inc. All rights reserved.