Linux – Scripting an sFTP batch upload every 1 minute

amazon-web-serviceslinuxsftpwebcam

I am wanting to record video of part of my house with a webcam while I am away for a few days to try and ensure any burgalers are photographed and the photo uploaded to my server before they even realise it.

I have setup a webcam to stream footage using mjpg_streamer. That works and is streaming OK.

It is possible to take a still capture from the stream whenever you want called 'FileName.jpg' :

wget http://127.0.0.1:8080/?action=snapshot -O FileName.jpg

I have an Amazon FTP server with Amazon Web Services and FTP access using sFTP. I am connected using Firefox's FireFTP plugin currently, so that works. The idea is to leave the computer running with the connection live.

I'd like to script the taking of a picture from the stream say every 1 minute and have the picture uploaded to my server via the live FTP connection, and either have the original file deleted off my PC so that the next one will save O or append a number to the end of each file and then FTP it up. e.g FileName1.jpeg, Filename2.jpeg.

I've Googled for hours and although there's loads of posts about scripting an FTP upload, I can't find any about a constant stream of uploading…i.e. "Watch this folder, upload the content every 1 minute and then a minute later upload whatever new content is in it".

I guess I need a bash script that will :

keep a counter so that each newly created file gets a different name
Send that filename to the "wget http://127.0.0.1:8080/?action=snapshot -O FileNameXX.jpg" every 30 seconds or 1 minute
Upload the FileNameXX.jpg to the FTP server

But I have no idea how to do that! Can anyone direct me? Or does anyone know of a way to do with FileZilla or something (which cant watch a folder AFAIK : https://forum.filezilla-project.org/viewtopic.php?t=41609?

Best Answer

My first tip would be to name the files using the date and time they were taken. That way you won't need to keep a counter anywhere, which would be difficult in a script which doesn't run continuously as its variables would get reset on each invocation. You could store the variables in files, but it's easier if you ensure the names won't collide. Something like wget http://127.0.0.1:8080/?action=snapshot -O "Snapshot-$(date).jpg" if you are using Bash. (Sorry if the syntax doesn't work, I'm no Bash expert and I'm typing this in my phone.)

Like you mentioned, there are several tutorials about scripting FTP uploads available. At least one of them should have included an example which uploads files by a pattern, such as "Snapshot-*.jpg", where the wildcard would match the timestamp. Or, you could point the FTP program (such as lftp or ncftp, which have binaries meant for scripting) to upload everything in a certain folder. Then wipe the folder if the program succeeded. That way you can run your script as often as you want using cron or a systemd timer, and have it be flexible enough to always try to upload any files which it didn't succeed with the later time it ran.

There's also software designed to do this task, and more, on their own. One such programs, which I've used myself, is simply called "motion" and is available for most distributions. It has built-in motion triggering (record and/or take snapshots) or continuous modes. It can be a bit CPU-intensive on systems like a Raspberry-Pi, but it certainly works.

If you want to step it up a bit, perhaps run multiple remote/local cameras, and have the motion detection offloaded to a more powerful central machine, look at Zoneminder. It takes longer to set up, and is in my experience picky about you manually setting the correct resolutions on your camera feeds, but it can be scripted to some degree.

Related Solutions

Linux: prevent SFTP users from deleting FTP’d files even straight after they upload them

Apologies i do not have sufficent reputation here to post all the links inline. I have prepared a gist which preserves them: https://gist.github.com/3590779

There's no way to achieve this with simple permissions. Due to OpenSSH's sftp-server you won't be able to implement the full requirements list but depending on the filesystem the files are being uploaded to, you can leverage [attributes] and [ACLs] to achieve some of your requirements.
Yes, the [sftp-server takes a -u parameter] (you can set this in your [sshd_config] on the Subsystem sftp line) which sets the umask for all uploads.
Yes, you can make use of inotify, one way may be with the [incron] tool although there are many ways to use inotify. Inotify allows you to have the kernel notify a userspace program on a filesystem event you identify, e.g. adding a file to a directory. You can then run a command on this event.

(3 Part 2) An alternative approach that may not be suitable for you is to use something like vsftpd with SSL protected FTP. This allows for encrypted FTP but because vsftpd is a full featured FTP server it provides simple configuration (see the [chown_uploads parameter] in vsftpd.conf) to match your exact needs.
Yes, via the inotify subsystem, you could register a watch for the [IN_CLOSE_WRITE] event.

How to setup ftp/sftp on aws

If you don't want to use AWS Transfer for SFTP, it is possible to set up your SFTP server directly from an EC2 instance.

If you follow correctly these instructions you should be able to create your SFTP users quite easily. In my specific case I used a micro T2 instance with Ubuntu 18.04

Let's install openSSH

sudo apt-get install openssh-server

You need to create a specific group where you will jail the users.

sudo groupadd sftpusers

Edit /etc/ssh/sshd_config using vim or nano
Comment out #Subsystem sftp /usr/lib/openssh/sftp-server
Then instead, add Subsystem sftp internal-sftp to allow SFTP connections into your server
Lastly, at the end of the file specify the new group configurations

Match group sftpusers
        ChrootDirectory %h
        X11Forwarding no
        AllowTcpForwarding no
        ForceCommand internal-sftp
        PasswordAuthentication yes

At this point your /etc/ssh/sshd_config should look like:

(...)

#Subsystem sftp /usr/lib/openssh/sftp-server

(...)

Subsystem sftp internal-sftp

Match group sftpusers
        ChrootDirectory %h
        X11Forwarding no
        AllowTcpForwarding no
        ForceCommand internal-sftp
        PasswordAuthentication yes

You need to restart the ssh service to apply the changes.

sudo service ssh restart

Now you should be set-up to create a new user.
Follow the different instructions of the command below and input the user password.

sudo adduser user1

Let's add our new user to the sftp group we created earlier.

sudo usermod -g sftpusers user1
sudo usermod -s /bin/nologin user1

At this point, the last thing we need to do is jail our user inside the /home/<user> directory.

sudo chown root:user1 /home/user1
sudo chmod 755 /home/user1

You can create new folders that belongs to the user using

sudo mkdir /home/user1/new_folder
sudo chown user1:user1 /home/user1/new_folder
sudo chmod 755 /home/user1/new_folder

I created this repo few days ago that automate this process

Best Answer

Related Solutions

Linux: prevent SFTP users from deleting FTP’d files even straight after they upload them

How to setup ftp/sftp on aws

Related Question