AWS Transfer Family SFTP Directories Are A Bit Weird

There are still a lot of people out there with SFTP (and even FTP!) based workflows. Amazon know this and have a dedicated product called AWS Transfer Family, which is basically an amazingly expensive SFTP wrapper that lives on top of S3.

If you don’t want the hassle of running SFTP on a $5/mo virtual server, then paying AWS on the order of USD$200/mo might be a good option.

There is some slightly weird behaviour compared to standard SFTP that caught me by surprise relating to directories.

(Note: I am doing this on a client’s SFTP setup, so I don’t know what it actually looks like on the S3 side.)

• If you try to rename a file into a directory that does not exist, you will not get an error – it will actually work, and create some sort of “virtual subdirectory” in the S3 bucket. e.g., if you do rename example.txt backup/example.txt, without the backup/ directory existing, and then do a directory listing, you’ll see there is a new backup/ directory that was created by that rename operation.
• If you then move the file back – rename backup/example.txt ./example.txt – the backup/ directory will disappear.
• If you create the backup/ directory first, and repeat the move in and out, the directory will persist.
• If the backup/ directory was created by the rename command, and you then try to do an ls * on the parent directory, it will return the files in backup/ as well – i.e., it will act like a recursive ls.

If you are trying to get closer to standard SFTP-based behaviour with directories, I suspect it’s safer to manually make the directories first (as you would normally) instead of relying on this weird automatic directory creation you get from the rename.