Jun 6, 2012
tom

S3fs performance improvements or alternative?

Question

I’ve been trying to use s3fs on an EC2 server but it’s really slow. I spent the last 5 hours uploading 100MB of small files.

Is there anything I can do to improve the performance or is it something related to S3fs? If not, what’s an alternative I could use?

Asked by MB.

Answer

S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I’d suggest using something like S3Curl

You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.

If you need to have it as a ‘mountable’ storage, the only alternatives to S3FS that I know of is S3Backer or s3ql

Answered by thinice

Related posts:

  1. How to force s3fs mount on boot
  2. s3fs and fuse: failed to open /dev/fuse: Permission denied
  3. Mounting an S3 bucket onto a AWS Ubuntu instance issues
  4. uploading to amazon s3 using lftp
  5. Store files in the cloud to be accessible by any VPS

Leave a comment