S3fs performance improvements or alternative?
I’ve been trying to use s3fs on an EC2 server but it’s really slow. I spent the last 5 hours uploading 100MB of small files.
Is there anything I can do to improve the performance or is it something related to S3fs? If not, what’s an alternative I could use?
S3FS may not be the best choice for a large amount of smaller files. The overhead with S3FS is pretty high as well. I’d suggest using something like S3Curl
You can even get parallel transfers going. Just remember it will never be fast like EBS / local storage.
If you need to have it as a ‘mountable’ storage, the only alternatives to S3FS that I know of is S3Backer or s3ql
Check more discussion of this question.
Related posts:
Leave a comment
Recent Posts
- Cron expression that runs every 5 minutes from 1:30 am – 6:00 am [duplicate]
- Understanding redundant power supplies
- Is there a way for administrators to disable users from installing Firefox extensions?
- Is there research material on NTP accuracy available?
- How to create a limited “domain admin” that does not have access to domain controllers?
Tags
active-directory
amazon-ec2
apache
apache2
backup
bash
centos
cisco
command-line
debian
dns
email
exchange
firewall
iis
iis7
iptables
linux
macosx
monitoring
mysql
networking
nginx
performance
permissions
php
postfix
raid
security
sql-server
sql-server-2005
sql-server-2008
ssh
ssl
ubuntu
unix
virtualization
vpn
webserver
windows
windows-7
windows-server-2003
windows-server-2008
windows-server-2008-r2
windows-xp





