FUSE-based file system backed by Amazon S3

Announcement: This s3fs project is moved to "s3fs-fuse" on GitHub December 15th, 2013. https://github.com/s3fs-fuse/s3fs-fuse

Announcement: Please submit usage/support questions to the Issues area instead of as a comment to this Wiki page. Thanks!

What's New

• s3fs-1.74.tar.gz (r478 - r498) fixes bugs(memory leak etc), IAM role, changes utility mode.
• s3fs-1.73.tar.gz (r465 - r474) fixes bugs, adds ahbe_conf option and samples.
• s3fs-1.72.tar.gz (r448 - r461) fixes bugs, adds multireq_max/parallel_count/fd_page_size option. And big change for performance.
• s3fs-1.71.tar.gz (r438,r439,r440,r441,r442,r444) fixes bugs, adds use_sse option.
• s3fs-1.70.tar.gz (r428,r429,r430,r431,r432,r433,r434) fixes bugs, adds nodnscache option.
• s3fs-1.69.tar.gz (r418,r419,r420,r421,r422,r423,r424) fixes bugs, supports enable_noobj_cache/enable_content_md5/uid/gid options, supports s3sync objects.
• s3fs-1.68.tar.gz (r408,r409,r410,r411,r412,r413,r414) fixes bugs, supports nonempty option, supports other S3 clients.
• s3fs-1.67.tar.gz (r405) fixes bugs, fixes memory leak.
• s3fs-1.66.tar.gz (r400) changes cache for performance, automatically checks xmlns, cleanups code, fixes bugs.
• s3fs-1.65.tar.gz (r396, r397) changes file layout of source codes, fixes a bug(r397).
• s3fs-1.64.tar.gz (r392, r393) changes dir object logic for compatibility, adds a option, fixes some bugs.
• s3fs-1.63.tar.gz (r390) adds stat_cache_expire option, enable permission checking, fixes some bugs.
• s3fs-1.62.tar.gz (r384, r385) fixes some bugs(memory leak, unclosed fd, etc), adds a few options for compatibility.
• s3fs-1.61.tar.gz (r368) Maintenance release for 1.60, fixes a compile-time bug in 32bit systems and a memory leak in md5sum().
• s3fs-1.60.tar.gz (r364) Support for files > 5GB. Fixed several memory leaks.
• s3fs-1.35.tar.gz (r306) Fixed issue with redirects and multipart upload works for files up to 64GB

Overview

s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files). Maximum file size=64GB (limited by s3fs, not Amazon).

s3fs is stable and is being used in number of production environments, e.g., rsync backup to s3.

Important Note: Your kernel must support FUSE, kernels earlier than 2.6.18-164 may not have FUSE support (see issue #140). Virtual Private Servers (VPS) may not have FUSE support compiled into their kernels.

To use it:

1. Get an Amazon S3 account! http://aws.amazon.com/s3/
2. Download, compile and install. (see InstallationNotes)
3. Specify your Security Credentials (Access Key ID & Secret Access Key) by one of the following methods:
   • using the passwd_file command line option
   • setting the AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables
   • using a .passwd-s3fs file in your home directory
   • using the system-wide /etc/passwd-s3fs file
4. do this:

/usr/bin/s3fs mybucket /mnt

That's it! the contents of your amazon bucket "mybucket" should now be accessible read/write in /mnt

The s3fs password file has this format (use this format if you have only one set of credentials):

accessKeyId:secretAccessKey

If have more than one set of credentials, then you can have default credentials as specified above, but this syntax will be recognized as well:

bucketName:accessKeyId:secretAccessKey

If you want to use IAM account, you can get AccessKey/secretAccessKey pair on AWS S3 console.

Note: The credentials files may not have lax permissions as this creates a security hole. ~/.passwd-s3fs may not have others/group permissions and /etc/passwd-s3fs may not have others permissions. Set permissions on these files accordingly:

% chmod 600 ~/.passwd-s3fs

% sudo chmod 640 /etc/passwd-s3fs

s3fs supports mode (e.g., chmod), mtime (e.g, touch) and uid/gid (chown). s3fs stores the values in x-amz-meta custom meta headers, and as such does "brute-force" re-uploads of s3 objects if/when mode and/or mtime changes. and uses x-amz-copy-source to efficiently change them.

s3fs has a caching mechanism: You can enable local file caching to minimize downloads, e.g., :

/usr/bin/s3fs mybucket /mnt -ouse_cache=/tmp

Hosting a cvsroot on s3 works! Although you probably don't really want to do it in practice. E.g., cvs -d /s3/cvsroot init. Incredibly, mysqld also works, although I doube you really wanna do that in practice! =)

Using rsync with an s3 volume as the destination doesn't quite work because of timestamp issues. s3fs does not (yet) support changing timestamps on files. I mean, it will work, as in it will copy files, but, the timestamps will just be current timestamps (rsync will complain about not being able to set timestamps but will continue).

s3fs works with rsync! (as of svn 43) Due to the way FUSE works and s3fs' "brute-force" support of mode (chmod) and mtime (touch), upon first sync, files are downloaded/uploaded more than once (because rsync does (a) chmod (b) touch and (c) rename), however, subsequent rsyncs are pretty much as fast as can be. If that's too much downloading/downloading for ya then try using the "use_cache" option to enable the local file cache... it will definitely minimize the number of downloads. as of https://code.google.com/p/s3fs/source/detail?r=152'>r152 s3fs uses x-amz-copy-source for efficient update of mode, mtime and uid/gid.

s3fs will retry s3 transactions on certain error conditions. The default retry count is 2, i.e., s3fs will make 2 retries per s3 transaction (for a total of 3 attempts: 1st attempt + 2 retries) before giving up. You can set the retry count by using the "retries" option, e.g., "-oretries=2".

Options

• default_acl (default="private")
  
  • the default canned acl to apply to all written s3 objects, e.g., "public-read"
    
    • see http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?RESTAccessPolicy.html "Canned Access Policies" for the full list of canned acls
      
  • any created files will have this canned acl
    
  • any updated files will also have this canned acl applied!

• prefix (default="") (coming soon!)
  
  • a prefix to append to all s3 objects

• retries (default="2")
  
  • number of times to retry a failed s3 transaction

• use_cache (default="" which means disabled)
  
  • local folder to use for local file cache

• use_rrs (default="" which means diabled)
  
  • use Amazon's Reduced Redundancy Storage

• use_sse (default="" which means disabled)
  
  • use Amazon’s Server Site Encryption.

• passwd_file (default="")
  
  • specify the path to the password file, over-rides looking for the password in in $HOME/.passwd-s3fs and /etc/passwd-s3fs

• ahbe_conf (default="" which means disabled)
  
  • This option specifies the configuration file path which file is the additional HTTP header by file(object) extension.

• public_bucket (default="" which means disabled)
  
  • anonymously mount a public bucket when set to 1, ignores the $HOME/.passwd-s3fs and /etc/passwd-s3fs files

• connect_timeout (default="10" seconds)
  
  • time to wait for connection before giving up

• readwrite_timeout (default="30" seconds)
  
  • time to wait between read/write activity before giving up

• max_stat_cache_size (default="10000" entries (about 4MB))
  
  • maximum number of entries in the stat cache

• url (default="http://s3.amazonaws.com")
  
  • sets the url to use to access amazon s3, e.g., if you want to use https then set url=https://s3.amazonaws.com

• stat_cache_expire (default is no expire)
  
  • specify expire time(seconds) for entries in the stat cache.

• enable_noobj_cache (default is disable)
  
  • enable cache entries for the object which does not exist.

• nodnscache
  • s3fs is always using dns cache, this option make dns cache disable.

• nomultipart
  • disable multipart uploads.

• multireq_max (default="500")
  
  • maximum number of parallel request for listing objects.

• parallel_count (default="5")
  
  • number of parallel request for downloading/uploading large objects. s3fs uploads large object(over 20MB) by multipart post request, and sends parallel requests. This option limits parallel request count which s3fs requests at once.

• fd_page_size (default="52428800"(50MB))
  
  • number of internal management page size for each file discriptor.

• enable_content_md5 (default is disable)
  
  • verifying uploaded data without multipart by content-md5 header.

• noxmlns
  • disable registing xml name space for response of ListBucketResult and ListVersionsResult etc. Default name space is looked up from "http://s3.amazonaws.com/doc/2006-03-01".

• nocopyapi
  • for a distributed object storage which is compatibility S3 API without PUT(copy api). If you set this option, s3fs do not use PUT with "x-amz-copy-source"(copy api).

• norenameapi
  • for a distributed object storage which is compatibility S3 API without PUT(copy api). This option is a subset of nocopyapi option.

Details

If enabled via "use_cache" option, s3fs automatically maintains a local cache of files in the folder specified by use_cache. Whenever s3fs needs to read or write a file on s3 it first downloads the entire file locally to the folder specified by use_cache and operates on it. When fuse release() is called, s3fs will re-upload the file to s3 if it has been changed. s3fs uses md5 checksums to minimize downloads from s3. Note: this is different from the stat cache (see below).

Local file caching works by calculating and comparing md5 checksums (ETag HTTP header).

The folder specified by use_cache is just a local cache. It can be deleted at any time. s3fs re-builds it on demand. Note: this directory grows unbounded and can fill up a file system dependent upon the bucket and reads to that bucket. Take precaution by using a quota system or routinely clearing the cache (or some other method).

s3fs supports chmod (mode) and touch (mtime) by virtue of "x-amz-meta-mode" and "x-amz-meta-mtime" custom meta headers. As well, these are supported in a brute-force manner. That is, changing any x-amz-meta headers requires re-uploading the s3 object. This is exactly what s3fs does. When changing mode or mtime, s3fs will download the s3 object, change the meta header(s) and re-upload the s3 object. Ditto for file rename. as of https://code.google.com/p/s3fs/source/detail?r=149'>r149 s3fs uses x-amz-copy-source, this means that s3fs no longer needs to operate in a brute-force manner; much faster now (one minor performance-related corner case left to solve... /usr/bin/touch)

The stat cache stores file information in memory and can improve performance. It's default setting is to store 10,000 entries which can account for about 4 MB of memory usage. When the stat cache fills up, entries with a low hit count are deleted first. The size of the stat cache is controllable with an option.

s3fs uses /etc/mime.types to "guess" the "correct" content-type based on file name extension. This means that you can copy a website to s3 and serve it up directly from s3 with correct content-types. Uknown file types are assigned "application/octet-stream".

Important Limitations

Eventual Consistency

Due to S3's "eventual consistency" limitations file creation can and will
occasionally fail. Even after a successful create subsequent reads can fail for an
indeterminate time, even after one or more successful reads. Create and read enough
files and you will eventually encounter this failure. This is not a flaw in s3fs and
it is not something a FUSE wrapper like s3fs can work around. The retries option does
not address this issue. Your application must either tolerate or compensate for these
failures, for example by retrying creates or reads. For more details,
see http://code.google.com/p/s3fs/wiki/EventualConsistency'>Eventual Consistency

libcurl version

s3fs runs with libcurl, then if you use libcurl with libnss, s3fs requires libcurl after version 7.21.5.
If you use lbcurl(with libnss) under version 7.21.5, s3fs leaks memory.
You don't mind about libcurl version when libcurl linked OpenSSL library instead of libnss.

Release Notes

• r497
  • Fixed a bug(Issue 389)
    
• r495, r496
  • Fixed a bug that object size is overflow on 32bit OS.
    
• r493
  • Changed utility mode and Supported removing multipart uploading object.
    
• r491, r492
  • Fixed a bug(Issue 371 #28, #32).
    
• r490
  • Supported IAM role(like s3fs-c).
    
• r489
  • Fixed a bug about public_bucket.
    
• r486
  • Fixed a bug(Issue 371).
    
• r483, r484, r487, r488, r493, r494, r495, r498
  • Fixed trifling bugs, and changed some codes.
    
• r482
  • Fixed memory leak, and changed parallel count 500 to 20 for multipart head, fixed some bugs.
    
• r481, r485
  • Fixed a bug(Issue 368).
    
• r480
  • Fixed a bug(Issue 321), and some bugs, strict checking option.
    
• r479
  • Calling curl_global_init function once, and Adds dynamic lock callback for OpenSSL.
    
• r478
  • for 32bit compiling.
    
• r465, r472, r473
  • Adds ahbe_conf option and samples, change mount point permission
    
• r466, r467, r468, r470, r471, r474
  • Fixes bugs and changes codes
    
• r448, r451, r452, r453, r454, r455, r460, r461
  • Fixes bugs and changes codes
    
• r449, r456, r457, r458, r459
  • Changes codes for performance, and adds options(multireq_max, parallel_count, fd_page_size)
    
• r442, r444
  • Fixed bugs and changed codes.
    
• r441
  • Refixed a bug( Issue 343, 235, 257, 265 )
    
• r440
  • Fixed a bug( Issue 342 )
    
• r 438, r439
  • Supports SSE and adds "use_sse" option( Issue 226 ).
    
• r434
  • Fixed a bug( Issue 235, Issue 257, Issue 265 )
    
• r432
  • Fixed a bug( Issue 241 )
    
• r 431, r433
  • Fixed bugs
    
• r430
  • Fixed a bug( Issue 235 )
    
• r429
  • Fixed a bug( Issue 429 )
    
  • Added nodnscache option
    
• r428
  • Fixed bugs( Issue 340, etc )
    
• r424
  • Supported uid/gid option
    
• r423
  • Added enable_content_md5 option
    
• r422
  • Fixes a bug(Issue 31)
    
• r420,r421
  • Fixes a bug(Issue 291)
    
  • Fixes a bug(Issue 240)
    
  • Added enable_noobj_cache option
    
• r418,r419
  • Fixed a bug that s3fs failed to get stats when max_stat_cache_size=0.
    
• r413, r414
  • Supports directory objects which are made by other S3 clients(Issue 27)
    
• r410
  • Supports "nonempty" fuse/mount option(Issue 265)
    
• r409
  • Fixes a bug(Issue 293)
    
• r408, r411, r412
  • Fixes a bug(Issue 320)
    
• r405
  • Fixes a bug(Issue: 328) - r404
  • Fixes a bug(Issue: 326) - r403
  • Fixes memory leak. - r403
• r400
  • Changes stat cache, it accumulates stat information and some headers.
    
  • Changes that s3fs can decide using the xmlns url automatically.
    
  • Fixes Issue 321 "no write permission for non-root user".
    
  • Cleanup code, fixes bougs.
    
• r397
  • Fixes a bug(can not make a directory which name is same as deleted directory name)
    
• r396
  • No changes for logic, only changes file layout of functions and valuables.
    
  • Adds s3fs_util.cpp/s3fs_util.h/common.h
    
• r392
  • Fixes bugs(memory leak, unclosed curl handle, etc).
    
  • Changes codes for cleaning up.
    
  • Adds noreanmeapi option.
    
  • Changes object name type for directory.
    
• r390
  • Adds stat_cache_expire option
    
  • Enable file permission checking.
    
  • Recognizes normal file object which has no x-amz-meta-mode header.
    
  • Adds "." and ".." directory when the files in the directory is listed.
    
  • Fixes bugs.
    
• r384
  • Fixes bugs(memory leak, unclosed fd, etc).
    
  • The mknod is changed that it always return error,.
    
  • The symbolic file attribute changed to S_IFLNK | S_IRWXU | S_IRWXG | S_IRWXO.
    
  • Fixes the truncate function to work.
    
  • Adds noxmlns and nocopyapi option.
    
• r302
  • fixed issue #144 - multipart upload works for files up to 64GB
    
• r301
  • fixed issue #145 - data corruption on redirections
    
• r298
  • implemented multipart upload for files > 20MB
    
• r289
  • fixed issue #17 - implement directory rename
    
  • fixed issue #18 - implement create() function
    
• r287
  • fixed issue #6 - accounted for short write on BIO_write
    
• r285
  • Fixed memory leak issues
    
• r281
  • minimum version for FUSE is now 2.8.4
    
  • changed curl write-to-memory callback function
    
  • eliminated a couple of memory leaks
    
  • added more debug code
    
• r280
  • service error codes from libcurl more appropriately
    
• r275
  • better error messages during check_service function
    
• r274
  • added FUSE's -s option back in
    
• r270
  • fixed issue #128 - problem with https on Fedora
    
• r269
  • fixed issue #125 - problem with https on CentOS
    
• r265
  • https - trap bad certificate issue during service_check
    
• r243
  • eliminate options for specifying credentials on the command line
    
• r241
  • more error checking - if connected to the network, check for valid credentials and the bucket name is valid
    
• r240
  • re-enable the the -f command line option (disables FUSE daemon mode)
    
• r235
  • add debug option to send debug messages to syslog
    
• r233
  • check for too liberal permissions on password file
    
• r231
  • more robust error checking
    
• r230
  • add "make check" target
    
• r229
  • added support for multiple credentials
    
• r227
  • basis for s3fs-1.10.tar.gz
    
• r217
  • basis for s3fs-1.0.tar.gz
    
• r213
  • added command line --version option
    
• r209
  • converted to Autotools
    
• r203
  • fail with error when bucket with upper case characters is attempted to be mounted
    
• r202
  • fixed bug associated with url = "https://"
    
• r201
  • rmdir now cannot delete a non-empty directory
    
• r200
  • support for Reduced Redundancy Storage
    
• r198
  • rsync without the --inplace option now creates the correct content-type metadata
    
• r197
  • removed compile time warning with respect to unused function
    
• r195
  • fixed argument ordering with initialization of fuse_operations data structure
    
• r166
  • case-insensitive lookup of content-type from /etc/mime.types
    
• r152
  • added support for symlinks... ln -s works!
    
• r151
  • use /etc/mime.types
    
• r150
  • added support for uid/gid... chown works!
    
• r149
  • support x-amz-copy-source... rsync much faster now!
    
• r145
  • log svn version at startup via syslog /var/log/messages
    
• r141
  • added "url" runtime parameter
    
• r136, r138
  • connect_timeout and readwrite_timeout
    
• r130
  • set uid/gid to whatever getuid()/getgid() returns
    
  • log some stuff to syslog (i.e., /var/log/messages)
    
  • fixed issue 14 (local file cache bug; fixed cp, rsync, etc...)
    
• r117
  • limit max-keys=20 (workaround for libcurl's 100% cpu issue?!?)
    
• r116
  • added crypto locking
    
• r114
  • curl_global_init
    
• r107
  • use CURLOPT_NOSIGNAL
    
• r106
  • rewind on retry
    
• r105
  • only send x-amz-acl and x-amz-meta headers
    
• r101, r102, r103
  • fixed curl_multi_timeout bug (found on mac)
    
• r99
  • added "default_acl" option
    
• r92
  • parallel-ized readdir(): getting a directory listing is now a lot faster
    
• r88
  • removed 10s read timeout that should not have been introduced
    
• r72 2008-02-18
  
  • use_cache now takes path to local file cache folder, e.g., /usr/bin/s3fs mybucket /s3 -ouse_cache=/tmp
    
• r66 2008-02-18
  
  • local file cache is back! however, it is disabled by default... use "use_cache" option, e.g., /usr/bin/s3fs mybucket /s3 -ouse_cache=1
• r57 2008-02-18
  
  • a few bug fixes:
    
    • touch x-amz-meta-mtime in flush()
      
    • use INFILE_LARGE (libcurl) (found on fc5/ppc)
      
  • tidyup
    
• r43 2008-02-17
  
  • mode (i.e., chmod), mtime and deep rename! rsync now works!
    
  • temporarily disabled local file cache (might not bring it back!)
    
• r28 2007-12-15
  
  • retry on 500 server error
    
• r27 2007-12-15
  
  • file-based (instead of memory-based)
    
    • this means that s3fs will no longer allocate large memory buffers when writing files to s3

FAQ

• What do I need to know?
  
  • /usr/bin/s3fs
    
  • /var/log/messages
    
  • an entry in /etc/fstab (optional - requires fuse to be fully installed issue #115)
    
  • the file $HOME/.passwd-s3fs or /etc/passwd-s3fs (optional)
    
  • the folder specified by use_cache (optional) a local file cache automatically maintained by s3fs, enabled with "use_cache" option, e.g., -ouse_cache=/tmp
    
  • the file /etc/mime.types
    
    • map of file extensions to Content-types
      
    • on Fedora /etc/mime.types comes from mailcap, so, you can either (a) create this file yourself or (b) do a yum install mailcap
      
  • stores files natively and transparently in amazon s3; you can access files with other tools, e.g., jets3t
    
• Why do I get "Input/output error"?
  
  • Does the bucket exist?
    
  • Are your credentials correct?
    
  • Is your local clock within 15 minutes of Amazon's? (RequestTimeTooSkewed)
    
• How do I troubleshoot it?
  
  • tail -f /var/log/messages
    
  • Use the fuse -f switch, e.g., /usr/bin/s3fs -f my_bucket /mnt
    
• Why do I see "Operation cannot be completed because you do not have sufficient privliges"
  • you'll see this when a program you're using (e.g., tar, rsync) is trying to explicitly set the modification time of a file. s3fs currently does not support this. Contents of the file are ok, its just that the timestamp might not be what you're expecting. I'm working to fix this. fixed in svn 43!
    
• Its still not working!
  
  • Try updating your version of libcurl: I've used 7.16 and 7.17
    
• Q: when I mount a bucket only the current user can see it; other users cannot; how do I allow other users to see it? ...why do I see "d?????????" in directory listings? A: use 'allow_other'
  
  • /usr/bin/s3fs -o allow_other mybucket /mnt
    
  • or from /etc/fstab: s3fs#mybucket /mnt fuse allow_other 0 0
    
• Q: How does the local file cache work?
  
  • A: It is unbounded! if you want you can use a cron job (e.g., script in /etc/cron.daily) to periodically purge "~/.s3fs"... due to the reference nature of posix file systems a periodic purge will not interfere with the normal operation of s3fs local file cache...!
    
• Q: How do I change the location of the "~/.s3fs" folder?
  • A: you don't (for now)... use a softlink -ouse_cache option is the path used for local file cache! e.g., /usr/bin/s3fs mybucket /s3 -ouse_cache=/tmp
    
• Q: s3fs uses x-amz-meta custom meta headers... will s3fs clobber any existing x-amz-meta custom header headers?
  
  • A: No!
    
• Q: I renamed a folder and now all of the files in that folder are gone! What the?!?
  
  • A: Rename it back and your files will be back. s3fs does not support deep directory rename and doesn't check for it either.

Limitations

• no permissions checking fixed in svn r390
• no chmod support: all files are 0775 fixed in svn 43!
  
• no symlink support added in r152
• rename is "supported" by virtue of returning EXDEV from rename() fixed in svn 43! svn 43 supports deep renaming of files
  
• when writing files: requires as much memory as the size of the largest file you're writing (this can be easily fixed) fixed (svn 27) you should now be able to copy, say, a 2GB file to s3 without having s3fs malloc 2GB of memory!
  
• deep rename directories?!?
• server side copies are not possible - due to how FUSE orchestrates the low level instructions, the file must first be downloaded to the client and then uploaded to the new location

ToDo

• support brute-force rename fixed in svn 43
  
• get symlinks working? added in r152
  • this would bog down performance: would have to do deep getattr() for every single object already doing this in svn 43... its not too bad!
    
• make install target
• get "-h" help working
• handle utime so that rsync works! fixed in svn 43!
  
  • probably a bad idea after all...
  • actually don't think it can be done: can't specify arbitrary create-time for PUT
• chmod support... acl
• permissions: using -o allow_other, even though files are owned by root 0755, another use can make changes
  
  • use default_permissions option?!?
    
• better error logging for troubleshooting, e.g., syslog...
  • need to parse response on, say, 403 and 404 errors, etc... and log 'em!
    
• use temporary file for flush() and then stream it to amazon

See Also

Here is a list of other Amazon S3 filesystems: