Found this: http://www.rsaccon.com/2007/10/mount-amazon-s3-on-your-mac.html But not tried it yet

Hi Jorang- I'm assuming your primary interest is recovering any data in case of disaster, e.g., across instance shutdowns

in my opinion for small mysql datasets it is reasonable to add a simple cron job to do a mysql dump to s3 (hourly, daily, whatever is acceptable to you); I've done this in the past myself (daily, because 'important' data changed rarely); using mysql innodb (instead of myisam) should minimize any database contention when the backup script is run

if you have a large dataset and/or your dataset changes often then maybe amazon simpledb might be a better route

I've recently checked in a version of s3fs that adds local file caching, so, for example, you could configure apache to serve files directly from a mounted s3fs volume; with the local file caching, with the exception of the initial warming up of the cache, you might be able to get decent local-file performance (though I have not tried this setup myself yet so I have no real world experience) (you definitely would not want to run mysql in this fashion!)

hope that helps!

Not sure if I'm allowed to distribute this, but here's a binary of the above source compiled on Mac OX 10.4.10: (expires march 1st, 2008), hosted on s3, of course ;) I followed the instructions in the above linked thread, which are also more cleanly duplicated here:http://www.rsaccon.com/2007/10/mount-amazon-s3-on-your-mac.html . The one thing I needed to also do to make it all work was to add /opt/local/bin to my PATH.

https://miradu.s3.amazonaws.com/s3fs?AWSAccessKeyId=0GWZZ6FN6895K2ETZ602&Expires=1204351195&Signature=LR8atfPW0XBjPMXIlXPLNZtAdIg%3D

Cheers,

-Michael Ducker miradu@miradu.com

tleasure- there is no provision for making a file "public-read" with s3fs; you can use a tool such as jets3t for that ... I was thinking of making "chmod" fiddle with the s3 permissions... chmod's "mode" user/group/other would map nicely, however, I think it would actually be a bad idea ... it would be an unexpected surprise to most people, I would think, not realizing that they're making their files world-readable! even a simple rsync of some 777 files would cause those 777 files to be world-readable by anyone on the Internet!

I've noticed that if you have pre-existing files that have a slash in the S3 filename (like a virtual directory), s3fs will not pick up on it until you create the directory locally. So if you have bucket:images/somefile.txt, you want to create a "images" directory locally in your s3 mount. Afterward, the files show up. Hope this helps.

rrizun - thanks for your response. I agree that mapping chmod to S3's ACL could yield an unexpected surprise. But I do think this functionality would be huge. Do you think there are any alternatives other than chmod to set the ACL initially rather than doing another request after the file transfer?

I see you're currently using curl "easy" functions. Have you looked into whether the "multi" and/or "share" functions would improve performance at all?

Not that it's neccessarily "bad" given the S3 architecture, but I'm just curious.

cblaise- use of libcurl's multi/share api by itself probably would not improve network performance all that much, however, use of FUSE's low-level asynchronous api in combination with libcurl's multi/share api would definitely improve the "responsiveness/robustness" of s3fs, e.g., hitting CTRL-C during an I/O operation should respond immediately due to async api vs sync api...

I'm trying this with the current (91) release code and with r88. Reads work fine, but writes are returning input/output error. A little testing and digging in the code shows that the my_curl_easy_perform is returning -EIO. (My test is just "touch /mnt/x" to create a new file)

Further, it seems that the EIO is being raised because I'm getting a 411 error back from AWS, which according to this indicates that the content-length header should be passed.

Since this is a list of errors from 2006, I'm guessing that I'm doing something wrong rather than the code or aws. Has anyone else had this problem? I assume that I'm authenticating ok to be able to read my own (private) buckets.

or better yet capture packets w/something like this:

tcpdump -s 1500 -A host s3.amazonaws.com

Hi there- I got this up and running yesterday on a MacBook? running OSX 10.4 and it worked quite well- I was able to weed out a few minor bugs in doing so. Having said that, try SVN version "107" which contains those bug fixes.

Are there files already in the bucket you're creating? Try creating and mounting an empty bucket and see if it still "sends Finder mental"! =)

I use jets3t (https://jets3t.dev.java.net) to manage buckets.

rrizun: I have the latest SVN and its not working mate. It seems to be deadlocking or something: I've got gdb on it waiting for it to crash out and Its not even letting me break in. I'm well up for getting this working so email me - jkp@kirkconsulting.co.uk if you want a debugger.

Jamie

Yup, unfortunately the local file caching issues still exist in the codebase... the symptoms are exactly as you describe... I'm sure its a simple fix but I just haven't been able to scrape up enough time to get around to fixing it!!!

Workaround: for now, disable local file cache! rsync should still work pretty good w/o it...

Once I fix it then I'll post a new src tarball...

Glad it works for ya!

The good news: it works, and works GREAT on Ubuntu 7.10 The bad news: There is one serious design glitch that kills its usability for my needs.

I'm using Bacula to write to an S3 bucket mounted by s3fs. I'm using local caching, and 100MB volumes in Bacula so not too much is uploaded when a volume changes.

Problem: Bacula writes that 100MB volume, and then closes the file. s3fs then starts uploading that file to S3, but it blocks until the file is uploaded, and Bacula can't create a second volume (file on disk) until the first one completes uploading. That means, I can't spool to the local cache at all if I want my backup job to run in a reasonable amount of time (we're talking multi-gigabytes on a 512kpbs uplink). Is there any way around this? I'm pretty sure you're going to tell me it involves threading s3fs, and I know that is a headache and a half, but it would be cool if it could be done. Thanks for the great project!

Hi pedahzur-

Ya, threading is not an issue... s3fs could do the s3 upload in a separate thread and return immediately to the caller, (essentially a write behind cache) however, but then there would be no way for s3fs to directly convey an error back to the caller, i.e., would not be able to return a bad error/status code/return value. Futhermore there are concurrency issues to consider, easiest solution would be to serialize the write behind thread.

So, ya, it could be done (quite easily, actually), but note without making it clear to the end user he ramifications and trade-offs, thru documentation I guess!

Feel free to add a new "Issue" to track this feature/enhancement!

Hi there, is it possible to store a relational database like MySQL directly into S3 using S3FS and run and update it on S3 via S3FS? Has anybody done this before? or its too risky? Thanks

Hi- it is possible, however I would assume performance would be terrible (haven't tried myself), based on the fact that s3fs operates in a "bruteforce" manner, re-uploading entire files on changes. ElasticDrive? and/or PersistentFS would probably be better candidates since they are really block devices.

FYI just for fun I tried running mysqld with datadir pointing to an s3 bucket and it worked! only issue is the "service mysqld start" timed out because mysqld created a 10MB ibdata and uploaded it; apparently the mysqld init script did not want to wait that long! =) I was able to do queries and it wasn't nearly as bad as I thought it would be; best bet would be to disable innodb and use myisam... =)

We are using it off an Amazon EC2 instance running Debian so the bandwidth should not be an issue since its all internal. I have a feeling that it could be the switch time between each file copy, what do you think? if it is, any way of reducing this? Thanks so much for your help.

Hi- I think this is a case of many small files vs one large file; in this case it ends up significant; on my machine (cable modem at home) using ethereal/wireshark , I can see about a one second "penalty" for setup before the file transfer actually takes place; that is, using "cp -p" preserve mode, ownership, timestamps, will cause s3fs to send a flurry of HTTP PUT requests to preserve mode, ownership, timestamps; all this takes time and adds up to the observed overhead that you're seeing; 76 files times approx 1 second overhead per file accounts for the discrepancy you're seeng: 58sec/76files=0.763 seconds overhead per file (I'm seeing just over 1 second per file, that's ec2 vs cable modem at home)

s3fs uses http keep-alive so it is probably as fast as can be; the only way to speed up copying of many small files would be to parallelize the copy operation; amazon s3 is very conducive toward parallelization (s3fs is fully multi-threaded and will use multiple simultaneous http connections)

so, all seems normal! hope that helps!

er, I guess the above calc is more like: (58sec-9sec)/76files=0.644 second overhead per file (I'm subtracting the 9 seconds it takes for the raw upload of the entire 80Mbyte file... you get the gist! =)

hi rrizun thanks! for s3fs as well, its better than elasticdrive if it matters anything, ed is slower than s3fs and just pretends to be faster (by returning instantly), under the hood is 2x or so slower, s3fs is great i wish i had more time to study the s3 api as well thanks ;-)

I am unable to compile. I get this:

buffy:~/s3fs# make g++ -ggdb -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -lfuse -lpthread -lcurl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lkrb5support -lresolv -lidn -ldl -lssl -lcrypto -lz -I/usr/include/libxml2 -L/usr/lib -lxml2 -lcrypto s3fs.cpp -o s3fs s3fs.cpp:1641:74: error: macro "fuse_main" passed 4 arguments, but takes just 3 s3fs.cpp: In function âint main(int, char)â: s3fs.cpp:1636: error: invalid conversion from âvoid ()(fuse_conn_info)â to âvoid ()()â s3fs.cpp:1639: error: âstruct fuse_operationsâ has no member named âutimensâ s3fs.cpp:1641: error: âfuse_mainâ was not declared in this scope s3fs.cpp: At global scope: s3fs.cpp:438: warning: âsize_t readCallback(void, size_t, size_t, void)â defined but not used make: all? Error 1

These files are not big ones. Sizes usually less than 10MB

I was able to get output from unsuccessful attempt to copy a SWF file (size: 2,605,315 bytes) Packets info were caught with the following command:

tcpdump -i eth0 -s 1500 -A -v host s3.amazonaws.com > info.txt

I found the xml message inside info.txt file:

..... HTTP/1.1 400 Bad Request^M
x-amz-request-id: E94B4261A9DD7418^M
x-amz-id-2: [removed_by_zelvlad]
Content-Type: application/xml^M
Transfer-Encoding: chunked^M
Date: Thu, 29 May 2008 19:04:58 GMT^M
Connection: close^M
Server: AmazonS3^M

...

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeout</Code>
<Message>Your socket connection to the server was not read from or written to within the timeout period. 
Idle connections will be closed.</Message><RequestId>E94B4261A9DD7418</RequestId>
<HostId>[removed_by_zelvlad]</HostId></Error>^M

Could it help to resolve the issue. Any suggestions?

Thank you very much for your help -Vlad

 Comment by jorang,  Dec 22, 2007

Anyone tried to use s3fs to make EC2 (MySQL and Apache) use files directly from S3? I read somewhere that the connection between EC2 and S3 would create problems in such a solution. Acording to your experience what could be reasonable to acomplish with EC2, S3 and s3fs when having a high-load website? Direct use, minute2minute backup, hourly backup, ...?

The easiest way to ensure all your ec2 dbs are save from ec2 crashes is to replicate your ec2 database on a regular machine

I currently replicate my ec2 dbs on my centos machine at home. The updates to the slave db is practically instantaneous so if ec2 every crashed I'd probably lose no more than a minute of data

Also you can make backups of your db by backing up the slave. No need to spend precious cpu power on database updates. Just run it on a utility computer sitting in your basement

Command "curl-config --version" returns libcurl 7.17.1. It looks like "100-continue" is present

About mounting/unmounting. As I mentioned I'm copying big structure with cp -r. I never managed to copy the whole directory without at least a dozen errors like this. I mounted/unmounted many times but some errors are always present.

E....H@.@....i.zH..B...P ....(..P.......PUT /A/B/C/introfinal2.swf HTTP/1.1
Host: s3.amazonaws.com
Accept: */*
Date: Thu, 29 May 2008 18:58:25 GMT
Content-Type:application/octet-stream
x-amz-acl:private
x-amz-meta-mode:33188
x-amz-meta-mtime:1212087505
Authorization: AWS [remove]
Content-Length: 2605315
Expect: 100-continue

14:58:25.108906 IP (tos 0x0, ttl 49, id 58818, offset 0, flags [DF], proto TCP (6), length 65) s3.amazonaws.com.http > [host].49798: P, cksum 0xd64d (correct), 4541:4566(25) ack 3101 win 1028
E..A..@.1...H..B.i.z.P...(.. ..AP....M..HTTP/1.1 100 Continue

...

14:58:52.173891 IP (tos 0x0, ttl 49, id 59119, offset 0, flags [DF], proto TCP (6), length 670) s3.amazonaws.com.http > [host].49798: P, cksum 0x06f6 (correct), 4566:5196(630) ack 857989 win 27854
E.....@.1...H..B.i.z.P...(.. &..P.l.....HTTP/1.1 400 Bad Request
x-amz-request-id: 14378A78D8316F17
x-amz-id-2: [removed]
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Thu, 29 May 2008 18:58:51 GMT
Connection: close
Server: AmazonS3

15c
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within 
the timeout period. Idle connections will be closed.</Message>
<RequestId>14378A78D8316F17</RequestId><HostId>[removed]</HostId></Error>

Thank you, Vlad

Yep: 526 MB in / (it's a linode, so I don't get to partition the root disk). Is there an argument to change where it creates those temp files?

hey. i'm having problems with mv, as opposed to cp. i have s3fs mounted at /var/n54data. when i do:

dd if=/dev/zero of=sux count=1024 bs=1024; cp sux /var/n54data

the resulting file (/var/n54data/sux) is 1M in size. however when i do:

dd if=/dev/zero of=sux count=1024 bs=1024; mv sux /var/n54data

the resulting file is 0 bytes in size.

p.z. s3fs is awesome.

s3fs uses posix http://linux.die.net/man/3/tmpfile to create its temporary local files... so, currently, no, there is no s3fs argument to control where temporary files are created

re: cp vs. mv ; yeah disk space was not a problem. also in the strace, it did try a rename and then fallback to cp. also it's using read/write, not mmap, so no fuse problem there. actually the straces look extremely similar; however the mv one has a SYS_320 line it (?), and of course also contains a trailing unlink :(

are you able to reproduce the problem?

Hello rrizun,

I'm logged in as the user "root".

I mount a directory as another user, e.g "apache" as seen in the code below:

su apache -c 's3fs mybucket /mymount'

I do this so my http server, running as user "apache", can write to the /mymount directory.

The problem is now the user "root" does not have access to the /mymount directory as seen below:

#cd /mymount
-bash: cd: /mymount: Permission denied

My question is how come the superuser "root" does not have access to a mount owned by "apache"

use fuse "allow_other" switch... search for "allow_other" on this page

I am new to s3fs. When I read that you mount your bucket with

/usr/bin/s3fs mybucket -o accessKeyId=aaa -o secretAccessKey=bbb /mnt

And what about file transfers? I'd assume they use http by default. Is there an option to change to https?

Hi- s3fs never transmits the secretAccessKey over the internet; s3fs (or any other s3 client for that matter) only transmits signatures created locally from the secretAccessKey; That is actually by amazon S3 design.

For the actual file transfer, "http" is used by default but can be overridden by usnig the s3fs "url" command line option and specifying, say, "https://s3.amazonaws.com"

hope that helps!

Has anyone gotten https access to work? http access works fine.

I keep getting the following error: ###problem with the SSL CA cert (path? access rights?)

i'm running s3fs r166, debian etch w/ libcurl 7.15.5, libfuse 2.7.1, openssl 0.9.8

the curl command line is able to access https://s3.amazonaws.com without error so i know my CA bundle is correct. it seems like the s3fs code may not be initializing the path to the CA bundle.

Running from the commandline: curl --trace foo https://s3.amazonaws.com yields: == Info: About to connect() to s3.amazonaws.com port 443 == Info: Trying 207.171.185.193... == Info: connected == Info: Connected to s3.amazonaws.com (207.171.185.193) port 443 == Info: successfully set certificate verify locations: == Info: CAfile: /etc/ssl/certs/ca-certificates.crt

Thanks Michael

I have https working on both lenny and hardy with libcurl4-openssl-dev 7.18.0 and ca-certificate.

I had to modify s3fs to use curl_easy_setopt(curl, CURLOPT_CAPATH, capath) with capath set to /etc/ssl/certs. See: http://curl.haxx.se/docs/sslcerts.html

Hope this helps. -Shane

Hi- yes, s3fs_rename() would need to recursively rename each sub-object and would typically be a long running operation (but should be able to be seamlessly resumed if interrupted); should be able to do deep directory rename now with some /usr/bin/find trickery, though I haven't looked into it myself; deep directory rename is one of the higher priorities, just need some spare time to implement it!

try "tail -f /var/log/messages" to see if it reveals anything; also, can you, e.g., mkdir and does the directory show up? as well, you can use another s3 client such as jets3t to inspect the s3 data/objects...

Please add http://s3backer.googlecode.com/ to the "See Also" list. Thanks.

symlinks are now added. The wiki still says otherwise. That was a big requirement for me. Do others a favor and update it please.

Hi Dylan- indeed, more docs would be good; as for NFS, I have not tried myself, nor have I heard of anyone else doing so; I'm sure there would be some level of performance degradation due to at least the file system layering... feel free to give it a try and post your results! =)

s3fs seems to be using a unique way of creating directories on S3. Is this ever going to change so as to be more compatible with other s3 tools?

S3Fox? and JungleDisk?(when in compatibility mode) both use a similar directory scheme and can read each others buckets. However neither of them can correctly parse the s3fs directory scheme.

I understand that there is no standard way of doing directories on s3, but I guess there must be a de facto standard arising as the jungledisk site states: "most S3 applications now use a simpler scheme that uses object names that resemble standard URLs, made possible by a feature called delimiters added to Amazon S3 last year."

Hi- Indeed I am interested in making s3fs compatible w/other s3 clients, as with everything its just a matter of finding time to do so; the "delimiters" feature (if it is what I'm thinking) is already being used by s3fs, but there are additional considerations wrt a complete "universal" folder representation convention; hope that helps!

I did some tests with some of the tools I'm considering. Seems that no tool is really very compatible with any other in regards to directory schemes. I made a comparison chart: http://www.coolbutuseless.com/post/40760186/s3-is-useful-as-a-backup-system-if-backups-are

I also tried the other 2 s3 fuse solutions and they aren't compatible with anything either.

>>> it is possible; try using tcpdump/ethereal/wireshark to inspect the packets to get an idea if there's any "thrashing" going on

Seems overly complex, I doubt I could interpret the results. I'm moving to the S3 storage platform, and the directory structure is a matrix. e.g.) a/b/0/0/0 , a/b/0/0/1 , to a/b/9/9/9. Quite a few folders and a great deal are populated.

When accessing my structure under s3fs, I do see delays the deeper I go. The delays are more noticeable when doing a "find", as opposed to a simple file read under PHP.

How does one "remap" under s3fs ? Would rebooting be the only option ?

Hi- looks like the "delay" that you're seeing is directly related to the depth of the directory structure; looks like fuse does a "stat" on each directory level before each file access

e.g., if I have a dir/file like "/a/b/c/d/e/f/g/abc.txt" then fuse (or something else higher up) will end up doing a stat on "/a" and then "/a/b" and then "/a/b/c" etc... before finally ending up at "/a/b/c/d/e/f/g/abc.txt"; these are all separate and independent S3 HTTP HEAD transactions as far as s3fs is concerned

don't think s3fs has much control over that since it is really just a "slave" to the fuse subsystem; s3fs could do some caching/second guessing, I guess, but then that might start to get a little bit tricky...

>>> The delays are more noticeable when doing a "find"

if you're interested in speeding that up... in the source code, in "s3fs_readdir()" search for "max-keys" and set that value to, say, 100 (instead of 20)... recompile and remount and see if that makes a difference; it might speed up "find" by 2x or so, ymmv

Anyone experienced high load burst on a server, using s3fs?

Hi- libcurl seems to have a "bug" wrt high numbers of "multi" handles... symptom is high cpu load... to "fix" it, in the source code, in s3fs_readdir(), search for "max-keys" and set the value to something less than 20, say, 5... recompile and remount and see if that makes a difference! (though that'll slow down readdir()... i.e., less parallel reads)

the other possibility is if local_cache is enabled? i.e., calculating local md5 checksums, etc...

judging by the "burst" observation, I'm guessing its the "max-keys" issue

Hi rrizun, I am having a heck of a time right now trying to figure out why I am getting IOEs. I have been building a test instance with FC9 using s3fs r166 and have the latest curl (7.18.2).. checking my logs, s3fs is returning ###response=403.. I have double checked my access keys and they are fine.. any idea of what might be causing this... also... the program is great and very much appreciate.. would like to make a donation to you if we can..

Hi- is the local time set correctly on your fc9 box? if that isn't it then try using tcpdump (e.g., "tcpdump -s 1500 -A host s3.amazonaws.com") to capture the xml document that s3 returns for the 403 response

rrizun... would like to report good things about the s3fs/NFS experiment but so far:

exportfs: Warning /mnt/s-three-test does not support NFS export.

Does some knucklehead/hacker or combination of both anticipate what it would take to get s3fs to "support NFS export"? I've yet to find anyone sharing about this, maybe it's too knuckleheaded I don't know.

btw... (note). anyone looking for some sick (in california that means 'good') Debian and Ubuntu AMIs should check out http://alestic.com. I only found these this morning and I am sad I didn't start using them earlier. that is a personal recommendation not some kind of commercial or anything, they definitely are in good shape to work with s3fs "out of the box". sometimes the kernal module thing can be weird on those xen-based AMIs, this has been my learning experience.

haven't tried exporting via nfs myself... see http://fuse.cvs.sourceforge.net/*checkout*/fuse/fuse/README.NFS

It'd be really great if it supported setxattr() and getxattr() to set metadata on files. Specifically, if there was a key for the canned ACL, you could easily write a tool to retrieve and set canned ACLs on files (eg "setacl public myfile") by calling setxattr().

It'd also be really nice if you could specify a transform - such as compression or encryption - for files to be stored with.

One other note: It'd be handy if one of the attributes you could retrieve via getxattr was the md5 sum for the file.

Hi rrizun,

I just tried to use s3fs with mpd (music player daemon) by pointing mpd at the s3fs mount. When mpd updates its database, it looks at the file modified times for everything in the directory (my bucket of lots of mp3 files). If the file is either non-existent in its database, or its modified time is newer than what mpd stored as its last modified time, it searches for ID3 tags in the file, extracts them and creates a database record.

When I did an mpd database update on my s3fs bucket, it basically started downloading my entire music collection, incurring quite a lot of OUT bandwidth. I am pretty certain that the ID3 tag algorithm only needs to read a part of the file, since on a local disk it can generate an entire database of 5000+ MP3s in about 2 minutes, which would be an extraordinary feat if the disk were reading all 25GB of data into memory.

But it seems that when the file is opened for reading, s3fs downloads the entire file. This is expensive and time-consuming. Of course, if mpd's file modified time algorithm does its job, this should be a one-time cost for each file.. it will skip over doing a read on files that have the same modified time it's expecting. And I don't modify my mp3 files very often ;)

Do you know of any way to permit the ID3 tag algorithm to do some seeking around in the file, find the start of the ID3 tags, and extract the needed few kilobytes, rather than downloading each 5+ MB mp3 file? It seems like this would more or less demand a large amount of round trips between the local system and s3, but if it could read small "pages" of the file (say, 64KB at a time) as the data is requested, this would make the ID3 tagging much more bandwidth-efficient.

Thanks,

Sean

SMcNam- as coded, s3fs follows a brute-force "all-or-nothing" strategy; indeed it will download the entire file even if ultimately only one byte needs to be read; the same phenomena that you're seeing can also be seen, e.g., by browsing an s3 mounted folder with GNOME Nautilus (or MacOSX Finder); they'll both wanna read the first few bytes of each file in order to determine their file types; I believe there is already an issue tracking this feature enhancement

rrizun, good point... It seems the two best FUSE-based s3 access methods each have a substantial weakness leading to undue bandwidth expense; let me explain.

s3fs is very good at uploading files to s3, whether small or large... it doesn't incur extraneous PUT/LIST/GETs because it seems to do a single PUT for an arbitrarily large file.

Of course, having a 1:1 mapping between actual files and s3 files has its drawbacks. Just as you said, to do any I/O on the file, it has to be retrieved in its entirety. So s3fs is very write-efficient, and very read-inefficient.

s3backer, on the other hand, reverses the problems. s3backer treats a ton of S3 files as a single virtual file in FUSE, which corresponds to a virtual hard disk drive. You then format a real filesystem on to, such as ext2. ext2 divides its data into blocks for the purpose of efficiency when reading; for the same reason, s3backer treats every few kilobytes of its virtual file as a separate S3 file. So if I want to read from a single byte inside a 20-meg file, s3backer asks the filesystem's allocation table where that byte is, and maps it to a file on S3, which is between 4k and 64k in size. So it's still getting one entire S3 file to provide your data, but that S3 file happens to be 64k instead of 20 megs.

Here comes s3backer's pitfall: when you start to deal with actual files that are several megabytes large, they are stored as tens, hundreds or thousands of different S3 files. Whenever you would like to read or write this entire file sequentially, in the background you are accruing tens, hundreds or thousands of GET or PUT requests into S3.

This has two major disadvantages: first, in the case of thousands of requests, even a multithreaded s3backer can still only upload your file at a maximum of 100 KB/s or so: the overhead of initiating a new GET/PUT request every 4k to 64k (depending on your configured block size) severely limits the amount of data that can be transferred at a time. There is a large amount of waiting done at the network layer when you're constantly initiating new TCP sockets. Second, those pennies really start adding up as your thousands of GET/PUT requests pile in.

Is it theoretically possible to design an s3-backed, Linux VFS mountable filesystem (whether FUSE or not) which is optimal for both sequential and random access? Can s3fs be improved to provide smarter random access without grabbing entire files? Or can s3backer be improved to provide smarter sequential access without creating thousands of individual TCP sockets (not to mention incurring many PUTs on your bill) for each block?

I am very interested in the direction ahead; perhaps we need guidance from Amazon to fully understand their intent with S3 and to help us come up with an optimal solution.

Thanks,

Sean

Just wanted to clarify the early parts of my last post: the issue is not reads vs. writes, but really sequential vs. random access. Random access (seek to a particular place, start reading) usually doesn't involve examining the entire file, so in that case we only want a small amount of the file to go over the network, regardless of whether it's an upload or a download. Sequential access is most efficient over S3 if it can be wrapped in a single GET or PUT request.

It seems like we can't have the best of both worlds unless S3 API allows us to read parts of an S3 file without downloading it.... if so, that might be the way to proceed: Go with s3fs, because it provides the 1:1 mapping, but try and be "smart" about VFS/FUSE requests for I/O, by only downloading pieces of the file that are requested.

Hmm: You might have to go with an adaptive algorithm that starts with small pieces when I/O begins, and "catches on" if sequential access continues in a predictable way, expanding the piece size exponentially to reduce the amount of PUTs or GETs as the I/O continues.

First: a moment of silence for yesterday's S3 incident.

Second: I found a really cool use for s3fs I'd like to share. Maybe others are doing this too but I'll share anyways. It is so simple.

Basically the challenge is migrating our initial data over to S3. So today I mounted the same S3 bucket four times:

/mnt/s3_1 /mnt/s3_2 /mnt/s3_3 /mnt/s3_4

then wrote a little multi-threaded uploader that mapped those s3fs directories to local directories and ran an rsync -ra SRC DEST on them simultaneously. Too cool, I thought. Really cut the time of transfer down.

This will only work for certain well defined situations where directory structure is such that the structure can be divided sensibly. And as you know, rrizun put in the work to support rsync, so that is great too.

Regarding the performance of s3backer's "block" access.

s3backer 1.0.x serviced each request to write a block within the FUSE thread that requested it. This ended up giving very slow write performance, because as it turns out the kernel only issues one write request at a time, and each block write would have to wait for the one before it to complete.

Version 1.1.x supports asynchronous parallel writes out of the block cache, which allows lots of blocks to be written simultaneously. Now you can saturate the network if you so choose, which is as it should be.

An interesting question is should you do the same thing, i.e., parallelization, on the read side? This would involve predictive caching ("read-ahead") of data. I.e., if you get a request to read block zero of a file, assume the next few blocks are going to be needed soon as well and read them all in parallel.

s3fs has these same issues in theory, but at the file level instead of the block level. E.g., one could imagine an application which reads or writes a bunch of files at once sequentially. Each operation would have to wait for the one before it. Of course s3fs could solve that problem the same way, i.e., using an asynchronous writer thread pool.

So there is a trade-off between file vs. block access and it probably depends on your particular situation which "granularity" is best.

Is anyone else having issues with the local cache? No matter what I try files are not being cached and constantly get re-downloaded from S3. Running on an Ubuntu EC2 AMI (feisty) with latest s3fuse (r166). Files are being read successfully (and served by apache), but not cached. Command:

/usr/local/bin/s3fs my_bucket /mnt/files -default_acl=public-read -ouse_cache=/mnt/cache -o allow_other

It looks like you're right as the problem is back. The weird thing is, some instances (1 out of 10 at the most) will cache properly yet most will fail to cache at all.

I am creating /mnt/cache on boot (in rc.local) and chmod'ing immediately afterwards to 777, so I don't think its a permissions problem on the directory - s3fs is running as root anyway.

Without local caching being consistent s3fs is pretty much unusable for me right now :(

Hi- just to clarify what's happening here... when you say "cached" and "no cache"... do you mean cache hit vs cache miss? that is, w/cache enabled, when s3fs downloads a file, it places it in the cache folder... however, that's not a "hit" or anything, only when the second and subsequent requests come in does s3fs look at the cache folder to see if there is a cache hit.

so, having said that, sounds like what you're seeing is (a) the local cache folder is being populated but (b) there are never any cache hits because, despite the fact that the file is in the local cache folder, s3fs still seems to download it (again) from s3

does that should about right?

By "no cache" I mean the file is not actually being saved to the cache folder.

Hi bmilleare- I've done a checkin of r177 that fixes a subtle stale curl handle/timeout issue; there could conceivably be some sort of interaction between that and local cache; so, if you're still interesting in resolving this, feel free to do a svn checkout of r177 and rebuild and retest and report your findings?!? I'm currently stumped on this one, so, even if this fix doesn't solve the local cache issue, it will still be addition info/help narrow things down... Thanks!

Building on CentOS 5.1 x86_64 results in gssapi_krb5 errors. s3fs.cpp: At global scope: s3fs.cpp:439: warning: ‘size_t readCallback(void, size_t, size_t, void)’ defined but not used /usr/bin/ld: skipping incompatible /usr/lib/libkrb5.so when searching for -lkrb5 /usr/bin/ld: skipping incompatible /usr/lib/libkrb5.a when searching for -lkrb5 /usr/bin/ld: cannot find -lkrb5 collect2: ld returned 1 exit status make: all? Error 1

dunno... are you running "configure" on libcurl? if so try --disable-krb4 --without-krb4 (just guessing)

any predictions when this fs will work with EU buckets? We have really slow connectivity to the US buckets from our ISP and EU buckets should be much faster, so it would be great if we could use those...

yes, EU bucket support is on the TODO list! no ETA, just hafta find time to have a look!

On Ubuntu do: sudo apt-get install fuse-utils

On my 8.04 desktop installation it is installed by default.

How do I stop it?

On OS X 10.5.4, using Finder to copy (drag and drop) gives an Error -36. ("The Finder cannot complete the operations because some data in 'YourFile?.txt' could not be read or written. (Error code -36).") The resulting file has file size zero.

However copying from the command line (using cp) works perfectly.

Sounds like this might be a MacFuse? issue, but I thought I'd mention it here just in case.

Thanks for the excellent work.

answer might be in  issue 30  http://code.google.com/p/s3fs/issues/detail?id=30

Hey rrizun, Does this work for Microsoft Windows?

Hi koficharlie- its Linux and MacOSX for now

Hi,

I've been using various S3 tools prior to s3fs, most recently "s3cmd" by Michal Ludvig (http://s3tools.logix.cz/s3cmd). It seems like s3fs would be a lot more convenient for me, but I'm having the issue that s3cmd and s3fs don't seem to be able to see each other's files.

I mount the s3fs volume, and there are no suspcious errors in /var/log/messages - I can mkdir a directory, unmount, remount and that directory is still there, along with its files. However, I can't see the files that s3cmd created in s3fs, and neither does s3cmd see the directory and/or files that s3fs created. I can't see any extra buckets being created in s3cmd either, so I don't think it's that the bucket name is wrong, although the name does include a hyphen - I don't know if this causes any problems somehow.

Any ideas what the problem might be? Or should I not be surprised that these two packages don't seem to read each other's entries?

Hi cartroo-

I've heard of s3cmd but have never used it until now; looks like s3cmd does not really have any concept of folders

in general, the various s3 client programs each have their own scheme for files and folders

s3cmd should be able to see files/folders created with s3fs, in raw form

s3fs should be able to see files create with s3cmd as long as the s3cmd commands issued are "compatible" with the way s3fs wants to view things; you probably don't really wanna do that though

(I'm typing this while listening to 3 other people, so, hope that makes sense!)

df shows that an s3fs filesystem is mounted; but it does not tell me which bucket it is.

How can I tell which bucket s3fs has mounted?

>>> How can I tell which bucket s3fs has mounted?

You can't if you're using df, unless, e.g., some sort of naming convention is followed.

You can, however, use something like "ps ax | grep s3fs".

I'm running an EC2 instance and using s3fs actively with 6 different buckets. The problem I'm seeing is that s3fs is consuming a great deal of memory, and does not seem to release it. I'm using the newest source, without caching.

Is this truly a memory leak, or some other component not working right ie) fuse or curl.

Here's the info from top:

2129 root 21 0 24164 1804 1292 S 0.0 0.1 0:00.03 s3fs 2135 root 20 0 343m 74m 2204 S 0.0 4.4 2:10.74 s3fs 2143 root 22 0 41784 3136 1848 S 0.0 0.2 0:02.57 s3fs 2151 root 24 0 24164 1804 1292 S 0.0 0.1 0:00.01 s3fs 2161 root 25 0 32568 2412 1804 S 0.0 0.1 0:00.01 s3fs 2168 root 20 0 60556 4196 1848 S 0.0 0.2 0:00.13 s3fs 2176 root 24 0 951m 472m 2184 S 0.0 27.8 0:23.31 s3fs

Any assistance would be appreciated.

Hi- what is the nature of the s3fs memory consumption? just trying to characterize the problem: does it slowly ramp up over time or does it consume that much memory right away? as well, are the files large in size? are there directories with lotsa files? Thanks

A quick summary would yes to all ! Files range 4-5mb for flv, lotsa php script, and we use a three tier folder lay out, so always 3 sub-directories in most cases.

Thanks for your quick response !

Hi- about the only s3fs resource I can think of that's unbounded is curl handles; s3fs maintains a pool of persistent curl handles; if s3fs needs a curl handle and the pool is empty then it allocates a new curl handle and then returns it to the pool; under normal use I can see the pool having, say 50 curl handles, however, under heavy concurrent use it could be 200-300+; not sure off hand how much memory a curl handle consumes; might want to monitor the s3fs process using "top" and then use s3fs in a highly concurrent manner and see if there is a continuous memory consumption ramp up

Hi Martin- I did have a peek at what it would take for EU bucket support; the original US bucket naming scheme that s3fs uses is different and not compatible with the new EU bucket naming scheme (e.g., mixed case, underscores, etc...), see http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?BucketRestrictions.html if s3fs were to use the new EU bucket naming scheme then that would break s3fs for existing buckets that were created using the original US bucket naming scheme; is the subcloud binary an option?

Hi voegtlin- indeed, sounds like a bug; s3fs sets content-type based on file name extension; rsync does its initial upload with a temporary file that ends with random characters, since those random characters do not match any known file extension from /etc/mime.types, s3fs sets the content=type to octet-stream; having said that, s3fs "rename" would need to be enhanced to re-lookup the content-type based on the new file extension during rename

thank you for the explanation. I have another question. I have noticed that 'stat' returns a size zero if a file has been recently created. I create a file (about 20K in size) and do a 'stat' right after it is created, to check that its size it correct. The returned size is zero. However, if I add a 'sleep 0.1' command between file creation and the 'stat' command, then the size is correctly returned. is this a bug ? is there a way to avoid this ?

Hi voegtlin- do you see this behavior 100% of the time? or is it intermittent? as well, are you doing the stat in parallel before the 20k file has finished uploading? or is everything being done serially? any feel for if it is "eventual-consistency" related?

no, I do it sequentially, and it happens 100% of the time. the file is a thumbnail of an image, that is created with 'convert'; perhaps it has to do with convert ?

dunno if this has anything to do with it http://developer.amazonwebservices.com/connect/thread.jspa?threadID=25535&tstart=0

can you post a script that reproduces the issue?

I've seen references to that before; don't know what its all about http://www.google.com/search?hl=en&q=fuse+BUG%3A+soft+lockup+detected+on+CPU%231!&btnG=Search

works for me on Ubuntu 8.10 server after loading the required packages. I note that AWS returns a "403 Access Denied" error if a nonexistent bucket name is specified - this caused me to spend a long time entering the authentication information in slightly different ways until I used the jets3t browser to note that the bucket I think of as "default" when using Jungle Disk has a much longer name from AWS' (and hence s3fs') perspective. with the bucket name fixed, everything's great.

Regarding the rsync performace: We have found that it is even up to 4 -5 times faster to mount a bucket on an EC2 instance and make a remote rsync to the instance (e.g. tunneled over ssh) than mount s3fs locally and make a rsync to the mounted directory. And less expensive too. I might write a simple howto about this method.

Michal Frackowiak http://michalfrackowiak.com

search this page for "apt-get"

Thanks mate!

Another question, Can I mount the same S3 bucket using S3fs on multiple servers? if so, any known problems?

Thanks!

no known problems other than the usual concurrency issues, e.g., no different than NFS mounting the same folder from more than one machine

Thanks!

don't know why files are not appearing from the perspective of the second instance... same accessKeyId? same bucket?

as far as s3fox goes, I believe s3fox uses a different scheme for representing files/folders that is incompatible with s3fs; I recommend using jets3t instead because it makes no assumptions about the contents of an s3 bucket

I'm able to mount my bucket and see my file listing in it, but I'm getting input/output errors. I'm assuming that this is not a problem with my credentials or bucket since I'm getting a listing, I don't have a clock skew. I'm seeing the following:

ls -l /mnt ls: cannot access /mnt/dir1: Input/output error ls: cannot access /mnt/list: Input/output error ls: cannot access /mnt/dir2: Input/output error total 0 ?????????? ? ? ? ? ? dir1 ?????????? ? ? ? ? ? list ?????????? ? ? ? ? ? dir2

I've reproduced this on two separate Ubuntu 8.10 machines. I'm not seeing anything in /var/log/messages. Any suggestions for some things I can try to get this working?

Sorry, let me reformat that output:

# ls -l /mnt
ls: cannot access /mnt/dir1: Input/output error
ls: cannot access /mnt/list: Input/output error
ls: cannot access /mnt/dir2: Input/output error
total 0
?????????? ? ? ? ?                ? dir1
?????????? ? ? ? ?                ? list
?????????? ? ? ? ?                ? dir2

probably trying to read a bucket w/objects that were created by another s3 tool? if so then s3fs is looking for additional meta data and not finding it

These files were uploaded by s3sync.rb, the Ruby rsync clone... Anyway to fix this?

no real way to fix it because it is not really a problem per-se; s3fs and s3sync.rb do not understand each other's formats; solution would be to re-upload files w/s3fs (alternatively, figure out which meta data needs to be set to make s3fs happy then use another s3 tool to set meta data on those files uploaded w/s3sync.rb)

So is the official answer, at least for now, that s3fs only works with objects uploaded by s3fs, and not by other tools like s3fox? Does anyone know an easy way to convert the objects?

Could somebody give a rough idea of costs please? Do you pay for data transferred, or by transaction? Does the storage cost depend on the amount of data in your bucket, or just the bucket size?

Can you join buckets for larger storage sizes?

http://aws.amazon.com/s3/#pricing

I'm seeing the same error as joeauty. I mount a new empty bucket and the process finishes with no error. However, when I try to go to the mounted directory, it tells me permission is denied.

ls -l returns

ls -l
ls: cannot access s3: Permission denied
total 0
d????????? ? ? ? ?                ? s3

Any clue what's going on here? S3FS is perfect for my needs and I'd really like to get it running.

Thanks

Does S3FS and/or the underlying S3 storage provide any form of integrity checking? I.e., is there any way a file can be corrupted during upload that would not be detected?

s3fs does not check Content-MD5

I'm trying to figure out why rsync'ed files to S3 wouldn't be viewable in Firefox/Google Chrome. For example, if I try to view a .htm or .jpg file, it prompts me to open or save the files. Oddly enough they view fine in IE. Any ideas? Maybe s3fs isn't recognizing my /etc/mime.types?

that is possible; in ff, what does "Tools -> Page Info" say about the Content-Type?

ah, I know what the problem is: s3fs does not consult mime.types on renames... when rsync copies, it copies to a tmp file and then renames it at the end, thus the loss of s3fs content-type... to fix it, just add these lines to s3f3_rename

meta["Content-Type"] = lookupMimeType(to); meta["x-amz-metadata-directive"] = "REPLACE";

(Note- NOT tested!)

Is the size of the local file cache constrained? Or will it just cache files until the disk fills up? What happens when the disk is full?

I'm mounting an S3 volume to multiple servers. When a user uploads an image to one of the servers, I want it stored on the S3 bucket so all servers can see the same images no matter which server uploaded it or which server is serving their page.

The bucket mounts to the instances without trouble, but I'm having trouble getting the permissions correct so that the apache user can write to the bucket. The bucket is symbolically linked from my web directory to /mnt/mybucket. I've also done the -o allow_other command when mounting the bucket. The bucket is set as world readable and I've tried everything from owner writable to world-writable. If I look at the directory permissions, it is showing rwxr-xr-x.

Is there a way to do this???

A workaround for the rsync/mime-type issue mentioned on "Feb 14, 2009" is to use the "--inplace" option with rsync. This forces it to write to the correct filename to start with, rather than writing to a temp file and renaming it. Which means the right content-type is set.

>>> The '.' and '..' entities are not listed with 'ls -a'. Is this the expected result?

yes

Hello! I've been using it for a few days in my EC2 server (An Ubuntu x86 Server instance). Yesterday, one of my s3fs mounts stopped working without reason, and would print "Transport endpoint is not connected" as a result of any try to access it. Re-mounting it fixed the problem (Temporarialy?).

Running ls -l in the parent folder displayed the "defective" mount point in red, with plenty of question marks:

ls: cannot access videos.example.com: Transport endpoint is not connected
total 0
d????????? ? ?    ?    ?                ? videos.example.com
drwxr-xr-x 1 root root 0 1969-12-31 19:00 www.example.com

In my /var/log/messages, I have a s3fs segfault (This one-line entry is the only interesting thing...)

Mar 10 13:48:49 domU-12-31-39-00-7D-13 kernel: [1132444.746495] s3fs[30280]: segfault at 0 ip b7bbc323 sp b6fe1fec error 4 in libc-2.8.90.so[b7b45000+158000]

This is everything I could find - Although it probably doesn't help much... :( Has anybody else had this issue? Any chances to get it fixed? (May I help? How?)

Anyway, Thanks for this great software ;)

I noticed that ls -l can return d????????? when a directory name contains a trailing /, which can be present if some other s3 clients have interacted with the bucket. I don't know if this applies in your case.

Hi - I would just like to add my voice to the requests for an Amazon S3 EU-compatible version of s3fs, please? :-)

Mieses, I didn't have any directory inside the crashed bucket (Just a long and messy list of files), so I think the "/" problem probably doesn't affect me.

And no other client was active when it crashed.

Crashed again :( Is anybody else running s3fs in a server? I'm getting these random crashes every 1-2 days. :(

I am backing up a lot of data with a script and rsync. All was well until i modified my script with nano and because the cache had filled my os drive (small 10gb) my edit wasnt saved and i lost my script. Currently disabled the cache. Is there anyway to manage the cache ? Is it really required if mainly writing data ?

I downloaded the "featured" code bundle s3fs-r177-source.tar.gz and compiled it on my Ubuntu 8.10 box. It never managed to establish a proper connection (I didn't get to the bottom of why) but my S3 Account Usage says I've made 1.3 million requests and will be charged accordingly! I imagine the code must have been looping, but that could prove to be a costly loop for me...

st...@bov.nu- there is no local cache management; you can use a cron job to periodically purge the local cache; in your case it sounds like you probably don't even need local cache-

paulo.raca- I'm not aware of any crash conditions in the s3fs code itself; wondering if its one of the libraries? can you get a coredump and invoke gdb on it for a traceback?

A cautionary tale (and some debugging tips) about using this with SSL: If you are going to use -ourl=https://s3.amazonaws.com, make sure you do not have a trailing slash on the end of the URL (I did this at first, and it took quite a lot of effort to work out why). If it works without SSL but not with it, a good thing to check is that you don't have a trailing slash on the end of your URL.

In case you are getting a different problem, here are some debugging tips I worked out while debugging this:

It writes information to syslog - check /var/log/syslog or where your system logs to for lines, which look like this:

Mar 21 16:47:40 amlap s3fs: ###timeout
Mar 21 16:47:40 amlap s3fs: ###retrying...
Mar 21 16:47:42 amlap s3fs: ###response=403

If you get an error response (like 403), AWS sends back a response in an XML language explaining the error. The problem is you can't see the error. If the problem occurs for HTTP too, the easiest way is to use wireshark or tcpdump or another packet capture program to spy on what s3fs is doing and read the error message. If, like I was, you are encountering a problem exclusively for SSL, however, read on.

My problem meant that readdir (triggered by ls), as well as practically every other call, were not working. For ease of debugging, I chose to debug what happened when readdir is called.

s3fs sets the curl option CURLOPT_FAILONERROR, which makes it hard to get the output. So I went into the function starting with: s3fs_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi) { and changed the following line: curl_easy_setopt(curl, CURLOPT_FAILONERROR, true); to: curl_easy_setopt(curl, CURLOPT_FAILONERROR, false);

Change this line and recompile by running make.

This is only a temporary change for debugging - it will cause problems if used in production, so don't forget to change the line back and recompile once you solve the problem.

s3fs forks when it runs, so the best way to debug it is to start it normally from the command line, and then attach to it with gdb... ps ax |grep s3fs From here, get the PID of s3fs, and run

gdb ./s3fs 1743

  break writeCallback
  break calc_signature
  cont

Now trigger the problematic request with the ls command from another shell, and change back to the gdb shell.

A breakpoint will hit in calc_signature as follows:

Breakpoint 3, calc_signature (method=
        {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x42037dc0 "h\006G\001"}}, content_type=
        {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x42037db0 "8\020���\177"}}, date=
        {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x42037da0 "�\005G\001"}}, headers=0x1479620, resource=
        {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x42037d90 "H\003G\001"}}) at s3fs.cpp:393
393	calc_signature(string method, string content_type, string date, curl_slist* headers, string resource) {
(gdb) next
394		string Signature;
(gdb) 
395		string StringToSign;
(gdb) 
396		StringToSign += method + "\n";
(gdb) 
397		StringToSign += "\n"; // md5
(gdb) 
398		StringToSign += content_type + "\n";
(gdb) 
399		StringToSign += date + "\n";
(gdb) 
400		int count = 0;
(gdb) 
401		if (headers != 0) {
(gdb) 
404				if (strncmp(headers->data, "x-amz", 5) == 0) {
(gdb) 
402			do {
(gdb) 
411		StringToSign += resource;
(gdb) 
413		const void* key = AWSSecretAccessKey.data();
(gdb) 
414		int key_len = AWSSecretAccessKey.size();
(gdb) print StringToSign.c_str()
$1 = 0x1470378 "GET\n\n\nSat, 21 Mar 2009 03:50:45 GMT\n/wwjcode"
(gdb) cont

Next, it will likely break in writeCallback, which you can debug like this...

Breakpoint 2, writeCallback (data=0x1484c8b, blockSize=1, numBlocks=707, userPtr=0x42037ec0) at s3fs.cpp:451
451	  string* userString = static_cast<string*>(userPtr);
(gdb) next
452	  (*userString).append(reinterpret_cast<const char*>(data), blockSize*numBlocks);
(gdb) 
453	  return blockSize*numBlocks;
(gdb) 
454	}
(gdb) printf "%s", (*userString).c_str()
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>47 45 54 0a 0a 0a 53 61 74 2c 20 32 31 20 4d 61 72 20 32 30 30 39 20 30 33 3a 35 31 3a 31 33 20 47 4d 54 0a 2f 2f 77 77 6a 63 6f 64 65</StringToSignBytes><RequestId>89EED394FEBE2205</RequestId><HostId>T+7tNvxde2qrUSUkIxY9vCv0uOeqDsa6yZgHdfR3Txuqs+Ql7YudvnanvDEOUDgt</HostId><SignatureProvided>nHjQiDuIpSUCWpCe2z6g8XBU2n4=</SignatureProvided><StringToSign>GET


Sat, 21 Mar 2009 03:51:13 GMT
//wwjcode</StringToSign><AWSAccessKeyId>1VBXKK4V5WVRSTTY0JG2</AWSAccessKeyId></Error>(gdb) 

In this case, you will see that the string we signed has /wwjcode in it, and the expected string has //wwjcode, because of the extra slash on the end of the URL. It is likely you will have some different type of error, but hopefully this is enough to get you start in debugging the problem.

Don't forget to change the

  curl_easy_setopt(curl, CURLOPT_FAILONERROR, false);

back to true and re-run make or you will get hung requests!

I am using rsnapshot with s3fs as the target for offsite backups. My rsnapshot logs show the following error:

[09/Jul/2009:23:50:45] /usr/bin/rsnapshot daily: ERROR: /bin/cp -al /mnt/s3/daily.0 /mnt/s3/daily.1 failed (result 256, exit status 1). Perhaps your cp does not support -al options?

Specifically, the -l is what's making it fail. The -l flag tells cp to link rather than copy, and a manually-invoked cp -l shows the following:

# cp -l /mnt/s3/test /mnt/s3/test2 cp: cannot create link `rsnap2': Operation not permitted

Hard links on s3fs are not supported. Symbolic linking, however, seems to be supported, with cp -s succeeding.

Is hard link support something that can be implemented?

Hi- hard links imply reference counting, so, unless amazon s3 supports references in the future, it is unlikely that s3fs will support hard links

Hello,

I am having some trouble with the current s3fs on MacFUSE 2.0.3 running on OS X 10.5.7. I can successfully mount a bucket and see its contents, but then can not copy new files into it. If I try to copy a file into S3, I will get an "Invalid argument" error and an empty file.

$ cp ~/Desktop/api-design.pdf .
cp: ./api-design.pdf: Invalid argument
cp: /Users/jchang/Desktop/api-design.pdf: could not copy extended attributes to\
 ./api-design.pdf: Attribute not found
$ ls -lh
total 48
[...]
-rw-r--r--  1 jchang  staff    -1B Jul 20 22:04 api-design.pdf

However, if I repeat the operation, the "Invalid argument" error goes away (although I still get an "Attribute not found" error, and the file now exists in S3.

$ cp ~/Desktop/api-design.pdf .
cp: /Users/jchang/Desktop/api-design.pdf: could not copy extended attributes to\
 ./api-design.pdf: Attribute not found
$ ls -lh
total 456
[...]
-rw-r--r--  1 jchang  staff   203K Jul 20 23:46 api-design.pdf

Furthermore, drag-and-drop from the Finder generates an error and an empty file:

The Finder cannot complete the operation because some data in
<filename> could not be read or written. (Error code -36)

Any ideas on why this might be happening? I have tried s3fs on the same Amazon bucket on a similar configured computer at work, and I do not see this error. How to diagnose? I'm wondering if it might be network related?

Jeff

What steps will reproduce the problem? 1. Compile s3fs.cpp with a modification : #define FUSE_USE_VERSION 26 #define off_t off_t 2. Mount s3fs bucket on amazon, using the command ./s3fs mybucketxxx ~/Desktop/CCCS3 -olocal,ping_diskarb,volname=CCCS3 3. hdiutil create MacHD -size 50G -type SPARSE -fs HFS+ -layout GPTSPUD -stretch 50G -volname MacintoshHD followed by an attempt to mount this : hdiutil attach MacHD.sparseimage

What is the expected output? What do you see instead?

Expected output - mounted sparseimage Seen :

$ hdiutil attach MacHD.sparseimage hdiutil: attach failed - Illegal seek

What version of the product are you using? On what operating system?

Please provide any additional information below.

Version 26 Operating system : Mac OSX Leopard 10.5.7

Yes, at work the computer is also running OS X with MacFUSE 2.0.3.

I have tried a third computer at home (also running OS X 10.5.7 with MacFUSE 2.0.3), and the third computer does not have the problem. So the problem seems to be isolated to that one computer and not the network. It is a mystery to me why different computers running the same software would respond differently.

Based on the error messages, I feel that maybe the issue has something to do with the way OS X deals with extended attributes (an issue that Linux doesn't have(?)). I'll dig deeper into this and post if I find anything useful.

My problem seems to be the same as  issue 49  "Can't upload files". Unfortunately, there's no solution for that one either.

Have been using s3fs for some time now, and have begun to have problems with folders with many files. In one case there are 10000 files in the bucket, and any filescan (dir/ls) results in an s3fs lockup, and in one case, server reboot. The command line used is: s3fs -o allow_other fmc_data -o retries=15 -o connect_timeout=8 -o readwrite_timeout=40 /data No caching is used, and have modified the readdir routine, increasing max-keys to 150 seeing a small performance increase. Am tempted to raise this even more, since there are so many files. Am i heading in the right direction ?

Things work beautifully for me except for any operation that involves permissions changes. For instance, a standard "cp filename dest" works fine, but "cp -p filename dest" does not. rsync, which is what I'm trying to ultimately use does a chmod automatically with the same results. A straight chmod errors out as well. All of these errors I believe are related so I think fixing one thing will most likely resolve everything.

# chmod 777 testfile.zip chmod: changing permissions of `testfile.zip': Input/output error

#/usr/bin/rsync -ru /u01/backup/exports/ /s3/backup/exports rsync: rename "/s3/backup/exports/.testfile.zip.FQOmch" -> "testfile.zip": Input/output error (5) rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]

The rsync is running as root and files are owned by a lesser privileged user, if it matters.

CentOS 5.3 s3fs r177 curl 7.15.5-2.1.el5_3.5

Any ideas?

Hi,

for those running into problems dealing with European buckets I made a small fork working with European buckets url schema.

Could be found at http://github.com/tractis/s3fs-fork

Hope it's useful!

Dave

great to see european bucket support - hope this will be merged back into main development.

anybody know why using rsync over s3fs is very (very) slow? thanks

try rsync --inplace

why does the filesize show up incorrectly on a zero byte file? the same for directories, the size that shows up on a directory is ridiculously huge. this seems to break some applications that aren't happy with opening a file for writing, running stat on the file, and finding out the file size is wrong.

for example ...

# touch foo # ls -l foo -rw-r--r-- 1 root root 18446744073709551615 Oct 18 01:51 foo # mkdir bar # ls -ld bar drwxr-xr-x 1 root root 18446744073709551615 Oct 18 01:53 bar

nevermind, I figured out the problem. it turns out that starting with curl 7.19.4, it returns -1 if the Content-Length is not known . See my post here for further details and a quick and dirty patch.

http://code.google.com/p/s3fs/issues/detail?id=50#c5

rrizun, can you please commit my patch to trunk (and cleanup the patch if necessary) for those of us running a newer version of curl?

Anybody successfully using tractis/s3fs-fork? I get Input/output error after mounting and doing an ls ...

TIA

When I try to install I get the following error:

root@localhost s3fs]# make Package 'libcurl' has no Version: field g++ -ggdb -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -pthread -L/lib64 -lfuse -lrt -ldl -I/usr/include/libxml2 -lxml2 -lz -lm -lcrypto s3fs.cpp -o s3fs s3fs.cpp:365:25: error: openssl/bio.h: No such file or directory s3fs.cpp:366:28: error: openssl/buffer.h: No such file or directory s3fs.cpp:367:25: error: openssl/evp.h: No such file or directory s3fs.cpp:368:26: error: openssl/hmac.h: No such file or directory s3fs.cpp:500:25: error: openssl/md5.h: No such file or directory

I have installed openssl. Does s3fs not know where my installation is?

I want to "publicly" serve files with obscure URLs that include a directory component, e.g. ab/cd/efgh.jpg but I don't want ab or ab/cd to be listable. It looks like I'm OK, in that when I try to access a directory via an s3 URL, what I get is an empty file. Am I right that the directories are inaccessible except to s3fs? So if I don't give public read access to the bucket, then nobody can list anything?

ourdoings:

I think the ability to list a buckets's contents is the property of the bucket's ACL, not the object's ACLs. If the bucket was created without specifying an ACL then I don't think anyone can get a listing of files, even if the files themselves are readable.

Caching doesn't seem to be working for me. I compiled and installed r177 without a problem on FC8 (on EC2). Everything other than caching seems to work fine. Here's a little shell dump:

# mkdir /s3
# chmod 777 /s3
# mkdir /s3tmp
# chmod 777 /s3tmp
# /usr/bin/s3fs testbucket2543 /s3 -ouse_cache=/s3tmp
# ls /s3
foo.txt
# ls /s3tmp
# cp /s3/foo.txt /tmp/
# cat /tmp/foo.txt
hai!
# ls -la /s3tmp
total 8
drwxrwxrwx  2 root root 4096 Nov 23 02:49 .
drwxr-xr-x 27 root root 4096 Nov 23 02:49 ..

Any ideas?

hi there,

Can anyone tell me where s3fs stores it's temporary files? E.g. I have a 3GB root disk (/) and and 300GB /mnt partition. When I am copying a 5GB file from or to S3 from my /mnt/test folder, I notice the disk use on the / filesystem goes up and up and then I run out of space. Is there anyway round this? I have just had a cp fail and now my / disk is at 100% usage but I cant find the files that s3fs has placed on the / file system.

Any help much appreciated.

s3fs uses tmpfile() to get temporary files; the os usually decides where it goes (though can typically be influenced via env vars)... try /tmp ?

I have checked /tmp and also checked the whole filesystem using a find script but can't see anything. On the machine (an EC2 instance) which had the failed copy, I had to unmount the s3 drive and the disk usage went back down to normal.

Lets say that the tmpfile() path was /tmp. Is it the case that s3fs would copy the file from /mnt/test/file.mpg to /tmp/file.mpg before copying it to /mnt/s3/file.mpg if I was to do:

% cp /mnt/test/file.mpg /mnt/s3/file.mpg

Or would it copy it directly to /mnt/s3/file.mpg ?

Disk usage on my box would indicate it does the first option.

currently no plans; I did have a version that did use range: headers exclusively, however, there was a significant performance hit; best solution would be a hybrid approach: try the full download first, and if app asks for any bytes 'out-of-order' then do specific range: gets

I like the hybrid approach. My application needs to read small parts of large images (think tens of GBs). We'd love to have that feature available to us.

I'd like to set-up an FTP user account with a mounted s3 bucket as the users home folder, Has anyone tried this or know if it would be possible. I have tried but the folder is not visible via ftp client. My knowledge does not extend to why this would not work, could any body shed some light?

Hi- try using the fuse allow_other option, e.g, -oallow_other

If you have a bucket with files and directories that were not created with s3fs, and you mount that bucket using s3fs, then list contents of that directory(bucket), you will only see files, not directories.

using s3fs you will only be able to see directories that were created using a mounted s3fs directory and mkdir. if you use another tool to delete that directory, listing the s3fs mounted directory will still show the deleted directory. you can even cd into it. it will have no contents. you can safely rmdir it.

Would you minnd adding http://code.google.com/p/s3ql/ to the list of other S3 file systems? I would also appreciate your input to http://code.google.com/p/s3ql/wiki/comparison

http://code.google.com/p/s3ql done

hi folks was wandering if anybody can point me in the right direction. I compiled the app with no problem and mounted the bucket. (now i am able to serve the images from web so permissions are correct) yet when i try to look with in my mounted drive i get nothing.

I triple checked all the my security codes and such and everything seems correct.

Any idea?

I updated from r188 to r191 and now I get I/O error on all operations with the syslog reporting response 403. If I switch back to r188 with the exact same mount command, it works.

Syslog contains the following (bucket name changed)

 URL is http://s3.amazonaws.com/MyBucket?delimiter=/&prefix=&max-keys=50
 URL changed is http://MyBucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=50
 connecting to URL http://MyBucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=50 
 ##response=403 

Any ideas?

the eu bucket changes in r191 break 'old style' bucket names (e.g., mixed case bucket names); for r191, bucket names must now conform to http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?BucketRestrictions.html

solution is either (a) stay with r188 or (b) copy contents of MyBucket? to a new dns compatible bucket

hope that helps!

Yes, that explains it. Thanks!

s3fs seems to log a massive amount of information to /var/log/messages, /var/log/syslog etc. Is it possible to somehow lessen the load on these logs?

Every single file operation is logged to messages and in syslog it complains about changing the URL (I have a EU based bucket, but a bucket name that is DNS valid fx. "xxxx-gggg")

(running debian 5.0)

We are trying to use s3fs mount only for writes and not reads. Has anyone experience with this kind of setup. Does it scale? Is it stable after say 100K files uploaded etc. Any suggestions would be appreciated.

Thanks

for Ubuntu Jaunty I had to install:

sudo apt-get install libfuse-dev libxml2-dev libssl-dev

I think I've found a problem with truncate() and size handling interacting with the use_cache.

I'm using rsync with the --inplace option to upload files to s3 via s3fs, and I have a file that has become smaller on my local machine. This file is appearing on S3 with the original size, rather than the new smaller size.

Under these conditions, rsync appears to do a truncate() down to the new size, but then when flush() is called, the file in the s3fs local cache is still the original size, so put_local_fd() uploads the entire original file size.

I would appear to have fixed this by making the following change in s3fs_truncate():

Instead of:

  auto_fd fd(fileno(tmpfile()));

I have:

  auto_fd fd(get_local_fd(path));
  ftruncate(fd.get(), size);

I'm not sure if there are any other side-effects that should be taken into account, but that seems to have fixed the problem I was seeing, anyway.

A simple question but: Can this be used by multiple machines at the same time. I would like to use this to share data between two load balances servers easily.

I found a problem in the latest code:

Mar 14 18:53:41 jajo s3fs: URL is http://s3.amazonaws.com/bucket?delimiter=/&prefix=&max-keys=50
Mar 14 18:53:41 jajo s3fs: URL changed is http://bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=50
Mar 14 18:53:41 jajo s3fs: connecting to URL http://bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=50
Mar 14 18:53:41 jajo s3fs: ###response=400

The missing '/' before '?' in the rewritten url is causing a 400 error for me.

Here is a little patch that fixes this:

--- s3fs/s3fs.cpp.orig	2010-02-05 01:45:59.000000000 +0100
+++ s3fs/s3fs.cpp	2010-03-15 09:46:56.000000000 +0100
@@ -296,7 +296,10 @@
 	int bucket_pos = url_str.find(token);
 	int bucket_size = token.size();
 
-	url_str = url_str.substr(0,7) + bucket + "." + url_str.substr(7,bucket_pos - 7)  + url_str.substr((bucket_pos + bucket_size));
+	if (url_str[bucket_pos + bucket_size] != '/') 
+		url_str = url_str.substr(0,7) + bucket + "." + url_str.substr(7, bucket_pos - 7) + "/" + url_str.substr((bucket_pos + bucket_size));
+	else
+		url_str = url_str.substr(0,7) + bucket + "." + url_str.substr(7, bucket_pos - 7) + url_str.substr((bucket_pos + bucket_size));
 
 
 	syslog(LOG_DEBUG, "URL changed is %s", url_str.c_str());

I get number (2^64)-1 for all directories and empty files when listing s3fs file system (ubuntu-amd64 r191):

# ls -l
drwxr-xr-x 1 root root 18446744073709551615 2010-03-25 08:57 artis

# cd artis/
# touch empty
# ls -l
-rw-r--r-- 1 root root 18446744073709551615 2010-03-25 08:57 empty

When trying to view a bucket using https I get an input/output error. The exact same configuration works fine for http. It works fine with r190 but r191 and above it fails. There's no information in the debug messages saying why the error occurs.

If I get a chance I'll try and fix it and provide a patch but if anyone else can figure it out before me that'd be great.

I'm using ubuntu 9.10 and libcurl 7.19.5 I get the same issue on ubuntu 9.04 with libcurl 7.18.2

is there any way to mount nested bucket? (like dev/sample_bucket)

patch against 191, preserves mount points owner information allowing you to mount other filesystems (like encrypted fuse filesystems) over s3fs

diff -r 8ae498df8035 s3fs.cpp
--- a/s3fs.cpp  Sun May 09 12:34:49 2010 -0500
+++ b/s3fs.cpp  Mon May 10 09:35:15 2010 -0500
@@ -345,7 +345,7 @@
 static string AWSAccessKeyId;
 static string AWSSecretAccessKey;
 static string host = "http://s3.amazonaws.com";
-static mode_t root_mode = 0;
+static struct stat root_stat;
 static string service_path = "/";
 
 // if .size()==0 then local file cache is disabled
@@ -807,8 +807,8 @@
        cout << "getattr[path=" << path << "]" << endl;
        memset(stbuf, 0, sizeof(struct stat));
        if (strcmp(path, "/") == 0) {
+               *stbuf = root_stat;
                stbuf->st_nlink = 1; // see fuse faq
-               stbuf->st_mode = root_mode | S_IFDIR;
                return 0;
        }
 
@@ -1278,6 +1278,10 @@
 s3fs_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi) {
        //cout << "readdir:"<< " path="<< path << endl;
 
+       // add .. and .
+       filler(buf, ".", NULL, 0);
+       filler(buf, "..", NULL, 0);
+
        string NextMarker;
        string IsTruncated("true");
 
@@ -1593,7 +1597,7 @@
                        struct stat buf;
                        // its the mountpoint... what is its mode?
                        if (stat(arg, &buf) != -1) {
-                               root_mode = buf.st_mode;
+                               root_stat = buf;
                        }
                }
        }

have not tried but it should (was able to compile/link/run on fedora/ppc and macosx/intel)

I've found the mount point doesn't have "." and ".." entries:

% mount | grep fuse                
/dev/fuse0 on /fs21 (fusefs, local, nosuid, synchronous, mounted by bacula)
% sudo -u bacula ls -al /fs21
total 0
% 

I think that's the reason why bacula doesn't want to use that directory:

ERR=dev.c:549 Could not open: /fs21/FULL4720, ERR=Not a directory

Hi,

I am trying to use S3FS r188 to mount my Eucalyptus Walrus buckets in the local machine. The mount succeeds, but when I do a cd to the mount point and try to do an ls, it does not return anything and simply hangs. I have to manually kill the S3FS daemon to come out of it. But, other commands like touch and create/save file with vim is working. I don't see any error messages in /var/log/messages and I am running the latest version of curl (7.20). Please help.

Regards, Saurav

When I sniff into the response of GET request(generated by ls command) with wireshark, I see that the response is correct with the listings of files. Thus walrus has responded with the file listings, but somehow S3FS is not able to decipher it and goes to a hang state. Any one encountered similar things ?

I added the following to get content encoding to work for gzipped javascript files:

Add this function anywhere toward the beginning of the code:

static string
get_contentEncoding(string to) {
	// returns whether or not to use content encoding
	string front = to;
	string lastdir = to;
	string extension;
	string::size_type pos = front.find_last_of('.');
	string::size_type pos2 = front.find_last_of('/');
	front = front.substr(0,pos2);
	pos2 = front.find_last_of('/');
	cout << "Front: " << front << endl;
	if (pos != string::npos) {
		extension = lastdir.substr(1+pos,string::npos);
		cout << "." << extension << " Extension Found" << endl;
	}
	if (pos2 != string::npos) {
		lastdir = front.substr(1+pos2,string::npos);
	}
	if (lastdir == "jsz" or extension == "jgz") {
		return "gzip";
	}
	return "";
}

Then add this directly below the line string contentType(lookupMimeType(path)); in function s3fs_mknod:

	string contentEncoding = get_contentEncoding(path);
	cout << "Setting content encoding to: " << contentEncoding << endl;
	if (contentEncoding == "gzip") {
		headers.append("Content-Encoding: "+contentEncoding);
	}

... and lastly, add the following to s3fs_rename() (note, this should also fix the contentType issue when using RSync:

    string contentType(lookupMimeType(to));
    meta["Content-Type"]=contentType;
    string contentEncoding = get_contentEncoding(to);
    cout << "Setting content encoding to: " << contentEncoding << endl;
    if (contentEncoding == "gzip") {
	    meta["Content-Encoding"]="gzip";
    }

Never mind my earlier post - after inspecting the packets with Wireshark I figured out what the problem was. I had to modify the way the string to sign is constructed to add the user token and product token.

I can provide the patch to add support for DevPay? buckets if anyone is interested.

Anyone know why a s3 mount wouldn`t see folders already in the bucket ?

What can this error: "AttributeError?: S3Connection? instance has no attribute 'aws_secret_access_key'." S3fs installed using the command "yum install fuse-s3fs" and when I tried riding my bucket appeared this error.

i know its confusing, but, fuse-s3fs (python?!?) is a different project !!

sorry .. I used "yum install fuse-s3fs" because I could not install S3FS in my fedora on Amazon EC2 instance ... I would spend a step-by-step instructions for installation?

Mr. Rizun,

S3FS works beautifully but two files are visible for each directory written to a bucket - the directory itself and a corresponding zero byte file. This is the case in the AWS S3 console, Cloudberry Explorer, s3//, and CrossFTP Pro. In the case of S3Fox? the zero byte file is actually displayed on top of the directory and must be deleted to expose the underlying directory. Is there any way to stop this using just your open source version of S3FS or do we need to buy your Subcloud product ?

Hi, I'm having an "Input/Output error".

I mount my one bucket on 3 servers, and the credentials haven't changed in a long time. The error only happens on one server, and it's only been like that for less than 24 hours. Before that everything was fine. I checked everything here: - the bucket exists (I checked manually) - date sync is fine (it's done hourly on all my servers) - credentials are ok (I checked manually)

Starting s3fs in the foreground (-f) does not show anything indicating something is wrong, I can see a log whenever I try to list files in my bucket, but that's it. Only after some time the ls command times out, and that's all I know. I don't see anything that catches my eye in /var/log/messages

I recompiled s3fs with the latest source, it's no help.

Any ideas for me ? I have no idea how to debug this, I'm sort of a newbie with networking processes and I don't know where to look.

Thanks, Mikael

Installing fuse on Ubuntu:

sudo apt-get install libfuse-dev 
sudo apt-get install fuse-utils

Why don't fix the directions?

Recently, I ran into the infamous "Input/output error" on a new install of r188. The S3 bucket was fine, and everything else correct, but still was not working.

It took quite a while to discover that my problem had to do with the "+" character that was part of the secretAccessKey. I created a new access key, and substituted that info into the /etc/passwd-s3fs file and all started working.

I just wanted to post this comment in case someone else runs into the same issue.

With v191 under Ubuntu 10.04 64Bit, whenever I transfer something off s3 to the local disk, s3fs appears to create huge cache files somewhere on the root fs, despite being mounted with no cache.

This is extremely frustrating as I almost exclusively have few large files (split tar archives with each part the size of 4GB) to transfer and I cannot throttle the bandwidth with rsync, as rsync only "sees" the file once it come out of the local cache.

Mount syntax was:

/usr/local/bin/s3fs my-archive \
-o accessKeyId=abcedfg \
-o secretAccessKey=1234567 \
-o allow_other \
-o retries=20 \
-o connect_timeout=20 \
-o readwrite_timeout=60 \
/s3archiv

What am I doing wrong here?

It seems downloaded data is being cached in a file which is immedately unlinked after creation, thus cannot be found with usual means.

Question:

• can I switch off local caching entirely when files are being downloaded? So that I can throttle download speeds on application level?
• if not, can I at least tell s3fs to store the cache elsewhere than in /tmp? Please remember, I haven't explicitely activated caching!

BTW: when I explicitely activate caching, I cannot give a max cache size. But this is necessary, as else s3fs would fill the entire filesystem the cache is on if enough data is copied to S3.

On Debian 5.0.6 I could get http but not https to work. Turns out it was rewriting the urls to "https:/bucket./s3.amazonaws.com?..." - an off by one error when assuming the url is always 7 chars long. Here's how I patched to account for "http://" in r191:

--- s3fs.cpp.orig	2010-09-08 05:59:04.000000000 +0000
+++ s3fs.cpp	2010-09-08 05:59:42.000000000 +0000
@@ -296,7 +296,8 @@
 	int bucket_pos = url_str.find(token);
 	int bucket_size = token.size();
 
-	url_str = url_str.substr(0,7) + bucket + "." + url_str.substr(7,bucket_pos - 7)  + url_str.substr((bucket_pos + bucket_size));
+	int url_pos = url_str.find("://") + 3 ;
+	url_str = url_str.substr(0,url_pos) + bucket + "." + url_str.substr(url_pos,bucket_pos - url_pos)  + url_str.substr((bucket_pos + bucket_size));
 
 
 	syslog(LOG_DEBUG, "URL changed is %s", url_str.c_str());

How do I create a mount point in /mnt WITHOUT using sudo/'su root'? It seems I need to have it exist, have write permissions, and specify it at the end of the command. i.e. s3fs blah blah /mnt/mountpoint It won't/can't create the direcotry. I have 'fusermount' suid'd.

Using Ubuntu Lucid Lynx 10.xx

I switched to using the /media directory. First /mnt didn't work at all, couldn't even chown or chmod a directory that I made there using sudo. Second, the latest filesystem specification that I could find:

http://www.pathname.com/fhs/pub/fhs-2.3.html#MNTMOUNTPOINTFORATEMPORARILYMOUNT

said that /mnt was for 'SYS ADMINS' to mount external file systems. That sounds like NOT for s3fs/fuse usage in intention.

I THINK that I am having mixed results with needing to pre create the directories for the mount points. I will wrote about it if it seems so. I don't have to mess with permissions or chown anything to make it work. So for now, I recommend /media on Ubuntu platforms as users, and /mnt for root to use for root purposes.

s3fs is a great project, I use it quite regularly. However, it seems as though the author has seemingly become somewhat disengaged. I say this because of a few observations:

- there hasn't been a release since February 2010 - the author maintains a non-free version of the software which seems to have since addressed some of the issues with the free version - several community members have submitted patches to resolve issues and those patches haven't been folded into a release.

Personally, I would like to see this software project go forward rather than stagnating. It is in pretty good shape and needs some polish. Rather than forking the code under the GPL and of course, giving credit to the original author, would it be possible for the members of the community to pick up maintenance?

hi mooredan- if you'd like to contribute then definitely feel free to pm me via rrizun@gmail.com tx

I noticed that a box on which i use s3fs with some huge buckets (regarding file count) recently slowed to a crawl - this was caused by debian's '/etc/cron.daily/locate' script, which walks through all of the buckets and didn't finish within 24 hours, so it got worse every night. I fixed this by adding the paths of the mounted buckets to the PRUNEPATHS option of that script.

how to connect to a bucket as anonymous?

It seems there is a leak somewhere when using 'https://' it consume little bit by little bit Situation: lot of small files, do 'find .' Option to set 'https': url=https://s3.amazonaws.com I tried to set option 'use_cache' this do not have any result fuse 2.8.5, s3fs 1.0, libcurl 7.19.7-15.16.amzn1, openssl 1.0.0a-1.2.amzn1 2.6.34.7-56.40.amzn1.x86_64

Ubuntu Lucid Lynx, built the code, install, tried to mount a freshly created amazon bucket. (i.e. s3fs mybucket /mnt/s3, using the /etc/passwd-s3fs file)

The bucket has content, but nothing shows up. An ls -l of the /mnt directory, shows that the mounted directory is owned by root.root (normally, that would be owned by tim.tim). The bucket itself is empty.

How do I debug this ?

Minor update to the above: I can create a file in this bucket, and it shows up on in the AWS console Just can't seem to read older files or directories

This is what I see in the syslog Oct 26 20:20:15 Walker s3fs: URL is http://s3.amazonaws.com/mybucket/Folder Oct 26 20:20:15 Walker s3fs: URL changed is http://mybucket.s3.amazonaws.com/Folder Oct 26 20:20:15 Walker s3fs: connecting to URL http://mybucket.s3.amazonaws.com/Folder

? connect to a bucket as anonymous ?

I don't think that you can. You'll need the accessKeyId and secretAccessKey to mount a bucket (or see it through any S3 client). However, you can publish the content to be readable through a URL (web address).

"leak"

Please open an issue providing as much detail as you can so that we can reproduce the issue. Include the full command line option or fstab entry. ...which release you're using too.

Here's a typical way that I mount a bucket (as me, a uid/gid other than root)

/etc/fstab:

s3fs#macbook.suncup.org /mnt/s3/macbook.suncup.org fuse netdev,use_cache=/tmp,use_rrs=1,allow_other,uid=1000,gid=1000 0 0

Note: creating files and/or directories with another S3 client may result in these not showing up correctly using s3fs Be consistent in your usage of a bucket. See  Issue #27 

? connect to a bucket as anonymous ?

? I don't think that you can. You'll need the accessKeyId and secretAccessKey to mount a bucket (or see it through any S3 client). However, you can publish the content to be readable through a URL (web address). ?

I use CyberDuck? on OSX and it can connect to a bucket as an anonymous, I just set up the permissions for "Everybody" to read and it's works, but s3fs want keys from me as I understand you don't need neither of them when you want to connect as an anonymous.

anonymous bucket. Learn something new everyday ;) Sure, I get it now. I'll add an enhancement issue to add support for this. Once well understood, it might be an easy thing to implement.

Hi - can someone explain how to connect to a European Bucket? I'm OK with the US one but what ever I try - I can't seem to get to the European one. Thanks!

EU bucket. I don't know - but I suspect that it might have something to do with how the URLs are structured (there are two ways), but I might be wrong. I'll add an issue for it.

How can I make s3fs stop logging every single request? It is just clogging up my /var/log and while I could adjust my syslog.conf to re-route or throw away any s3fs log data, I would rather like to prevent these excessive log data to be produced at all. Only errors and other important stuff should be logged.

Hello ,

I'm new to this Amazon S3 world. Here is my Scenario and goal. 1) I'm mounted my Amazon bucket to Red hat Server using s3fs . 2) But when i'm trying to use samba share for that mounted partition then it fails .

Goal : I want to use samba share for mounted amazon(fuse) partition .

Please help me out for the same.

Thanks in Advance.

I just wanted to say this is a pretty amazing piece of code. I just started pushing 2.5 million documents to S3 on Centos using s3fs. Looks like it will take a month. My only complaint is that it dies a couple times a day on average with "###curlCode: 7 msg: couldn't connect to server". Unfortunately that does cause s3fs to go bye-bye and my program has to umount and mount again.

I think I see above thats a curl bug, but even so, it would be nice if it didn't dismount the file system. :-)

But overall this has been a real easy way to get S3 into some legacy java code intended for a local file, albeit with some caveats. Thanks

I've installed s3fs on two servers running CentOS 5.5 - it works on one but NOT the other. Both servers are located on different networks. It seems to work on the one with the fresh install of CentOS 5.5 (while the other was upgraded from v5.0) - I keep getting on s3fs: "HTTP: 403 Forbidden - it is likely that your credentials are invalid" even though I've used the same setup as with the working server. It's mind boggling. Anyone has any ideas what I should look for?

Ok solved it. It had to do with the clock not matching the amazon server clocks. s3fs should be better at parsing the XML error message received from server.

I managed to mount an S3 repo but after a few minutes of rsyncing files, the mount becomes no longer available and I start received the following error if I try to 'cd' to it: ls: /mnt/myreponame/: Transport endpoint is not connected

The only way to get it working again is to unmount, and re-run s3fs. Anyone else experiencing this problem?

Another error after rsyncing for a few minutes:rsync: writefd_unbuffered failed to write 2 bytes sender?: Broken pipe (32) rsync: open "/mnt/crepo123/p/n101.jpg" failed: No such file or directory (2) rsync: close failed on "/mnt/crepo123/p/": No such file or directory (2) rsync error: error in file IO (code 11) at receiver.c(628) [receiver=2.6.8] rsync: connection unexpectedly closed (28218 bytes received so far) sender? rsync error: error in rsync protocol data stream (code 12) at io.c(463) [sender=2.6.8]

Just got a notice from s3. They said they have increase the object size limit from 5gb to 5tb. Anyone know if this will work with the s3fs or any changes needed? Thanks you. "We are excited to announce that Amazon S3 has increased the maximum size of an object from 5 gigabytes to 5 terabytes. You can now easily store and reference high resolution videos, large backup files, scientific instrument data or other large datasets as single objects. To store objects larger than 5 gigabytes, use the Multipart Upload feature, which allows parallel uploads and streaming of large objects into Amazon S3 as they are being created."

Hi, I have the following problems when I try a rsync -avz:

rsync: failed to set times on "/sonoro/.": Input/output error (5) rsync: recv_generator: failed to stat "/sonoro/3537_conferenciapordoctorrudolffgrossmann_pte1a_ladob.mp3": Transport endpoint is not connected (107) rsync: connection unexpectedly closed (107 bytes received so far) sender? rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]

Im using Ubuntu 10.10

Thanks

I've started to see this issue today too. It appears when I am transferring files of about 50 MB. I'm seeing the issue on Ubuntu and Centos 5.5.

i'm facing the same problem of giovangonzalez , i can reproduce in Centos 5.5 with 2 differents versions :

s3fs 1.19 fuse-2.7.4-8.el5 s3fs s3f2-svn r284 fuse-2.8.4

although i tried with different rsync commands :

rsync -vv -Pr --delete --no-whole-file /mnt/nfs /mnt/s3 rsync -vv -aP --no-whole-file --link-dest=/mnt/s3/archive/backup-date "+%Y-%m-%d" /mnt/nfs /mnt/s3

P.S: i'm sending files bigger than 1Gb because i want to use delta feature and incremental ..

if i split files in smaller chucks, won't allow to use delta and incremental features in my understanding.

Maybe related to :

http://code.google.com/p/s3fs/issues/detail?id=84

Great tool anyway , keep going guys !

My apologies in advance if I overlooked this in the FAQ & docs - but how do I mount a bucket of a different S3 user? In other words, how do I with my access ID & key mount a bucket owned by a different S3 user -- a bucket that I have permissions to see? I'm hoping that I just don't have the syntax right, but maybe s3fs cannot do this yet?

Hi,

Firstly - nice implementation, works a treat... mostly!

Having a rather peculiar issue when attempting to ls a large directory (thousands of files).

s3fs successfully chews through the first few dozen sets of parallel requests for individual file info, however then fails at the same point each time, like so

Jan 13 15:06:20 nfsuk001 s3fs: readdir: remaining_msgs: 30 code: 7  msg: Couldn't connect to server
Jan 13 15:06:20 nfsuk001 s3fs: readdir: remaining_msgs: 29 code: 7  msg: Couldn't connect to server
Jan 13 15:06:20 nfsuk001 s3fs: readdir: remaining_msgs: 28 code: 7  msg: Couldn't connect to server

s3fs then dies with CURLE_COULDNT_RESOLVE_HOST, which remains the case even after increasing all tcp limits in sysctl, and placing the IP for the aws host in /etc/hosts

Attempted amending the curl error handling within s3fs to retry in the case of CURLE_COULDNT_RESOLVE_HOST, but it simply fails repeatedly with the same issue.

Versions : 
ubuntu/maverick 10.10
libcurl 7.21.0-1ubuntu1
fuse 2.8.4-1ubuntu1
s3fs 1.33

Happy to provide any further info which helps isolate this, although thus far I've drawn a blank.

Further bit of information - the point at which it usually dies is when there are 1024 or thereabouts ESTABLISHED connections according to netstat.

ipv4 section in sysctl is

net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768    61000

One last tidbit - doesn't occur when repeatedly listing a smaller (240 files) directory.

If I increase ulimit, the behaviour changes - it gets through a few thousand files, then just freezes, and returns Software caused connection abort on the ls command, without displaying anything else.