By default the Fuse mount option "-o direct_io" is not used. Advantages are that mmap on the file system does work (files can get executed) and the Linux page cache is used for reading files. The page cache allows the integrated Fuse read ahead feature to be used which improves the data rate of read()s in presence of high latencies.

The downside of the disabled "-o direct_io" option is that the interruption of read() requests is not possible. By default, mount.xtreemfs will retry requests which failed due to communication errors, redirections (if the client contacted a backup primary and was redirected to the master) or internal server errors (for instance if the state of the replication is unknown and a retry is required) infinitely until the server/a replica did reply. In consequence, the client is very robust against small outages of any XtreemFS service. If the user does not want to wait for the completion of an operation, he is able to interrupt the request. We use the Fuse interruption support to implement this in Linux and it works every well, except for read() requests.

If you interrupt a read() request (e.g. by pressing Ctrl+C) and -o direct_io is disabled, the read syscall itself will get interrupted and your application does unblock. But the XtreemFS client itself won't get informed about the interruption and will go on retrying the read(). For this reason we reduced the default number of retries for read() operations to 40 attempts (see command line parameter --max-read-tries). In the worst case, the client will go on retrying the operation up to "--max-read-tries * --request-timeout" seconds (by default 40 * 60, 240 seconds or 40 minutes). While the retrying read() is active in the background, the file will be held open by Fuse and it will not be possible to unmount the file system as long as the file is open due to the retrying read().

If you have long running XtreemFS clients, this issue shouldn't bother you. If the retrying read() in the background prevents you from unmounting the file system, consider using -o direct_io (which implies the lack of advantages stated above) or reduce the number of --max-read-tries and/or the values --connection-timeout or --request-timeout. However, keep in mind that the retry functionality is essential if you use replication or your network connection is unstable. For this reason we did not disable the retry of read() requests at all and left it at 40 tries by default.

I asked at the Fuse mailing list about this issue and got a reply: http://sourceforge.net/mailarchive/message.php?msg_id=28208370

Miklos Szeredi told me that "It is not a fuse thing, it's a core kernel thing" and therefore neither XtreemFS nor Fuse will be able to fix this issue. Instead, we have to cope with the available workarounds.